Lane line detection method, lane line detection device, electronic apparatus, computer program, and computer-readable recording medium

Abstract

Disclosed are a lane line detection method, a lane line detection device, an electronic apparatus, a computer program, and a computer-readable recording medium. The invention discloses a lane line detection method using a processor. This method comprises: a step for acquiring a traveling image captured by an imaging device of a traveling vehicle; detecting an edge corresponding to the lane line in the acquired traveling image, and generating an edge image on the basis of the detected edge; a step of detecting a linear component on the basis of the detected edge and generating an edge linear processed image on the basis of the detected linear component; a step of detecting a lane line point corresponding to the lane line using the generated edge image and the edge straight line processed image; and a step of detecting a lane line based on the detected lane line point.

Description

[0001] This application is a divisional application of the invention patent application with the application date of April 9, 2021, application number "202110384407.9", and invention name "Lane line detection method, lane line detection device, electronic device, computer program and computer readable recording medium". Technical Field

[0002] The present invention relates to a lane line detection method, a lane line detection device, a computer program and a computer-readable recording medium that utilize a vehicle's driving image to detect lane lines displayed on a road.

[0003] Furthermore, the present invention relates to an electronic device that provides guidance for assisting a driver based on detected lane lines.

[0004] Furthermore, the present invention relates to an automatic driving system for controlling automatic driving of a vehicle based on detected lane lines. Background Art

[0005] The most important thing when driving a vehicle is to drive safely and prevent traffic accidents. For this purpose, the vehicle is equipped with various auxiliary devices, safety devices such as seat belts and air bags to control the vehicle's posture and the functions of the vehicle's components.

[0006] Furthermore, in recent years, a black box or the like is installed on a vehicle to store the driving images of the vehicle and the data transmitted from various sensors, and a device for identifying the cause of the vehicle accident is installed on the vehicle in the event of a breakdown. Portable terminals such as smartphones and tablets can also be equipped with a black box or a navigation application, etc., and used as such a vehicle device.

[0007] Therefore, in recent years, advanced driver assistance systems (ADAS) have been developed and popularized, which use driving images captured during the driving of a vehicle to assist the driver of the vehicle and promote safe driving and driver convenience.

[0008] In addition, with the rapid development of IT technology in recent years, interest in autonomous vehicles integrated with vision systems has gradually increased.

[0009] In order to accurately determine lane departure and lane maintenance in advanced driver assistance systems (ADAS) and autonomous vehicles, it is very important to accurately identify the lane line of the lane the vehicle is traveling in from the driving images captured during the vehicle's driving. If the lane line of the lane the vehicle is traveling in is not accurately identified and the vehicle deviates from the lane line, it may directly lead to an accident.

[0010] In addition, in recent years, augmented reality navigation that displays various guidance information on a screen including the real world that the user actually views has also been rapidly developing. The augmented reality navigation uses existing constructed route data to generate augmented reality route guidance lines indicating the route from the current position of the vehicle to the destination, and displays it in augmented reality.

[0011] However, in augmented reality navigation, the actual lane line and the augmented reality route guide line are consistent and invisible depending on the GPS reception speed, which will become a factor that reduces the sense of reality of the advantage of augmented reality. In addition, it is difficult to display the augmented reality route guide line in an environment where there is no or no route data.

[0012] Therefore, even for augmented reality navigation, the lane lines of the lane in which the vehicle is traveling must be accurately identified and the augmented reality route guidance lines must be realistically displayed.

[0013] In addition, according to the existing lane line recognition technology, information such as color and edge is used to detect curved lanes, but there is a problem of a high misdetection rate of curved lanes due to cars and other obstacles on the side roads and road markings. Summary of the invention

[0014] The present invention is proposed to solve the above-mentioned problem, and its purpose is to provide a vision-based lane line detection method, lane line detection device, computer program and computer-readable recording medium that can accurately detect straight lane lines and curved lane lines from driving images taken while the vehicle is driving.

[0015] In addition, an object of the present invention is to provide an augmented reality-based electronic device that accurately identifies lane lines of a lane in which a vehicle is traveling and realistically displays augmented reality guidance objects (particularly route guidance objects and lane line departure guidance objects).

[0016] Another object of the present invention is to provide an automatic driving system that controls automatic driving of a vehicle by accurately recognizing a lane line of a lane in which the vehicle is traveling.

[0017] To achieve the above object, a lane line detection method using a program according to an embodiment of the present invention includes: a step of acquiring a driving image captured by a camera of a moving vehicle; a step of detecting edges corresponding to lane lines in the acquired driving image, and generating an edge image based on the detected edges; a step of detecting straight line components based on the detected edges, and generating an edge straight line processed image based on the detected straight line components; a step of detecting lane line points corresponding to the lane lines using the generated edge image and the edge straight line processed image; and a step of detecting the lane lines based on the detected lane line points.

[0018] Moreover, the method further includes the step of generating a bird's-eye view image by transforming the viewpoint of the front view image corresponding to the acquired driving image, and the step of detecting the edge is to detect the edge corresponding to the lane line in the bird's-eye view image.

[0019] Furthermore, it also includes the step of dividing the overhead image into a first area and a second area according to the distance from the vehicle, and the step of generating the edge straight line processing image is to respectively detect the straight line components in a first edge image corresponding to the divided first area and a second edge image corresponding to the second area, and the edge straight line processing image can be generated based on the detected straight line components.

[0020] Moreover, in the step of detecting the lane line points, the lane line points corresponding to the left lane line and the lane line points corresponding to the right lane line can be detected respectively.

[0021] Moreover, the step of detecting the lane line points may include: the step of detecting initial lane line points using at least one of the generated edge image and the edge straight line processed image; and the step of generating a search window, applying the generated search window to the generated edge image and the edge straight line processed image, respectively, and detecting subsequent lane line points that meet preset conditions in the applied search window.

[0022] Moreover, the step of determining the initial lane line point may include: the step of detecting the left initial lane line point whose brightness value becomes the largest on the left side at the lower end of the edge image; and the step of detecting the right initial lane line point whose brightness value becomes the largest on the right side at the lower end of the edge image.

[0023] Moreover, the step of detecting the subsequent lane line point may include: a step of generating a first search window based on the determined initial lane line point; a step of detecting a first subsequent lane line point that meets a preset condition by applying the generated first search window to the edge image and the edge straight line processed image respectively; a step of generating a second search window with the column located in the first search window as the center column based on the first subsequent lane line point; and a step of arranging the generated second search window on the upper part of the first search window and applying it to the edge image and the edge straight line processed image respectively to detect a second subsequent lane line point that meets the preset condition.

[0024] Moreover, lane line points that meet the preset conditions may be detected only in the middle row of the search window.

[0025] Furthermore, the row size of the search window may be adaptively changed according to the inclination of the straight line component.

[0026] Moreover, the method further includes a step of calculating an energy value of an active contour model (ACM), and the preset condition may be a lane line point that minimizes the energy value of the active contour model (ACM).

[0027] Moreover, the step of calculating the energy value of the active contour model (ACM) may include: a step of calculating the internal energy value using the average distance between multiple previous lane line points and the distance between the current lane line point and the previous lane line point; and a step of calculating the external energy value using the brightness values ​​of the lane line points detected respectively in the edge image and the edge straight line processed image.

[0028] Moreover, the step of detecting the lane line may further include: a step of calculating a curve equation formed by the detected lane line points; and a step of fitting the lane line points into a curve based on the calculated curve equation.

[0029] Moreover, the step of detecting the lane line may include: the step of detecting the lane line points of the driving image corresponding to the lane line points of the overhead image by transforming the lane line points detected in the overhead image into the viewpoint of the acquired driving image; and the step of detecting the lane line based on the detected lane line points of the driving image.

[0030] On the other hand, in order to achieve the above-mentioned purpose, a lane line detection device according to an embodiment of the present invention includes: an image acquisition unit, which acquires a driving image captured by a camera of a moving vehicle; an edge detection unit, which detects edges corresponding to lane lines in the acquired driving image, and generates an edge image based on the detected edges; a straight line component detection unit, which detects straight line components based on the detected edges, and generates an edge straight line processed image based on the detected straight line components; a lane line point detection unit, which uses the generated edge image and the edge straight line processed image to detect lane line points corresponding to the lane lines; and a lane line detection unit, which detects the lane lines based on the detected lane line points.

[0031] Moreover, it also includes an image viewpoint conversion unit that generates a bird's-eye view image by converting the viewpoint of a front view image corresponding to the acquired driving image, and the edge detection unit can detect an edge corresponding to the lane line in the bird's-eye view image.

[0032] Furthermore, it also includes an image area segmentation unit that segments the overhead image into a first area and a second area according to the distance from the vehicle. The straight line components can be respectively detected in a first edge image corresponding to the segmented first area and a second edge image corresponding to the second area, and an edge straight line processing image can be generated based on the detected straight line components.

[0033] Moreover, the lane line point detection unit can detect the lane line point corresponding to the left lane line and the lane line point corresponding to the right lane line respectively.

[0034] Moreover, the lane line point detection unit may include: an initial lane line point detection unit, which detects initial lane line points using at least one of the generated edge image and the edge straight line processed image; and a subsequent lane line point detection unit, which generates a search window and applies the generated search window to the generated edge image and the edge straight line processed image, respectively, and detects subsequent lane line points that meet preset conditions in the applied search window.

[0035] Moreover, the initial lane line point detection unit can detect the left initial lane line point with the maximum brightness value at the lower end of the edge image, and can detect the right initial lane line point with the maximum brightness value at the lower end of the edge image.

[0036] Moreover, the subsequent lane line point detection unit can generate a first search window based on the determined initial lane line point, and detect the first subsequent lane line point that meets the preset conditions by applying the generated first search window to the edge image and the edge straight line processed image respectively. The first subsequent lane line point generates a second search window with the column located in the first search window as the center column, and the generated second search window is arranged on the upper part of the first search window, and is applied to the edge image and the edge straight line processed image respectively to detect the second subsequent lane line point that meets the preset conditions.

[0037] Moreover, lane line points that meet the preset conditions may be detected only in the middle row of the search window.

[0038] Furthermore, the row size of the search window may be adaptively changed according to the inclination of the straight line component.

[0039] Moreover, it also includes an active contour model energy calculation unit for calculating the energy value of an active contour model (ACM), and the preset condition may be a lane line point that minimizes the energy value of the active contour model (ACM).

[0040] Moreover, the active contour model energy calculation unit uses the average distance between multiple previous lane line points and the distance between the current lane line point and the previous lane line point to calculate the internal energy value, and can use the brightness values ​​of the lane line points detected respectively in the edge image and the edge straight line processed image to calculate the external energy value.

[0041] Moreover, the method further includes a lane line point fitting unit which calculates a curve equation formed by the detected lane line points and fits the lane line points into a curve based on the calculated curve equation.

[0042] Moreover, the lane line detection unit detects the lane line points of the driving image corresponding to the lane line points of the overhead image by transforming the lane line points detected in the overhead image into the viewpoints of the acquired driving image, and can detect lane lines based on the detected lane line points of the driving image.

[0043] On the one hand, in order to achieve the above purpose, an electronic device is provided according to an embodiment of the present invention for providing guidance for assisting a driver by detecting lane lines, wherein the electronic device includes: a display unit for displaying guidance information that can be confirmed by the driver; an image acquisition unit for acquiring a driving image captured by a camera; an edge detection unit for detecting edges corresponding to lane lines in the acquired driving image and generating an edge image based on the detected edges; a straight line component detection unit for detecting straight line components based on the detected edges and generating an edge straight line processed image based on the detected straight line components; a lane line point detection unit for detecting lane line points corresponding to the lane lines from the generated edge image and the edge straight line processed image; a lane line detection unit for detecting the lane lines from the detected lane line points; and a control unit for controlling the display unit to provide guidance based on the detected lane lines.

[0044] Furthermore, it also includes an augmented reality providing unit, which generates a route guidance object for guiding a route from the current position of the vehicle to a destination, and generates an augmented reality image displayed on the lane of the captured driving image based on the generated route guidance object and the detected lane line. The control unit can control the display unit to display the generated augmented reality image.

[0045] Moreover, it also includes an augmented reality providing unit, which generates a lane line departure guidance object for guiding the vehicle to depart from the lane line, and generates an augmented reality image displayed on the lane line based on the generated lane line departure guidance object, and the control unit can control the display unit to display the generated augmented reality image.

[0046] On the one hand, in order to achieve the above object, a computer-readable recording medium according to an embodiment of the present invention records a program for executing the lane line detection method.

[0047] On the one hand, a program according to an embodiment of the present invention to achieve the above-mentioned purpose is stored in a computer-readable recording medium to execute the above-mentioned lane line detection method.

[0048] Effects of the Invention

[0049] According to the various embodiments of the present invention described above, both straight lane lines and curved lane lines can be accurately detected from the driving images captured when the vehicle is driving.

[0050] In addition, according to various embodiments of the present invention, since the straight line component is used as the main factor, misidentification caused by vehicles on the side roads and other obstacles can be reduced, and both the straight lane lines and the curved lane lines can be accurately detected.

[0051] In addition, according to various embodiments of the present invention, by accurately identifying the lane lines of the lane in which the vehicle is traveling, augmented reality guidance objects displayed based on the lane lines (e.g., route guidance objects and lane line departure guidance objects) can be realistically displayed on the augmented reality screen.

[0052] In addition, according to various embodiments of the present invention, even in an environment where route data cannot be provided or route data has not been established due to national characteristics (such as China), an augmented reality guidance object based on lane lines is displayed on an augmented reality screen by accurately identifying lane lines.

[0053] In addition, according to the present invention, by accurately recognizing the lane line of the lane in which the vehicle is traveling, the problem of automatic driving error caused by misrecognition of the lane line can be solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 FIG. 4 is a block diagram showing a lane line detection device according to an embodiment of the present invention.

[0055] Figure 2 FIG. 4 is a block diagram showing a lane point detection unit according to an embodiment of the present invention in more detail.

[0056] Figure 3 It is a diagram for explaining image viewpoint transformation according to an embodiment of the present invention.

[0057] Figure 4 It is a diagram for explaining image region segmentation according to an embodiment of the present invention.

[0058] Figure 5It is a diagram for explaining a linear component detection process according to an embodiment of the present invention.

[0059] Figure 6 FIG. 1 is a diagram showing an edge image and an edge line processed image according to an embodiment of the present invention.

[0060] Figure 7 It is a diagram for explaining the determination of initial lane points according to an embodiment of the present invention.

[0061] Figure 8 2 is a diagram for illustrating a subsequent lane line point determination method according to an embodiment of the present invention.

[0062] Fig. 9 FIG. 4 is a diagram showing a lane point detection result according to an embodiment of the present invention.

[0063] Fig.10 The diagram is used to illustrate a process of detecting lane lines from a front view image using a top view image according to an embodiment of the present invention.

[0064] Fig.11 is a flow chart showing a lane line detection method according to an embodiment of the present invention.

[0065] Fig.12 FIG. 4 is a flowchart specifically illustrating a lane line detection method according to an embodiment of the present invention.

[0066] Fig.13 Detailed description of the process of detecting lane points (S206, S207) according to an embodiment of the present invention.

[0067] Fig.14 is a flowchart showing the lane line detection step (S208) according to an embodiment of the present invention in more detail.

[0068] Fig.15 is a block diagram showing an electronic device according to an embodiment of the present invention.

[0069] Fig.16 It is a diagram for explaining a method for generating a route guidance object according to an embodiment of the present invention.

[0070] Fig.17 FIG. 1 is a diagram for explaining a system network connected to electronic devices according to an embodiment of the present invention.

[0071] Fig.18 is a diagram illustrating an augmented reality guide screen displaying a route guide object according to an embodiment of the present invention.

[0072] Fig.19is a diagram illustrating an augmented reality guidance screen displaying a lane line departure guidance object according to an embodiment of the present invention.

[0073] Fig. 20 This is a diagram showing an implementation mode in which the electronic device according to one embodiment of the present invention does not include a camera unit.

[0074] Fig.21 This is a diagram showing an implementation mode in which an electronic device according to an embodiment of the present invention includes a camera unit.

[0075] Fig. 22 FIG. 1 is a diagram showing an implementation using a head-up display (HUD) according to an embodiment of the present invention.

[0076] Fig.23 is a block diagram showing an automatic driving system according to an embodiment of the present invention.

[0077] Fig.24 Detailed description is a block diagram showing a structure of an autonomous vehicle according to an embodiment of the present invention.

[0078] In the figure:

[0079] 10: Lane line detection device, 11: Image acquisition unit, 12: Image viewpoint conversion unit, 13: Image region segmentation unit, 14: Edge detection unit, 15: Line component detection unit, 16: Lane line point detection unit, 17: Lane line point fitting unit, 18: Lane line detection unit, 100: Electronic device, 110: Storage unit, 120: Input unit, 130: Output unit, 140: Lane line detection unit, 160: Augmented reality providing unit, 170: Control unit, 180: Communication unit, 190: Sensing unit, 195: Power supply unit DETAILED DESCRIPTION

[0080] The following content is only an example of the principle of the invention. Therefore, a person skilled in the art of the invention can invent various devices that are not explicitly described or shown in the specification but embody the principle of the invention and are included in the concept of the invention. In addition, all conditional terms and embodiments listed in this specification are only used to clearly understand the concept of the invention, and are not limited to the specifically listed embodiments and states as described above.

[0081] Furthermore, it should be understood that not only the principles, viewpoints and embodiments of the present invention, but also all detailed descriptions listing specific embodiments include structural and functional equivalents of the matters described. Furthermore, it should be understood that the equivalents include not only currently known equivalents, but also equivalents to be developed in the future, that is, all elements invented in a way to perform the same function regardless of the structure.

[0082] Thus, for example, it should be understood that the block diagrams in this specification show conceptual views of exemplary circuits that embody the principles of the present invention. Similarly, it should be understood that all flow charts, state transition diagrams, pseudo codes, etc. can be substantially embodied by computer-readable media, whether or not a computer or processor is explicitly shown, and represent various processing that can be executed by a computer or processor.

[0083] The functions of the various elements shown in the figure, including functional modules representing processing or similar concepts, are not only related to dedicated hardware, but also to appropriate software, so that hardware with the ability to execute software can be used. When provided by a processor, the functions can be provided by a single dedicated processor, a single shared processor, or multiple individual processors, some of which can be shared.

[0084] Furthermore, the explicit use of terms such as processing, control, or concepts similar to processing and control cannot be interpreted as exclusively referring to hardware capable of executing software, but should be understood to implicitly include digital signal processor (DSP) hardware, read-only memory (ROM) for storing software, random access memory (RAM) and non-volatile memory without limitation. Other conventional hardware may also be included.

[0085] In the scope of the invention claims of this specification, the structural elements that are expressed as means for executing the functions described in the detailed description are intended to include, for example, all methods of the functions of software in all forms including a combination of circuit elements that execute the functions or firmware / microcode, etc., in a manner to execute the functions in combination with appropriate circuits for executing the software. In the present invention defined by the scope of the invention claims, the functions provided by a plurality of means are combined and combined in the manner required by the scope of the invention claims, and therefore, it should be understood that any means that can provide the functions are also equivalent to the functions grasped from this specification.

[0086] The above objects, features and advantages will become more clearly understood through the following detailed description in conjunction with the accompanying drawings, so that a person skilled in the art can easily implement the technical concept of the present invention. In addition, in the process of describing the present invention, if it is determined that the specific description of the known technology related to the present invention will make the main idea of ​​the present invention unclear, the detailed description thereof will be omitted.

[0087] Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0088] Figure 1 FIG. 4 is a block diagram showing a lane line detection device according to an embodiment of the present invention. Figure 2 FIG. 4 is a block diagram showing a lane line detection device according to an embodiment of the present invention in more detail. Figure 1 and 2As shown, the lane line detection device 10 may include an image acquisition unit 11, an image viewpoint conversion unit 12, an image region segmentation unit 13, an edge detection unit 14, a straight line component detection unit 15, a lane line point detection unit 16, a lane line point fitting unit 17, and all or part of a lane line detection unit 18. In addition, the lane line point detection unit 16 may include an initial lane line point detection unit 16-1, a subsequent lane line point detection unit 16-2, and an ACM energy calculation unit 16-3.

[0089] The lane marking detection device 10 can detect lane markings of a lane in which the vehicle is traveling from the driving image by performing image processing on the driving image captured by a camera of the vehicle.

[0090] Here, the driving of the vehicle refers to a state in which the vehicle is driven by a driver and is located on a road, and may be a concept including a driving state in which the vehicle is driving on a road, a parked state in which the vehicle is parked on the road, and a parked state in which the vehicle is parked.

[0091] In addition, the driving image captured by the camera of the vehicle may include a front driving image captured in front of the vehicle and a rear driving image captured in rear of the vehicle.

[0092] The lane line detection device 10 can be implemented using software, hardware or a combination thereof. As an example, according to hardware implementation, it can be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, and micro-processors for performing other functions.

[0093] For the convenience of explanation, the following will take the case where the driving image is a front driving image as an example to explain in more detail each component module constituting the lane line detection device 10 .

[0094] The image acquisition unit 11 can acquire the driving image captured by the camera of the vehicle. Specifically, the image acquisition unit 11 can acquire the driving image captured by the camera installed in the vehicle in real time when the vehicle is driving.

[0095] Here, the acquired driving image may include a plurality of lane lines differentiated according to lane lines, a road consisting of a plurality of lane lines, and a plurality of vehicles traveling on the road.

[0096] Lane markings may refer to lines on both sides of a lane that a vehicle is in. In addition, lanes may refer to a road such as a first lane, a second lane, ... an Nth lane, etc., formed by lane markings on which a vehicle is traveling.

[0097] In addition, the acquired driving image may be a color image or a grayscale image.

[0098] When the acquired driving image is a color image, the image acquisition unit 11 may generate a grayscale image from the color image through grayscale processing. However, when the acquired driving image is a grayscale processed image, the image acquisition unit 11 may not perform other grayscale processing.

[0099] According to the present invention, by using grayscale processed images to execute the lane line detection algorithm described later, images can be processed in real time even in low-end terminals.

[0100] However, the present invention is not limited thereto. When the driving image acquired by the image acquisition unit 11 is a color image, the image acquisition unit 11 may not perform grayscale processing but use the color image to execute the lane line detection algorithm described later. In this case, according to the present invention, the RGB color space of the color image may be transformed into the CIELUV color space, the YCbCr color space, or the HSV color space, and the lane line detection algorithm described later may be executed using the color image after the color space transformation.

[0101] For the convenience of explanation, the following will take the case where the driving image is a grayscale processed image as an example.

[0102] The image viewpoint conversion unit 12 may convert the viewpoint of the running image acquired by the image acquisition unit 11 to generate a running image with the viewpoint converted.

[0103] As an example, when the viewpoint of a camera installed in a vehicle faces the front of the vehicle, the driving image acquired by the image acquisition unit 11 may be a forward-looking image corresponding to the viewpoint of the camera. However, if the lane line detection algorithm described later is executed through the forward-looking image, the accuracy of detecting the lane line may be reduced. Therefore, according to the present invention, the image viewpoint transformation unit 12 receives the forward-looking image acquired by the image acquisition unit 11, and transforms the viewpoint of the received forward-looking image to generate a top view image. Here, the top view image may refer to an image of a viewpoint from above to below.

[0104] The image viewpoint conversion unit 12 may use a conversion matrix (or homography) of the following mathematical formula 1 to convert the viewpoint of the driving image.

[0105]

Mathematical formula 1

[0106]

[0107] Here, W is the weight, (x, y, 1) is the original position, (x', y', 1) is the transformed position, and h 11 to h 33 is the transformation matrix.

[0108] That is, the image viewpoint conversion unit 12 may set a region of interest (ROI) of the front view image and apply the transformation matrix to all pixels of the ROI to generate a bird's-eye view image. Here, the ROI may be a preset region or may be changed according to the specifications of the camera.

[0109] In addition, the above-mentioned transformation matrix may be a matrix that assigns preset values ​​to the storage unit. Alternatively, the image viewpoint transformation unit 12 calculates the transformation matrix based on the forward image acquired by the shooting device by turning on the device power, and then generates the overhead image based on the calculated transformation matrix. When the power is turned off, the transformation matrix may be deleted.

[0110] Reference Figure 3 The video viewpoint conversion unit 12 of the present invention will be described in more detail.

[0111] Figure 3 FIG. 2 is a diagram for explaining image viewpoint transformation according to an embodiment of the present invention. Figure 3 The driving image acquired by the image acquisition unit 11 may be a front view image corresponding to the viewpoint of the camera (refer to Figure 3 a). In this case, the image viewpoint conversion unit 12 selects the region of interest 3001 from the front view image, applies the selected region of interest 3001 to the transformation matrix, and can generate an image of the viewpoint from above to the bottom, that is, a bird's-eye view image ( Figure 3 (b)).

[0112] In addition, the image viewpoint conversion unit 12 according to the present invention can generate a front view image from a bird's-eye view image by reversely applying the above conversion.

[0113] For the convenience of explanation, the following description will be made by taking the case where the driving image is a bird's-eye view image as an example.

[0114] The image region dividing unit 13 may divide the overhead image generated by the image viewpoint conversion unit 12 into a first region and a second region according to the distance from the vehicle. Figure 4 The image area dividing unit 13 will be described in more detail.

[0115] Figure 4 FIG. 2 is a diagram for explaining image region segmentation according to an embodiment of the present invention. Figure 4 The overhead image generated by the image viewpoint conversion unit 12 can be Figure 4 (a) is shown. The image region segmentation unit 13 is as follows Figure 4 As shown in (b), it can be divided into a first area 3101-2, which corresponds to an area close to the vehicle according to the distance from the vehicle; and a second area 3101-1, which corresponds to an area far from the vehicle.

[0116] The edge detection unit 14 can detect the edge corresponding to the lane line in the driving image. Specifically, the edge detection unit 14 can detect the edge corresponding to the lane line in the overhead image generated by the image viewpoint transformation unit 12. Here, the edge refers to the edge or contour, and can refer to the part existing at the point where the brightness of the image changes from a low value to a high value or vice versa. For example, the edge detection unit 14 can use various technologies such as Sobel edge extraction technology, Prewitt edge extraction technology, Roberts edge extraction technology, Compass edge extraction technology, etc. to detect the edge corresponding to the lane line in the driving image.

[0117] In addition, the edge detection unit 14 may generate an edge image based on the detected edge. Here, the edge image generated by the edge detection unit 14 may be an image that distinguishes the detected edge area from the non-edge area. Figure 6 This edge image is described.

[0118] The linear component detection unit 15 can detect the linear component based on the edge detected by the edge detection unit 14. Specifically, the linear component detection unit 15 applies the linear component detection algorithm to each edge image in the first area and the second area segmented by the image area segmentation unit 13, and can detect the linear component from the edge included in each first area and the second area. Here, the linear component detection unit 15 detects the linear component for the first area 3101-2 corresponding to the close area and the second area 3101-1 corresponding to the far area in the area segmented by the image area segmentation unit 13, and thus can represent the curve corresponding to the curved vehicle line.

[0119] The linear component detection unit 15 may detect the linear component based on the edge detected by the edge detection unit 14 using, for example, Hough transform. The linear component detected by the linear component detection unit 15 may be one or more.

[0120] Furthermore, the straight line component detection unit 15 may generate an edge straight line processed image based on the detected straight line component. Here, the edge straight line processed image generated by the straight line component detection unit 15 may be an image that distinguishes the detected straight line component area from other areas. Figure 6 The edge straight line processing image is described.

[0121] In addition, refer to Figure 5 The operation of the linear component detection unit will be described in more detail.

[0122] Figure 5 It is a diagram for explaining a linear component detection process according to an embodiment of the present invention. Figure 5 (a) is an edge image generated by the edge detection unit 14, and the linear component detection unit 15 may apply Hough Transform to the first edge image 3201 corresponding to the first area and the second edge image 3202 corresponding to the second area divided by the image area division unit 13. In this case, Figure 5 As shown in (b), the linear component detection unit 15 can detect first linear components 3201 - 1 and 3201 - 2 from the edges included in the first edge image 3201 , and detect second linear components 3202 - 1 and 3202 - 2 from the edges included in the second edge image 3202 .

[0123] Figure 6 1 is a diagram showing an edge image and an edge straight line processed image according to an embodiment of the present invention. The edge detection unit 14 may generate an edge image based on the detected edges. Figure 6(a) The edge image generated by the edge detection unit 14 may include the detected edge region (white display region) and the non-edge region (black display region), and the edge region can be distinguished from the non-edge region in the edge image.

[0124] In addition, the linear component detection unit 15 may generate an edge linear processed image based on the linear component detected from the edge image. Figure 6 (b) The edge straight line processed image generated by the straight line component detection unit 15 may include the detected straight line component area (white display area) and other areas (black display area), and the straight line component area in the edge straight line processed image can be distinguished from the other areas.

[0125] In addition, the lane line point detection unit 16 may detect lane line points corresponding to lane lines using the generated edge image and edge line processed image.

[0126] Usually, the lane line on which the vehicle travels consists of a left lane line and a right lane line, so the lane line point detection unit 16 can use the generated edge image and edge straight line processed image to respectively detect the lane line points corresponding to the left lane line and the lane line points corresponding to the right lane line.

[0127] However, the present invention is not limited thereto. When the lane in which the vehicle is traveling consists of only one lane line, the lane line point detection unit 16 may also use the generated edge image and edge line processed image to detect only the lane line points corresponding to the lane line on one side.

[0128] Here, the lane point detection unit 16 may include an initial lane point detection unit 16 - 1 , a subsequent lane point detection unit 16 - 2 , and an ACM energy calculation unit 16 - 3 .

[0129] The initial lane line point detection unit 16-1 can determine the initial lane line point using at least one of the edge image generated by the edge detection unit 14 and the edge straight line processed image generated by the straight line component detection unit 15. Specifically, the initial lane line point detection unit 16-1 detects two or more points with the largest brightness values ​​at the bottom end of at least one of the edge image and the edge straight line processed image, and can determine the point with the largest brightness value on the left side of the points as the left initial lane line point, and can determine the point with the largest brightness value on the right side of the points as the right initial lane line point. Figure 7 The method of determining these preliminary lane points is described in more detail.

[0130] Figure 7 1 is a diagram for explaining how to determine an initial lane line point according to an embodiment of the present invention. Figure 7The initial lane line point detection unit 16-1 is a diagram showing the use of the edge image 3301 to determine the initial lane line point. The initial lane line point detection unit 16-1 sets the lower end portion 3302 of the edge image 3301, analyzes the histogram of the lower end portion 3302 to detect two or more points with the maximum brightness value, and can determine the initial left lane line point V with the maximum brightness value on the left side among the points. 1 L , and it can be determined that the right side of the point has the largest brightness value and the right initial lane line point V 1 R .

[0131] in addition, Figure 7 In the above, the initial lane line point detection unit 16-1 analyzes the edge image generated by the edge detection unit 14 to detect the initial lane line points. However, the present invention is not limited thereto. According to another embodiment of the present invention, the edge straight line processed image generated by the straight line component detection unit 15 may be analyzed to detect the initial lane line points. Alternatively, the average position of the initial lane line points detected in the edge image and the initial lane line points detected in the edge straight line processed image may be detected as the initial lane line points.

[0132] On the other hand, when the initial lane line point is determined, the subsequent lane line point detection unit 16-2 generates a search window, applies the generated search window to the edge image and the edge straight line processed image, and determines the point that meets the preset condition within the applied search window as the subsequent lane line point. In this case, the subsequent lane line point detection unit 16-2 can perform the generation and application of the search window from the bottom end to the top end of each of the edge image and the edge straight line processed image to determine multiple subsequent lane line points.

[0133] Specifically, the subsequent lane line point detection unit 16-2 generates a first search window based on the determined initial lane line point, and the generated first search window can be applied to each of the edge image and the edge straight line processed image to determine the first subsequent lane line point that meets the preset conditions. In addition, the subsequent lane line point detection unit 16-2 generates a second search window with the column where the first subsequent lane line point is located in the first search window as the center column, and arranges the generated second search window at the upper part of the first search window, and applies it to each of the edge image and the edge straight line processed image to determine the second subsequent lane line point that meets the preset conditions. Then, the subsequent lane line point detection unit 16-2 generates the third, fourth, and Nth search windows in the same manner as the generation of the second search window, until the end of the edge image and the edge straight line processed image, and applies the multiple third, fourth, and Nth search windows generated in sequence to the upper end of the edge image and the edge straight line processed image in sequence to determine multiple subsequent lane line points.

[0134] Here, the search window applied to the edge image and the search window applied to the edge line processed image may have the same size, and may be the search window arranged at the same position in each image.

[0135] In addition, the preset condition may be a condition for determining a point that minimizes the energy value of an active contour model (ACM) as a subsequent line point.

[0136] ACM is a model that repeatedly performs a process of minimizing internal energy and external energy and searches for a contour, and may be composed of internal energy and external energy.

[0137] The ACM energy calculation unit 16 - 3 according to the present invention can calculate the energy value of the ACM based on the following mathematical formula 2.

[0138]

Mathematical formula 2

[0139]

[0140] Here, E int Can represent the internal energy value, E ext Can represent the external energy value, E ACM Can represent ACM energy value.

[0141] *Internal energy

[0142] Specifically, the ACM energy calculation unit 16-3 can calculate the internal energy value using the average distance between multiple previous points and the distance between the current point and the previous point. Here, when the distance between the current lane line point and the previous lane line point is close to the average distance between multiple previous lane line points, the energy decreases and converges. Specifically, the ACM energy calculation unit 16-3 can calculate the internal energy value based on the following mathematical formula 3.

[0143]

Mathematical formula 3

[0144]

[0145] Here, v i represents the i-th lane line point vector, Indicates the average distance to the i-1th lane line point. Here, the initial lane line point i can be 0, and the subsequent lane line points can be assigned 1, 2...n in sequence.

[0146] *External energy

[0147] Specifically, as shown in the following mathematical formula 4, the ACM energy calculation unit 16-3 can use the brightness value of the lane line point detected in each of the edge image and the edge straight line processed image to calculate the external energy value. In particular, when only the edge image is used to calculate the external energy, since the lane line point cannot be accurately detected, according to the present invention, the edge straight line processed image and the edge image are used at the same time, but a larger weight value is applied to the straight line processed image to calculate the external energy, so that the lane line point can be detected more accurately.

[0148] [Formula 4]

[0149]

[0150] Here, f edge (v i ) can represent the brightness value of the edge image in the i-th lane line point (value: 0 to 255), f hough (v i ) can represent the brightness value of the edge line processing image in the i-th lane line (value: 0 to 255), fE max It can represent the maximum brightness of the edge image (value: 255), fH max It can represent the maximum brightness value of the edge straight line processed image (value: 255).

[0151] Below, refer to Figure 8 The following describes a method for detecting subsequent lane line points using the active contour model (ACM).

[0152] Figure 8 2 is a diagram for illustrating a subsequent lane line point detection method according to an embodiment of the present invention. Figure 8 , the initial lane line point V is detected by the initial lane line point detection unit 16-1 0 L 、V 0 R When the subsequent lane line point detection unit 16-2 generates the first search windows 3401-1 and 3401-2 based on the initial lane line points, the first search windows 3401-1 and 3401-2 may be arranged to be located at the center of the last row of the first search windows 3401-1 and 3401-2.

[0153] Then, the subsequent lane line point detection unit 16-2 may apply the generated first search windows 3401-1, 3401-2 to each of the edge image and the edge line processed image, and calculate the internal energy value and the external energy value of each pixel of the first search windows 3401-1, 3401-2 based on the above mathematical formula. In addition, the subsequent lane line point detection unit 16-2 may apply the calculated internal energy value and external energy value to the above mathematical formula 2 to calculate the ACM energy value of each pixel of the first search window 3401. In addition, the subsequent lane line point detection unit 16-2 may detect the point with the smallest calculated ACM energy value as the first subsequent lane line point V 1 L 、V 1 R .

[0154] Thereafter, the subsequent lane line point detection unit 16-2 may generate a first subsequent lane line point V 1 L 、V 1 R The columns located in the first search windows 3401-1, 3401-2 are the second search windows 3402-1, 3402-2 as the center columns. In addition, the subsequent lane line point detection unit 16-2 may arrange the generated second search windows 3402-1, 3402-2 on top of the first search windows 3401-1, 3401-2. In this case, the subsequent lane line point detection unit 16-2 arranges the second search windows 3402-1, 3402-2 on top of the first search windows 3401-1, 3401-2 at a position where the second search windows 3402-1, 3402-2 overlap with a portion of the first search windows 3401-1, 3401-2, or may arrange the second search windows 3401-1, 3401-2 on top of the first search windows 3401-1, 3401-2 at a non-overlapping position.

[0155] Thereafter, the subsequent lane line point detection unit 16-2 may apply the generated second search windows 3402-1 and 3402-2 to the edge image and the edge line processed image, respectively, and calculate the internal energy value and the external energy value of each pixel of the second search windows 3402-1 and 3402-2 based on the above mathematical formula. Then, the subsequent lane line point detection unit 16-2 may apply the calculated internal energy value and the external energy value to the above mathematical formula 2 to calculate the ACM energy value of each pixel of the second search windows 3402-1 and 3402-2. In addition, the subsequent lane line point detection unit 16-2 may detect the point with the smallest calculated ACM energy value as the second subsequent lane line point V 2 L 、V 2 R .

[0156] Here, the expression of the second search window and the second subsequent lane line point is used to distinguish the first search window from the first subsequent lane line point, and the second search window and the second subsequent lane line point can be the concept of including all search windows after the first search window and all lane line points after the first subsequent lane line point.

[0157] That is, the subsequent lane line point detection unit 16-2 according to an embodiment of the present invention can repeat the above-mentioned second search window setting and second subsequent lane line point detection process, set multiple second search windows 3403-1, 3403-2, ..., 340N-1, 340N-2, and detect multiple second subsequent lane line points V 3 L 、V 3 R , ..., V N L 、V N R .

[0158] In addition, the search window applied to the edge image and the search window applied to the edge line processed image have the same size and may be the search window arranged at the same position in each image.

[0159] In addition, the search window may be of a preset size, and the lane point detection unit 16 may use the search window of the preset size to detect lane points. Figure 8 As shown, the size of the search window can be preset to 5 pixels in width and 5 pixels in height.

[0160] However, the present invention is not limited thereto, and the size of the search window of the lane point detection unit 16 according to an embodiment of the present invention may be adaptively changed to detect lane points. For example, the lane point detection unit 16 according to an embodiment of the present invention may adaptively change the size of the search window according to the inclination of the straight line component of the edge line processed image generated by the straight line component detection unit 15.

[0161] That is, in the curve section, the smaller the curve curvature radius, the greater the risk of the curve, and the inclination of the straight component corresponding to the curve lane line will be smaller. Therefore, as the inclination of the straight component is smaller, the lane line point detection unit 16 can increase the row size of the search window, so that a wider area can be searched to the left and right.

[0162] In addition, according to the present invention, the subsequent lane line point determination unit 16-2 can detect points that meet the preset conditions only in the middle row of the first search window and the second search window. That is, according to the present invention, by detecting points that meet the preset conditions only in the pixels of the middle row of the search window, it is not necessary to calculate the ACM of all pixels in the search window, thereby improving the operation processing speed, and the image can be processed in real time even on a low-end terminal.

[0163] Fig. 9 FIG. 2 is a diagram showing a lane line point detection result according to an embodiment of the present invention. Fig. 9 The subsequent lane line point detection unit 16-2 generates a search window and applies the generated search window to each of the edge image and the edge line processed image, so as to detect all subsequent lane line points that meet the preset conditions within the applied search window.

[0164] In addition, the lane line point fitting unit 17 may calculate a curve equation formed by the detected lane line points, and fit the lane line points into a curve based on the calculated curve equation.

[0165] That is, if the lane line points detected by the lane line point detection unit 16 are connected as they are, the shape may be different from the actual curve. Therefore, the lane line point fitting unit 17 according to the present invention calculates the coefficients a, b, c of the quadratic equation described in the following mathematical formula 5, calculates the curve equation, and applies all the lane line points detected in the calculated curve equation to fit the lane line points into a curve.

[0166]

Mathematical formula 5

[0167] x=ay 2 +by+c

[0168] The lane line detection unit 18 can detect lane lines based on the lane line points detected by the lane line point detection unit 16. Specifically, the lane line detection unit 18 transforms the lane line points detected from the overhead image into the viewpoint of the acquired driving image (here, the driving image is the front view image), so that the lane line points of the front view image corresponding to the lane line points of the overhead image can be detected. Here, the lane line points detected in the overhead image can be the lane line points detected by the lane line point detection unit 16 or the lane line points curve-fitted by the lane line point fitting unit 17. In addition, the transformation matrix of the image viewpoint transformation unit 12 can be used to perform the transformation from the overhead image viewpoint to the front view image viewpoint.

[0169] In addition, the lane line detection unit 18 can detect lane lines from lane line points in the front view image.

[0170] Reference Fig.10 The operation of the lane detection unit 18 will be described in more detail.

[0171] Fig.10 2 is a diagram for explaining a process of detecting lane lines in a front view image using a top view image according to an embodiment of the present invention. Fig.10 The image viewpoint conversion unit 12 can generate the following from the front view image: Fig.10 (a) is a top view. Fig.10 As shown in (b), the lane point detection unit 16 can detect all lane points using the overhead image.

[0172] On the other hand, Fig.10 As shown in (c), when all lane line points are detected, the lane line detection unit 18 transforms the lane line points detected in the overhead image into the front view image viewpoints, thereby detecting the lane line points of the front view image corresponding to the lane line points of the overhead image. In addition, the lane line detection unit 18 can detect lane lines 3501 and 3502 from the left and right lane line points of the front view image.

[0173] Below, refer to Figures 11 to 13 The lane line detection method according to an embodiment of the present invention is described in more detail.

[0174] Fig.11 FIG. 1 is a flow chart showing a lane line detection method according to an embodiment of the present invention. Fig.11 The lane line detection device can obtain a driving image captured by a shooting device of a moving vehicle (S101).

[0175] In addition, the lane line detection device can detect edges corresponding to the lane lines in the acquired driving image, and generate an edge image based on the detected edges (S102).

[0176] Furthermore, the lane line detection apparatus may detect a straight line component based on the detected edge, and may generate an edge straight line processed image based on the detected straight line component ( S103 ).

[0177] In addition, the lane line detection device may detect a lane line point corresponding to the lane line using the generated edge image and edge line processed image (S104).

[0178] Then, the lane line detection device may detect the lane line based on the detected lane line points (S105).

[0179] Fig.12 FIG. 1 is a flowchart showing a lane line detection method according to an embodiment of the present invention in more detail. Fig.12 The lane line detection device can obtain a driving image captured by a camera of a moving vehicle (S201).

[0180] In addition, the lane line detection device may transform the viewpoint of the front view image corresponding to the acquired driving image to generate a top view image (Top View) (S202).

[0181] In addition, the lane line detection device may divide the overhead image into a first area and a second area according to the distance from the vehicle (S203).

[0182] In addition, the lane line detection device can detect edges corresponding to the lane lines in the overhead image, and generate an edge image based on the detected edges (S204).

[0183] In addition, the lane line detection device can detect a straight line component in each of the first edge image corresponding to the segmented first area and the second edge image corresponding to the second area, and generate an edge straight line processed image based on the detected straight line component (S205).

[0184] In addition, the lane line detection device may detect an initial lane line point using at least one of the generated edge image and edge line processed image (S206).

[0185] In addition, the lane line detection device can generate a search window and apply the generated search window to each of the edge image and the edge line processed image, so that subsequent lane line points that meet the preset conditions can be detected in the applied search window (S207).

[0186] Then, the lane line detection device may detect the lane line based on the detected lane line points (S208).

[0187] Fig.13 Detailed description of the lane line point detection step (S206, S207) according to one embodiment of the present invention. Fig.13 First, the lane line detection device can detect an initial lane line point where the brightness value at the lower end of the edge image or the edge line processed image becomes the maximum (S301).

[0188] In addition, the lane line detection device can generate a first search window based on the detected initial lane line points (S302).

[0189] In addition, the lane line detection device may apply the generated first search window to each of the edge image and the edge line processed image, and determine a point that meets a preset condition as a first subsequent lane line point (S303).

[0190] In addition, the lane line detection device may generate a second search window with the column where the first subsequent lane line point is located in the first search window as the center column (S304).

[0191] In addition, the lane line detection device can arrange the generated second search window on the upper part of the first search window, and apply it to each of the edge image and the edge line processed image to determine a second subsequent lane line point that meets the preset conditions (S305).

[0192] Here, the preset condition may be a lane line point that minimizes the energy value of an active contour model (ACM). Specifically, step (S305) may include a step of calculating an internal energy value using an average distance between a plurality of previous lane line points and a distance between a current lane line point and a previous lane line point; and a step of calculating an external energy value using a brightness value of each lane line point detected in an edge image and an edge line processed image.

[0193] In addition, the lane line detection device according to the present invention can detect all lane line points by repeatedly executing steps (S304) and (S305) until the upper end of the edge image and the edge line processed image.

[0194] In addition, when the above-mentioned lane line points are detected, the lane line points corresponding to the left lane line and the lane line points corresponding to the right lane line can be detected respectively.

[0195] In addition, the lane line detection device can detect points that meet preset conditions only in the middle row of the search window.

[0196] Furthermore, the line size of the search window can be adaptively changed according to the inclination of the linear component.

[0197] Fig.14 2 is a flowchart showing the lane line detection step (S208) according to an embodiment of the present invention in more detail. Fig.14 The lane line detection device can calculate a curve equation formed by the detected multiple lane line points (S401).

[0198] Then, the lane line detection device may fit the detected lane line points into a curve based on the calculated curve equation (S402).

[0199] In addition, the lane line detection device can transform the plurality of lane line points detected from the overhead image into the viewpoints of the front view image (S403).

[0200] In addition, the lane line detection device can detect multiple lane line points of the front view image according to the above transformation (S404).

[0201] In addition, the lane line detection device can detect the lane line based on the multiple lane line points detected in the driving image (S405).

[0202] In addition, the lane line detection device 10 can be implemented by a module of an electronic device that outputs various guidance information for assisting the driver's driving, and performs a route guidance function. Figures 15 to 22 Further details are given.

[0203] Fig.15 is a block diagram showing an electronic device according to an embodiment of the present invention. Fig.15 The electronic device 100 includes all or part of a storage unit 110 , an input unit 120 , an output unit 130 , a lane line detection unit 140 , an augmented reality providing unit 160 , a control unit 170 , a communication unit 180 , a sensing unit 190 , and a power supply unit 195 .

[0204] Among them, the electronic device 100 can be embodied by a variety of devices such as a smart phone, a tablet computer, a laptop computer, a personal digital assistant (PDA), a portable multimedia player (PMP), smart glass, engineering glass, navigation, a car dash cam and a car video recorder that provide driving-related guidance to the driver of the vehicle, and can be installed in the vehicle.

[0205] Driving-related guidance may include various guidance such as route guidance, lane line departure guidance, lane line maintaining guidance, front vehicle departure guidance, signal light change guidance, front vehicle collision prevention guidance, lane line change guidance, lane line guidance, curve guidance, etc. to assist vehicle drivers in driving.

[0206] Among them, route guidance may include: augmented reality route guidance, which performs route guidance by combining the image in front of the moving vehicle with various information such as the user's location and direction; and two-dimensional or three-dimensional route guidance, which performs route guidance by combining two-dimensional (2D, 2-Dimensional) or three-dimensional (3D, 3-Dimensional) map data with various information such as the user's location and direction.

[0207] Furthermore, the route guidance may include an aerial map route guidance that performs route guidance by combining the aerial map data with various information such as the user's location, direction, etc. The route guidance includes route guidance when the user is driving in a vehicle and when the user is walking or running.

[0208] In addition, lane line departure guidance can be used to guide whether a moving vehicle deviates from a lane line.

[0209] In addition, lane keeping guidance can be used to guide the vehicle back to the lane it was originally traveling in.

[0210] Furthermore, the front vehicle departure guidance may be used to guide whether the vehicle in front of the parked vehicle should depart.

[0211] And, the signal light change guidance can be used to guide whether the signal light in front of the vehicle being parked changes the signal. As an example, if the green signal light representing the departure signal is changed under the state of the red signal light representing the stop signal, it is guided.

[0212] Furthermore, the front vehicle collision prevention guidance provides guidance to prevent a collision with the front vehicle if the distance to the vehicle in front of the parked or moving vehicle is within a predetermined distance.

[0213] The lane change guidance is to guide the vehicle to change its lane from the lane it is in to another lane in order to guide the route to the destination.

[0214] And, lane guidance can be used to guide the lane where the current vehicle is located.

[0215] Furthermore, the curve guidance may be used to guide the vehicle to a curve on a road after a predetermined time.

[0216] As described above, driving-related images such as the front image of the vehicle that can provide various guidance can be captured by a camera installed in the vehicle or a camera of a smart phone. Among them, the camera can be a camera that is integrated with the electronic device 100 installed in the vehicle to capture the front of the vehicle.

[0217] As another example, the camera may be a camera that is separately installed in the vehicle with the electronic device 100 to shoot the front of the vehicle. In this case, the camera may be another vehicle image shooting device that is installed toward the front of the vehicle, and the electronic device 100 may receive the captured image through wired or wireless communication with the separately installed vehicle image shooting device, or if a storage medium for storing the captured image of the vehicle image shooting device is inserted into the electronic device 100, the electronic device 100 may receive the captured image.

[0218] Based on the above content, the electronic device 100 according to an embodiment of the present invention is described in more detail below.

[0219] The storage unit 110 is used to store various data and applications required for the operation of the electronic device 100. In particular, the storage unit 110 can store data required for the operation of the electronic device 100, such as an operating system, a route search application, map data, etc. In addition, the storage unit 110 can store data generated by the operation of the electronic device 100, such as searched route data, received images, etc.

[0220] The storage unit 110 can be embodied by a built-in storage device such as a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electronically erasable and programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a memory card, a universal subscriber identity module (USIM), etc., and can also be embodied by a removable storage device such as a USB memory.

[0221] The input unit 120 has a function of converting a physical input from the outside of the electronic device 100 into a specific electrical signal. The input unit 120 may include all or part of the user input unit 121 and the microphone unit 123 .

[0222] The user input unit 121 can receive user input such as touch, push action, etc. The user input unit 121 can be implemented by at least one of various button forms, a touch sensor for receiving touch input, and a proximity sensor for receiving proximity motion.

[0223] The microphone unit 123 can receive the user's voice and sounds generated inside and outside the vehicle.

[0224] The output unit 130 is a device that outputs data of the electronic device 100 to the user in the form of images and / or voices. The output unit 130 may include all or part of the display unit 131 and the audio output unit 133 .

[0225] The display unit 131 is a device that outputs visually recognizable data to the user. The display unit 131 can be embodied by a display unit disposed on the front of the housing of the electronic device 100. In addition, the display unit 131 is formed as a whole with the electronic device 100 and can output visually recognizable data, such as a head up display (Head Up Display), which is separately provided with the electronic device 100 and can output visually recognizable data.

[0226] The audio output unit 133 is a device that outputs data that can be auditorily recognized by the electronic device 100. The audio output unit 133 may be implemented by a speaker that expresses the data to be notified to the user of the electronic device 100 in the form of sound.

[0227] The lane detection unit 140 can perform the functions of the lane detection device 10 described above.

[0228] The augmented reality providing unit 160 may provide an augmented reality visual mode. Augmented reality is a method of visually superimposing additional information (e.g., graphic elements indicating points of interest (POI), graphic elements indicating the risk of collision with the vehicle ahead, graphic elements indicating the distance between vehicles, graphic elements indicating curves, and various additional information for assisting the driver in driving safely) on a screen that displays the real world actually viewed by the user.

[0229] The augmented reality providing unit 160 may include all or part of a calibration unit, a three-dimensional space generating unit, an object generating unit, and a matching unit.

[0230] The calibration unit may perform calibration of camera parameters corresponding to the camera from the captured image captured by the camera. The camera parameters are parameters constituting the camera matrix as information representing the relationship between the real space image and the photo, and may include camera extrinsic parameters and camera intrinsic parameters.

[0231] The three-dimensional space generation unit may generate the virtual three-dimensional space based on the captured image captured by the camera. Specifically, the three-dimensional space generation unit may apply the camera parameters estimated by the calibration unit to the two-dimensional captured image to generate the virtual three-dimensional space.

[0232] The object generation unit may generate objects for guidance in augmented reality, such as a forward vehicle collision prevention guidance object, a route guidance object, a lane line change guidance object, a lane line departure guidance object, a curve guidance object, etc. At this time, the object generation unit may generate a plurality of objects for guidance in augmented reality based on the lane line detected by the lane line detection unit 140 .

[0233] Specifically, the object generation unit may generate a route guidance object for guiding the route to the destination based on the lane line detected by the lane line detection unit 140. Fig.16 This is explained in more detail.

[0234] Fig.16 FIG. 2 is a diagram for explaining a method for generating a route guidance object according to an embodiment of the present invention. Fig.16, the object generation unit can detect multiple points constituting the left lane line 3601 detected by the lane line detection unit 140. Moreover, the object generation unit can detect multiple points constituting the right lane line 3602 detected by the lane line detection unit 140. Among them, the detected multiple points can be a curve equation formed by calculating the multiple lane line points detected in the overhead image in the lane line point detection unit 16, and the detected multiple lane line points are fitted into a curve based on the calculated curve equation, and the fitted lane line points are transformed into the viewpoints of the front view image to match the lane line points of the front view image.

[0235] In addition, the object generation unit may determine a plurality of virtual points located between the plurality of points of the left lane line 3601 and the plurality of points of the right lane line 3602, and generate a virtual guide line 3603 connecting the plurality of virtual points. In this case, the object generation unit may generate the virtual guide line 3603 by calculating a curve equation formed by the plurality of virtual points and fitting the plurality of lane line points detected based on the calculated curve equation into a curve.

[0236] In addition, the object generation unit may generate a route guidance object 3604 having its center at the virtual guide line 3603. As an example, the generated route guidance object 3604 has an arrow shape, and the arrow may indicate a route that the vehicle should travel.

[0237] However, this is only one embodiment of the present invention, and the route guidance object of the present invention is not limited thereto. According to another embodiment of the present invention, the object generation unit can generate a route guidance object arranged on the lane where the vehicle must travel, in the left lane line and the right lane line, by using only the multiple points constituting the detected left lane line 3601 and the multiple points constituting the detected right lane line 3602 without calculating the virtual guide line 3603.

[0238] Moreover, the object generation unit may generate a lane line departure guidance object based on the lane line detected by the lane line detection unit 140. Specifically, when the moving vehicle departs from the lane line, the object generation unit may use the lane line detected by the lane line detection unit 140 to generate a lane line departure guidance object arranged on the lane line from which the vehicle departs. That is, the lane line departure guidance object generated by the object generation unit may be generated in the same shape as the lane line detected by the lane line detection unit 140 (that is, the lane line curve-fitted by the lane line detection unit 140).

[0239] On the one hand, the communication unit 180 can be used to enable the electronic device 100 to communicate with other devices. The communication unit 180 can include all or part of the location data unit 181, the wireless network unit 183, the broadcast transceiver unit 185, the mobile communication unit 186, the short-range communication unit 187, and the wired communication unit 189.

[0240] The location data unit 181 is a device for acquiring location data through a global navigation satellite system (GNSS). The global navigation satellite system is a navigation system that uses radio wave signals received from artificial satellites to calculate the location of a receiving terminal. As specific examples of global navigation satellite systems, according to their operating systems, there can be cited the Global Positioning System (GPS), Galileo, GLONASS, COMPASS, IRNSS, and QZSS. The location data unit 181 of the electronic device 100 according to an embodiment of the present invention can acquire location data by receiving a global navigation satellite system signal that provides services in the area where the electronic device 100 is used. Alternatively, in addition to the global navigation satellite system, the location data unit 181 can acquire location data by communicating with a base station or an access point (AP).

[0241] The wireless network unit 183 connects to a wireless network to obtain or send data. The wireless network unit 183 can connect to the network through a variety of communication protocols defined as wireless data transmission and reception for wireless local area network (WLAN), wireless broadband (Wibro), world interoperability for microwave access (Wimax), and high speed downlink packet access (HSDPA).

[0242] The broadcast transceiver 185 transmits and receives broadcast signals through various broadcast systems. The broadcast systems that can be transmitted and received by the broadcast transceiver 185 may be terrestrial digital multimedia broadcasting (DMBT), digital multimedia broadcasting satellite (DMBS), forward link media (MediaFLO), handheld digital video broadcast (DVBH), terrestrial integrated services digital broadcast (ISDBT), etc. The broadcast signals transmitted and received by the broadcast transceiver 185 may include traffic data, life data, etc.

[0243] The mobile communication unit 186 can connect to a mobile network to perform voice and data communications according to various mobile communication standards such as the third generation (3G), the third generation partnership project (3GPP), and the long term evolution (LTE).

[0244] The short-distance communication unit 187 is a device for short-distance communication. As described above, the short-distance communication unit 187 can communicate through Bluetooth (Bluetooth), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra WideBand (UWB), ZigBee, Near Field Communication (NFC), Wireless-Fidelity (Wi-Fi), etc.

[0245] The wired communication unit 189 is an interface device that can connect the electronic device 100 to other devices by wire. The wired communication unit 189 can be a USB module that communicates through a USB port.

[0246] As described above, the communication unit 180 may communicate with other devices using at least one of the location data unit 181 , the wireless network unit 183 , the broadcast transceiver unit 185 , the mobile communication unit 186 , the short-range communication unit 187 , and the wired communication unit 189 .

[0247] As an example, when the electronic device 100 does not include a camera function, at least one of the short-range communication unit 187 and the wired communication unit 189 can be used to receive images captured by a vehicle imaging device such as a car dash cam or a car video recorder.

[0248] As another example, when communicating with a plurality of devices, one may communicate via the short-range communication unit 187 , and another may communicate via the wired communication unit 119 .

[0249] The sensing portion 190 is a device that can sense the current state of the electronic device 100. The sensing portion 190 may include all or part of the motion sensing portion 191 and the light sensing portion 193.

[0250] The motion sensing unit 191 can detect the motion of the electronic device 100 in the three-dimensional space. The motion sensing unit 191 can include a three-axis geomagnetic sensor and a three-axis acceleration sensor. The motion data obtained by the motion sensing unit 191 is combined with the position data obtained by the position data unit 181 to more accurately calculate the trajectory of the vehicle to which the electronic device 100 is attached.

[0251] The light sensor 193 is a device for measuring the ambient illuminance of the electronic device 100. The illuminance data acquired by the light sensor 193 can be used to change the brightness of the display 131 to correspond to the ambient brightness.

[0252] The power supply unit 195 is a device that supplies power required for the operation of the electronic device 100 or the operation of other devices connected to the electronic device 100. The power supply unit 195 may be a device that receives power from a battery built into the electronic device 100 or an external power source such as a vehicle. In addition, the power supply unit 195 is embodied by the wired communication module 119 or by a wireless receiving device according to the form of receiving power.

[0253] The control unit 170 controls the overall operation of the electronic device 100. Specifically, the control unit 170 may control all or part of the storage unit 110, the input unit 120, the output unit 130, the lane detection unit 140, the augmented reality providing unit 160, the communication unit 180, the sensing unit 190, and the power supply unit 195.

[0254] Fig.17 FIG. 1 is a diagram for explaining a system network connected to an electronic device according to an embodiment of the present invention. Fig.17According to an embodiment of the present invention, the electronic device 100 can be embodied by various devices installed in the vehicle, such as a navigation device, a vehicle image capture device, a smart phone or other vehicle augmented reality interface providing device, and can be connected to a variety of communication networks and other electronic devices 61, 62, 63, 64.

[0255] Furthermore, the electronic device 100 may be linked with the GPG module according to the radio wave signal received from the artificial satellite 70 to calculate the current location and the current time period.

[0256] Each artificial satellite 70 can transmit L-band frequencies of different frequency bands. The electronic device 100 calculates the current position based on the time required for the L-band frequencies transmitted from each artificial satellite 70 to reach the electronic device 100.

[0257] On the other hand, the electronic device 100 can be wirelessly connected to the network 90 through the communication unit 180, the control station 80 (ACR), the base station 85 (RAS), the access point (AP, Access Point), etc. If the electronic device 100 is connected to the network 90, it is also indirectly connected to other electronic devices 61, 62 connected to the network 90 to exchange data.

[0258] On the other hand, the electronic device 100 can be indirectly connected to the network 90 through other devices 63 with communication functions. For example, if the electronic device 100 does not have a module that can connect to the network 90, it can communicate with other devices 63 with communication functions through a short-range communication module or the like.

[0259] Fig.18 FIG. 2 is a diagram showing an augmented reality guide screen displaying a route guide object according to an embodiment of the present invention. Fig.18 The augmented reality providing unit 160 can generate route guidance objects 3701, 3702, 3703 that guide the route from the current position of the vehicle to the destination, and generate an augmented display image that displays the generated route guidance objects 3701, 3702, 3703 on the lane of the driving image captured based on the lane line detected by the lane line detection unit 140 and displays them on a plane.

[0260] That is, the display unit 131 may display an arrow-shaped first route guidance object 3701 in the augmented reality image, wherein the first route guidance object 3701 has a virtual guide line pointing to the route that the vehicle should travel at the center of the two lane lines detected by the lane line detection unit 140. The virtual guide line may be a virtual line connecting multiple virtual points located in the middle of multiple points of the left lane line and multiple points of the right lane line.

[0261] In addition, the display unit 131 may display a second route guidance object 3702 and a third route guidance object 3703 on the left and right sides of the first route guidance object 3701, respectively, and the second route guidance object 3702 and the third route guidance object 3703 may be displayed on the driving lane of the vehicle without departing from the left and right lane lines detected by the lane line detection unit 140. The second route guidance object 3702 and the third route guidance object 3703 may be carpet-shaped lane recognition objects displayed only on the driving lane of the vehicle.

[0262] In addition, the augmented reality image according to an embodiment of the present invention may include an area 3704 for displaying the location of the vehicle on the map. In addition, the augmented reality image according to an embodiment of the present invention may also include driving information of the vehicle, for example, the driving speed of the vehicle, the speed limit of the road on which the vehicle is traveling, the current time, the distance to the destination, the remaining time, the direction in which the vehicle needs to travel at the U-turn point, and an area 3705 indicating the distance to the U-turn point.

[0263] Fig.19 FIG. 2 is a diagram showing an augmented reality guidance screen showing a lane line deviation guidance object according to an embodiment of the present invention. Fig.19 The augmented reality providing unit 160 generates a lane line departure guidance object 3801 for guiding the vehicle to depart from the lane line, and the generated lane line departure guidance object 3801 can generate an augmented reality image displayed on the lane line based on the lane line detected by the lane line detecting unit 140.

[0264] As an example, Fig.19 As shown, when the vehicle deviates from the left lane line, the display unit 131 displays a left lane line deviation guide object 3801 having the same shape as the lane line detected by the lane line detection unit 140 (i.e., the lane line curve fitted by the lane line detection unit 140), and can display an augmented reality image of the generated left lane line deviation guide object 3801 on the lane line detected by the lane line detection unit 140.

[0265] Fig. 20 FIG. 2 is a diagram showing an embodiment of an electronic device according to an embodiment of the present invention without a camera unit. Fig. 20 The vehicle image capturing device 200 which is separately provided with the electronic device 100 can constitute an electronic device according to an embodiment of the present invention through wired or wireless communication.

[0266] The electronic device 100 may include a display portion 131 , a user input portion 121 , and a microphone 123 disposed on the front of a housing 191 .

[0267] The vehicle imaging device 200 may include a camera 222 , a microphone 224 , and an attachment portion 281 .

[0268] Fig.21 FIG. 2 is a diagram showing an embodiment of an electronic device including a camera unit according to an embodiment of the present invention. Fig.21 In the case where the electronic device 100 includes the camera 150, the camera 150 of the electronic device 100 captures the front of the vehicle and allows the user to recognize the display portion of the electronic device 100. Thus, an electronic device according to an embodiment of the present invention can be embodied.

[0269] Fig. 22 FIG. 2 is a diagram showing an embodiment of a head-up display (Head-Up Display) according to an embodiment of the present invention. Fig. 22 , the head-up display can display an augmented reality guidance screen on the head-up display through wired and wireless communication with other devices.

[0270] As an example, augmented reality can be provided by using a head-up display on the front glass of the vehicle or by using an image overlap of an additional image output device, and as described above, the augmented reality providing unit 160 can generate an interface image overlapped on the augmented reality image or glass, etc. Thus, augmented reality navigation or vehicle infotainment system, etc. can be embodied.

[0271] In addition, the lane line detection device 10 can be implemented as a module of a system for automatic driving to perform a route guidance function. Figure 23 to Figure 24 This is explained in more detail.

[0272] Reference Fig.23 According to this embodiment, the automatic driving vehicle 2000 may include a control device 2100, sensing modules 2004a, 2004b, 2004c, 2004d, an engine 2006 and a user interface 2008.

[0273] The self-driving vehicle 2000 may have an automatic driving mode or a manual mode. As an example, according to a user input received through the user interface 2008, the manual mode may be switched to the automatic driving mode, or the automatic driving mode may be switched to the manual mode.

[0274] When the vehicle 2000 is operated in the automatic driving mode, the automatic driving vehicle 2000 may be operated under the control of the control device 2100 .

[0275] In this embodiment, the control device 2100 may include a controller 2120 having a memory 2122 and a processor 2124 , a sensor 2110 , a communication device 2130 , and an object detection device 2140 .

[0276] In the present embodiment, the object detection device 2140 is a device for detecting an object located outside the vehicle 2000. The object detection device 2140 detects the object located outside the vehicle 2000 and can generate object information according to the detection result.

[0277] The object information may include information on the presence or absence of the object, position information of the object, distance information between the vehicle and the object, and relative speed information between the vehicle and the object.

[0278] The object may include various objects located outside the vehicle 2000, such as lane lines, other vehicles, pedestrians, traffic signals, lights, roads, structures, speed bumps, terrain objects, and animals. Here, the traffic signal may be a concept including a traffic light, a traffic sign, and a pattern or text drawn on the road surface. In addition, the light may be light generated from lights of other vehicles, light generated from street lamps, or sunlight.

[0279] In addition, structures can be objects located around the road and fixed on the ground. For example, structures can include street lights, street trees, buildings, telephone poles, traffic lights, and bridges. Terrain objects can include mountains, hillsides, etc.

[0280] The object detecting device 2140 may include a camera module. The controller 2120 may extract object information from an external image photographed by the camera module and cause the controller 2120 to process the information.

[0281] In addition, the object detection device 2140 may also include an imaging device for identifying the external environment. In addition to LIDAR, RADAR, GPS equipment, odometer (Odometry) and other computer vision equipment, ultrasonic sensors, infrared sensors, etc. may also be used, and these devices may be selected or started simultaneously as needed for more precise detection.

[0282] In addition, the sensor 2110 can be connected to the sensing modules 2004a, 2004b, 2004c, and 2004d to obtain various sensing information of the internal / external environment of the vehicle. Among them, the sensor 2110 may include a posture sensor (for example, a yaw sensor, a roll sensor, a pitch sensor, a collision sensor, a wheel sensor, a speed sensor, a tilt sensor, a weight sensor, a heading sensor, a gyro sensor, a positioning module, a vehicle forward / backward sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor rotated by a steering wheel, a vehicle internal temperature sensor, a vehicle internal humidity sensor, an ultrasonic sensor, an illumination sensor, an accelerator pedal position sensor, a brake pedal position sensor, etc.

[0283] Thus, sensor 2110 can obtain sensing signals of vehicle posture information, vehicle collision information, vehicle direction information, vehicle position information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle tilt information, vehicle forward / backward information, battery information, fuel information, tire information, headlight information, in-vehicle temperature information, in-vehicle humidity information, steering wheel rotation angle, outside vehicle illumination, pressure applied to the accelerator pedal, pressure applied to the brake pedal, and the like.

[0284] In addition, sensor 2110 may also include an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake air temperature sensor (ATS), a water temperature sensor (WTS), a throttle position sensor (TPS), a TDC sensor, a crank angle sensor (CAS), etc.

[0285] In this way, the sensor 2110 can generate vehicle status information based on the sensing data.

[0286] The wireless communication device 2130 is used to implement wireless communication between the automatic driving vehicles 2000. For example, the user's mobile phone, other wireless communication devices 2130, other vehicles, central devices (traffic control devices), servers, etc. can communicate with the automatic driving vehicle 2000. The wireless communication device 2130 can send and receive wireless signals according to the connection wireless protocol. The wireless communication protocol can be Wireless Fidelity (Wi-Fi), Bluetooth (Bluetooth), Long-Term Evolution (Long-Term Evolution, LTE), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (WCDMA), Global Systems for Mobile Communications (Global Systems for Mobile Communications, GSM), but the communication protocol is not limited to this.

[0287] Furthermore, in the present embodiment, the automatic driving vehicle 2000 can realize communication between vehicles through the wireless communication device 2130. That is, the wireless communication device 2130 can communicate with other vehicles on the road and multiple other vehicles through vehicle-to-vehicle (V2V) communication. The automatic driving vehicle 2000 can send and receive information such as driving warnings and traffic information through communication between vehicles, and can also request information from other vehicles or receive information from other vehicles. For example, the wireless communication device 2130 can perform vehicle-to-vehicle communication through a dedicated short-range communication (DSRC) device or a cellular vehicle-to-vehicle communication (C-V2V) device. In addition, in addition to communication between vehicles, communication (V2X, Vehicle to Everything communication) between vehicles and other things (for example, electronic devices carried by pedestrians, etc.) can also be performed through the wireless communication device 2130.

[0288] In this embodiment, the controller 2120 is a unit that controls the overall operation of each unit in the vehicle 2000, and is manufactured by the vehicle manufacturing company or is additionally configured after manufacturing in order to perform the function of automatic driving. Alternatively, it may include a structure that continuously performs additional functions by updating the controller 2120 configured during manufacturing. The controller 2120 may also be named ECU (Electronic Control Unit).

[0289] The controller 2120 collects various data from the connected sensors 2110, the object detection device 2140, the communication device 2130, etc., and can transmit control signals based on the collected data to the sensors 2110, the engine 2006, the user interface 2008, the communication device 2130, and the object detection device 2140 included as other structures in the vehicle. In addition, although not shown in the figure, control signals can also be transmitted to the acceleration device, the braking system, the steering device, or the navigation device related to the driving of the vehicle.

[0290] In this embodiment, the controller 2120 can control the engine 2006. For example, the automatic vehicle 2000 senses the speed limit of the road on which it is traveling and controls the engine 2006 to prevent the driving speed from exceeding the speed limit, and can control the engine 2006 to accelerate the driving speed of the automatic vehicle 2000 within a range that does not exceed the speed limit.

[0291] In addition, when the automatic driving vehicle 2000 approaches or leaves the lane line while the automatic driving vehicle 2000 is driving, the controller 2120 can determine whether the approach and departure of the lane line are caused by normal driving conditions or other driving conditions, and can control the engine 2006 to control the driving of the vehicle according to the judgment result. Specifically, the automatic driving vehicle 2000 can be equipped with a lane line detection device 10 according to an embodiment of the present invention, and the lane line detection device 10 can detect the lane lines formed on both sides of the lane in which the vehicle is driving. In this case, the controller 2120 determines whether the automatic driving vehicle 2000 approaches or leaves the lane line. If it is determined that the automatic driving vehicle 2000 approaches or leaves the lane line, it can be determined whether it is caused by normal driving conditions or other driving conditions. Among them, as an example of a normal driving condition, there will be a situation where the vehicle needs to change lanes. In addition, as an example of other driving conditions, there will be a situation where the lane line of the vehicle does not need to be changed. When the vehicle does not need to change lanes, if the controller 2120 determines that the vehicle is approaching or leaving the lane line, the automatic driving vehicle 2000 can be controlled to drive normally without leaving the lane line.

[0292] The lane line detection device 10 may be constituted by a module in the control device 2100 of the automatic driving vehicle 2000. That is, the memory 2122 and the processor 2124 of the control device 2100 may implement the lane line detection method according to the present invention through software.

[0293] In the case where there are other vehicles or obstacles in front of the vehicle, the controller 2120 can control the engine 2106 or the brake system to slow down the vehicle. In addition to the speed, the track, the route, and the steering angle can be controlled. The controller 2120 can generate the required control signal and control the vehicle's travel based on the recognition information of other external environments such as the vehicle's driving lane line and driving signals.

[0294] In addition to generating the self-control signal, the controller 2120 communicates with surrounding vehicles or a central server and transmits instructions for controlling surrounding devices through the received information, thereby controlling the driving of the vehicle.

[0295] Furthermore, in the case where the position of the camera module 2150 is changed or the shooting angle is changed, it is difficult for the controller 2120 to accurately identify the vehicle or the lane line according to the present embodiment. Therefore, in order to prevent the above phenomenon, a control signal is generated in a manner of performing calibration of the camera module 2150. Therefore, in the present embodiment, the controller 2120 generates a calibration control signal through the camera module 2150, thereby, even if the installation position of the camera module 2150 is changed due to vibration or impact, etc., which occurs with the movement of the automatic driving vehicle 2000, the normal installation position, direction, shooting angle, etc. of the camera module 2150 can be continuously maintained. When the initial installation position, direction, shooting angle information of the camera module 2120 stored in advance and the initial installation position, direction, shooting angle information of the camera module 2120 measured during the driving process of the automatic driving vehicle 2000 become above a critical value, the controller 2120 can generate a control signal to perform calibration of the camera module 2120.

[0296] In this embodiment, the controller 2120 may include a memory 2122 and a processor 2124. The processor 2124 may execute software stored in the memory 2122 according to a control signal of the controller 2120. Specifically, the controller 2120 stores data and instructions for executing the lane line detection method according to the present invention in the memory 2122. In order to implement one or more methods described herein, the instructions may be executed by the processor 2124.

[0297] Here, the memory 2122 may be stored in a recording medium that can be executed in the non-volatile processor 2124. The memory 2122 may store software and data through appropriate internal and external devices. The memory 2122 may be composed of a random access memory (RAM), a read only memory (ROM), a hard disk, and a memory 2122 device connected to a software protector.

[0298] The memory 2122 can at least store an operating system (OS), user applications, and executable instructions. The memory 2122 can also store application data and arrange data structures.

[0299] Processor 2124 is a microprocessor or a suitable electronic processor, and may also be a controller, a microcontroller or a state machine.

[0300] The processor 2124 may be embodied by a combination of computing devices, which may be constituted by a digital signal processor, a microprocessor, or a suitable combination of these.

[0301] In addition, the automatic driving vehicle 2000 may also include a user interface 2008 for the user to input information to the control device 2100. The user interface 2008 may allow the user to input information through appropriate interaction. For example, it may be embodied by a touch screen, a keyboard, an operation button, etc. The user interface 2008 transmits the input or instruction to the controller 2120, and the controller 2120 may execute the control action of the vehicle in response to the input or instruction.

[0302] Furthermore, the user interface 2008, as an external device of the automatic driving vehicle 2000, can communicate with the automatic driving vehicle 2000 through the wireless communication device 2130. For example, the user interface 2008 can establish linkage with a mobile phone, a tablet computer or other computer devices.

[0303] Furthermore, in the present embodiment, the case where the self-driving vehicle 2000 includes the engine 2006 is described, but other types of propulsion systems may also be included. For example, the vehicle may be operated by electric energy, hydrogen energy, or a hybrid system combining these. Therefore, the controller 2120 includes a propulsion mechanism based on the propulsion system of the self-driving vehicle 2000, and the control signal may be provided to the structure of each propulsion mechanism.

[0304] Below, refer to Fig.24 The detailed structure of the control device 2100 that executes the lane line display method according to this embodiment is described in more detail.

[0305] The control device 2100 includes a processor 2124. The processor 2124 may be a common single or multi-core microprocessor, a dedicated microprocessor, a microcontroller, a programmable gate array, etc. The processor may be referred to as a central processing unit (CPU). In addition, in this embodiment, the processor 2124 may be used in combination with multiple processors.

[0306] The control device 2100 further includes a memory 2122. The memory 2122 may be any electronic component that stores electronic information. In addition to a single memory, the memory 2122 may also include a combination of multiple memories 2122.

[0307] Data and instruction language 2122a for executing the lane line detection method according to the present invention may also be stored in the memory 2122. When the processor 2124 executes a plurality of instruction language 2122a, all or part of the instruction language 2122a and the data 2122b required for executing the instruction may be loaded into 2124a, 2124b.

[0308] The control device 2100 may include a transmitter 2130a, a receiver 2130b, or a transceiver 2130c for transmitting and receiving signals. One or more antennas 2132a, 2132b may be electrically connected to the transmitter 2130a, the receiver 2130b, or each transceiver 2130c, and multiple antennas may be additionally included.

[0309] The control device 2100 may further include a digital signal processor (DSP) 2170. The vehicle may process a digital signal more quickly through the digital signal processor 2170.

[0310] The control device 2100 may further include a communication interface 2180. The communication interface 2180 may include one or more ports and / or communication modules for connecting other devices to the control device 2100. The communication interface 2180 may interact with the control device 2100 of the user.

[0311] The various structures of the control device 2100 may be connected via one or more buses 2190 , which may include a power bus, a control signal bus, a status signal bus, a data bus, etc. Under the control of the processor 2124 , the multiple structures may communicate information to each other via the bus 2190 and perform the intended functions.

[0312] On the one hand, the terms "first", "second", "third" and "fourth" in the specification and claims are used, if recorded, to distinguish between similar constituent elements, and are not necessarily used to describe a specific order or order of occurrence. Terms used in the same manner should be understood to be interchangeable under appropriate circumstances so that an embodiment of the present invention recorded herein can be executed in other orders not shown or described herein, for example. Similarly, when it is recorded herein that a method includes a series of steps, the order of the steps disclosed herein is not necessarily the order in which the steps are executed, and any recorded steps can be omitted and / or any other steps not recorded herein can be added to the method. For example, without departing from the scope of the present invention, the first constituent element can be named the second constituent element, and similarly, the second constituent element can be named the first constituent element.

[0313] In addition, the terms "left side", "right side", "front", "rear", "top", "bottom", "above", "below", etc. in the specification and claims are used for explanation purposes and do not necessarily describe unchanged relative positions. Terms used in the same manner should be understood to be interchangeable under appropriate circumstances so that an embodiment of the present invention described herein can, for example, move in other directions not shown or described herein. The term "connected" used herein is defined as being directly or indirectly connected by electrical or non-electrical means. Objects described herein as being "adjacent" to each other may appropriately be objects that are in physical contact with each other or adjacent to each other or in the same general range or area as appropriate, depending on the article in which the phrase is used. The phrases "in one embodiment" present herein refer to the same embodiment, but are not certain.

[0314] In addition, in the specification and claims, various variations of expressions such as "connected", "connected", "coupled", "coupled", "combined", and "combined" are used to mean direct connection with other components or indirect connection through other components.

[0315] In contrast, when it is mentioned that a certain structural element is “directly connected to” or “directly coupled to” another structural element, it should be understood that there are no other structural elements intervening.

[0316] In addition, the suffixes "module" and "unit" of the constituent elements used in this specification are given or used interchangeably only for the convenience of writing the specification, and they themselves do not have mutually distinguishing meanings or functions.

[0317] In addition, the terms used in this specification are intended to illustrate the embodiments and are not intended to limit the present invention. In this specification, when not specifically mentioned in the text, the singular includes the plural. The terms "consisting of" or "including" in the specification shall not be interpreted as necessarily including the various constituent elements or various steps mentioned in the specification, may not include some of the constituent elements or steps, or should be interpreted as also including other additional constituent elements or steps.

[0318] The present invention has been described above with its preferred embodiment as the center. All embodiments and conditional examples disclosed in this specification are recorded by those skilled in the art to facilitate readers to understand the principles and concepts of the present invention. Those skilled in the art should understand that the present invention can be embodied in a deformed form within the scope of the essential characteristics of the present invention.

[0319] Therefore, the disclosed embodiments should be considered in an illustrative rather than a limiting sense. The scope of the present invention is described in the scope of the patent claims rather than the description, and all differences within the same scope should be construed as being included in the present invention.

[0320] On the one hand, the lane line detection method according to various embodiments of the present invention can be embodied by a program and provided to a server or device. Thus, each device is connected to a server or device storing the program, so that the program can be downloaded.

[0321] In addition, the control methods according to various embodiments of the present invention described above can be embodied by programs and stored in various non-transitory computer readable media (non-transitory computer readable medium) for provision. Non-transitory computer readable media refers to a medium that stores data semi-permanently and can be read by a device rather than a medium that stores data for a short period of time such as a register, cache, memory, etc. Specifically, the various applications or programs described above can be stored in non-transitory readable media such as CD, DVD, hard disk, Blu-ray disc, USB, memory card, ROM, etc. for provision. In addition, the preferred embodiments of the present invention are shown and described above, but the present invention is not limited to the above-mentioned specific embodiments. Those skilled in the art can perform various modified implementations within the spirit of the present invention as requested by the scope of the claims. Moreover, the modified implementation should not be understood separately from the technical idea or prospect of the present invention.

Claims

1. A lane line detection method, which is a lane line detection method using a processor, characterized in that: include: The step of acquiring a driving image captured by a camera of a moving vehicle; A step of detecting edges corresponding to lane lines in the acquired driving image, and generating an edge image based on the detected edges; A step of detecting straight line components based on the detected edge, and generating an edge straight line processed image based on the detected straight line components; A step of detecting lane line points corresponding to the lane line using the generated edge image and the edge line processed image; as well as The step of detecting the lane line according to the detected lane line point.

Images