Method and device for determining road markings

Abstract

A method and device for determining road markings are provided. The method includes: performing detection of two road markings of a lane in which a vehicle is traveling from a driving image; determining whether at least one of the two road markings is not detected; if it is determined that at least one road marking is not detected, determining at least one road marking for the at least one undetected road marking based on first information related to a drivable road area including the road; and controlling the driving of the vehicle based on the at least one determined road marking.

Description

[0001] This application claims the priority benefit of Korean Patent Application No. 10-2018-0100318 filed in the Korean Intellectual Property Office on August 27, 2018, the disclosure of which is hereby incorporated by reference in its entirety. Technical Field

[0002] Apparatuses and methods consistent with exemplary embodiments of the present inventive concept relate to determining road markings in a drivable road area. Background Art

[0003] Visual information enhancement technology is provided to assist the driving of vehicles and other vehicles. In such technology, various methods are used to extract lane markings or road information from driving images.

[0004] For example, it is difficult to detect road markings from images when there is insufficient pixel information about long-distance images, when lane markings are obscured by various objects in the road environment, when road markings are not well detected due to shadows during driving on the shoulder of the road, when road markings are worn, or when road markings are unclearly merged or separated.

[0005] When accurate lane detection is not performed, difficulties may arise in providing accurate information for driving of the vehicle (such as vehicle control, route determination, etc.). Summary of the invention

[0006] The present inventive concept provides various exemplary methods and apparatuses for determining road markings for undetected road markings of a road in consideration of a drivable road area when a vehicle is driven along the road.

[0007] According to an exemplary embodiment, a method for determining road markings of a road along which a vehicle is traveling is provided. The method may include: performing detection of two road markings of a lane in which the vehicle is traveling from a traveling image; determining whether at least one of the two road markings is not detected; if it is determined that at least one road marking is not detected, determining at least one road marking for the at least one undetected road marking based on first information related to a drivable road area including the road; and controlling the traveling of the vehicle based on the at least one determined road marking.

[0008] According to an exemplary embodiment, a device for determining road markings of a road along which a vehicle is traveling is provided. The device may include: a sensor configured to acquire a driving image; a processor configured to: perform detection of two road markings of a lane in which the vehicle is traveling from the driving image; determine whether at least one of the two road markings is not detected; if it is determined that at least one road marking is not detected, determine at least one road marking for the at least one undetected road marking based on first information related to a drivable road area; and control the driving of the vehicle based on the at least one determined road marking.

[0009] According to an exemplary embodiment, a method for controlling driving of a vehicle is provided, the method may include: determining whether at least one road marking is not detected when driving the vehicle; if it is determined that at least one road marking is not detected when driving the vehicle, predicting at least one road marking for the at least one undetected road marking using information about the road obtained from a driving image and at least one of pre-stored information related to the road, and correcting the at least one predicted road marking to be included in a drivable road area; determining the at least one corrected road marking as the at least one undetected road marking; and controlling driving of the vehicle based on the at least one determined road marking, wherein the drivable road area is determined based on pixel values ​​of a driving image captured by a sensor installed in the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1A and Figure 1B An example of a problem that occurs when road markings are determined without considering a drivable road area is shown.

[0011] Figure 2 An example of a method of determining road markings is shown.

[0012] Figure 3A and Figure 3B An example of a method of acquiring the first information is shown.

[0013] Figure 4 An example of a method of determining an undetected road marking when one of two road markings is not detected is shown.

[0014] Figure 5 An example of a method of determining an undetected road marking when one of two road markings is detected is shown.

[0015] Figure 6 An example of a method of determining an undetected road marking when two road markings are not detected is shown.

[0016] Figures 7 to 9 An example of a method of determining two undetected road markings is shown.

[0017] FIG. 10A to FIG. 10C An example of a method of acquiring the second information is shown.

[0018] Fig.11 An example of a method of determining two road markings of a lane in which a vehicle is traveling from an image is shown.

[0019] Fig.12 An example of a method of calculating the offset parameter and the curvature parameter of the first reticle regression function is shown.

[0020] Fig.13 and Fig.14 An example of a method of determining road markings is shown.

[0021] Fig.15 An example of a device for determining road markings is shown.

[0022] Throughout the drawings and detailed description, unless otherwise described or provided, the same reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be drawn to scale, and the relative sizes, proportions, and depictions of the elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0023] The following detailed description of various embodiments is provided to help the reader gain a comprehensive understanding of the inventive concept. However, from these descriptions, various permutations, modifications, and equivalents of the embodiments will be apparent. For example, the order of operations described herein is merely exemplary and is not limited to the order set forth herein, but may be changed as clearly described except for operations that must occur in a particular order. In addition, for greater clarity and simplicity, descriptions of features known in the art may be omitted. The embodiments presented are exemplary.

[0024] Although terms such as "first", "second", and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions are not limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion referred to may also be referred to as the second member, second component, second region, second layer, or second portion.

[0025] The terms used herein are only used to describe various examples and are not intended to limit the present disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0026] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs. Unless explicitly defined as such herein, terms (such as those defined in general dictionaries) will be interpreted as having a meaning consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal manner.

[0027] The examples set forth below can be used to display virtual routes or generate visual information to assist the driving of an autonomous vehicle in an augmented reality (AR) navigation system of an intelligent vehicle. These examples can be used to interpret visual information and assist safe and pleasant driving in a device including an intelligent system (such as a head-up display (HUD) installed for driving assistance or fully autonomous driving of a vehicle). These examples can be applied to, for example, autonomous vehicles, intelligent vehicles, smart phones, and mobile devices. Hereinafter, these examples will be described in detail with reference to the accompanying drawings, wherein the same reference numerals are used for the same elements.

[0028] Figure 1A and Figure 1B An example of a problem that occurs when road markings are determined without taking into account the drivable road area is shown. Figure 1A An image 110 showing a road including a lane in which a vehicle is traveling (hereinafter also referred to as a “travel lane”), Figure 1B A diagram 130 is shown which displays drivable areas 133 and non-drivable areas 136 on a road.

[0029] For example, when the weather is cloudy, when lane recognition is not possible on the road ahead due to snow, rain, fog, etc., when vehicles are complexly entangled ahead, when road markings are obscured by shadows and cannot be clearly recognized, when road markings are damaged by cars and become invisible, and when parts of road markings are missing, it may be difficult or impossible to identify the road ahead. Figure 1A The image 110 shown in FIG.

[0030] When no road markings are detected, they can typically be determined by assuming the width of the current driving lane and mirroring the road markings detected from one side to the road markings on the other side or considering the most recently detected road markings, e.g., virtual markings estimated from the last markings.

[0031] However, if Figure 1B As shown in the diagram 130 of FIG. 1 , the actual road area 139 ahead of the vehicle may be different from the drivable area 133 in which the vehicle is traveling. When the actual road area 139 is different from the drivable area 133 and when the drivable area 133 is not considered when determining the road markings, the road markings determined for the road ahead may be generated in the non-drivable area 136 outside the driving lane, which may cause a serious accident.

[0032] By considering the shape of the driving lane and / or the drivable area 133 (or the non-drivable area 136) when determining the road markings, the road markings can be stably detected even when a portion of the road markings is missing or is obscured by shadows and is not visible. In addition, the accuracy of the road marking determination can be improved.

[0033] Hereinafter, a drivable area refers to a road area designated as a place where vehicles are allowed to travel under applicable laws or regulations and some areas of pedestrian paths where vehicles are allowed to enter and leave, and / or an area where vehicles are customarily allowed to travel even if not permitted by such laws or regulations (e.g., an open space next to a road). A drivable area may also be referred to as a drivable road area. Hereinafter, the term "drivable road" and the term "drivable road area" are understood to have the same meaning.

[0034] A non-drivable area refers to an area where vehicle driving is not permitted by applicable laws or regulations, and / or an area where a vehicle cannot be driven due to natural terrain (such as mountains, valleys, waterfalls, oceans, etc.).

[0035] Figure 2 An example of a method for determining road markings is shown. Figure 2In operation 210, the apparatus for determining road markings according to an embodiment performs detection of two road markings of a lane in which the vehicle is traveling from an image. Here, the image is, for example, a driving image acquired using a capture device mounted on the vehicle. The driving image may include a single driving image or multiple driving images. The driving image includes multiple frames. The capture device may be fixed to a predetermined area of ​​the vehicle (such as a windshield, a dashboard, a rearview mirror, etc.) to capture a driving image around the vehicle (for example, in front of the vehicle). The capture device includes, for example, a visual sensor, an image sensor, or a device performing a similar function. The capture device captures a single image and also captures an image of each frame as the case may be. The driving image may also be an image captured by a device other than the detection device. For example, the driving image may be Figure 3A Image 310.

[0036] In operation 210, the apparatus may perform detection of two road markings of a lane in which the vehicle is traveling based on, for example, a neural network, wherein the neural network is trained to identify a road marking area including the road markings. The convolutional neural network is trained to verify, for example, a bounding box of a road marking to be detected from a driving image and a type of road marking to be detected. Fig.11 and Fig.12 An example of a method of performing detection of two road markings of a lane using the apparatus is further described.

[0037] In operation 220, the device determines whether at least one of the two road markings is not detected. For example, in operation 220, the device may determine whether two road markings are detected, whether at least one of the two road markings is detected, or whether two road markings are not detected.

[0038] In operation 230, based on the determination result of operation 220, the apparatus determines at least one road marking for at least one undetected road marking using the first information related to the drivable road area. For example, the first information is mapping information indicating the probability that a pixel in the image corresponds to the drivable road area. The mapping information indicating the probability that a pixel in the image corresponds to the drivable road area may also be referred to as road probability mapping information. The first information is obtained by segmenting the image of the driving lane. Referring to Figure 3A and Figure 3B An example of a method of acquiring first information using the device is further described.

[0039] The device determines at least one road marking for at least one undetected road marking by generating a virtual road marking based on, for example, road marking information detected so far, road marking information currently detected, and a drivable road area currently detected. In addition, the device corrects a detected road marking on one side of two road markings (hereinafter, also referred to as "one marking") based on the first information, and determines a road marking on the other side corresponding to the one side (hereinafter, also referred to as "another marking") based on the corrected one marking.

[0040] Will refer to Figure 4 and Figure 5 Further described is a method for using the device to determine the other road marking when one of the two road markings is detected. Figures 6 to 9 A method of using the apparatus to determine two road markings when the two road markings are not detected is further described.

[0041] The apparatus controls the travel of the vehicle based on the road marking determined for at least one undetected road marking in operation 230. For example, the apparatus generates control parameters for controlling a steering wheel, a brake, and an acceleration / deceleration pedal of the vehicle based on the road marking determined for at least one undetected road marking in operation 230, and controls the travel of the vehicle using the control parameters.

[0042] In addition, the device is provided in a display device (e.g. Fig.15 At least one undetected road marking determined in operation 230 is displayed on a display 1550).

[0043] Here, the term "vehicle" refers to a vehicle traveling on a road, and includes, for example, an automatic or automated driving vehicle and an intelligent or smart vehicle equipped with an advanced driver assistance system (ADAS). The term "road" refers to a road on which a vehicle travels, and includes various types of roads (such as, for example, expressways, national roads, local roads, expressways, and intercity highways). A road includes one or more lanes. The term "lane" refers to a road space distinguished by markings marked on the surface of the road. A single lane is distinguished by a left marking and a right marking or its boundary line. The term "marking" refers to road markings and includes various types of markings, for example, solid lines, dotted lines, curves, and zigzag lines marked in white, blue, or yellow on the surface of the road. The markings may correspond to the markings on one side that distinguish a single lane, or may correspond to a pair of lines (for example, the left and right markings that distinguish a single lane).

[0044] Figure 3A and Figure 3B An example of a method of acquiring first information for predicting a road marking for an undetected road marking is shown. Figure 3AA driving image 310 is shown, Figure 3B Shown with Figure 3A The segmented image 330 corresponds to the driving image 310 .

[0045] The driving image 310 includes a road image and a road surface image, wherein the road surface image includes, for example, other vehicles, road markings, curbs, sidewalks, and surroundings. The driving image 310 may include a road marking area including vehicles and markings and a surrounding area including buildings, trees, and the sky.

[0046] The apparatus for determining road markings according to an embodiment separates objects from the driving image 310, and determines what the separated objects (e.g., road markings or roads) refer to in units of pixels, thereby generating a segmented image 330. Objects are divided in units of classes. The classes used here may include various semantic units (e.g., roads, road markings, vehicles, sidewalks, people, animals, skies, and buildings).

[0047] The device generates a segmented image 330 through a classification network, wherein the classification network includes a convolutional layer and a fully connected layer of several stages. The device uses a classifier model trained to output a training output from a training image to segment the driving image 310 into a plurality of regions. For example, the classifier model is a convolutional neural network. The training image can be a color image, and the training output can indicate a regional image obtained by segmenting the training image (e.g., training input). For example, the training output can be a regional image pre-segmented based on attributes or classes (e.g., road markings, roads, vehicles, people, objects, and background) corresponding to each pixel of the training image and manually specified.

[0048] The device uses a well-known classification network (such as AlexNet, VGGNet, and GoogleNET) to separate objects from the driving image 310 in semantic units, obtains the semantics of each segmented area in pixel units, and labels each segmented area for each class, thereby generating a segmented image 330. For example, the device operates to distinguish between pixels belonging to a class corresponding to a drivable road and pixels not belonging to a class corresponding to a drivable road, rather than classifying the classes of all pixels.

[0049] The device obtains first information including a probability that a pixel corresponds to a class (e.g., a road, a road marking, and a drivable area on a sidewalk) previously classified as a drivable area 335 in the segmented image 330. According to an example, the device obtains first information including information related to pixels having a probability greater than or equal to a preset threshold. For example, when a probability of 0.9 corresponding to a road class or a road marking class has a probability value greater than a preset threshold of 0.758, the device obtains a probability value of 0.9 of an area corresponding to the corresponding classification (road class) as the first information.

[0050] Figure 4 An example of a method of determining a road marking for an undetected road marking when one of two road markings is not detected is shown. Figure 4 A diagram 400 showing a non-drivable area 410 , a drivable area 430 , a left marking line 450 detected from a driving image, and a right marking line 470 predicted and corrected from the left marking line 450 is shown.

[0051] When the driving image (such as Figure 3A When a left marking line 450 is detected from a driving image 310 of a vehicle, the apparatus for determining road marking lines according to an embodiment predicts the shape of a right marking line based on the detected left marking line 450. For example, when the detected left marking line 450 has an S shape, the apparatus predicts that the right marking line has an S shape.

[0052] The device corrects the position of the predicted right line marking so that the corrected right line marking is within the drivable area 430 instead of within the non-drivable area 410. The device determines the corrected right line marking as the right line marking 470, and controls the vehicle to travel along the detected left line marking 450 and the corrected right line marking 470.

[0053] Figure 5 An example of a method of determining a road marking for an undetected road marking when one of two road markings is detected is shown. Figure 5 Shown in Figure 2 In operation 230 of the present invention, when it is determined that one of the two road markings is detected and the other road marking is not detected, a method for determining a road marking for the undetected road marking is provided.

[0054] In operation 510, the device for determining road markings according to an embodiment generates a first marking regression function corresponding to a detected marking. The device selects a plurality of path points on a map that are within a predetermined distance of the current position of the vehicle, and calculates a first marking regression function representing the shape of a detected marking of the road on the map by performing a linear regression analysis on the path points. The current position of the vehicle is measured using, for example, a global positioning system (GPS) sensor, an accelerometer sensor, and a compass sensor included in the device. For example, map information includes a map that uses symbols, characters, and images to represent geographic features on the surface, map coordinates including the latitude, longitude, and altitude of the geographic features, and a plurality of path points represented on a path on the map. Here, the plurality of path points are a collection of coordinates obtained at preset intervals when the vehicle is traveling on a single lane on the road. The map information corresponds to map information of a general map level or map information of a general navigation level. The map information is stored in a memory of the device (e.g., Fig.15 1530), or through a communication interface of the device (e.g., Fig.15 The communication interface 1540) receives from a source external to the device.

[0055] In operation 520, the device predicts a second line marking regression function corresponding to the other line marking of the two road markings based on the first line marking regression function. The device searches for first information associated with a drivable road area corresponding to the other line marking based on the first line marking regression function, and predicts the second line marking regression function based on the first information having a value higher than a preset threshold value among the searched first information. In addition, the device predicts the second line marking regression function corresponding to the other line marking (virtual line) using an equation obtained by adding s times (s is a predetermined number, for example, 2 times) the width of the previous driving lane as an offset to the first line marking regression function.

[0056] In operation 530 , the apparatus corrects the second line marking regression function based on the first information so that another line marking is included in the drivable road area.

[0057] In operation 540 , the apparatus determines another road marking based on the second road marking regression function, that is, determines a road marking for an undetected road marking.

[0058] Figure 6 An example of a method of determining a road marking for an undetected road marking when two road markings are not detected is shown. Figure 6 A diagram 600 is shown showing a non-drivable area 610 , a drivable area 630 , and right and left marking lines 650 , 670 predicted and corrected as described below.

[0059] When two road markings are not shown in the driving image (such as Figure 3AWhen a driving image 310 of the vehicle is detected, the device for determining road markings according to the embodiment predicts the shapes of the right marking and the left marking based on the shape of the drivable area 630. The device predicts that the right marking has a shape that bends from a straight line to the left based on the shape of the drivable area 630. The device predicts the shape of the left marking by generating a left marking at a left position corresponding to a predetermined width distance based on the predicted right marking. The device corrects the positions of the predicted right marking and the left marking so that the predicted right marking and the left marking are included in the drivable area 630. The device controls the vehicle to travel along the corrected right marking 650 and left marking 670.

[0060] When two road markings are not detected from the driving image, the apparatus predicts the road markings on one side based on the boundary between the drivable area 630 and the non-drivable area 610 , and predicts the road markings on the other side based on the predicted road markings on one side.

[0061] The device generates a road marking regression function based on the road marking information saved for each frame until the road marking is recognized from the driving image.

[0062] The device predicts the position of a marking line adjacent to the non-drivable area 610 using a pixel having a value higher than a preset threshold value among a plurality of pixels included in the first information corresponding to a marking line adjacent to the non-drivable area 610 (e.g., a right marking line) and a marking line regression function. The device adjusts or moves the marking line regression function so that the marking line regression function is included in the region of pixels having a value higher than the preset threshold value and is not included in the non-drivable area 610. In this way, the device predicts a marking line (e.g., a right marking line adjacent to the non-drivable area 610).

[0063] The device predicts the position of another marking line corresponding to one marking line adjacent to the non-drivable area 610 based on the position of one marking line adjacent to the non-drivable area 610. For example, the device predicts the left position at a certain distance (or a predetermined lane width) from the position of the right marking line adjacent to the non-drivable area 610 as the position of the other marking line.

[0064] Will refer to Figures 7 to 9 The description further describes an example of using the apparatus to determine road markings for two undetected road markings.

[0065] Figure 7 An example of a method of determining road markings for two undetected road markings is shown. Figure 7 Shows that when Figure 2 In operation 230 of the present invention, when two road markings are not detected, a method for determining road markings for two undetected road markings using a boundary of a road area is provided.

[0066] When two road markings are not detected, the apparatus for determining road markings according to an embodiment predicts one of the two road markings based on first information, and predicts another road marking corresponding to the one road marking based on the one road marking.

[0067] In operation 710, the device searches for a boundary of a drivable road area from the first information. Figure 6 The device searches for a non-drivable area from the first information, and searches for a boundary of a drivable area based on the non-drivable area.

[0068] In operation 720 , the apparatus generates a first road marking regression function corresponding to a road marking on one side based on a boundary of the drivable area.

[0069] In operation 730, the apparatus determines a road marking on one side using a first marking regression function. Since the description of operations 520 to 540 is also applicable to operations 740 to 760, repeated description will be omitted.

[0070] Figure 8 An example of a method of determining road markings for two undetected road markings is shown. Figure 8 Shows that when Figure 2 In operation 230 of the present invention, when two road markings are not detected, a method for determining a road marking for an undetected road marking using a previously traveled road is provided.

[0071] According to an embodiment, a device for determining a road marking predicts one of two road markings based on pre-stored information associated with the shape of the one road marking, and corrects the one road marking based on first information. The device determines the one of the two road markings, and then determines another road marking corresponding to the one road marking based on the one road marking.

[0072] In operation 810, the device generates a first line regression function corresponding to one of two road line markings based on pre-stored information associated with the shape of the one line marking. The device generates an extension line extending from the line marking on which the vehicle has traveled, and calculates a line regression function corresponding to the extension line, thereby calculating the first line regression function corresponding to the one line marking.

[0073] In operation 820, the device corrects the first marking regression function based on the first information so that the one marking is included in the drivable road area. The device corrects the first marking regression function to be included in the drivable road area in an area or position corresponding to a pixel having a value higher than a preset threshold among a plurality of pixels included in the first information.

[0074] In operation 830, the apparatus determines the one reticle using the first reticle regression function corrected in operation 820. Since the description of operations 520 to 540 is also applicable to operations 840 to 860, a repeated description will be omitted.

[0075] Fig. 9 An example of a method of determining road markings for two undetected road markings is shown. Fig. 9 A method of determining a road marking for an undetected road marking by setting a search range when at least one road marking is not detected in operation 230 is shown.

[0076] In operation 910, the apparatus for determining road markings according to an embodiment acquires second information about the current driving lane. The apparatus acquires the second information by generating an extension line extending from the markings of the lane in which the vehicle has been driving. The apparatus determines the second information based on any one or any combination of, for example, information about road markings detected from a driving image, information about the shape of a previously driven road, information about the shape of a drivable road detected from the first information, and path point information corresponding to the current position of the vehicle. An example of a method of determining the second information using the apparatus will be further described with reference to FIG. 10.

[0077] In operation 920, the device sets a search range in the first information based on the second information. The device sets an offset range indicating the search range based on a first road marking regression function generated according to the second information. Here, the search range includes pixels corresponding to at least one candidate road marking among a plurality of pixels included in the first information.

[0078] In operation 930, the device generates at least one virtual road marking of at least one undetected road marking by performing a search on the first information within the search range. The device extracts pixels having a value higher than a preset threshold value from among a plurality of pixels included in the search range. The device generates at least one virtual road marking by generating a second marking regression function based on the extracted pixels.

[0079] In operation 940 , the apparatus determines a road marking for at least one undetected road marking based on the at least one virtual road marking generated in operation 930 .

[0080] FIG. 10A to FIG. 10C An example of a method of acquiring the second information is shown. Fig. 10A Road marking information related to road markings recognized from the driving image is shown.

[0081] The apparatus for determining a road marking according to an embodiment acquires road marking information related to a road marking 1010 from a driving image using a neural network, wherein the neural network is trained to recognize a road marking area including a road marking from the driving image. The neural network distinguishes between the road marking information related to the road marking 1010 and an object 1020 (such as a vehicle that is not the road marking 1010), and verifies the type of the road marking 1010. The apparatus acquires second information related to the shape of a lane (e.g., the position and shape of the left and right markings of the lane and the width of the lane) from the road marking information related to the road marking 1010.

[0082] Fig. 10B Waypoint information 1040 related to waypoints W1, W2, W3, W4, W5, W6, and W7 acquired from a map 1030 is shown. The apparatus for determining road markings according to an embodiment determines or predicts a marking regression function (e.g., a linear equation, a quadratic equation, a cubic equation, or a curve (clothoid) equation) corresponding to a road on which the vehicle is traveling based on map information corresponding to the position of the vehicle, and detects road markings from a driving image using the marking regression function.

[0083] The apparatus regresses waypoint information 1040 of waypoints W1, W2, W3, W4, W5, W6, and W7 extracted from a map 1030 based on the current position of the vehicle, thereby recognizing that the road is located in a curved shape or an S shape.

[0084] The device determines a line regression function suitable for expressing the shape of the road (e.g., an S-curve shape). The device converts the quadratic equation (e.g., c 2 x 2 +c 1 x+c 0 ) is determined as the line regression function, and the cubic equation (e.g., c 3 x 3 +c 2 x 2 +c 1 x+c 0 ) is determined as the line regression function.

[0085] The device assigns a greater regression weight to the reticle regression function of waypoint W3 than to the reticle regression function of waypoint W1 so that the curve is properly expressed.

[0086] The device identifies the shape of a road on the map 1030 based on the waypoint information, and determines a line marking regression function corresponding to the lane based on the shape of the road. The device assigns a regression weight corresponding to the lane by, for example, performing a regression analysis on the waypoint information. The device determines second information associated with the shape of the lane using the line marking regression function corresponding to the lane.

[0087] In addition, the device determines second information with increased accuracy and speed by detecting lanes using a line marking regression function and a regression weight so that the shape of the road indicated by the waypoint information is most appropriately expressed, wherein the line marking regression function considers the number of times determined for a predetermined interval of the waypoint information or each coordinate point of the waypoint information.

[0088] Fig. 10C Virtual road markings or extensions 1060 are shown that are determined based on markings 1050 (eg, shape information) of a lane in which the vehicle has traveled.

[0089] The device for determining a road marking generates an extension line extending from the road marking along which the vehicle has traveled to obtain second information. The device determines a partial path corresponding to the marking line 1050 of the lane along which the vehicle has traveled and extends the marking line of the lane based on the partial path, thereby generating an extension line 1060. The device sets a search range in the first information so that the extension line 1060 is included in the drivable road area. The device extracts pixels having a value higher than a preset threshold value from among a plurality of pixels included in the search range, and determines the second information using a regression function generated based on the extracted pixels.

[0090] Fig.11 An example of performing a method of detecting two road markings of a lane in which a vehicle is traveling from a driving image is shown. Fig.12 An example of a method of obtaining the offset parameter and the curvature parameter of the first reticle regression function is shown.

[0091] Reference Fig.11 and Fig.12 In operation 1110, the apparatus for determining road markings acquires an image of a lane in which a vehicle is traveling from a driving image.

[0092] In operation 1120, the device obtains Fig.121210 and the path point information corresponding to the current position of the vehicle 1230. In this example, the path point information corresponds to a portion of points on a road on a map corresponding to a road 1250 on which the vehicle 1230 is traveling, and / or the coordinates of a portion of points. The device uses a plurality of path points to determine curvature information related to the road on which the vehicle 1230 is traveling, wherein the plurality of path points are located within a path point on the map corresponding to a look-ahead distance from a path point adjacent to the current position of the vehicle 1230. The term "look-ahead distance" refers to a distance required for information on the travel of the vehicle 1230, and is determined based on any one or any combination of, for example, the speed of the vehicle 1230, the visible distance, the weather, the driving conditions, and the road conditions of the vehicle 1230. In addition, the look-ahead distance corresponds to a distance appropriately expressing a plurality of path points based on the degree of a line regression function (e.g., a polynomial equation).

[0093] Two road markings may be detected based on road marking information detected from the driving image including the lane image acquired in operation 1110 and the waypoint information acquired in operation 1120. In operation 1130, the apparatus obtains a curvature parameter of a first marking regression function corresponding to a road marking on one side based on the waypoint information.

[0094] The device calculates a first line regression function (eg, c 2 x 2 +c 1 x), and use the first line regression function to determine the curvature information C of the road 1250 1 and C 2 .

[0095] In operation 1140, the apparatus obtains an offset parameter of a first marking regression function based on road marking information detected from the driving image. The apparatus detects a marking area including road markings from the driving image, and estimates a lateral offset C of the vehicle 1230 using a reference line based on curvature information and the marking area. 0 . When the curvature information is 0, the reference line may be a straight line. When the curvature information is greater than 0, the reference line may be a curve. For example, the curve includes a curve having a curvature in one direction, a curve having a curvature in at least two directions, a curve corresponding to at least a portion of a circle, and a curve corresponding to at least a portion of an ellipse. The lateral offset of the vehicle 1230 corresponds to the distance from the center line of the road 1250 to the center of the vehicle 1230.

[0096] In operation 1150, the device detects two road markings of the lane using the curvature parameter of the first line marking regression function and the offset parameter of the first line marking regression function. The device obtains a line marking regression function (such as, c 2 x 2 +c 1 x+c 0 ). The device detects two road markings of a lane using a lane width obtained by multiplying an offset in an acquired marking regression function by, for example, 2.

[0097] In addition to the examples described above, various schemes for detecting road markings may be used. When at least one road marking is not detected, the road marking may be determined for the undetected road marking based on the above examples.

[0098] Fig.13 An example of a method for determining road markings is shown. Fig.13 In operation 1350, the device for determining road markings acquires a driving image. In operation 1310, the device extracts the probability of each pixel from the driving image and generates a segmented image. In this example, the device generates the segmented image by distinguishing objects or road marking areas from the driving image through pixel-level classification. In operation 1315, the device converts the segmented image with the distinguished road markings into a top view.

[0099] In operation 1320, the device identifies the current position of the vehicle based on, for example, GPS information and map information. In operation 1325, the device searches for a plurality of waypoints located within a waypoint corresponding to a forward-looking distance from the current position of the vehicle identified in operation 1320. In operation 1330, the device calculates a map pattern using the plurality of waypoints found by searching in operation 1325.

[0100] In operation 1335 , the apparatus determines whether a road marking of a lane in which the vehicle is traveling is detected using the top view generated in operation 1315 and the map pattern calculated in operation 1330 .

[0101] When a road marking is detected in operation 1335, the apparatus determines a polynomial equation corresponding to the detected road marking in operation 1345. For example, the polynomial equation corresponding to the detected road marking is c 2 x 2 +c 1 x+c 0 .like Fig.12 As shown in FIG. 1 , the device moves the reference line in units of pixels (eg, a single pixel) while calculating the curvature information C 1 and C2 The apparatus detects the road markings based on the calculated fitting scores and determines two road markings adjacent to the lateral position of the vehicle. The apparatus detects the road markings based on the calculated fitting scores and determines two road markings adjacent to the lateral position of the vehicle. The apparatus detects the road markings based on the lateral offset C of the road previously acquired. 0 and curvature information C 1 With C 2 A polynomial equation is determined. The determined polynomial equation may correspond to a line-marking regression function representing a shape of a road corresponding to a driving path of the vehicle and a position of the vehicle in a driving lane.

[0102] In operation 1350 , the apparatus determines a road marking using the polynomial equation determined in operation 1345 .

[0103] In addition, the device generates various control parameters for controlling the travel of the vehicle (eg, a steering wheel, brake, acceleration / deceleration pedal of the vehicle) and outputs the control parameters using the polynomial equation determined in operation 1345. The device determines the path of the vehicle using a line regression function.

[0104] When the road marking is not detected in operation 1335, the apparatus re-detects the road marking that failed to be detected in operation 1335 from the road probability mapping information indicating the probability that the pixel in the driving image corresponds to the drivable road area in operation 1340. The aforementioned example for predicting the road marking for the undetected road marking may be applied to operation 1340.

[0105] In operation 1350 , the apparatus determines the road markings re-detected from the road probability mapping information as final road markings.

[0106] In addition to autonomous driving systems, in robot systems that require precise control in various industrial environments (such as smart factories) and indoor robot applications with map information, the method of providing information for vehicle driving can also be applied to provide control information and detect control lines for path control of the robot.

[0107] Fig.14 An example of a method for determining road markings is shown. Fig.14 In operation 1410, the apparatus for determining road markings recognizes a driving image based on a camera. In operation 1420, the apparatus acquires road probability mapping information from the recognized image.

[0108] In operation 1430 , the apparatus detects road markings based on the road probability map information.

[0109] In operation 1440, the apparatus determines whether road markings on both sides are detected from the lane in which the vehicle is traveling. When it is determined in operation 1440 that road markings on both sides are detected, in operation 1480 the apparatus maintains the road markings so that the vehicle travels in the lane.

[0110] When it is determined in operation 1440 that both road markings are not detected, the apparatus determines whether a road marking on one side is detected in operation 1450. When it is determined in operation 1450 that a road marking on one side is detected, the apparatus generates a virtual line on the other side corresponding to the detected road marking on one side in operation 1460.

[0111] When it is determined in operation 1450 that the road markings on either side are not detected, in operation 1470 , the apparatus determines the road markings on both sides using a road probability map based on pre-stored road marking information.

[0112] Although not shown, an embodiment of determining a road marking for an undetected road marking based on information associated with a drivable road area may be applied to operations 1460 and 1470 .

[0113] Fig.15 An example of an apparatus for determining road markings is shown. Fig.15 , the device 1500 for determining road markings includes a sensor 1510 and a processor 1520. The device 1500 also includes a memory 1530, a communication interface 1540, and a display 1550. The sensor 1510, the processor 1520, the memory 1530, the communication interface 1540, and the display 1550 communicate with each other via a communication bus 1505.

[0114] The sensor 1510 includes, for example, an image sensor, an accelerometer sensor, a compass sensor, a GPS sensor, a gyroscope sensor, an odometer, and a geomagnetic sensor. The sensor 1510 acquires positioning information indicating the position of the vehicle. In addition, the sensor 1510 captures a driving image of the vehicle.

[0115] The processor 1520 detects two road markings of a lane in which the vehicle is traveling from an image (such as, but not limited to, a driving image). The processor 1520 determines a road marking for at least one undetected road marking using first information associated with a drivable road area based on a result obtained by determining whether at least one of the two road markings is not detected.

[0116] The device 1500 receives sensing information through the communication interface 1540. In one example, the communication interface 1540 receives sensing information from other sensors external to the device 1500. In another example, the communication interface 1540 receives driving images, map information, and GPS information from a source external to the device 1500.

[0117] The processor 1520 outputs information for driving of the vehicle through the communication interface 1540. The information for driving of the vehicle includes control parameters for controlling any one or any combination of a driving direction, an angle of a steering wheel, and a speed of the vehicle based on a local path of the vehicle.

[0118] The processor 1520 executes the operation with reference to FIGS. 1 to 1 Fig.14 At least one of the described methods or an algorithm corresponding to at least one of the methods.

[0119] As used herein, the term "processor" is a hardware-implemented data processing device having physically constructed circuits to perform desired operations. For example, the desired operations include codes or instructions contained in a program. Hardware-implemented data processing devices include, but are not limited to, for example, microprocessors, central processing units (CPUs), processor cores, multi-core processors, multiprocessors, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0120] The processor 1520 executes a program and controls the device 1500. Codes of the program executed by the processor 1520 are stored in the memory 1530.

[0121] The memory 1530 stores various images including driving images and map information. The memory 1530 stores various information generated during the processing operation of the processor 1520. In addition, the memory 1530 stores various data and programs. The memory 1530 includes a volatile memory or a non-volatile memory. The memory 1530 includes a large-capacity storage medium (such as a hard disk) to store various data.

[0122] The display 1550 displays the road markings and map information determined by the processor 1520 alone or together.

[0123] The equipment, unit, module, device and other components described herein are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include: controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer can be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a limited manner to achieve the desired result). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) to perform the operations described in this application. Hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, although the singular term "processor" or "computer" can be used in the description of the examples described in this application, in other examples, multiple processors or computers can be used, or a processor or computer can include multiple processing elements or multiple types of processing elements or apply the above two forms. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can implement a single hardware component or two or more hardware components. Hardware components can have any one or more of different processing configurations, wherein the examples of different processing configurations include: a single processor, an independent processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, multiple instructions single data (MISD) multiprocessing, and multiple instructions multiple data (MIMD) multiprocessing.

[0124] The method for performing the operations described in the present application is performed by computing hardware (e.g., by one or more processors or computers), wherein the computing hardware is implemented as the above-described execution for performing the instructions or software for the operations performed by the method described in the present application. For example, a single operation or two or more operations can be performed by a single processor or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors, or a processor and a controller, and one or more other operations can be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can perform a single operation or two or more operations.

[0125] The instructions or software for controlling a processor or computer to implement a hardware component and perform the method as described above are written as a computer program, a code segment, an instruction, or any combination thereof, for individually or collectively indicating or configuring a processor or computer to operate as a machine or a special-purpose computer to perform the operations performed by the hardware component and the method as described above. In one example, the instructions or software include machine code (such as, machine code generated by a compiler) directly executed by a processor or computer. In another example, the instructions or software include high-level code executed by a processor or computer using an interpreter. A programmer of ordinary skill in the art can easily write instructions or software based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing the operations performed by the hardware component and the method as described above.

[0126] The instructions or software that controls the processor or computer to implement the hardware components and perform the methods described above, as well as any associated data, data files, and data structures, are recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as a multimedia mini card or card (for example, Secure Digital (SD) or Extreme Digital (XD)), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device configured to store instructions or software and any associated data, data files and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the instructions.

[0127] Although the present disclosure includes specific examples, it will be clear to those skilled in the art that various changes in form and detail may be made to these examples without departing from the spirit and scope of the inventive concept and its equivalents. The examples described herein are considered to be descriptive only and not for limiting purposes. The description of the features or aspects in each example will be considered to be applicable to similar features or aspects in other examples. If the described techniques are performed in a different order, and / or if the components in the described system, architecture, device, or circuit are combined in a different manner and / or the components in the described system, architecture, device, or circuit are replaced or supplemented with other components or their equivalents, appropriate results can be obtained. Therefore, the scope of the inventive concept is not limited by specific embodiments, but by the claims and their equivalents, and all changes within the scope of the inventive concept and its equivalents will be interpreted as included in the inventive concept.

Claims

1. A method for determining a road marking along a road along which a vehicle is traveling, the method comprising: Detecting two road markings of a lane in which the vehicle is traveling from a driving image; determining whether at least one of the two road markings is not detected; If it is determined that at least one road marking is not detected, determining at least one road marking for at least one undetected road marking based on first information related to a drivable road area including the road; Controlling the movement of the vehicle based on at least one determined road marking, The first information includes mapping information indicating the probability that a pixel in the driving image corresponds to a drivable road area, If it is determined that two road markings are not detected, the step of determining at least one road marking for the at least one undetected road marking includes: acquiring second information related to the shape of a lane in which the vehicle is traveling; Setting a search range in the first information based on the second information, wherein the step of setting the search range includes: setting an offset range indicating the search range based on a first road marking line regression function generated according to the second information, wherein the search range includes pixels corresponding to candidates of at least one candidate road marking line among a plurality of pixels included in the first information; By performing a search on the first information within a search range, at least one virtual road marking corresponding to the at least one undetected road marking is generated, wherein the step of generating the at least one virtual road marking comprises: extracting pixels having a probability value higher than a preset threshold from a plurality of pixels included in the search range, and generating the at least one virtual road marking by generating a second marking regression function based on the extracted pixels; For the at least one undetected road marking, the at least one virtual road marking is determined as the at least one road marking.

2. The method according to claim 1, wherein: If it is determined that one of the two road markings is detected and the other of the two road markings is not detected, the step of determining at least one road marking for the at least one undetected road marking comprises: predicting the other road marking based on the one road marking; The predicted road marking is corrected based on the first information.

3. The method according to claim 2, wherein: The step of predicting the other road marking includes: predicting a fourth marking regression function corresponding to the other road marking based on a third marking regression function corresponding to the one road marking, The step of correcting the other road marking includes: correcting a fourth road marking regression function so that the other road marking is included in the drivable road area.

4. The method according to claim 1, wherein: If it is determined that two road markings are not detected, the step of determining at least one road marking for the at least one undetected road marking comprises: predicting one of the two road markings based on the first information; The other road marking of the two road markings corresponding to the one road marking is predicted based on the predicted one road marking.

5. The method according to claim 4, wherein: The step of predicting the one road marking comprises: searching for a boundary of a drivable road area from the first information; A third road marking regression function corresponding to the one road marking is generated based on the boundary.

6. The method according to claim 5, wherein: The search steps include: searching for a non-drivable area based on the first information; Search boundaries based on no-drivable areas.

7. The method according to claim 1, wherein: If it is determined that the two road markings are not detected, the step of determining at least one road marking for the at least one undetected road marking includes: predicting one of the two road markings based on pre-stored information related to the shape of the road; The predicted road marking is corrected based on the first information.

8. The method according to claim 7, wherein: The step of predicting the one road marking includes: generating a third marking regression function corresponding to the one road marking based on pre-stored information, The step of correcting the predicted road markings includes: correcting a third marking regression function so that the predicted road markings are included in the drivable road area.

9. The method according to claim 1, wherein: The steps for detecting two road markings include: Acquire a lane image of a lane in which the vehicle is traveling from the driving image; Obtaining waypoint information corresponding to the current position of the vehicle; The two road markings are detected based on the road marking information and the waypoint information detected from the lane image.

10. The method according to claim 9, wherein: The step of detecting the two road markings comprises: Obtaining a curvature parameter of a third road marking regression function corresponding to one of the two road markings based on the path point information; An offset parameter of a third road marking regression function is obtained based on the road marking information.

11. The method according to claim 1, wherein: The step of acquiring the second information includes at least one of the following steps: determining second information based on road marking information related to the road; determining second information based on waypoint information corresponding to the current position of the vehicle; determining second information from shape information related to the road; The second information is determined based on shape information about the drivable road area detected from the first information.

12. The method according to claim 1, further comprising: The first information related to the drivable road area is acquired by segmenting the driving image based on the pixel values ​​of the driving image.

13. The method according to claim 1, further comprising: A map indicating the probability that pixels in the image of the lanes in the driving image correspond to the drivable road area is generated by segmenting the driving image.

14. The method according to claim 1, further comprising: The at least one determined road marking is displayed on a display installed in the vehicle.

15. A device for determining road markings along a road along which a vehicle is traveling, the device comprising: A sensor configured to acquire a driving image; The processor is configured as: Detecting two road markings of a lane in which the vehicle is traveling from a driving image; determining whether at least one of the two road markings is not detected; If it is determined that at least one road marking is not detected, determining at least one road marking for at least one undetected road marking based on first information related to the drivable road area; Controlling the movement of the vehicle based on at least one determined road marking, The first information includes mapping information indicating the probability that a pixel in the driving image corresponds to a drivable road area, If it is determined that two road markings are not detected, the step of determining at least one road marking for the at least one undetected road marking includes: acquiring second information related to the shape of a lane in which the vehicle is traveling; Setting a search range in the first information based on the second information, wherein the step of setting the search range includes: setting an offset range indicating the search range based on a first road marking line regression function generated according to the second information, wherein the search range includes pixels corresponding to candidates of at least one candidate road marking line among a plurality of pixels included in the first information; By performing a search on the first information within a search range, at least one virtual road marking corresponding to the at least one undetected road marking is generated, wherein the step of generating the at least one virtual road marking comprises: extracting pixels having a probability value higher than a preset threshold from a plurality of pixels included in the search range, and generating the at least one virtual road marking by generating a second marking regression function based on the extracted pixels; For the at least one undetected road marking, the at least one virtual road marking is determined as the at least one road marking.

16. The device according to claim 15, wherein: The processor is configured to: if it is determined that one of the two road markings is detected and the other of the two road markings is not detected, predict the other road marking based on the one road marking, and correct the predicted road marking based on the first information.

17. The apparatus according to claim 15, wherein: The processor is configured to: if it is determined that the two road markings are not detected, predict one of the two road markings based on the first information, and predict another of the two road markings corresponding to the one road marking based on the predicted one road marking.

18. A method for controlling the travel of a vehicle, the method comprising: determining whether at least one road marking is not detected while driving the vehicle; If it is determined that at least one road marking is not detected when the vehicle is driven, predicting at least one road marking for at least one undetected road marking using at least one of information about the road obtained from the driving image and pre-stored information about the road, and correcting the at least one predicted road marking to be included in a drivable road area based on first information, wherein the first information includes mapping information indicating a probability that a pixel in the driving image corresponds to the drivable road area; determining at least one corrected road marking as the at least one undetected road marking; Controlling the movement of the vehicle based on at least one determined road marking, The drivable road area is determined based on pixel values ​​of a driving image captured by a sensor installed in the vehicle. Wherein, if it is determined that two road markings are not detected when driving the vehicle, the step of predicting at least one road marking for the at least one undetected road marking comprises: acquiring second information related to the shape of a lane in which the vehicle is traveling; Setting a search range in the first information based on the second information, wherein the step of setting the search range includes: setting an offset range indicating the search range based on a first road marking line regression function generated according to the second information, wherein the search range includes pixels corresponding to candidates of at least one candidate road marking line among a plurality of pixels included in the first information; By performing a search on the first information within a search range, at least one virtual road marking corresponding to the at least one undetected road marking is generated, wherein the step of generating the at least one virtual road marking comprises: extracting pixels having a probability value higher than a preset threshold from a plurality of pixels included in the search range, and generating the at least one virtual road marking by generating a second marking regression function based on the extracted pixels; For the at least one undetected road marking, the at least one virtual road marking is determined as the at least one road marking.

19. The method according to claim 18, further comprising: If the probability value that a pixel of the driving image is a drivable road area is higher than a preset threshold, the pixel is determined to be a drivable road area.

Images