Apparatus and method for generating a lane polyline using a neural network model

A neural network model processes sensor data to generate accurate lane polylines for autonomous vehicles, addressing the challenge of complex road environments and enhancing safety through direct learning and control.

KR103045229B1Active Publication Date: 2026-11-1042DOT INC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
KR1020220077498
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-09-07
Filing Date
2022-06-24
Publication Date
2026-11-10
Estimated Expiration
2042-06-24

AI Technical Summary

Technical Problem

Existing lane detection technologies in autonomous vehicles face challenges in accurately generating lane polylines, which are crucial for safe driving, due to the complexity of road environments and the need for precise sensor data processing.

Method used

A neural network model is employed to generate lane polylines by acquiring base images from vehicle sensors, extracting multi-scale image features, transforming them into bird's-eye view features, and training the model using lane polylines as output data, eliminating the need for separate processing steps.

Benefits of technology

The method achieves high-accuracy lane polyline generation directly from bird's-eye view features through end-to-end learning, enabling precise vehicle control without additional processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112022066268017-PAT00040_ABST
    Figure 112022066268017-PAT00040_ABST
Patent Text Reader

Abstract

The present disclosure relates to a method and apparatus for generating a lane polyline using a neural network model. A method according to one embodiment of the present disclosure can train a neural network model by generating a bird's-eye view feature based on a basic image obtained from at least one sensor mounted on a vehicle, using the bird's-eye view feature as input data for the neural network model, and using a lane polyline for a predetermined road as output data. In the present disclosure, the lane polyline obtained from the above-described neural network model can be utilized for vehicle control without undergoing a separate processing step.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to an apparatus and method for generating a lane polyline using a neural network model. Background Technology

[0002] The smartification of vehicles is rapidly progressing due to the convergence of information and communication technology and the automotive industry. As a result of this smartification, vehicles are evolving from simple mechanical devices into smart cars, and autonomous driving is receiving particular attention as a core technology for smart cars. Autonomous driving is a technology that allows a vehicle to find its way to a destination on its own without the driver having to operate the steering wheel, accelerator pedal, or brakes.

[0003] Recently, due to technological advancements, safety driving assistance systems such as lane departure warning systems and lane keeping, as well as automatic vehicle control systems, have been developed and are being rapidly commercialized. In particular, lane detection is one of the core technologies for solving major challenges in autonomous vehicles, and extensive research is being actively conducted amidst international interest.

[0004] Since the detection of driving lanes has a significant impact on safe driving, various sensors are utilized to accurately detect driving lanes by estimating and determining the position of the lane. For example, various sensors such as image sensors, radar, or lidar are used individually or in a fused form to implement autonomous driving vehicle control systems for lane detection or object recognition in front of the vehicle.

[0005] The aforementioned background technology is technical information that the inventor possessed for the derivation of the present invention or acquired during the process of deriving the present invention, and it cannot be considered as prior art disclosed to the general public prior to the filing of the present invention. The problem to be solved

[0006] The present invention provides an apparatus and method for generating a lane polyline using a neural network model. The problems to be solved by the present invention are not limited to those mentioned above, and other problems and advantages of the present invention not mentioned can be understood from the following description and will be more clearly understood by the embodiments of the present invention. Furthermore, it will be understood that the problems and advantages to be solved by the present invention can be realized by the means and combinations thereof set forth in the claims. means of solving the problem

[0007] As a technical means for achieving the technical problem described above, the first aspect of the present disclosure provides a method for generating a lane polyline using a neural network model, comprising: acquiring a base image of a predetermined road obtained from at least one sensor mounted on a vehicle; extracting a multi-scale image feature using the base image; performing a view transformation on the extracted multi-scale image feature to generate a bird's-eye view feature (BEV feature); and training the neural network model by using the bird's-eye view feature as input data for the neural network model and using the lane polyline of the predetermined road as output data.

[0008] A second aspect of the present disclosure provides an apparatus for generating a lane polyline using a neural network model, comprising: a memory storing at least one program; and a processor for driving a neural network by executing the at least one program, wherein the processor acquires a base image of a predetermined road from a camera mounted on a vehicle, extracts a plurality of image features using the base image, performs a view transformation on the extracted plurality of image features to generate a bird's-eye view feature, uses the bird's-eye view feature as input data for a neural network model, and trains the neural network model by using the lane polyline of the predetermined road as output data.

[0009] A third aspect of the present disclosure may provide a computer-readable recording medium that records a program for executing a method according to a first aspect on a computer.

[0010] In addition to this, other methods for implementing the present invention, other systems, and computer-readable recording media storing a computer program for executing said methods may be further provided.

[0011] Other aspects, features, and advantages other than those described above will become clear from the following drawings, claims, and detailed description of the invention. Effects of the invention

[0012] According to the means for solving the problem of the present disclosure described above, the present disclosure can obtain a lane polyline for a predetermined road with high accuracy from a bird's-eye view feature through end-to-end learning of a neural network model.

[0013] In addition, the lane polyline obtained from the neural network model can be utilized for vehicle control without undergoing a separate processing step. Brief explanation of the drawing

[0014] FIGS. 1 to 3 are drawings for explaining an autonomous driving method according to one embodiment. FIG. 4 is an exemplary drawing for explaining a method of performing image encoding and view conversion according to one embodiment. FIG. 5 is an exemplary drawing for explaining a method of generating a bird's-eye view feature according to one embodiment. FIG. 6 is an exemplary diagram illustrating the operation method of a neural network model according to one embodiment. FIGS. 7a and 7b are exemplary drawings for illustrating a seed probability loss according to one embodiment. FIGS. 8a through 8c are exemplary drawings for illustrating embedding offset loss. FIG. 9 is an exemplary drawing for illustrating a sequence loss according to one embodiment. FIG. 10 is a flowchart illustrating the generation of a lane polyline using a neural network model according to one embodiment. FIG. 11 is a block diagram of a lane polyline generating device according to one embodiment. Specific details for implementing the invention

[0015] The advantages and features of the present invention, and the methods for achieving them, will become clear by referring to the embodiments described in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments presented below, but can be implemented in various different forms and should be understood to include all modifications, equivalents, and substitutions that fall within the spirit and scope of the present invention. The embodiments presented below are provided to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention. In describing the present invention, detailed descriptions of related known technologies are omitted if it is determined that such detailed descriptions may obscure the essence of the present invention.

[0016] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to specify the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0017] Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various numbers of hardware and / or software configurations that execute specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a specific function. Additionally, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented as algorithms executed on one or more processors. Furthermore, the present disclosure may employ prior art for electronic configuration, signal processing, and / or data processing, etc. Terms such as "mechanism," "element," "means," and "configuration" may be used broadly and are not limited to mechanical and physical configurations.

[0018] Furthermore, the connecting lines or connecting members between the components depicted in the drawings are merely illustrative of functional connections and / or physical or circuit connections. In the actual device, connections between components may be represented by various alternative or added functional connections, physical connections, or circuit connections.

[0019] In the following, 'vehicle' may mean any type of means of transportation used to move people or goods using an engine, such as automobiles, buses, motorcycles, kickboards, or trucks.

[0020] The present disclosure will be described in detail below with reference to the attached drawings.

[0021] FIGS. 1 to 3 are drawings for explaining an autonomous driving method according to one embodiment.

[0022] Referring to FIG. 1, an autonomous driving device according to one embodiment of the present invention can be mounted on a vehicle to implement an autonomous driving vehicle (10). The autonomous driving device mounted on the autonomous driving vehicle (10) may include various sensors for collecting surrounding situational information. For example, the autonomous driving device may detect the movement of a preceding vehicle (20) operating in front through an image sensor and / or event sensor mounted on the front of the autonomous driving vehicle (10). The autonomous driving device may further include sensors for detecting not only the front of the autonomous driving vehicle (10), but also other vehicles (30) operating in adjacent lanes and pedestrians around the autonomous driving vehicle (10).

[0023] At least one of the sensors for collecting situational information around an autonomous vehicle may have a predetermined field of view (FoV) as illustrated in FIG. 1. For example, when a sensor mounted on the front of an autonomous vehicle (10) has a field of view (FoV) as illustrated in FIG. 1, information detected at the center of the sensor may have relatively high importance. This may be because the information detected at the center of the sensor contains most of the information corresponding to the movement of the preceding vehicle (20).

[0024] The autonomous driving device processes information collected by the sensors of the autonomous driving vehicle (10) in real time to control the movement of the autonomous driving vehicle (10), while at least some of the information collected by the sensors can be stored in a memory device.

[0025] Referring to FIG. 2, the autonomous driving device (40) may include a sensor unit (41), a processor (46), a memory system (47), and a vehicle body control module (48). The sensor unit (41) includes a plurality of sensors (42-45), and the plurality of sensors (42-45) may include an image sensor, an event sensor, an illuminance sensor, a GPS device, an acceleration sensor, etc.

[0026] Data collected by the sensors (42-45) can be transmitted to the processor (46). The processor (46) can store the data collected by the sensors (42-45) in the memory system (47) and determine the movement of the vehicle by controlling the vehicle body control module (48) based on the data collected by the sensors (42-45). The memory system (47) may include two or more memory devices and a system controller for controlling the memory devices. Each of the memory devices may be provided as a single semiconductor chip.

[0027] In addition to the system controller of the memory system (47), each of the memory devices included in the memory system (47) may include a memory controller, and the memory controller may include an artificial intelligence (AI) computation circuit such as a neural network. The memory controller may generate computation data by assigning a predetermined weight to data received from the sensors (42-45) or the processor (46), and may store the computation data in a memory chip.

[0028] FIG. 3 is a diagram showing an example of image data acquired by a sensor of an autonomous vehicle equipped with an autonomous driving device. Referring to FIG. 3, the image data (50) may be data acquired by a sensor mounted on the front of the autonomous vehicle. Accordingly, the image data (50) may include the front of the autonomous vehicle (51), a preceding vehicle (52) in the same lane as the autonomous vehicle, a driving vehicle (53) around the autonomous vehicle, a background (54), and lanes (55, 56), etc.

[0029] In the image data (50) according to the embodiment illustrated in FIG. 3, the data in the area where the front part (51) of the autonomous vehicle and the background (54) appear may be data that is unlikely to affect the operation of the autonomous vehicle. In other words, the front part (51) of the autonomous vehicle and the background (54) may be considered as data of relatively low importance.

[0030] On the other hand, the distance from the preceding vehicle (52) and the lane change movement of the driving vehicle (53) can be very important factors for the safe operation of the autonomous vehicle. Therefore, the data in the area of ​​the video data (50) that includes the preceding vehicle (52) and the driving vehicle (53) can have relatively high importance in the operation of the autonomous vehicle.

[0031] The memory device of the autonomous driving device can store image data (50) received from the sensor by assigning different weights to each region. For example, a high weight can be assigned to the data in the region including the preceding vehicle (52) and the driving vehicle (53), and a low weight can be assigned to the data in the region where the front part (51) and background (54) of the autonomous driving vehicle appear.

[0032] Additionally, the autonomous driving device can detect lanes (55, 56) included in the road on which the vehicle is currently driving. For example, the autonomous driving device can detect lane pixels that are likely to correspond to lanes in image data (50) through image processing, and determine the lane arrangement situation ahead by fitting the detected lane pixels to a specific lane model.

[0033] In one embodiment, the autonomous driving device can detect lane pixels in image data (50) using a neural network model. The neural network model is a model trained to detect lane pixels corresponding to lanes in the image data (50). Image data (50) is input to the neural network model, and the neural network model can output probability information indicating the probability that each image pixel included in the image data (50) corresponds to a lane (55, 56). The autonomous driving device can determine lane pixels based on the probability information per pixel. For example, the autonomous driving device can determine image pixels among the image pixels of the image data (50) that have a probability value determined by the neural network model greater than a threshold value as lane pixels.

[0034] FIG. 4 is an exemplary drawing for explaining a method of performing image encoding and view conversion according to one embodiment.

[0035] Referring to FIG. 4, the lane polyline generating device may include an image encoder (410) and a view transformer (420).

[0036] The image encoder (410) may use a base image (411) as input data. The base image (411) may be an image of the road on which the vehicle is driving, obtained from at least one sensor mounted on the vehicle. For example, at least one sensor mounted on the vehicle may also be implemented as the sensor unit (41) described in 2.

[0037] The image encoder (410) can apply the base image (411) to a feature pyramid network to extract multi-scale image features (421).

[0038] The feature pyramid network may consist of a bottom-up process (412) and a top-down process (413). The bottom-up process (412) may be a forwarding process that generates multiple first feature maps by reducing the resolution of the base image (411) by half using a convolutional neural network (CNN). For example, in the bottom-up process (412), ResNet50 may be used, and first feature maps of 1 / 8, 1 / 16, 1 / 32, and 1 / 64 sizes relative to the base image (411) may be generated.

[0039] The top-down process (413) may be a process of up-sampling the final first feature map (414) generated in the bottom-up process (412) by a factor of two to generate a plurality of second feature maps, and combining the plurality of second feature maps and the plurality of first feature maps to extract a multi-scale image feature (421).

[0040] The view converter (420) can generate a bird's-eye view feature (BEV feature) (422) by performing a view conversion on a multi-scale image feature (421) extracted from an image encoder (410). The method for generating the bird's-eye view feature (422) will be described later in FIG. 5.

[0041] FIG. 5 is an exemplary drawing for explaining a method of generating a bird's-eye view feature according to one embodiment.

[0042] Referring to FIG. 5, a view converter (500) included in a lane polyline generating device is shown. The view converter (500) receives an image feature (510) and can generate a bird's-eye view feature (520) through a predetermined step.

[0043] Specifically, the view converter (500) can generate a bird's-eye view feature (520) by collapsing the image feature (510) in the height dimension in the first step (531), stretching the multi-scale image feature containing the image feature (510) in the depth dimension in the second step (532), and resampling using camera parameters in the third step (533).

[0044] Meanwhile, the lane polyline generating device has a bird's-eye view feature (520) f BEVc Homography transformation H c It can be remapped to canonical coordinates by applying . Specifically, the lane polyline generating device generates a bird's-eye view feature (520) f according to Equation 1. BEVc A homography transformation can be applied to it. In Equation 1, I is the base image, is an image encoder function, represents the view transformer, K represents camera-related parameters for capturing a given road (e.g., focal length, camera principal point, etc.), R represents the rotation transformation between the vehicle coordinate system and the camera coordinate system, and t represents the translation transformation between the vehicle coordinate system and the camera coordinate system. Additionally, z=0 indicates the transformed bird's-eye view feature f, which is the result value according to Equation 1. BEV It means that it is a value on the XY plane.

[0045]

[0047] The lane polyline generation device is a converted bird's-eye view feature f BEVThe neural network model can be trained by using the polyline of a lane on a predetermined road as input data and using the polyline of the lane on the road as output data. The training method of the neural network model will be described later in Fig. 6.

[0048] FIG. 6 is an exemplary diagram illustrating the operation method of a neural network model according to one embodiment.

[0049] Referring to FIG. 6, a neural network model (600) is illustrated. A lane polyline generating device can train the neural network model (600) by using a converted bird's-eye view feature (610) as input data for the neural network model (600) and using a lane polyline (640) for a predetermined road as output data.

[0050] A converted bird's-eye view feature (610) can be input as input data to the neural network model (600). The converted bird's-eye view feature (610) may be a result value calculated through the above mathematical formula 1. By applying at least one pre-set layer (620) and a loss function (630) to the converted bird's-eye view feature (610), a lane polyline (640) can be output as the output value of the neural network model (600).

[0051] In one embodiment, the neural network model (600) has a loss function according to Equation 2 below It can be trained so that the value is minimized. In Equation 2, I is the base image, represents a neural network model (600), K represents camera-related parameters for capturing a given road (e.g., focal length, camera principal point, etc.), R represents the camera coordinate system, and t represents the vehicle coordinate system. is the ground truth value of the lane polyline for a given road, and represents a value predicted through a neural network model (600) of a lane polyline for a given road.

[0052]

[0054] In the present disclosure, a lane polyline for a given road can be obtained through end-to-end learning of a neural network model (600) without using a homography function. In the present disclosure, at least one loss function used for learning the neural network model (600) can be set for this purpose. Specific details regarding the loss function will be described below.

[0055] Meanwhile, in one embodiment, the neural network model (600) described in FIG. 6 can also perform image encoding and view conversion described in FIG. 4 and FIG. 5.

[0056] The lane polyline generation device can train the neural network model (600) by using a basic image as input data for the neural network model (600) and using a lane polyline for a predetermined road as output data.

[0057] Specifically, the lane polyline generating device can input a base image as input data to a neural network model (600). Additionally, the lane polyline generating device can drive the neural network model (600) to generate a bird's-eye view feature from the base image and obtain a lane polyline as output data based on the generated bird's-eye view feature. More specifically, the lane polyline generating device can drive the neural network model (600) to extract a multi-scale image feature using the base image, perform a view transformation on the extracted multi-scale image feature to generate a bird's-eye view feature, and obtain a lane polyline for a predetermined road as output data based on the generated bird's-eye view feature.

[0058] FIGS. 7a and 7b are exemplary drawings for illustrating a seed probability loss according to one embodiment.

[0059] Referring to FIGS. 7a and 7b, a pixel coordinate system (700) is illustrated. A lane polyline generating device can generate a pixel coordinate system (700) by converting the spatial coordinate system of a bird's-eye view feature into an embedding coordinate system.

[0060] Referring to FIG. 7a, the point mark (710) included in the pixel coordinate system (700) represents a pixel, and the straight line mark (720) represents a lane.

[0061] The bird's-eye view image generating device may set a seed probability loss as at least one loss function (or loss) applied to a neural network model. The seed probability value varies depending on whether each of the multiple pixels in the pixel coordinate system (700) belongs to the foreground or background category. Here, the foreground refers to objects such as vehicles or lanes, and the background refers to the background. If a certain pixel in the pixel coordinate system (700) is determined to be in the background, the lane polyline generating device may determine the seed probability of that pixel to be 0. Additionally, the higher the probability that a certain pixel in the pixel coordinate system (700) belongs to the foreground, the lane polyline generating device may determine the seed probability of that pixel to have a value closer to 1.

[0062] Multiple pixels in the pixel coordinate system (700) are classified into foreground pixels representing the foreground or background pixels representing the background. Referring to FIG. 7b, the first pixel group (730) of the multiple pixels in the pixel coordinate system (700) corresponds to foreground pixels, and the second pixel group (740) corresponds to background pixels.

[0063] In one embodiment, the lane polyline generating device can set a seed probability loss composed of the first formula and the second formula by applying the first formula to the foreground pixel corresponding to the foreground among a plurality of pixels and applying the second formula to the background pixel corresponding to the background.

[0064] A bird's-eye view image generating device can calculate the probability that each foreground pixel corresponds to a predetermined cluster among multiple clusters corresponding to each of multiple lanes included in a predetermined road. A lane polyline generating device can set a first formula based on the pixel value of the foreground pixel and the calculated probability. Meanwhile, a method for calculating the probability of corresponding to a cluster will be described later in FIGS. 8a to 8c.

[0065] In addition, the lane polyline generating device can set the first formula by applying a scaling factor to correct the imbalance between the pixel values ​​of the foreground pixels and the pixel values ​​of the background pixels.

[0066] In addition, the lane polyline generation device can set a second formula based on the pixel value of the background pixel.

[0067] The seed probability loss including the aforementioned first and second formulas can be expressed by the following Equation 3. In one embodiment, the neural network model has a seed probability loss according to the following Equation 3. It can be learned so that the value is minimized.

[0068] In mathematical formula 3, the first term on the right-hand side may be the first formula described above, and the second term may be the second formula described above. Additionally, in mathematical formula 3, N is the number of pixels included in the pixel coordinate system (700). is the scaling factor, q i is the seed predicted value of the i-th pixel, S k is foreground, b g is the background, is the i-th embedding coordinate e in the pixel coordinate system (700). i It represents the probability of belonging to the k-th cluster.

[0069]

[0071] FIGS. 8a through 8c are exemplary drawings for illustrating embedding offset loss.

[0072] Referring to FIGS. 8a through 8c, a pixel coordinate system (800) is illustrated. Referring to FIG. 8a, the first straight line mark (811) and the second straight line mark (812) included in the pixel coordinate system (800) represent different lanes.

[0073] The lane polyline generating device can set an embedding offset loss as at least one loss function (or, loss) applied to a neural network model. The embedding offset is the i-th embedding coordinate e in the pixel coordinate system (800). i represents the probability that it corresponds to the k-th cluster. Here, clusters can be set for each of the multiple lanes included in a given road. The closer the i-th pixel in the pixel coordinate system (800) is located to the center of the k-th cluster, the more the lane polyline generating device [represents] the corresponding probability value It can be determined to be close to 1.

[0074] Referring to FIG. 8b, each pixel shading visualizes the probability of belonging to a specific cluster, and the darker the shading, the greater the probability value of belonging to a specific cluster. In FIG. 8a, the first pixel (821) located close to the first straight line mark (811) and the second pixel (825) located close to the second straight line mark (812) have a very high probability value of belonging to a specific cluster, so it can be seen that the shading is darkly displayed in FIG. 8b. Meanwhile, the lane polyline generating device can calculate the probability of belonging to a specific cluster only for foreground pixels and can omit the calculation for background pixels.

[0075] Referring to FIG. 8c, pixels clustered by cluster are illustrated. The above-described bird's-eye view feature has spatial coordinates, and the lane polyline generation device can convert the bird's-eye view feature having spatial coordinates into two-dimensional embedding coordinates (850) by running a neural network model.

[0076] The lane polyline generating device can set the first pixel (821) on the embedding coordinates (850) as the centroid for the first cluster (831) and the second pixel (825) as the centroid for the second cluster (835).

[0077] In one embodiment, the lane polyline generation device can extract foreground pixels corresponding to the foreground from among a plurality of pixels included in a bird's-eye view feature. Additionally, the lane polyline generation device can calculate the probability that each foreground pixel corresponds to a predetermined cluster among a plurality of clusters on embedding coordinates (850) corresponding to each of a plurality of lanes included in a predetermined road. Furthermore, the lane polyline generation device can set an embedding offset loss using the calculated probability and train a neural network model using the embedding offset loss.

[0078] Additionally, the lane polyline generating device can set a center point for each of the plurality of clusters on the embedding coordinates (850) and a fixed margin for forming the cluster. Using the center point and the fixed margin, the lane polyline generating device can calculate the probability that each of the foreground pixels corresponds to a predetermined cluster among the plurality of clusters.

[0079] Additionally, the lane polyline generating device may set a clustering threshold probability value to calculate the probability that each foreground pixel corresponds to a predetermined cluster among multiple clusters on the embedding coordinates (850). The lane polyline generating device may calculate the probability that each foreground pixel corresponds to a predetermined cluster among multiple clusters using a center point, a fixed margin, and a clustering threshold probability value.

[0080] The neural network model embedding offset loss according to Equation 4 below It can be trained to minimize the value. In Equation 4, K is the number of clusters, represents the support vector machine-related loss (e.g., Lovasz hinge loss).

[0081]

[0083] In addition, the probability that each of the foreground pixels corresponds to a predetermined cluster among a plurality of clusters on the embedding coordinates (850). can be calculated according to the following mathematical formula 5. According to mathematical formula 5 It follows a Gaussian distribution.

[0084]

[0086] In addition, of mathematical formula 5 can be calculated according to the following mathematical formula 6. In mathematical formula 6, R is a fixed margin for forming a cluster in the embedding space (800), P r-represents the clustering threshold probability value.

[0087]

[0089] FIG. 9 is an exemplary drawing for illustrating a sequence loss according to one embodiment.

[0090] The lane polyline generation device may set an ordinal loss as at least one loss function (or, loss) applied to a neural network model. Here, ordinal refers to the order of the start and end of the lane corresponding to the polyline. The lane polyline generation device may determine the ordinal value to be closer to 0 as a certain pixel in the pixel coordinate system is closer to the starting point of the lane, and the ordinal value to be closer to 0 as it is closer to the ending point of the lane.

[0091] Referring to FIG. 9, the order in which each pixel in the pixel coordinate system is arranged between the start and end points of a lane is illustrated. Order value 1 and order value 2 in FIG. 9 represent result values ​​for different lanes (or clusters), respectively. In FIG. 9, the darker the shading, the closer a given pixel is to the end point of the lane.

[0092] In one embodiment, the lane polyline generation device can extract foreground pixels corresponding to the foreground from among a plurality of pixels included in a bird's-eye view feature. Additionally, the lane polyline generation device can set an order loss using the order of each of the foreground pixels for a lane included in a predetermined road, and train a neural network model using the order loss.

[0093] Additionally, to set the order loss, the lane polyline generation device may identify a cluster containing each of the foreground pixels and determine the order value of the foreground pixels for the lane corresponding to the identified cluster. The lane polyline generation device may set the order loss by using the order value for each of the foreground pixels.

[0094] Additionally, the lane polyline generation device can determine the order values ​​of foreground pixels such that foreground pixels closer to the start point of the lane have a value closer to the first value, and foreground pixels closer to the end point of the lane have a value closer to the second value.

[0095] The lane polyline generation device may set the order loss using a smooth L1 algorithm, but the algorithm used is not limited thereto.

[0096] The neural network model has order loss according to Equation 7 below. It can be learned so that the value is minimized. In Equation 7, d n represents the order value of the nth pixel and has a value between 0 and 1. represents the ground truth value.

[0097]

[0099] Meanwhile, the smooth L1 algorithm can be expressed by mathematical equation 8.

[0100]

[0102] The lane polyline generation device can train a neural network model such that at least one of the embedding offset loss, seed probability loss, and order loss is minimized.

[0103] In one embodiment, the lane polyline generation device can train a neural network model such that the final loss according to Equation 9 below is minimized. In Equation 9 It can be set differently depending on the importance of each term.

[0104]

[0106] FIG. 10 is a flowchart illustrating the generation of a lane polyline using a neural network model according to one embodiment.

[0107] Referring to FIG. 10, in step 1010, the lane polyline generating device can obtain a basic image of a predetermined road obtained from at least one sensor mounted on a vehicle.

[0108] In step 1020, the lane polyline generation device can extract multi-scale image features using a base image.

[0109] In step 1030, the lane polyline generation device can generate a bird's-eye view feature by performing a view transformation on the extracted multi-scale image feature.

[0110] In step 1040, the lane polyline generation device can train the neural network model by using a bird's-eye view feature as input data for the neural network model and using a lane polyline for a predetermined road as output data.

[0111] In one embodiment, the lane polyline generation device extracts foreground pixels corresponding to the foreground from among a plurality of pixels included in a bird's-eye view feature, calculates the probability that each foreground pixel corresponds to a predetermined cluster among a plurality of clusters corresponding to each of a plurality of lanes included in a predetermined road, sets an embedding offset loss using the calculated probability, and trains a neural network model using the embedding offset loss.

[0112] Additionally, the lane polyline generation device may set a center point for each of the plurality of clusters and a fixed margin for forming the clusters in order to calculate the probability. Using the center point and the fixed margin, the lane polyline generation device may calculate the probability that each foreground pixel corresponds to a predetermined cluster among the plurality of clusters.

[0113] Additionally, the lane polyline generation device can set a clustering threshold probability value to calculate the probability. The lane polyline generation device can calculate the probability that each foreground pixel corresponds to a predetermined cluster among a plurality of clusters by using a center point, a fixed margin, and a clustering threshold probability value.

[0114] In one embodiment, the lane polyline generation device sets a seed probability loss based on whether each of the plurality of pixels included in the bird's-eye view feature belongs to the foreground or background category, and can train a neural network model using the seed probability loss.

[0115] In addition, the lane polyline generation device can set a seed probability loss composed of the first formula and the second formula by applying the first formula to foreground pixels corresponding to the foreground among a plurality of pixels and applying the second formula to background pixels corresponding to the background in order to set a seed probability loss.

[0116] Additionally, the lane polyline generation device can calculate the probability that each foreground pixel corresponds to a predetermined cluster among a plurality of clusters corresponding to each of a plurality of lanes included in a predetermined road. The lane polyline generation device can set a first formula based on the pixel value of the foreground pixel and the calculated probability.

[0117] In addition, the lane polyline generation device can set a second formula based on the pixel value of the background pixel.

[0118] In addition, the lane polyline generating device can set the first formula by applying a scaling factor further to correct the imbalance between the pixel values ​​of the foreground pixels and the pixel values ​​of the background pixels.

[0119] In one embodiment, the lane polyline generation device extracts foreground pixels corresponding to the foreground from among a plurality of pixels included in a bird's-eye view feature, sets an order loss using the order of each foreground pixel for a lane included in a predetermined road, and can train a neural network model using the order loss.

[0120] Additionally, the lane polyline generation device can identify clusters containing each of the foreground pixels and determine the order values ​​of the foreground pixels for the lanes corresponding to the identified clusters. The lane polyline generation device can set an order loss using the order values ​​for each of the foreground pixels.

[0121] In addition, the lane polyline generation device can determine the order values ​​of foreground pixels such that foreground pixels closer to the start point of the lane have a value closer to the first value, and foreground pixels closer to the end point of the lane have a value closer to the second value.

[0122] In one embodiment, the lane polyline generation device can train a neural network model such that at least one of the embedding offset loss, seed probability loss, and order loss is minimized.

[0123] FIG. 11 is a block diagram of a lane polyline generating device according to one embodiment.

[0124] Referring to FIG. 11, the lane polyline generating device (1100) may include a communication unit (1110), a processor (1120), and a DB (1130). Only the components related to the embodiment are shown in the lane polyline generating device (1100) of FIG. 11. Therefore, a person skilled in the art will understand that other general-purpose components may be included in addition to the components shown in FIG. 11.

[0125] The communication unit (1110) may include one or more components that enable wired / wireless communication with an external server or external device. For example, the communication unit (1110) may include at least one of a short-range communication unit (not shown), a mobile communication unit (not shown), and a broadcast receiving unit (not shown).

[0126] DB (1130) is hardware that stores various data processed within the lane polyline generation device (1100) and can store a program for processing and controlling the processor (1120).

[0127] DB (1130) may include RAM (random access memory), such as DRAM (dynamic random access memory) and SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), CD-ROM, Blu-ray or other optical disc storage, HDD (hard disk drive), SSD (solid state drive), or flash memory.

[0128] The processor (1120) controls the overall operation of the lane polyline generating device (1100). For example, the processor (1120) can control the input unit (not shown), display (not shown), communication unit (1110), DB (1130), etc., by executing programs stored in DB (1130). The processor (1120) can control the operation of the lane polyline generating device (1100) by executing programs stored in DB (1130).

[0129] The processor (1120) can control at least some of the operations of the lane polyline generating device (1100) described in FIGS. 1 to 10. The lane polyline generating device (1100) and the autonomous driving device (40) may be the same device, or at least some of the operations performed in each device may be the same.

[0130] The processor (1120) may be implemented using at least one of ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and other electrical units for performing functions.

[0131] In one embodiment, the lane polyline generating device (1100) may be a mobile electronic device. For example, the lane polyline generating device (1100) may be implemented as a smartphone, tablet PC, PC, smart TV, PDA (personal digital assistant), laptop, media player, navigation, a device equipped with a camera, and other mobile electronic devices. Additionally, the lane polyline generating device (1100) may be implemented as a wearable device such as a watch, glasses, a hair band, and a ring equipped with communication functions and data processing functions.

[0132] In another embodiment, the lane polyline generating device (1100) may be an electronic device embedded in a vehicle. For example, the lane polyline generating device (1100) may be an electronic device inserted into a vehicle through tuning after the production process.

[0133] In another embodiment, the lane polyline generating device (1100) may be a server located outside the vehicle. The server may be implemented as a computer device or a plurality of computer devices that communicate via a network to provide commands, code, files, content, services, etc. The server receives data necessary to generate a lane polyline from devices mounted on the vehicle and can determine the vehicle's movement path based on the received data.

[0134] In another embodiment, the process performed in the lane polyline generating device (1100) may be performed by at least some of a mobile electronic device, an electronic device embedded in a vehicle, and a server located outside the vehicle.

[0135] An embodiment according to the present invention may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded on a computer-readable medium. In this case, the medium may include a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a hardware device specifically configured to store and execute program instructions, such as a ROM, RAM, or flash memory.

[0136] Meanwhile, the above-mentioned computer program may be one specifically designed and configured for the present invention, or one known and available to those skilled in the art of computer software. Examples of computer programs may include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0137] According to one embodiment, the method according to various embodiments of the present disclosure may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices. In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0138] Unless explicitly stated or contrary to the order of the steps constituting the method according to the present invention, said steps may be performed in a suitable order. The present invention is not necessarily limited by the order in which said steps are described. The use of all examples or exemplary terms (e.g., etc.) in the present invention is merely for the purpose of describing the present invention in detail, and the scope of the present invention is not limited by said examples or exemplary terms unless limited by the claims. Furthermore, those skilled in the art will understand that various modifications, combinations, and changes may be made according to design conditions and factors within the scope of the claims or equivalents to which they are added.

[0139] Accordingly, the scope of the present invention should not be limited to the embodiments described above, and all scopes equivalent to or equivalently modified from the claims set forth below, as well as the claims set forth below, shall be considered to fall within the scope of the concept of the present invention.

Claims

Claim 1 A method performed by a device for generating a lane polyline using a neural network model, comprising: acquiring a base image of a predetermined road obtained from at least one sensor mounted on a vehicle; extracting a multi-scale image feature using the base image; performing a view transformation on the extracted multi-scale image feature to generate a bird's-eye view feature (BEV feature); and training the neural network model by using the bird's-eye view feature as input data for the neural network model and using the lane polyline of the predetermined road as output data; wherein the step of training the neural network model comprises: setting an embedding offset loss using the probability that each of the foreground pixels among the plurality of pixels included in the bird's-eye view feature corresponds to a predetermined cluster among a plurality of clusters corresponding to each of the plurality of lanes included in the predetermined road; and training the neural network model using the embedding offset loss. Claim 2 delete Claim 3 A method according to claim 1, wherein the step of setting the embedding offset loss comprises: a step of setting a centroid for each of the plurality of clusters and a fixed margin for forming the cluster; and a step of calculating the probability that each of the foreground pixels corresponds to a predetermined cluster among the plurality of clusters using the centroid and the fixed margin. Claim 4 In claim 3, the step of calculating the probability comprises: a step of setting a clustering threshold probability value; and a step of calculating the probability that each of the foreground pixels corresponds to a predetermined cluster among the plurality of clusters using the center point, the fixed margin, and the clustering threshold probability value. Claim 5 The method of claim 1 further comprises: a step of inputting the bird's-eye view feature as input data for the neural network model learned by the method according to claim 1; and a step of generating a lane polyline for the predetermined road as output data for the neural network model. Claim 6 In claim 5, the method further comprises the step of generating a control signal for controlling a vehicle traveling on the predetermined road based on a lane polyline for the predetermined road. Claim 7 An apparatus for generating a lane polyline using a neural network model, comprising: a memory storing at least one program; and a processor for driving a neural network by executing the at least one program, wherein the processor acquires a base image of a predetermined road from a camera mounted on a vehicle, extracts a plurality of image features using the base image, performs a view transformation on the extracted plurality of image features to generate a bird's-eye view feature, uses the bird's-eye view feature as input data for the neural network model, and uses the lane polyline of the predetermined road as output data to train the neural network model, wherein training the neural network model involves setting an embedding offset loss using the probability that each of the foreground pixels among the plurality of pixels included in the bird's-eye view feature corresponds to a predetermined cluster among a plurality of clusters corresponding to each of the plurality of lanes included in the predetermined road, and training the neural network model using the embedding offset loss. Claim 8 A computer-readable recording medium having a program for executing the method of claim 1 on a computer. Claim 9 A method performed by a device for generating a lane polyline using a neural network model, comprising: acquiring a base image of a predetermined road obtained from at least one sensor mounted on a vehicle; inputting the base image as input data to a neural network model; and driving the neural network model to generate a bird's-eye view feature from the base image and acquiring a lane polyline as output data based on the generated bird's-eye view feature; wherein the neural network model is learned through an embedding offset loss set using the probability that each of the foreground pixels corresponding to the foreground among the plurality of pixels included in the bird's-eye view feature corresponds to a predetermined cluster among a plurality of clusters corresponding to each of the plurality of lanes included in the predetermined road.

Citation Information

Patent Citations

  • Method and apparatus of indicating lane

    KR1020180050848A

  • Lane detection method, device, electronic device and readable storage medium

    KR1020210080459A