A lane line detection method, device, equipment and storage medium

Abstract

The application discloses a lane line detection method, device and equipment and a storage medium. The method comprises the following steps: inputting a lane line image into a preconfigured lane line detection model; performing feature extraction processing on the lane line image by using a feature extraction submodel in the lane line detection model to obtain a lane line feature map of a target lane line; performing coordinate prediction processing on the lane line feature map by using a coordinate prediction submodel in the lane line detection model to obtain a predicted coordinate position of the target lane line; performing offset prediction processing on the lane line feature map by using an offset prediction submodel in the lane line detection model to obtain an offset prediction result of the target lane line; and determining a lane line point chain corresponding to the target lane line based on the predicted coordinate position and the offset prediction result. The method disclosed by the application can perform offset prediction on the coordinate prediction result in the coordinate prediction process of the lane line, so that a more accurate lane line position is obtained, and the accuracy of lane line detection is improved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving or driver assistance technology, and in particular to a lane line detection method, device, equipment and storage medium. Background Technology

[0002] With the development of autonomous and intelligent driving technologies, lane detection has become one of the core tasks in autonomous driving and advanced driver assistance systems. It helps vehicles identify road boundaries, enabling functions such as autonomous navigation, lane keeping, and collision avoidance. It also plays a crucial role in intelligent driving, including vehicle localization and navigation, obstacle detection and avoidance, traffic rule recognition, adaptive cruise control (ACC), lane departure assist (LDA), intelligent lane changing, and overtaking.

[0003] In existing technologies, deep learning algorithms are typically used to learn lane line feature representations directly from the original image through semantic segmentation to achieve lane line detection. However, in real-world driving conditions, lane line detection suffers from poor adaptability to complex driving environments, such as changes in lighting, shadows, and inclement weather, which in turn reduces the accuracy of lane line detection. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a lane line detection method, apparatus, device, and storage medium, with the aim of improving the accuracy of lane line detection results.

[0005] The embodiments of this application disclose the following technical solutions: In a first aspect, this application provides a lane line detection method, the method comprising: The lane line image is input into a pre-configured lane line detection model, and the feature extraction sub-model in the lane line detection model is used to perform feature extraction processing on the lane line image to obtain the lane line feature map of the target lane line. The coordinate prediction sub-model in the lane line detection model is used to perform coordinate prediction processing on the lane line feature map to obtain the predicted coordinate position of the target lane line. The offset prediction sub-model in the lane line detection model is used to perform offset prediction processing on the lane line feature map to obtain the offset prediction result of the target lane line; Based on the predicted coordinates and the offset prediction results, the lane line point chain corresponding to the target lane line is determined.

[0006] Optionally, in the method described above, the step of using the coordinate prediction sub-model in the lane detection model to perform coordinate prediction processing on the lane feature map to obtain the predicted coordinate position of the target lane line includes: The lane line feature map is input into the coordinate prediction sub-model, and the lane line feature map is scaled to obtain the target lane line feature map to be predicted; the target lane line feature map is a lane line feature map whose vertical dimension size matches the preset vertical anchor point parameters. Based on the preset vertical anchor point parameters, the feature map to be predicted is divided in the vertical dimension and in the horizontal dimension to obtain a set of anchor point coordinate positions; the set of anchor point coordinate positions includes multiple vertical anchor point positions and multiple horizontal positions corresponding to each vertical anchor point position. The convolutional layer of the coordinate prediction sub-model is used to perform feature extraction processing on the feature map to be predicted to obtain target output data; each row of the target output data corresponds to a vertical anchor point position; each column of the target output data corresponds to a horizontal position; each element in the target output data represents the lane line prediction probability at the combination of the vertical anchor point position and the horizontal position. From each row of the target output data, determine the first target element with the maximum lane line prediction probability, and use the longitudinal anchor point position and lateral position corresponding to the first target element as the predicted coordinate position of the target lane line.

[0007] Optionally, in the method described above, the step of using the deviation prediction sub-model in the lane line detection model to perform offset prediction processing on the lane line feature map to obtain the offset prediction result of the target lane line includes: The first predicted position is obtained by using the deviation prediction sub-model in the lane line detection model to perform coordinate prediction processing on the lane line feature map. The first predicted position is processed using the activation function in the deviation prediction sub-model to obtain the second predicted position of the target lane line; The deviation prediction result of the target lane line is obtained by using the loss function in the deviation prediction sub-model to calculate the deviation of the second prediction position.

[0008] Optionally, in the method described above, determining the lane line point chain corresponding to the target lane line based on the predicted coordinate position and the offset prediction result includes: Based on the predicted coordinate position and the offset prediction result, the lane line point chain optimization sub-model in the lane line detection model is used for processing to obtain the lane line point chain corresponding to the target lane line; the lane line point chain optimization sub-model is trained by the cross ratio loss function, and the lane line point chain optimization sub-model is used to integrate the predicted coordinate position and the offset prediction result to output the point chain that fits the actual lane line direction.

[0009] Optionally, in the method described above, the lane detection model further includes a lane line type recognition sub-model, and the method further includes: Using the lane line type recognition sub-model, the lane line feature map of the target lane line is convolved to obtain a three-dimensional feature map with a first value for the number of channels; the first value is the defined number of lane line types, and each channel in the channel corresponds to a feature map of one lane line type. The three-dimensional feature map is unfolded to obtain a one-dimensional feature vector; The one-dimensional feature vector is mapped through the fully connected layer of the lane line type recognition sub-model to obtain the target matrix; wherein, the row dimension of the target matrix is ​​each lane line, the column dimension of the target matrix is ​​each lane line type, and each element in the target matrix represents the probability that the lane line corresponding to the element in the row dimension is classified into the lane line type corresponding to the element in the column dimension. The second target element that has the maximum value in the column dimension of the target lane line is determined from the target matrix, and the lane line type corresponding to the second target element is used as the type prediction result of the target lane line.

[0010] Optionally, the method described above further includes: According to a preset mapping table, the three-dimensional coordinate point corresponding to each coordinate point in the lane line point chain is determined; the mapping table contains the correspondence between two-dimensional coordinate points and three-dimensional plane coordinate points; Based on the three-dimensional coordinate points corresponding to each coordinate point, a three-dimensional lane line point chain is constructed.

[0011] Optionally, in the method described above, the pre-configured lane detection model is obtained by training an initial lane detection model. The initial feature extraction sub-model in the initial lane detection model includes a global feature extraction model and a lane segmentation model. The method further includes: The global feature extraction model is used to extract features from the acquired lane line annotation map to obtain a lane line image. The lane line image is trained using the lane line segmentation model to obtain trained parameters, which are then applied to the pre-configured lane line detection model. These parameters are used to enhance the supervision of lane line features in the lane line feature map.

[0012] Secondly, this application provides a lane line detection device, comprising: The feature extraction module is used to input the lane line image into the pre-configured lane line detection model, and use the feature extraction sub-model in the lane line detection model to perform feature extraction processing on the lane line image to obtain the lane line feature map of the target lane line. The coordinate prediction module is used to perform coordinate prediction processing on the lane line feature map using the coordinate prediction sub-model in the lane line detection model to obtain the predicted coordinate position of the target lane line. The offset prediction module is used to perform offset prediction processing on the lane line feature map using the deviation prediction sub-model in the lane line detection model to obtain the offset prediction result of the target lane line. The lane line point chain determination module is used to determine the lane line point chain corresponding to the target lane line based on the predicted coordinate position and the offset prediction result.

[0013] Thirdly, this application provides an electronic device, the device including: a processor, and a memory communicatively connected to the processor; The memory stores the instructions that the computer executes; The processor executes computer execution instructions stored in the memory to implement any of the lane line detection methods described in the above embodiments.

[0014] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement any of the lane detection methods described in the above embodiments.

[0015] Compared with the prior art, this application has the following advantages: The method of this application inputs a lane line image into a pre-configured lane line detection model. The feature extraction sub-model within the lane line detection model performs feature extraction on the lane line image, effectively filtering out irrelevant noise to obtain a lane line feature map of the target lane line. Then, the coordinate prediction sub-model within the lane line detection model performs coordinate prediction on the lane line feature map to obtain the predicted coordinate position of the target lane line. Next, the offset prediction sub-model within the lane line detection model performs offset prediction on the lane line feature map to obtain the offset prediction result of the target lane line, thereby correcting errors in coordinate prediction and improving detection accuracy. Finally, by combining the predicted coordinate position and the offset prediction result, the lane line point chain corresponding to the target lane line is determined. The coordinate results are dynamically corrected by predicting the offset, enabling more precise lane line positioning. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic flowchart of an embodiment of the lane line detection method provided in this application; Figure 2 A schematic flowchart illustrating another embodiment of the lane line detection method provided in this application; Figure 3 A schematic flowchart of another embodiment of the lane line detection method provided in this application; Figure 4 A schematic diagram illustrating the predicted coordinate position and offset prediction results provided in this application; Figure 5 A schematic diagram illustrating the principle of lane intersection ratio loss calculation provided in this application; Figure 6 A schematic diagram of a preset mapping table provided in this application; Figure 7 A schematic diagram of one embodiment of a lane line detection device provided in this application; Figure 8 This is a schematic diagram of the structure of one embodiment of an electronic device provided in this application. Detailed Implementation

[0018] As described above, current lane detection methods typically classify individual pixels in the acquired image to separate lane lines from the background, thereby achieving the purpose of detecting and distinguishing lane lines.

[0019] Through research, the inventors have proposed a lane line detection method, device, equipment, and storage medium to solve the technical problem of inaccurate lane line position detection results in the prior art.

[0020] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0021] See Figure 1 This figure is a schematic flowchart of an embodiment of a lane line detection method provided in this application. Figure 1 As shown, the method includes: 101: Input the lane line image into the pre-configured lane line detection model, and use the feature extraction sub-model in the lane line detection model to perform feature extraction processing on the lane line image to obtain the lane line feature map of the target lane line.

[0022] In this embodiment, before inputting the lane line image into the pre-configured lane line detection model, necessary preprocessing can be performed on real-time video frames or static images captured by the vehicle's camera. This includes, but is not limited to, image distortion correction, removal of sky and other irrelevant parts, and scaling of the captured image, thereby obtaining the lane line image. The pre-processed lane line image is then input into the pre-configured lane line detection model, where a feature extraction sub-model performs feature extraction. For example, the feature extraction sub-model can be a common backbone network structure such as ResNet (Residual Network). Specifically, features in the image are extracted using the backbone network structure, and the lane line image is input into the input layer of the ResNet network. After a series of residual block processing steps, feature extraction is achieved, generating a lane line feature map of the target lane line.

[0023] 102: The coordinate prediction sub-model in the lane line detection model is used to perform coordinate prediction processing on the lane line feature map to obtain the predicted coordinate position of the target lane line.

[0024] In this embodiment, the lane line feature map is scaled using a coordinate prediction sub-model to adjust its size, resulting in a feature map to be predicted for the target lane line. This feature map is then partitioned along both the vertical and horizontal dimensions to obtain a set of anchor point coordinates. The feature map is then input into the convolutional layer of the coordinate prediction sub-model. The convolutional layer performs convolution operations on the feature map using a series of filters, extracting the feature information corresponding to each anchor point coordinate, thus obtaining the target output data. From each row of the target output data, the first target element with the highest lane line prediction probability is determined, and the anchor point coordinates corresponding to this first target element are used as the predicted coordinates of the target lane line. It can be understood that the predicted coordinates of the target lane line can be summarized and described as a chain of lane line points with multiple predicted coordinates.

[0025] 103: The offset prediction sub-model in the lane line detection model is used to perform offset prediction processing on the lane line feature map to obtain the offset prediction result of the target lane line.

[0026] In this embodiment, the deviation prediction sub-model in the lane line detection model is used to perform coordinate prediction processing on the lane line feature map to obtain the first predicted position; then the activation function in the deviation prediction sub-model is used to process the first predicted position to obtain a more accurate second predicted position of the target lane line; then the loss function in the deviation prediction sub-model is used to calculate the deviation of the second predicted position to obtain the offset prediction result of the target lane line.

[0027] 104: Based on the predicted coordinate position and offset prediction results, determine the lane line point chain corresponding to the target lane line.

[0028] In this embodiment, the predicted coordinate position is combined with the offset prediction result. The offset prediction result can be applied to the predicted coordinate position through operations such as addition or multiplication. The trained lane line point chain optimization sub-model is then used to process the result, making the corrected lane line coordinate position closer to the actual coordinate position, thereby obtaining the lane line point chain corresponding to the target lane line.

[0029] In this embodiment, the lane line image is input into a pre-configured lane line detection model. The feature extraction sub-model within the lane line detection model performs feature extraction processing on the lane line image, effectively filtering out irrelevant noise, thereby obtaining the lane line feature map of the target lane line. Then, the coordinate prediction sub-model within the lane line detection model performs coordinate prediction processing on the lane line feature map to obtain the predicted coordinate position of the target lane line. Next, the offset prediction sub-model within the lane line detection model performs offset prediction processing on the lane line feature map to obtain the offset prediction result of the target lane line, thereby correcting errors in coordinate prediction and improving detection accuracy. Finally, by combining the predicted coordinate position and the offset prediction result, the lane line point chain corresponding to the target lane line is determined. Dynamically correcting the coordinate results through predicted offset allows for more precise lane line positioning, improving the accuracy of lane line detection.

[0030] See Figure 2 This figure is a schematic flowchart of another embodiment of a lane line detection method provided in this application. Figure 2 As shown, based on the above embodiments, a specific implementation of S102, "using the coordinate prediction sub-model in the lane line detection model to perform coordinate prediction processing on the lane line feature map to obtain the predicted coordinate position of the target lane line," includes: S1021: Input the lane line feature map into the coordinate prediction sub-model, and scale the lane line feature map to obtain the target lane line feature map to be predicted. The target lane line feature map is a lane line feature map whose vertical dimension matches the preset vertical anchor point parameters.

[0031] S1022: Based on preset vertical anchor point parameters, the feature map to be predicted is partitioned in the vertical dimension and in the horizontal dimension to obtain a set of anchor point coordinate positions. The set of anchor point coordinate positions includes multiple vertical anchor point positions and multiple horizontal positions corresponding to each vertical anchor point position.

[0032] S1023: Use the convolutional layer of the coordinate prediction sub-model to perform feature extraction processing on the feature map to be predicted, and obtain the target output data.

[0033] In this data, each row of the target output data corresponds to a vertical anchor point position; each column of the target output data corresponds to a horizontal position; and each element in the target output data represents the lane line prediction probability at the combination of the corresponding vertical anchor point position and the horizontal position.

[0034] S1024: Determine the first target element with the maximum lane line prediction probability from each row of the target output data, and use the longitudinal anchor point position and lateral position corresponding to the first target element as the predicted coordinate position of the target lane line.

[0035] In this embodiment, after inputting the lane line feature map into the coordinate prediction sub-model, the lane line feature map is scaled to further adjust its size, resulting in a feature map to be predicted for the target lane line, thus more accurately focusing on the effective area of ​​the target lane line. The feature map to be predicted is then divided to obtain a set containing anchor point coordinate positions. For example, it can be divided according to a custom Row anchor parameter. The Row anchor parameter can be defined as the vertical dimension of the image, and the vertical anchor point positions obtained by dividing the feature map along the column dimension can be N, i.e., the ordinate of the predicted coordinate position. Furthermore, the feature map to be predicted is further divided along the row dimension a preset number of times, which can be M, thus obtaining a set of anchor point coordinate positions including M vertical anchor point positions and N horizontal positions corresponding to each vertical anchor point position. In layman's terms, based on preset vertical anchor point parameters, N vertical anchor points or anchors are divided into the vertical dimension of the feature map to be predicted. Then, this division is repeated M times in the horizontal dimension, resulting in a grid of lane lines. This divides the effective lane line region of the feature map into M×N grids, yielding the coordinate positions corresponding to each anchor. The convolutional layer of the coordinate prediction sub-model then performs feature extraction on the feature map, resulting in target output data of size (M+1)×T×N. Here, T represents the total number of lane lines. For each of the T lane lines, the maximum value among the M+1 horizontal coordinate positions corresponding to each of the N vertical coordinate positions in the target output data is determined, thus obtaining the N predicted coordinate positions on the target lane line. For example, assuming the coordinate prediction sub-model predicts 4 lane lines in the feature map, with 16 Row anchor parameters, 16 vertical anchor points are divided in the vertical column dimension, and the horizontal division can be repeated 100 times. To avoid situations where there are no lane lines at the anchor, we increase the value by 1 from 100. If the maximum value appears in the 101st grid, it means that there are no points in the lane lines at that location.

[0036] Understandably, during the training of the coordinate prediction sub-model, the classification loss during coordinate prediction can be calculated using, but is not limited to, the softmax function.

[0037] In this embodiment, the lane line feature map is input into the coordinate prediction sub-model, and the lane line feature map is scaled to obtain the target lane line feature map to be predicted. The target lane line feature map is a lane line feature map whose vertical dimension matches the preset vertical anchor point parameters, which can reduce the interference of irrelevant information and thus improve the accuracy of lane line detection. Based on the preset vertical anchor point parameters, the target lane line feature map is divided in the horizontal dimension and in the vertical dimension to obtain a set of anchor point coordinate positions. The set of anchor point coordinate positions includes multiple vertical anchor point positions and multiple horizontal positions corresponding to each vertical anchor point position. The convolutional layer of the coordinate prediction sub-model is used to perform feature extraction processing on the target lane line feature map to obtain target output data. From each row of elements in the target output data, the first target element with the maximum lane line prediction probability is determined, and the vertical anchor point position and horizontal position corresponding to the first target element are used as the predicted coordinate position of the target lane line. Lane line prediction can be completed in a shorter time, improving processing efficiency.

[0038] In practice, because the size of the lane line feature map is not fixed, if the image length is 1024, then dividing the horizontal dimension 100 times, each coordinate still corresponds to 10 pixels. Such prediction will introduce a large error. Therefore, it is necessary to improve the prediction accuracy by performing lane line offset prediction. See also Figure 3 This figure is a schematic flowchart of another embodiment of the lane line detection method provided in this application. Figure 3 As shown, based on the above embodiments, a specific implementation of S103, "using the deviation prediction sub-model in the lane line detection model to perform offset prediction processing on the lane line feature map to obtain the offset prediction result of the target lane line," includes: S1031: The deviation prediction sub-model in the lane line detection model is used to perform coordinate prediction processing on the lane line feature map to obtain the first predicted position.

[0039] In this embodiment, for example, when the lane line feature map is input into the deviation prediction sub-model, the lane line feature map is first processed for coordinate prediction. Specific steps include: scaling the lane line feature map to obtain the feature map to be predicted for the target lane line; dividing the feature map to be predicted into row and column dimensions to obtain a set of anchor point coordinate positions; using the convolutional layer of the deviation prediction sub-model to extract features from the feature map to obtain target output data; determining the first target element with the highest lane line prediction probability from each row of the target output data, and using the anchor point coordinate position corresponding to the first target element as the first predicted position of the target lane line. Figure 4 As shown, for example, one of the coordinate positions in the first predicted position determined by coordinate prediction of the lane line feature map is as follows: Figure 4The position of the loc point is shown, but the position of the pred point in the grid cannot be further determined. Therefore, it is necessary to further predict the offset between the two points pred and loc.

[0040] S1032: The first predicted position is processed using the activation function in the deviation prediction sub-model to obtain the second predicted position of the target lane line.

[0041] In this embodiment, for example, the first predicted position is processed by using the sigmoid activation function in the deviation prediction sub-model, and the output is normalized to the interval [0,1). Finally, a more accurate second predicted position can be obtained for the coordinates of each point in the lane line point chain.

[0042] S1033: Use the loss function in the deviation prediction sub-model to calculate the deviation at the second prediction position and obtain the target lane line offset prediction result.

[0043] In this embodiment, the second predicted position is used as input and passed to the loss function in the deviation prediction sub-model to calculate the deviation and obtain the deviation prediction result of the target lane line.

[0044] Understandably, during the training of the offset prediction sub-model, the SmoothL1 loss function can be used, but is not limited to, to calculate the deviation between the predicted coordinates and the actual coordinates. The SmoothL1 loss function takes the predicted position and the actual lane line position as input, calculates the difference between them, and outputs a deviation value. This deviation value reflects the degree of deviation between the predicted and actual coordinates.

[0045] In this embodiment, the deviation prediction sub-model in the lane line detection model is used to perform coordinate prediction processing on the lane line feature map to obtain the first predicted position; the activation function in the deviation prediction sub-model is used to process the first predicted position to obtain the second predicted position of the target lane line; the loss function in the deviation prediction sub-model is used to calculate the deviation of the second predicted position to obtain the offset prediction result of the target lane line, thereby improving the accuracy of predicting the lane line position.

[0046] Furthermore, based on the above embodiments, a specific implementation of S104, "determining the lane line point chain corresponding to the target lane line based on the predicted coordinate position and offset prediction results," includes: Based on the predicted coordinate position and offset prediction results, the lane line point chain optimization sub-model in the lane line detection model is used for processing to obtain the lane line point chain corresponding to the target lane line.

[0047] Among them, the lane line point chain optimization sub-model is trained by the cross ratio loss function; the lane line point chain optimization sub-model is used to integrate the predicted coordinate position and offset prediction results, and output the point chain that fits the actual lane line direction.

[0048] In this embodiment, the predicted coordinate position is combined with the offset prediction result. The offset prediction result can be applied to the predicted coordinate position through operations such as addition or multiplication. The trained lane line point chain optimization sub-model is then used to process the result, making the corrected lane line coordinate position closer to the actual coordinate position, thereby obtaining the lane line point chain corresponding to the target lane line.

[0049] In lane detection, a lane line point chain optimization sub-model is used to describe the geometry and trend of lane lines. To train this sub-model to accurately fit lane lines, a suitable loss function is needed to guide the training process. For example, the lane line point chain optimization sub-model can be trained using the cross-ratio loss function. The principle is as follows... Figure 5 As shown, in Figure 5 In the two lane lines drawn, a preset width is given for each lane line to calculate the intersection ratio at each horizontal position. The calculation rule is shown in the formula below:

[0050] Where p is Figure 5 For the shorter lane line, q is Figure 5 For the longer lane lines, w is half the width W of each lane line; x is a point on the lane line, then xw is the left endpoint of the lane line, and x+w is the right endpoint of the lane line.

[0051] This represents the horizontal overlap ratio of two lane lines, reflecting the degree of offset between the two points. When two lane lines overlap, A positive value indicates a positive value, otherwise a negative value. The degree of deviation of the entire lane line is indicated by the sum of the intersection-combination ratios of all points on the target lane line; if The larger the value, the closer the two lane lines are; if The smaller the value, the farther apart the two lane lines are.

[0052] Because lane line widths appear wider at closer objects and narrower at farther objects in camera-captured images, and because lane lines at farther distances have a higher effective pixel density, different lane widths can be set at different locations. The lane line width is determined by… Figure 5 The hypotenuse and the value of Y in the middle triangle are determined as shown in the following formula:

[0053] in, and These represent points on the lane lines. shaft and Local changes in the axis, It can represent the distance between two adjacent anchors, thus implementing and compensating for changes in lane line inclination. When the lane line is perpendicular, At this point, the value of w is W / 2, and the value of w increases with the inclination of the lane line. The value of w also serves as a control. Parameters for calculating the strictness.

[0054] Understandably, besides using the Cross-Union Ratio (CIRR) loss function to train the model and improve detection accuracy, the relationship between the prediction results of adjacent anchors can be used to further improve detection accuracy. Since the predicted coordinate distance between two adjacent anchors should not be too far apart, and the slope of the line segment formed by the output points of adjacent anchors should not differ too much, the L1 loss function and the SmoothL1 loss function can be used to calculate the losses for these two parts, respectively.

[0055] In this embodiment, based on the predicted coordinate position and offset prediction results, the lane line point chain optimization sub-model in the lane line detection model is used for processing to obtain the lane line point chain corresponding to the target lane line. The lane line point chain optimization sub-model is trained by the cross ratio loss function. The lane line point chain optimization sub-model is used to integrate the predicted coordinate position and offset prediction results and output a point chain that fits the actual lane line direction. This can reduce the error between the lane line coordinate position and the actual position, thereby improving the accuracy of generating the lane line point chain.

[0056] Furthermore, based on the above embodiments, the lane line detection model also includes a lane line type recognition sub-model, and the method further includes: Using the lane line type recognition sub-model, the lane line feature map of the target lane line is convolved to obtain a three-dimensional feature map with the number of channels being the first value.

[0057] The first value represents the number of defined lane line types, and each channel in the channel corresponds to a feature map of one lane line type.

[0058] The three-dimensional feature map is unfolded to obtain a one-dimensional feature vector.

[0059] The one-dimensional feature vector is mapped through the fully connected layer of the lane line type recognition sub-model to obtain the target matrix.

[0060] In this matrix, the row dimension represents each lane line, the column dimension represents each lane line type, and each element in the target matrix represents the probability that the lane line corresponding to the element in the row dimension is classified into the lane line type corresponding to the element in the column dimension.

[0061] The second target element with the largest value in the column dimension of the target lane line is determined from the target matrix, and the lane line type corresponding to the second target element is used as the type prediction result of the target lane line.

[0062] For example, if classified by lane line shape, lane line types can include, but are not limited to, solid lines, dashed lines, single lines, double lines, and forked lines. If classified by lane line color, lane line types can include, but are not limited to, white lines and yellow lines. However, in practical applications, lane line types do not appear singly. For example, a combination of yellow, double lines, and solid lines constitutes a double yellow line type; a combination of white, solid lines, and single lines constitutes a regular lane line type. Therefore, in this embodiment, different types can be merged according to their practical application meaning, combining multiple categories into a single category for classification, totaling K categories.

[0063] In this embodiment, taking T lane lines as an example, the lane line type recognition sub-model is used to classify the lane line types in the lane line image. First, the sub-model performs convolution processing on the lane line feature map of the target lane line to obtain a three-dimensional feature map with K channels; each channel corresponds to a specific lane line type. Then, the three-dimensional feature map with K channels is unfolded, transforming it into a one-dimensional feature vector with a feature size of K×T×N. Next, using the fully connected layer in the model, weights are learned to convert the extracted features into a score for each lane line type, mapping the unfolded one-dimensional feature vector into a target matrix of size T×K. Finally, the lane line type corresponding to the maximum value among the K scores for each lane line is determined from the target matrix as the predicted type of the current lane line.

[0064] Understandably, during the training of the lane line type recognition sub-model, cross-entropy can be used to calculate the classification loss.

[0065] In this embodiment, a lane line type recognition sub-model is used to perform convolution processing on the lane line feature map of the target lane line to obtain a three-dimensional feature map with a channel number of a first value. The three-dimensional feature map is then unfolded to obtain a one-dimensional feature vector. This one-dimensional feature vector is then mapped through a fully connected layer of the lane line type recognition sub-model to obtain a target matrix. From the target matrix, a second target element is determined, which is the element with the maximum value in the column dimension of the target lane line. The lane line type corresponding to the second target element is used as the type prediction result of the target lane line. This achieves the classification and recognition of lane line types, improves the accuracy of lane line recognition, and thus accurately determines the driving rules of the current road, such as which areas allow lane changes and which areas prohibit crossing, thereby regulating driving behavior and improving road driving safety.

[0066] Furthermore, the above embodiment involves lane line detection on a two-dimensional image captured by a camera, and the resulting detection is a chain of lane line points in a two-dimensional plane. To improve the practicality of lane line detection in real-world applications, the two-dimensional lane line point chain obtained from the two-dimensional image captured by the camera is converted to a three-dimensional bird's-eye view (BEV) plane. Based on the above embodiment, the method may further include: Based on the preset mapping table, determine the three-dimensional coordinate point corresponding to each coordinate point in the lane line point chain.

[0067] The mapping table contains the correspondence between two-dimensional coordinate points and three-dimensional plane coordinate points.

[0068] A three-dimensional lane line point chain is constructed based on the three-dimensional coordinate points corresponding to each coordinate point.

[0069] In this embodiment, for example, a preset mapping table is used to store the correspondence between two-dimensional coordinate points and three-dimensional coordinate points calibrated by a camera calibration method based on camera intrinsic and extrinsic parameters. Its representation can be as follows: Figure 6 The depth visualization lookup table is shown below; the lower part of the lookup table has a lower color depth, corresponding to the near part of the lane line in the 2D image; the upper part has a higher color depth, corresponding to the far part of the lane line in the 2D image. Then, based on each coordinate point in the lane line point chain, the corresponding 3D coordinate point in the BEV plane is obtained by looking up a preset mapping table.

[0070] Understandably, to avoid the problem of inaccurate monocular depth estimation, all point chains in the preset mapping table are preset to be on the ground plane, i.e., their height is 0.

[0071] In this embodiment, the three-dimensional coordinates corresponding to each coordinate point in the lane line point chain are determined according to a preset mapping table. The mapping table contains the correspondence between two-dimensional coordinates and three-dimensional plane coordinates. Based on the three-dimensional coordinates corresponding to each coordinate point, a three-dimensional lane line point chain is constructed, which realizes the mapping of the lane line point chain from the two-dimensional plane to the three-dimensional plane, improves the practicality of lane line detection results, and ensures the safety of vehicle driving.

[0072] Furthermore, based on the above embodiments, the pre-configured lane detection model is obtained by training an initial lane detection model. The initial feature extraction sub-model in the initial lane detection model includes a global feature extraction sub-model and a lane segmentation sub-model. The method may also include: The lane line image is obtained by performing feature extraction processing on the acquired lane line annotation map through a global feature extraction sub-model.

[0073] The lane line image is trained using a lane line segmentation sub-model to obtain trained parameters, which are then applied to a pre-configured lane line detection model. These parameters are used to enhance the supervision of lane line features in the lane line feature map.

[0074] In this embodiment, during the training of the feature extraction sub-model, the lane line segmentation sub-model can also be used to process the multi-scale features extracted by the global feature extraction sub-model at different stages, thereby enhancing the feature information of the lane lines. Specifically, in the lane line segmentation sub-model, the first feature with a size larger than a preset size in the multi-scale features is further convolved with several layers, and the second feature with a size smaller than the preset size in the multi-scale features is convolved once and upsampled and interpolated once, so that the second feature smaller than the preset size is transformed into a feature with the same size as the second feature. These features are then merged by channel superposition to predict the lane line segmentation result. The predicted segmentation result is then used to calculate the cross-entropy with the lane lines marked in the lane line image used for model training, and the calculated cross-entropy is used to supervise the lane line feature extraction process during actual model application. Each channel corresponds to one lane line. For example, if the predicted feature size is fW×fH and the number of predicted lane lines is T, then the size of the lane line segmentation output is T×fW×fH. It should be noted that this lane line segmentation sub-model is only used as an auxiliary training module to help the model learn lane line feature information more accurately. Its training process also includes a loss calculation step for pixel-by-pixel alignment of segmentation results. This lane line segmentation sub-model is only enabled during the model training phase. After the model training is completed and the parameters are fixed, there is no need to load this lane line segmentation sub-model when deploying a real vehicle. The lane line detection task can be completed solely through the feature extraction sub-model and the subsequent prediction sub-model.

[0075] Understandably, lane lines marked in lane line images can be annotated as continuous chains of points in three-dimensional space using laser point clouds that have been stacked, and then projected onto the image; alternatively, pixels in the front view image can be annotated frame by frame, and the information obtained after annotation includes lane line type information and coordinate position information for each chain of points.

[0076] In this embodiment, the acquired lane line annotation map is processed by a global feature extraction sub-model to obtain a lane line image; then, the lane line image is trained by a lane line segmentation sub-model to obtain trained parameters, which are then applied to a pre-configured lane line detection model. The parameters are used to enhance the supervision of lane line features in the lane line feature map, thereby achieving accurate identification of lane line features and improving the accuracy of lane line detection.

[0077] See Figure 7 This figure is a schematic diagram of the structure of one embodiment of a lane line detection device provided in this application. Figure 7 As shown, the device includes a feature extraction module 71, a coordinate prediction module 72, an offset prediction module 73, and a lane line point chain determination module 74.

[0078] The feature extraction module 71 is used to input the lane line image into the pre-configured lane line detection model, and use the feature extraction sub-model in the lane line detection model to perform feature extraction processing on the lane line image to obtain the lane line feature map of the target lane line.

[0079] The coordinate prediction module 72 is used to perform coordinate prediction processing on the lane line feature map using the coordinate prediction sub-model in the lane line detection model to obtain the predicted coordinate position of the target lane line.

[0080] The offset prediction module 73 is used to perform offset prediction processing on the lane line feature map using the deviation prediction sub-model in the lane line detection model to obtain the offset prediction result of the target lane line.

[0081] The lane line point chain determination module 74 is used to determine the lane line point chain corresponding to the target lane line based on the predicted coordinate position and offset prediction results.

[0082] The lane line detection device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0083] Further, based on the above embodiments, the coordinate prediction module 72 is specifically used to input the lane line feature map into the coordinate prediction sub-model, scale the lane line feature map to obtain the target lane line feature map to be predicted; the target lane line feature map is a lane line feature map whose vertical dimension size matches the preset vertical anchor point parameters; based on the preset vertical anchor point parameters, the target lane line feature map is divided in the vertical dimension and in the horizontal dimension to obtain a set of anchor point coordinate positions; the set of anchor point coordinate positions includes multiple vertical anchor point positions and multiple horizontal positions corresponding to each vertical anchor point position; the convolutional layer of the coordinate prediction sub-model is used to perform feature extraction processing on the target lane line feature map to obtain target output data; each row of the target output data corresponds to a vertical anchor point position; each column of the target output data corresponds to a horizontal position, and each element in the target output data represents the lane line prediction probability at the combination of the corresponding vertical anchor point position and the horizontal position; the first target element with the maximum lane line prediction probability is determined from each row of the target output data, and the vertical anchor point position and the horizontal position corresponding to the first target element are used as the predicted coordinate position of the target lane line.

[0084] The lane line detection device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0085] Furthermore, based on the above embodiments, the offset prediction module 73 is specifically used to perform coordinate prediction processing on the lane line feature map using the deviation prediction sub-model in the lane line detection model to obtain a first predicted position; to process the first predicted position using the activation function in the deviation prediction sub-model to obtain a second predicted position of the target lane line; and to calculate the deviation of the second predicted position using the loss function in the deviation prediction sub-model to obtain the offset prediction result of the target lane line.

[0086] The lane line detection device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0087] Furthermore, based on the above embodiments, the lane line point chain determination module 74 is specifically used to process the lane line point chain optimization sub-model in the lane line detection model based on the predicted coordinate position and offset prediction results to obtain the lane line point chain corresponding to the target lane line; the lane line point chain optimization sub-model is obtained by training through the cross ratio loss function, and the lane line point chain optimization sub-model is used to integrate the predicted coordinate position and offset prediction results to output a point chain that fits the actual lane line direction.

[0088] The lane line detection device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0089] like Figure 7 As shown in the dashed line module, based on the above embodiment, the device may also include a lane line type recognition module 75.

[0090] The lane line type recognition module 75 uses the lane line type recognition sub-model to perform convolution processing on the lane line feature map of the target lane line to obtain a three-dimensional feature map with a channel number of a first value; the first value is the defined number of lane line types, and each channel in the channel corresponds to a feature map of one lane line type; the three-dimensional feature map is unfolded to obtain a one-dimensional feature vector; the one-dimensional feature vector is mapped through the fully connected layer of the lane line type recognition sub-model to obtain a target matrix; wherein, the row dimension of the target matrix is ​​each lane line, the column dimension of the target matrix is ​​each lane line type, and each element in the target matrix represents the probability that the lane line corresponding to the element in the row dimension is classified into the lane line type corresponding to the element in the column dimension; the second target element with the maximum value in the column dimension of the target lane line is determined from the target matrix, and the lane line type corresponding to the second target element is used as the type prediction result of the target lane line.

[0091] The lane line detection device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0092] like Figure 7 As shown in the dashed line module, based on the above embodiment, the device may also include a three-dimensional lane line determination module 76.

[0093] The three-dimensional lane line determination module 76 is used to determine the three-dimensional coordinate point corresponding to each coordinate point in the lane line point chain according to a preset mapping table; the mapping table contains the correspondence between two-dimensional coordinate points and three-dimensional plane coordinate points; and constructs a three-dimensional lane line point chain based on the three-dimensional coordinate point corresponding to each coordinate point.

[0094] The lane line detection device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0095] like Figure 7 As shown in the dashed line module, based on the above embodiment, the pre-configured lane line detection model is obtained by training the initial lane line detection model. The initial feature extraction sub-model in the initial lane line detection model includes a global feature extraction model and a lane line segmentation model. Therefore, the device may also include a model training module 77.

[0096] The model training module 77 is used to perform feature extraction processing on the acquired lane line annotation map through a global feature extraction model to obtain a lane line image; to train the lane line image through a lane line segmentation model to obtain trained parameters, and to apply the parameters to a pre-configured lane line detection model; the parameters are used to enhance the supervision of lane line features in the lane line feature map.

[0097] The lane line detection device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0098] See Figure 8 This figure is a schematic diagram of the structure of an embodiment of an electronic device provided in this application. The electronic device 80 may include a processor 81 and a memory 82.

[0099] The processor 81 is communicatively connected to the memory 82, which stores computer execution instructions. The processor 81 is configured to execute the technical solutions in any of the foregoing method embodiments by executing the computer execution instructions stored in the memory 82.

[0100] Optionally, the memory 82 can be either standalone or integrated with the processor 81. Optionally, when the memory 82 is a device independent of the processor 81, the electronic device 80 may further include a bus for connecting the aforementioned devices.

[0101] The electronic device is used to execute the technical solutions in any of the foregoing method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0102] This application also provides a computer-readable storage medium storing computer-executable instructions. When these instructions are executed by a processor, they are used to implement the above-described method. The implementation principle and technical effects are similar and will not be repeated here.

[0103] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. The components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment solution according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0104] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A lane line detection method, characterized in that, The method includes: The lane line image is input into a pre-configured lane line detection model, and the feature extraction sub-model in the lane line detection model is used to perform feature extraction processing on the lane line image to obtain the lane line feature map of the target lane line. The coordinate prediction sub-model in the lane line detection model is used to perform coordinate prediction processing on the lane line feature map to obtain the predicted coordinate position of the target lane line. The offset prediction sub-model in the lane line detection model is used to perform offset prediction processing on the lane line feature map to obtain the offset prediction result of the target lane line; Based on the predicted coordinates and the offset prediction results, the lane line point chain corresponding to the target lane line is determined.

2. The method according to claim 1, characterized in that, The step of using the coordinate prediction sub-model in the lane line detection model to perform coordinate prediction processing on the lane line feature map to obtain the predicted coordinate position of the target lane line includes: The lane line feature map is input into the coordinate prediction sub-model, and the lane line feature map is scaled to obtain the target lane line feature map to be predicted; the target lane line feature map is a lane line feature map whose vertical dimension size matches the preset vertical anchor point parameters. Based on the preset vertical anchor point parameters, the feature map to be predicted is divided in the vertical dimension and in the horizontal dimension to obtain a set of anchor point coordinate positions; the set of anchor point coordinate positions includes multiple vertical anchor point positions and multiple horizontal positions corresponding to each vertical anchor point position. The convolutional layer of the coordinate prediction sub-model is used to perform feature extraction processing on the feature map to be predicted to obtain target output data; each row of the target output data corresponds to a vertical anchor point position; each column of the target output data corresponds to a horizontal position; each element in the target output data represents the lane line prediction probability at the combination of the vertical anchor point position and the horizontal position. From each row of the target output data, determine the first target element with the maximum lane line prediction probability, and use the longitudinal anchor point position and lateral position corresponding to the first target element as the predicted coordinate position of the target lane line.

3. The method according to claim 2, characterized in that, The step of using the deviation prediction sub-model in the lane detection model to perform offset prediction processing on the lane feature map to obtain the offset prediction result of the target lane line includes: The lane line detection model uses the deviation prediction sub-model to perform coordinate prediction processing on the lane line feature map to obtain the first predicted position. The first predicted position is processed using the activation function in the deviation prediction sub-model to obtain the second predicted position of the target lane line; The deviation prediction result of the target lane line is obtained by using the loss function in the deviation prediction sub-model to calculate the deviation of the second prediction position.

4. The method according to claim 3, characterized in that, The step of determining the lane line point chain corresponding to the target lane line based on the predicted coordinate position and the offset prediction result includes: Based on the predicted coordinate position and the offset prediction result, the lane line point chain optimization sub-model in the lane line detection model is used for processing to obtain the lane line point chain corresponding to the target lane line; the lane line point chain optimization sub-model is trained by the cross ratio loss function, and the lane line point chain optimization sub-model is used to integrate the predicted coordinate position and the offset prediction result to output the point chain that fits the actual lane line direction.

5. The method according to any one of claims 1-4, characterized in that, The lane line detection model also includes a lane line type recognition sub-model, and the method further includes: Using the lane line type recognition sub-model, the lane line feature map of the target lane line is convolved to obtain a three-dimensional feature map with a first value for the number of channels; the first value is the defined number of lane line types, and each channel in the channel corresponds to a feature map of one lane line type. The three-dimensional feature map is unfolded to obtain a one-dimensional feature vector; The one-dimensional feature vector is mapped through the fully connected layer of the lane line type recognition sub-model to obtain the target matrix; wherein, the row dimension of the target matrix is ​​each lane line, the column dimension of the target matrix is ​​each lane line type, and each element in the target matrix represents the probability that the lane line corresponding to the element in the row dimension is classified into the lane line type corresponding to the element in the column dimension. The second target element that has the maximum value in the column dimension of the target lane line is determined from the target matrix, and the lane line type corresponding to the second target element is used as the type prediction result of the target lane line.

6. The method according to claim 5, characterized in that, The method further includes: According to a preset mapping table, the three-dimensional coordinate point corresponding to each coordinate point in the lane line point chain is determined; the mapping table contains the correspondence between two-dimensional coordinate points and three-dimensional plane coordinate points; Based on the three-dimensional coordinate points corresponding to each coordinate point, a three-dimensional lane line point chain is constructed.

7. The method according to claim 1, characterized in that, The pre-configured lane detection model is obtained by training an initial lane detection model. The initial feature extraction sub-model in the initial lane detection model includes a global feature extraction sub-model and a lane segmentation sub-model. The method further includes: The global feature extraction sub-model is used to perform feature extraction processing on the acquired lane line annotation map to obtain the lane line image; The lane line image is trained using the lane line segmentation sub-model to obtain trained parameters, which are then applied to the pre-configured lane line detection model. These parameters are used to enhance the supervision of lane line features in the lane line feature map.

8. A lane line detection device, characterized in that, include: The feature extraction module is used to input the lane line image into the pre-configured lane line detection model, and use the feature extraction sub-model in the lane line detection model to perform feature extraction processing on the lane line image to obtain the lane line feature map of the target lane line. The coordinate prediction module is used to perform coordinate prediction processing on the lane line feature map using the coordinate prediction sub-model in the lane line detection model to obtain the predicted coordinate position of the target lane line. The offset prediction module is used to perform offset prediction processing on the lane line feature map using the deviation prediction sub-model in the lane line detection model to obtain the offset prediction result of the target lane line. The lane line point chain determination module is used to determine the lane line point chain corresponding to the target lane line based on the predicted coordinate position and the offset prediction result.

9. An electronic device, characterized in that, The device includes: a processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

Images