A lane line edge real-time detection method and system based on uneven anchors

Abstract

The application discloses a lane line edge real-time detection method and system based on uneven anchors and belongs to the field of lane line detection. The method comprises the following steps: inputting a to-be-detected road scene image into a backbone network in a trained detection model to obtain N lane line edge images; the detection model is trained in the following manner: training the detection model by using a training data set; the training data set comprises a plurality of road scene images, and the target label of each road scene image is N annotation images generated by the edge lines of all lane lines in the image under the driving of uneven anchors; and the target label has a dimension of N x total column anchor number x total row anchor number. The application adopts the setting of multiple unevenly distributed anchors, the anchor distribution is denser closer to the top of the ROI or closer to the lane line area, and the anchor distribution is sparser farther away from the top of the ROI or farther away from the lane line area, so that the shape of the lane edge line can be better expressed, and the detection result accuracy for the lane line far away from the vehicle and the curved lane line can be improved.

Description

Technical Field

[0001] This invention belongs to the field of lane line detection, and more specifically, relates to a method and system for real-time detection of lane line edges based on non-uniform anchors. Background Technology

[0002] Lane detection is one of the key technologies in autonomous driving systems. It is integrated into the environmental perception module of autonomous vehicles to provide the vehicle with information about the environment around the vehicle, including the type of lane lines, the precise edge position of the lane lines, and the drivable area of ​​the vehicle.

[0003] However, lane detection faces challenges in both accuracy and frame rate. Firstly, there's the challenge of accuracy. Vehicles travel in diverse areas, resulting in varying numbers of lanes (single-lane and multi-lane) and different types of lanes (yellow and white, straight and curved, solid and dashed). Furthermore, lane lines in the same area appear differently at different times and in different weather conditions. For example, lane lines are clear during the day but difficult to identify at night due to insufficient light, and glare from lane lines under strong sunlight can also cause overexposure. All these factors affect lane detection accuracy. Secondly, a high frame rate is also crucial, as the lane edge detection module needs to not only acquire the lane line's position but also provide the results to other modules for further processing to arrive at the final vehicle control decision.

[0004] Deep learning-based methods can fully extract semantic features of images at various levels using deep neural networks, thus exhibiting good robustness. However, existing deep learning-based lane detection methods have the following defects and shortcomings: (1) Most existing algorithms are based on lane detection from the driver's perspective or the perspective of a dashcam. The detection result is the approximate position of the lane line rather than the precise edge position. However, cameras installed on other parts of autonomous vehicles, such as the left and right sides of the vehicle, need to detect the precise edge position of the lane line to prevent lane crossing. Most mainstream algorithms may not have good applicability. (2) Anchor-based lane detection methods, such as UFLD and LaneATT, are newly proposed real-time lane detection algorithms in recent years. However, the anchors they use are evenly distributed, resulting in a small number of anchor points used to describe lane lines that are far from the center on the side of the field of view and the part with large curvature at the top of the curved lane line. This makes it difficult to describe the shape of the lane line well, resulting in low detection accuracy. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a method and system for real-time lane line edge detection based on non-uniform anchors, which aims to achieve accurate detection of lane line edge positions and solve the problem of low accuracy for lane lines and curved lane lines with large distances to the side of the field of view under the drive of uniform anchor distribution.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a method for real-time detection of lane line edges based on non-uniform anchors, the method comprising:

[0007] The road scene image to be detected is input into the backbone network of the trained detection model to obtain N lane line edge images, where N represents the predefined maximum number of lane line edges.

[0008] The detection model includes a backbone network;

[0009] The backbone network includes an image visual feature extraction module and a dimension transformation module; wherein, the image visual feature extraction module is used to extract features at various levels of the image; and the dimension transformation module is used to transform the dimension of high-level features, i.e. semantic features, to the dimension of the target label.

[0010] The detection model is trained using a training dataset. The training dataset includes multiple road scene images. The target label for each road scene image is N labeled images generated under anchor-driven conditions, representing the edges of all lane lines in that image. The target label dimension is N × total column anchors × total row anchors. The anchors are set to be non-uniform using any of the following methods:

[0011] Method 1: Extract the ROI region of the original image, divide the ROI region into several regions according to the height direction of the image, take a row anchor with the same number of rows between each region, and take a row anchor with different number of rows between different regions, and the number of rows between them becomes smaller as they are higher up, so as to form an uneven row anchor with denser rows at the top and sparser rows at the bottom.

[0012] Method 2: The lane line is located on the left side of the image. The ROI region of the original image is cropped and divided into several regions according to the width direction of the image. Within the same region, a column anchor is taken with the same number of columns. The number of columns between different regions is different, and the number of rows between them is smaller as they are further to the left, so as to form an uneven column anchor with denser columns on the left and sparser columns on the right.

[0013] Method 3: The lane line is located on the right side of the image. The ROI region of the original image is cropped and divided into several regions according to the width direction of the image. Within the same region, a column anchor is taken with the same number of columns. The number of columns between different regions is different, and the number of rows between them is smaller as they are further to the right, so as to form an uneven column anchor with denser columns on the right and sparser columns on the left.

[0014] Method 4: Simultaneously satisfying both Method 1 and Method 2;

[0015] Method 5: Simultaneously satisfy both Method 1 and Method 3.

[0016] Preferably, the number of regions and the number of rows / columns in each region are adjusted according to the resolution of the ROI region of the input image, as follows:

[0017] Let the resolution of the ROI region of the input image be W×H;

[0018] Divided vertically from top to bottom into Total k row There are 10 regions, each of which is evenly distributed with respect to 12 regions. ( row -1)k row ... k row Anchor row, k row Indicates the preset number of row anchor regions;

[0019] Divided into the width direction Total k col When the lane lines are located on the left side of the lane, each lane is evenly distributed within that zone. m(k col -1)k col ..., mk col Each anchor is evenly distributed within its designated area when the lane line is on the right. The number of column anchors is m, which adjusts the number of column anchors within the region, and k... col This indicates the preset number of column anchor areas.

[0020] It should be noted that k row The larger the value, the more anchor regions there are, with denser anchors closer to the top and sparser anchors closer to the bottom; k col The larger the value of m, the more anchor areas there are, and the denser the areas are closer to the lane lines, and the sparser the areas are further away from the lane lines.

[0021] Preferably, the image visual feature extraction module is a lightweight ResNet18 network;

[0022] The lightweight ResNet18 network replaces all standard convolutions in all stages of the standard ResNet18 network, except for the convolution operations in the first stage and residual connections, with depthwise separable convolutions.

[0023] It should be noted that using depthwise separable convolution reduces the number of parameters in the image visual feature extraction module and improves the speed of lane line edge detection.

[0024] Preferably, the backbone network further includes a channel attention module, which is used to assign different channel attention scores to different channels of the semantic feature map output by the image visual feature extraction module, wherein the attention score represents the degree of attention the neural network pays to different channels.

[0025] It should be noted that the channel attention module added to the backbone network further improves the detection accuracy of the backbone network.

[0026] Preferably, the dimension transformation module includes a multilayer perceptron and a rearrangement module, and the final detection result is as follows:

[0027]

[0028] Among them, the output result Y i From N sheets n row_grid ×n col_grid The non-uniform anchor-driven mesh representation means that the points on the lane line edges to be detected fall within these meshes, n col_grid n represents the number of cells in each column of the final output tensor, i.e., the number of row anchors. row_grid This represents the number of cells per row in the final output tensor, i.e., the number of column anchors. This represents the output feature map obtained by sequentially processing the i-th input image through the image visual feature extraction module and the channel attention module; Flatten represents the flattening operation; MLP represents a multilayer perceptron, consisting of 3 fully connected layers, with an output format of 1×(N·(n row_grid +1)·n col_grid ), where +1 indicates that each row has an additional grid to represent lane lines not existing in that row; Reshape indicates a rearrangement operation, rearranging the output of the fully connected layer into N×(n row_grid +1)×n col_grid The format.

[0029] Preferably, the model further includes: an auxiliary branch that participates only in the training phase.

[0030] The auxiliary branches include: an upsampling module, a spatial attention module, a second channel attention module, a feature fusion module, a channel adjustment module, and a Softmax function;

[0031] The upsampling module is used to upsample the output feature maps of the second, third and fourth stages of the residual block using bilinear interpolation.

[0032] The spatial attention module is used to assign different spatial attention scores to different positions on each feature map output by the upsampling module.

[0033] The second channel attention module is used to assign different channel attention scores to different channels of each feature output by the spatial attention module;

[0034] The feature fusion module is used to stack each feature map output by the second attention module to perform feature fusion;

[0035] The channel adjustment module is used to adjust the number of fused feature channels to be the same as the target label;

[0036] The Softmax function is used to map the feature map output by the channel adjustment module to a probability representation between 0 and 1 for training and optimization.

[0037] The attention score represents the degree of attention the neural network pays to different channels or regions.

[0038] It should be noted that this module is only used during the training process to assist in optimizing the weights of the image visual feature extraction module using a multi-task training approach, and is not enabled in the testing or deployment environment to improve the actual detection frame rate.

[0039] Preferably, the channel attention module is ECANet.

[0040] Preferably, the spatial attention module is CANet.

[0041] Preferably, the N detection results are post-processed as follows:

[0042] For the non-uniform anchor drive n row_grid ×n col_grid The grid is divided into two parts. The grid points with a probability less than a preset threshold in each row are marked as "0", the grid point with the highest probability in each row is marked as "1", and the values ​​of the other positions are marked as "0". The lane line edge is determined by the presence of "1" in the detection results.

[0043] A lane line is defined by the edges of every two adjacent lane lines.

[0044] To achieve the above objectives, in a second aspect, the present invention provides a lane line edge real-time detection system based on non-uniform anchors, comprising a processor and a memory; the memory is used to store computer execution instructions; the processor is used to execute the computer execution instructions, causing the method described in the first aspect to be executed.

[0045] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0046] This invention proposes a real-time lane line edge detection method and system based on non-uniform anchors, employing five non-uniform anchor distribution settings. The non-uniform anchors are denser closer to the top or lane line area because, due to perspective effects, the top of the lane line edge is more curved when curved lane lines appear; denser anchors can better represent the curved shape of the lane line edge. Furthermore, lane lines furthest to the side of the field of view can be represented using more anchor points, ensuring detection accuracy. The anchor distribution is sparser further away from the top or lane line area because this area is closest to the camera, the lane line is clearest in the field of view, and the edge shape is straight; for straight lines, only a smaller number of points are needed to accurately represent their shape. Compared to current anchor-based algorithms that generally use uniform anchor distribution, the non-uniform anchor distribution designed in this invention can better represent the shape of the lane line edge, improve the detection accuracy of lane lines far from vehicles and curved lane lines, and reduce the output size without affecting accuracy, thus increasing the detection frame rate. Attached Figure Description

[0047] Figure 1 The flowchart of a real-time lane line edge detection method based on non-uniform anchors provided by the present invention is shown.

[0048] Figure 2 This is a schematic diagram of uneven anchor distribution provided in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0050] Figure 1 The present invention provides a flowchart of a real-time lane line edge detection method based on non-uniform anchors. The method includes: inputting the road scene image to be detected into the backbone network of a trained detection model to obtain N lane line edge images, where N represents a predefined maximum number of lane line edges.

[0051] like Figure 1 As shown, the detection model includes a backbone network; the backbone network includes an image visual feature extraction module and a dimension transformation module; wherein, the image visual feature extraction module is used to extract features at various levels of the image; the dimension transformation module is used to transform the dimension of high-level features, i.e., semantic features, to the dimension of the target label.

[0052] The detection model is trained using the following method: A training dataset is used to train the detection model; the training dataset includes multiple road scene images. The target label for each road scene image is N labeled images generated under anchor drive, representing the edges of all lane lines in that image. The dimension of the target label is N × total column anchors × total row anchors. The anchors are set to be non-uniform using any of the following methods: Method 1: The ROI region of the original image is extracted, and the ROI region is divided into several regions along the image height direction. A row anchor is taken within the same region with the same number of rows between them. The number of rows between different regions is different, and the number of rows between them decreases as they rise, forming a non-uniform row anchor with denser anchors at the top and sparser ones at the bottom; Method 2: Lane lines are located in… Method 1: The left side of the image is cropped, and the ROI region of the original image is divided into several regions along the image width direction. Within the same region, a column anchor is taken with the same number of columns spaced apart. The number of columns spaced apart varies between different regions, and the number of rows spaced apart decreases as it moves to the left, to form an uneven column anchor with denser columns on the left and sparser columns on the right. Method 2: The lane lines are located on the right side of the image. The ROI region of the original image is cropped, and the ROI region is divided into several regions along the image width direction. Within the same region, a column anchor is taken with the same number of columns spaced apart. The number of columns spaced apart varies between different regions, and the number of rows spaced apart decreases as it moves to the right, to form an uneven column anchor with denser columns on the right and sparser columns on the left. Method 3: Both methods 1 and 2 are satisfied. Method 4: Both methods 1 and 3 are satisfied.

[0053] The input image is preprocessed, including ROI region cropping, normalization, and standardization.

[0054] The Region of Interest (ROI) is defined as the portion of the input image above that lacks lane lines, leaving only the portion below with lane lines. This is because inputting the portion without lane lines into a deep neural network for feature extraction is meaningless; cropping the ROI reduces computational load and improves the detection frame rate. In this embodiment, the chosen method is to crop the upper 1 / 3 of the input image, leaving the lower 2 / 3.

[0055] The normalization and standardization process involves mapping the input image to a range of 0 to 1 by dividing it by 255, then subtracting the mean of the dataset and dividing it by the standard deviation of the dataset to reduce the scale difference between features, resulting in less oscillation in the direction of gradient descent during training and faster convergence.

[0056] Preferably, the number of regions and the number of rows / columns in each region are adjusted according to the resolution of the ROI region of the input image, as follows:

[0057] Let the resolution of the ROI region of the input image be W×H. row This indicates the preset number of anchor regions, which are then divided vertically from top to bottom as follows: Total k rowThere are 10 regions, each of which is evenly distributed with respect to 12 regions. ( row -1)k row ... k row Individual anchor. k col Given the preset number of column anchor areas, the width direction is divided into... Total k col When the lane lines are located on the left side of the lane, each lane is evenly distributed within that zone. m(k col -1)k col ..., mk col Each anchor is evenly distributed within its designated area when the lane line is on the right. There are column anchors, where m is used to adjust the number of column anchors within a region. row The larger the value (ranging from 3 to 6), the more anchor regions there are, with denser anchors closer to the top and sparser anchors closer to the bottom; k col The larger the value (ranging from 2 to 5) or the larger the value m (ranging from 3 to 10), the more anchor regions there are, and the denser the regions are closer to the lane lines, and the sparser the regions are further away from the lane lines.

[0058] Figure 2 This is a schematic diagram of uneven anchor distribution provided for an embodiment of the present invention. Figure 2 As shown, in this embodiment, a non-uniform anchor designed using non-uniform distribution method 4 is employed, where k row =k col =3, m=5.

[0059] Preferably, considering the real-time requirements of detection in low-computing-power devices such as embedded devices, the image visual feature extraction module is a lightweight ResNet18 network; the lightweight ResNet18 network replaces all standard convolutions in all stages of the standard ResNet18 network except for the convolution operation of the first stage and residual connection with depthwise separable convolutions.

[0060] Specifically, the standard ResNet18 network's main structure is divided into six parts. The first part uses 7×7 large convolutions and max pooling for downsampling, aiming to rapidly reduce image resolution while preserving as much original information as possible. The subsequent second to fifth parts use 3×3 convolutional kernels for feature extraction in small regions. After each part, the output image resolution is halved, and the number of channels is increased to obtain more feature information. The last part consists of fully connected layers and global average pooling; this part is not for feature extraction. Considering that the first part aims to preserve as much original information as possible, and that the summation of channels in standard convolution can be seen as a feature enhancement operation, the standard convolution in the first part is retained. The second to fifth parts are residual blocks, the main part for feature extraction. These parts contain layers with equal input and output channels, so depthwise separable convolutions can be used instead of standard convolutions. The first convolutional layer and the convolutional operations of the residual connections in the second to fifth parts are downsampling operations. Similarly, to preserve more information, these convolutional layers retain the original standard convolution operations.

[0061] Preferably, the backbone network further includes a channel attention module, used to assign different channel attention scores to different channels of the semantic feature map output by the image visual feature extraction module, wherein the attention score represents the degree of attention the neural network pays to different channels. This further improves the detection accuracy of the backbone network.

[0062] Preferably, the channel attention module is ECANet. ECANet proposes a local cross-channel interaction strategy without dimensionality reduction, which effectively avoids the impact of dimensionality reduction on the channel attention learning effect. This module achieves significant gains with only a small increase in parameters. Furthermore, the appropriate cross-channel interaction proposed by ECANet can significantly reduce model complexity while maintaining performance. Top-1 accuracy results on the ImageNet dataset classification task show that ECANet outperforms SENet and CBAM. Therefore, this invention prefers ECANet for channel attention score allocation.

[0063] In the backbone of the network, considering the need for real-time detection, X4i only uses a lightweight network ECANet to add channel attention to the output of the feature extraction network, in order to improve the detection accuracy of the network backbone while ensuring the detection frame rate:

[0064]

[0065] The feature map for assigning attention scores includes the outputs of the third to fifth stages of the lightweight ResNet18 network, namely the outputs of the second, third, and fourth stages of the residual block. Where i represents the output result of the i-th sample as input. The segmentation-aided learning module needs to utilize... These three feature maps come from different scales; and the backbone needs to utilize... The attention scores are then assigned to the data before it is fed into subsequent layers.

[0066] Preferably, the dimension transformation module includes a multilayer perceptron and a rearrangement module, and the final detection result is as follows:

[0067]

[0068] Where i represents the i-th input and the output result Y i From N sheets n row_grid ×n col_grid The non-uniform anchor-driven mesh representation means that the points on the lane line edges to be detected fall within these meshes, n col_grid n represents the number of cells in each column of the final output tensor, i.e., the number of row anchors. row_grid This represents the number of cells per row in the final output tensor, i.e., the number of column anchors. This represents the output feature map obtained by sequentially processing the i-th input image through the image visual feature extraction module and the channel attention module; Flatten represents the flattening operation; MLP represents a multilayer perceptron, consisting of 3 fully connected layers, with an output format of 1×(N·(n row_grid +1)·n col_grid Reshape represents a rearrangement operation, rearranging the output of the fully connected layer into N×(n) arrays. row_grid +1)×n col_grid The format.

[0069] The final output of the main component is trained under supervised training using a loss function. First, the Softmax function is used to transform the output into a probability distribution Pr between 0 and 1. i Then, the cross-entropy loss function LCE is used to calculate Pr in the channel direction. i and Losses:

[0070]

[0071] in, It is to manually annotate pixels. The result is obtained by quantizing the non-uniform anchor-driven mesh that falls into the corresponding channel according to different lane line edge markings. Final result Y i shape and Consistent, each channel stores information about the edge of a lane line, and the detection result of the lane line edge is the result of selecting cells on a predefined uneven anchor.

[0072] Preferably, the model further includes: an auxiliary branch that participates only in the training phase; the auxiliary branch includes: an upsampling module, a second channel attention module, a spatial attention module, a feature fusion module, a channel adjustment module, and a Softmax function. The upsampling module is used to upsample the output feature maps of the residual block in the second, third, and fourth stages using bilinear interpolation. The spatial attention module is used to assign different spatial attention scores to different positions on each feature map output by the upsampling module. The second attention module is used to assign different channel attention scores to different channels of each feature output by the spatial attention module. The feature fusion module is used to stack each feature map output by the second attention module for feature fusion. The channel adjustment module is used to adjust the number of fused feature channels to be the same as the target label. The Softmax function is used to map the feature maps output by the channel adjustment module to a probability expression between 0 and 1 for training and optimization. The attention score represents the degree of attention the neural network pays to different channels or regions.

[0073] Preferably, the spatial attention module is CANet. CANet models long-distance dependencies between channels by extracting attention feature maps in the horizontal and vertical directions. Moreover, compared with the global pooling method used by CBAM, the horizontal and vertical attention method can provide more accurate spatial location information more effectively. Therefore, this invention prefers CANet to allocate spatial attention scores.

[0074] Since the segmentation-assisted learning module is only enabled during training to help optimize the performance of the feature extraction network, and is not enabled during model testing and actual deployment, there is no need to consider the complexity of this part of the network. In this part, channel and spatial attention scores are assigned to the feature map. The three feature maps have different resolutions than the ground truth values ​​of the labels, so they need to be upsampled to the same size first. Then, spatial attention and channel attention are added using CANet and ECANet.

[0075]

[0076] f ECA (x)=σ{Conv 1×1 [GAP(x)]}*x

[0077] f CA (x)=σ{Conv 2group[BN(Concat(AVP x (x), AVP y (x)))]}*x

[0078] Among them, Up bilinear (·) indicates that upsampling is performed using bilinear interpolation, f CA (·) Apply CANet to the feature map to assign spatial attention scores, f ECA (·) ECANet is used to assign channel attention scores to the feature maps, where GAP represents global average pooling and AVPx represents average pooling along the x-direction. y This represents average pooling along the y-direction, BN represents batch normalization, Concat represents feature stacking, σ is the Sigmoid function, and Conv... 2group This indicates that the convolution is performed in two groups.

[0079] Furthermore, a loss function needs to be designed for the segmentation task auxiliary learning module with added attention mechanism to supervise the training of the model.

[0080] In the segmentation task-assisted learning module, the first step is to upsample the feature maps and assign attention decentralization. Feature stacking is performed, followed by 1×1 convolution for dimensionality adjustment and feature fusion. The softmax function is used to map the behavior combination to a probability representation between 0 and 1, and finally the cross-entropy loss function is used for training and optimization.

[0081]

[0082]

[0083]

[0084] Where N represents the predefined maximum number of lane line edges, Pr i The result after converting the output into probabilities. To convert manually segmented and labeled pixel-level ground truth into the result of one-hot encoding of each pixel in the channel direction, the cross-entropy loss function LCE is used to calculate Pr in the channel direction. i and The loss.

[0085] The total loss function is a weighted sum of the main branch loss function and the auxiliary branch loss function:

[0086] L total =αL main +βL seg

[0087] Here, α and β are hyperparameters used to adjust the loss weights.

[0088] Preferably, the obtained N detection results are post-processed as follows: for the n non-uniform anchor driven row_grid ×n col_grid The grid is divided into two parts. The grid points with a probability less than a preset threshold in each row are marked as "0", the grid point with the highest probability in each row is marked as "1", and the values ​​of the other positions are marked as "0". The lane line edge is determined by the presence of "1" in the detection result. A lane line is determined by every two adjacent lane line edges.

[0089] Furthermore, the present invention provides a lane line edge real-time detection system based on non-uniform anchors, including a processor and a memory; the memory is used to store computer execution instructions; the processor is used to execute the computer execution instructions, causing the above method to be executed.

[0090] The effectiveness of this invention is demonstrated through experiments. The BJ-ULL dataset was used for performance testing. The BJ-ULL dataset, short for Beijing-Urban Lane Line, consists of lane line images collected from streets in Beijing's urban area. Compared to the existing publicly available lane line dataset TuSimple, BJ-ULL is the first dataset that can be used for lane line edge detection and lane line retroreflection coefficient calculation. Furthermore, BJ-ULL includes challenging detection scenarios such as curves, low-light conditions at night, and strong light. The original BJ-ULL dataset consists of 10 five-minute color videos (including both daytime and nighttime scenes) and 4000 grayscale lane line images, all with a resolution of 4096×2160 pixels. The video images were extracted as follows: for common straight-line scenes, one image was extracted every 10 frames; for less common scenes such as curves, low-light conditions at night, and strong light, one image was extracted every 5 frames. To reduce the dataset size, all training images were downsampled to a resolution of 1280×720 pixels. The processed dataset contains a total of 8191 images, of which 7191 were randomly selected for the training set and the remaining 1000 were used as the test set. The specific composition of the dataset is shown in Table 1.

[0091] Table 1. Composition of the BJ-ULL (Beijing-Urban Lane Line) dataset

[0092]

[0093]

[0094] Before introducing the specific evaluation metrics, the following criteria need to be explained: TP (TruePositive): The predicted result is true and the actual result is also true; TN (True Negative): The predicted result is false and the actual result is also false; FP (False Positive): The predicted result is true and the actual result is false; FN (False Negative): The predicted result is false and the actual result is true.

[0095] The specific criteria are as follows:

[0096] The algorithm compares the points in the i-th column (i.e., the i-th lane edge line as perceived by the algorithm) with the points in the i-th row of the ground truth annotation file. If the ordinate of the j-th row anchor point in the algorithm's output matches the ordinate of the k-th row anchor point in the ground truth file, then the j-th row anchor point in the i-th column of the algorithm's output belongs to the lane edge line. Further, the x-coordinates are evaluated. If the difference in x-coordinates is less than a preset threshold (thresh), the result is TP; otherwise, it is FP.

[0097] If the algorithm outputs that the k-th row anchor point in the i-th column does not belong to the lane edge (i.e., the output is 0), and there is a point with that row anchor point as its ordinate in the i-th row of the ground truth, then it indicates a missed detection and is judged as FN. If after traversing the i-th row there is no point with that row anchor point as its ordinate, then it indicates that the row anchor point is indeed not the lane edge and the detection is correct, and it is judged as TN.

[0098] If the algorithm outputs a lane edge line at the j-th row of the i-th column, and the value of i exceeds the number of rows in the ground truth annotation (e.g., the algorithm outputs a 3rd lane edge line, but the ground truth only contains 2 lane edge lines), then the result is classified as FP. If the algorithm outputs a lane edge line at the j-th row of the i-th column, and the value of i exceeds the number of rows in the ground truth annotation (i.e., there is no annotation in the ground truth), but because the point is not a lane edge line, then the result is classified as TN.

[0099] Based on these four judgment results, three evaluation indicators can be constructed: Precision, Recall, and F1. score :

[0100]

[0101]

[0102]

[0103] Precision refers to the percentage of correctly detected line anchors out of all detected line anchors; Recall refers to the percentage of correctly detected line anchors out of all line anchors along the lane edges in the validation set; F1 score To comprehensively evaluate the indicators, the results of accuracy and recall were considered together; therefore, the experiment adopted F1. score As an evaluation indicator.

[0104] This invention, along with two open-source and real-time lane detection algorithms, UFLD and LaneATT, were used to achieve F1 scores in various scenarios on the BJ-ULL dataset. score The comparison results are shown in Table 2. The results show that the method of this invention is optimal in terms of average results compared to the other two algorithms, proving the effectiveness of the method of this invention.

[0105] Table 2. F1 of the present invention, UFLD, and LaneATT in various scenarios in BJ-ULL. score contrast

[0106]

[0107] Table 3 shows the comparison results of this invention and two open-source, real-time lane detection algorithms, UFLD and LaneATT, in straight and curved scenarios. Position 1 represents the lane closest to the vehicle, position 4 represents the outermost lane, and positions 2 and 3 are in between. For straight scenarios, only the results of the outermost lane at position 4 are compared; for curved scenarios, the results of all four lanes are compared. The results show that the method of this invention achieves optimal detection results for the outermost lane and curved lane scenarios, demonstrating the effectiveness of this method.

[0108] Table 3. F1 of the present invention, UFLD, and LaneATT in different scenarios in BJ-ULL. score contrast

[0109]

[0110] Table 4 shows the comparison results of the present invention with two open-source and real-time lane detection algorithms, UFLD and LaneATT, in terms of model size and inference frame rate. The inference frame rate was obtained by testing on a Tesla V100.

[0111] Table 4 Comparison of this invention with UFLD and LaneATT in terms of model size and inference frame rate.

[0112] Comparison items UFLD LaneATT This invention Inference frame rate (FPS) 280 153 350 Model size (MB) 340 240 301

[0113] The results show that, in terms of inference frame rate, the proposed method has the fastest inference frame rate; in terms of model size, LaneATT's model is the smallest, followed by the proposed method. Overall, the proposed method's model outperforms both UFLD and LaneATT algorithms in terms of inference frame rate and F1 score accuracy. Although the proposed method's model size is not the smallest, its 301MB size is still acceptable.

[0114] In summary, this invention has certain advantages compared to existing open-source deep learning-based lane detection algorithms. This invention acquires visual features of the input image through a deep neural network and quickly filters out valuable information by introducing channel and spatial attention mechanisms to obtain more accurate feature representations. Simultaneously, this invention introduces a non-uniform anchoring method, making the shape representation of lane edges more accurate. Furthermore, by introducing depthwise separable convolution, this invention effectively improves the frame rate of model detection, meeting the frame rate requirements of practical model deployment.

[0115] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for real-time detection of lane line edges based on non-uniform anchors, characterized in that, The method includes: The image of the road scene to be detected is input into the backbone network of the trained detection model to obtain... Lane edge image, This indicates the predefined maximum number of lane line edges; For the aforementioned non-uniform anchor drive The grid, This represents the number of cells in each column of the final output tensor, i.e., the number of row anchors. This represents the number of cells per row in the final output tensor, i.e., the number of column anchors. The Softmax function is used to convert the output into a probability distribution between 0 and 1. The values ​​of grid points with a probability less than a preset threshold in each row are marked as "0", the value of the grid point with the highest probability in each row is marked as "1", and the values ​​of the remaining positions are marked as "0". The presence of "1" in the detection results determines the lane line edge; every two adjacent lane line edges determine a lane line. The detection model includes a backbone network; The backbone network includes an image visual feature extraction module and a dimension transformation module; wherein, the image visual feature extraction module is used to extract features at various levels of the image; and the dimension transformation module is used to transform the dimension of high-level features, i.e. semantic features, to the dimension of the target label. The detection model is trained using a training dataset; the training dataset includes multiple road scene images, and the target label for each road scene image is generated by anchor-driven edge lines of all lane lines in that image. There are labeled images, and the target label dimension is... × Total number of column anchors × Total number of row anchors, wherein the anchors are set to be non-uniform in any of the following ways: Method 1: Extract the ROI region of the original image, divide the ROI region into several regions according to the height direction of the image, take a row anchor with the same number of rows between each region, and take a row anchor with different number of rows between different regions, and the number of rows between them becomes smaller as they are higher up, so as to form an uneven row anchor with denser rows at the top and sparser rows at the bottom. Method 2: The lane line is located on the left side of the image. The ROI region of the original image is cropped and divided into several regions according to the width direction of the image. Within the same region, a column anchor is taken with the same number of columns. The number of columns between different regions is different, and the number of rows between them is smaller as they are further to the left, so as to form an uneven column anchor with denser columns on the left and sparser columns on the right. Method 3: The lane line is located on the right side of the image. The ROI region of the original image is cropped and divided into several regions according to the width direction of the image. Within the same region, a column anchor is taken with the same number of columns. The number of columns between different regions is different, and the number of rows between them is smaller as they are further to the right, so as to form an uneven column anchor with denser columns on the right and sparser columns on the left. Method 4: Simultaneously satisfying both Method 1 and Method 2; Method 5: Simultaneously satisfy both Method 1 and Method 3.

2. The method as described in claim 1, characterized in that, Adjust the number of regions and the number of rows / columns in each region according to the resolution of the ROI region of the input image, as follows: Let the resolution of the ROI region of the input image be... ; Divided vertically from top to bottom into , ... common There are 10 regions, each of which is evenly distributed with respect to 12 regions. , ... Anchor line, Indicates the preset number of row anchor regions; Divided in the width direction into , ... common When the lane lines are located on the left side of the lane, each lane is evenly distributed within that zone. , ... Each anchor is evenly distributed within its designated area when the lane line is on the right. , ... Each column anchor, Used to adjust the number of column anchors within a region. This indicates the preset number of column anchor areas.

3. The method as described in claim 1, characterized in that, The image visual feature extraction module is a lightweight ResNet18 network; The lightweight ResNet18 network replaces all standard convolutions in all stages of the standard ResNet18 network, except for the convolution operations in the first stage and residual connections, with depthwise separable convolutions.

4. The method as described in claim 3, characterized in that, The backbone network also includes a channel attention module, which is used to assign different channel attention scores to different channels of the semantic feature map output by the image visual feature extraction module. The attention score represents the degree of attention the neural network pays to different channels.

5. The method as described in claim 4, characterized in that, The dimension transformation module includes a multilayer perceptron and a rearrangement module. The final detection result is as follows: Among them, the output results From N sheets The non-uniform anchor-driven mesh representation means that the points on the lane line edges that need to be detected fall within these meshes. Indicates the first The output feature map is obtained by sequentially processing the input image through the image visual feature extraction module and the channel attention module; This indicates a flattening operation; MLP stands for Multilayer Perceptron, which consists of three fully connected layers, and its output format is... Reshape indicates a rearrangement operation, which rearranges the output of the fully connected layer into... The format.

6. The method as described in claim 5, characterized in that, The model also includes an auxiliary branch that participates only in the training phase. The auxiliary branches include: an upsampling module, a spatial attention module, a second channel attention module, a feature fusion module, a channel adjustment module, and a Softmax function; The upsampling module is used to upsample the output feature maps of the second, third and fourth stages of the residual block using bilinear interpolation. The spatial attention module is used to assign different spatial attention scores to different positions on each feature map output by the upsampling module. The second channel attention module is used to assign different channel attention scores to different channels of each feature output by the spatial attention module; The feature fusion module is used to stack each feature map output by the second attention module to perform feature fusion; The channel adjustment module is used to adjust the number of fused feature channels to be the same as the target label; The Softmax function is used to map the feature map output by the channel adjustment module to a probability representation between 0 and 1 for training and optimization. The attention score represents the degree of attention the neural network pays to different channels or regions.

7. The method as described in claim 5 or 6, characterized in that, The channel attention module is ECANet.

8. The method as described in claim 6, characterized in that, The spatial attention module is CANet.

9. A real-time lane line edge detection system based on non-uniform anchors, characterized in that, Including processor and memory; The memory is used to store computer-executed instructions; The processor is configured to execute the computer execution instructions, causing the method described in any one of claims 1 to 8 to be executed.

Images