Lane detection method based on learnable line anchor and KAN network detection head

Abstract

The present invention discloses a lane line detection method based on learnable line anchors and a KAN network detection head. First, a backbone image feature extraction model is used to perform multi-stage feature extraction on the input lane line image. In the first few stages, the focus is on the learning and adjustment of the line anchors, and the predefined line anchors are used to perform line anchor pooling on the feature map output by the backbone image feature extraction model, and a detection head based on the KAN network is used to generate preliminary prediction results of the lane lines, and the line anchor positions are adjusted to perform line anchor pooling on the feature map of the next stage, and the pooling results of the previous stage are fused as global information. After repeated multiple times, the adjusted line anchors are used to perform line anchor pooling on the feature map of the last stage, and the final lane line prediction results are output. In addition, during the training process of the model, the segmented intersection-union ratio is proposed as part of the loss calculation to more effectively supervise model learning. This method can improve the speed and accuracy of lane line detection.

Description

Technical Field

[0001] The present invention belongs to the field of machine vision technology and relates to lane line detection in autonomous driving, and in particular to a lane line detection method based on a learnable line anchor and a KAN network detection head. Background Art

[0002] Lane detection is an important technology in autonomous driving and is the basis for road marking and vehicle navigation. With the rapid development of deep learning, lane detection technology has made significant progress, with improvements in detection accuracy and real-time performance. Lane detection methods based on deep learning can be divided into three categories: segmentation-based methods, anchor-based methods, and parameter representation-based methods.

[0003] The segmentation-based method separates the lane line area in the image from the background to achieve detection. Due to the complex image processing involved, this method has a slow detection speed and is difficult to meet real-time requirements. The parameter representation-based method usually uses mathematical models, such as polynomial functions to fit the shape of the lane line. Due to the oversimplification of the assumption of the lane line shape, the detection accuracy is not high under complex road conditions. The anchor-based method uses a predefined set of anchor points to represent the potential position of the lane line, and combines other features to predict the final lane line. This method achieves a good balance between accuracy and real-time performance, and has obvious advantages over the other two methods.

[0004] Although anchor-based methods have performed well in lane detection, they still face several challenges. First, such methods are highly dependent on predefined anchor points. The quality of anchor point definition affects the accuracy of detection, which makes such methods limited in flexibility. In addition, dense predefined anchor points will also affect the speed of detection. Second, anchor-based methods have low detection accuracy under complex lane conditions (such as lane lines are partially blocked, dim light, strong light, etc.). Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a lane line detection method based on learnable line anchors and KAN network detection head. The KAN network is used as the detection head, and multiple line anchor pooling operations are performed on the feature maps of different stages output by the feature extraction network according to the detection results to achieve learnable line anchor optimization, obtain line anchors that better match the real lane lines, and improve detection accuracy and efficiency.

[0006] The lane line detection method based on learnable line anchors and KAN network detection head includes the following steps:

[0007] Step 1: Collect lane line images, perform data augmentation operations, mark lane line positions, and form a training set.

[0008] Step 2: Input the training set data into the backbone image feature extraction model, perform multi-stage feature extraction, and obtain feature maps of different scales. , p=1,2,…P, P represents the feature extraction stage.

[0009] Step 3: Define N rays as predefined line anchors, where the starting point of the i-th line anchor is , angle is .

[0010] Minimum scale feature map output by the backbone image feature extraction model First, we divide it into two equal parts in the vertical direction by k straight lines parallel to the x-axis. Count the intersection points of k horizontal lines with each predefined line anchor as the sampling points of the line anchor . Use grid sampling technique to extract Extract the feature information of the sampling point coordinates and obtain the pooling features of the line anchor .

[0011] Step 4: Pool features for line anchors through cross-stage attention Add global information:

[0012]

[0013] in, is the global feature output of cross-stage attention. for The feature map after convolution and resizing, C is the number of channels of the pooled feature.

[0014] Step 5: The global features obtained in step 4 The classification detection head and position detection head based on KAN (Kolmogorov-Arnold Network) network are input respectively, and the probability of the current prediction containing the lane line and the predicted lane line position are output respectively. The predicted lane line position includes the position and length of the predicted line anchor , and line anchor sampling points exist The offset in the axis direction.

[0015] Step 6: Update the line anchor according to the line anchor position obtained by the position detection head. Return to step 3 and use the updated line anchor to Perform line anchor pooling to obtain pooling features , and the pooling features of the previous scale Fusion is performed, and then global features are added through cross-stage attention:

[0016]

[0017] =conv( )

[0018] Then, through the classification detection head and position detection head based on the KAN network, the predicted position and length of the line anchor and the line anchor sampling point are output. The offset in the axis direction.

[0019] Step 7: Repeat step 6 until the maximum scale feature map is completed. After pooling and adding global information, the classification detection head and position detection head of the KAN network are used to predict the probability and position of the lane line.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] 1. By using a mechanism that can learn line anchors, the dependence on the quality of predefined line anchors is reduced, and the generalization ability and flexibility of the method are improved.

[0022] 2. The use of cross-stage attention and KAN network-based detection head not only enhances the feature extraction effect, but also improves the accuracy of the final prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flow chart of the lane line detection method based on learnable line anchors and KAN network detection head.

[0024] Figure 2 This is a schematic diagram of the model structure of DLA (deep layer aggregation).

[0025] Figure 3 Schematic diagram of the lane detection model in the embodiment.

[0026] Figure 4 Schematic diagram of the structure of cross-stage attention in the embodiment.

[0027] Figure 5 This is a schematic diagram of the KAN (Kolmogorov-Arnold NetWork) network structure. DETAILED DESCRIPTION

[0028] In order to make the technical solution and effects of the present invention clearer and more explicit, the technical solution of the present invention is described in detail below with reference to the accompanying drawings.

[0029] like Figure 1As shown in the figure, the lane line detection method based on learnable line anchors and KAN network detection head can improve the detection accuracy because high-quality line anchors can better match the real lane lines; the detection head is a key component for generating the final prediction, and its performance will directly affect the accuracy of the entire model; at the same time, a well-designed loss function can more effectively guide the model to learn key features. Therefore, this application makes improvements from the above three directions to improve the accuracy and efficiency of predicting lane lines. The specific steps are as follows:

[0030] Step 1: This example uses the lane line public dataset Train and test the model. The dataset covers lane line images in various road environments, including urban roads, highways, and rural roads. At the same time, the dataset contains lane line images in various scenarios, such as lane lines partially blocked by vehicles, curved lane lines, lane lines in poor lighting conditions at night, lane lines in shadows, etc. The dataset fully covers various situations that may be encountered in actual applications.

[0031] Since the upper boundary of the image in the dataset usually does not contain lane lines, this part of the content is not only unhelpful for the model to predict lane lines but may also interfere with the prediction of the lane line position. Therefore, the fixed height part of the top of the image is cropped and the position of the true value of the lane line is adjusted. A series of data augmentation operations are performed on the cropped image, including randomly offsetting the image horizontally and vertically, flipping the image horizontally, rotating the image at a set angle, applying Gaussian blur to the image with a set probability, etc. Finally, the resulting image is rescaled to keep the size consistent with the original image and randomly divided into training and test sets.

[0032] Step 2: Input the training set data into the backbone image feature extraction model to perform multi-stage feature extraction.

[0033] In this embodiment, And feature pyramid as the backbone network for feature extraction. Figure 2 As shown, It includes four stages, each of which is a tree structure. The output of the aggregation node is fed back to the trunk as the input of the next stage, and all previous blocks are aggregated in each stage. DLA uses iterative deep aggregation and hierarchical deep aggregation to aggregate information at different stages and block levels. DLA divides the network into different stages according to resolution, and uses hierarchical deep aggregation in each stage. Iterative deep aggregation propagates the aggregated features of shallower stages to deeper stages.

[0034] The outputs of the last three stages are selected as the input of the feature pyramid, and feature maps of different scales from small to large are obtained through top-down fusion. , , ,like Figure 3 shown.

[0035] Step 3: Define 172 rays as predefined line anchors , where the starting point of the i-th line anchor is , angle is . For the minimum scale feature map output by the backbone image feature extraction model First, we divide it into two equal parts in the vertical direction by k straight lines parallel to the x-axis. Count the intersection points of k horizontal lines with each predefined line anchor as the sampling points of the line anchor :

[0036]

[0037] in, is the downsampling scale of the backbone network, which is used to scale the line anchor to the current feature map scale. is the ordinate of k horizontal lines. Then grid sampling technique is used to extract the feature map After the pooling operation, each line anchor can obtain a dimension of The feature vector of , c is the number of feature map channels before the pooling operation, and the pooling features of all line anchors are .

[0038] Step 4: To further improve the quality of learnable line anchors, especially under complex conditions such as partial occlusion of lane lines, cross-stage attention is used to add rich global information to the pooled features, so that the pooled features of each line anchor interact with the entire feature map and enhance the feature representation of the line anchor:

[0039]

[0040] in, is the global feature output of cross-stage attention. express function, for After convolution and resizing, C is the number of channels of the pooled feature, such as Figure 4 shown.

[0041] Step 5: Figure 5As shown in the figure, the KAN network is a nested combination of nonlinear activation and linear combination. Each box in the figure is a nonlinear activation function. The input is summed after multiple nonlinear activation functions and is used as an output of the current layer. In this embodiment, a two-layer KAN network is used as the detection head to transform the global feature obtained in step 4 into The output of the classification detection head predicts the probability of containing lane lines. The position detection head outputs the predicted lane line position, including the position and length of the predicted line anchor , and line anchor sampling points exist Offset in the axis direction .

[0042] Step 6: Anchor the wire Updated to . Return to step 3 and use the updated line anchors to Perform line anchor pooling and combine it with the pooling features of the previous scale Fusion is performed, and then global features are added through cross-stage attention:

[0043]

[0044]

[0045] Among them, conv() represents the convolution operation, and cat() represents the splicing in the channel dimension. Then, through the classification detection head and position detection head based on the KAN network, the predicted position and length of the line anchor, as well as the line anchor sampling point in The offset in the axis direction.

[0046] Step 7. Repeat step 6 and update the line anchor to , and the maximum scale feature map Perform line anchor pooling and add global information, and use the classification detection head and position detection head of the KAN network to predict the probability and position of the lane line.

[0047] During the training phase, the lane line predictions obtained by the detection head need to be paired with the true lane line to guide model learning. Since the pairing quality will affect the effect of model learning, it is proposed to use dynamic matching of positive samples in the training phase. The allocation method sets an unfixed number of predicted lane lines for each true lane line, which can prevent the prediction with a low matching degree from being allocated to the true value and affect the training effect. The dynamic allocation method is:

[0048] Calculate the segmented IOR as the IOR :

[0049]

[0050]

[0051]

[0052] The segmentation method of segmented intersection-and-union ratio is consistent with the segmentation of line anchor pooling, using k horizontal straight lines for segmentation. and represents the predicted half-width of the jth lane line and the half-width of the true lane line. When the lane line is vertical, the half-width of the lane line w= Pixels, when the lane line is tilted, , is the angle between the lane line and the horizontal direction, calculated based on the upper and lower adjacent points. , For the upper and lower adjacent points Coordinate difference, For the upper and lower adjacent points Coordinate difference. Predict the lane line for the jth segment coordinate, is the jth true lane line coordinate, is the prediction interval of the jth segment, is the true value interval of the jth segment. min() and max() represent the minimum and maximum values ​​respectively.

[0053] The calculation steps of the segment intersection-to-union ratio are as follows: select the smaller of the right boundary of the true value interval and the right boundary of the prediction interval as the right boundary of the intersection, select the larger of the left boundary of the true value interval and the left boundary of the prediction interval as the left boundary of the intersection, and when the right boundary of the intersection is greater than or equal to the left boundary of the intersection, the intersection of the current segment is determined, otherwise the intersection of the current segment is 0. Select the larger of the right boundary of the true value interval and the right boundary of the prediction interval as the right boundary of the union, and select the smaller of the left boundary of the true value interval and the left boundary of the prediction interval as the left boundary of the union, thereby determining the union of the current segment. The sum of all segment intersections divided by the sum of all segment unions is the intersection-to-union ratio of the entire true value lane line and the predicted lane line.

[0054] Filter out the one with the largest intersection-combination ratio with the current true value predictions, The maximum number of predictions actually assigned to the current true value is obtained by adding the intersection and union of the predictions and rounding them up. . Calculate the matching cost matrix between the predicted lane line and the true lane line, and select the one with the highest matching degree in the matching cost matrix Each prediction is used as a match for a true value, while ensuring that each prediction is assigned to at most one true lane line. Each position in the matching cost matrix represents the matching degree between a prediction and a true value. Includes similarity costs and classification costs:

[0055]

[0056] in, Represents the distance cost between the predicted lane line and the true value, and represents the starting point and angle cost of the line anchor, Represents classification cost. and are the weights of similarity cost and classification cost respectively. In this embodiment, , .

[0057] Setting the total loss function for:

[0058]

[0059] in, , , They represent classification loss, line anchor loss, and intersection-over-union loss respectively. The calculation method of intersection-over-union loss is the same as that of segmentation intersection-over-union loss. Calculation is consistent. Weight value , , .

[0060] During the testing phase, the lane line prediction obtained by the detection head may include multiple predictions of the same real lane line. These repeated predictions need to be filtered to ensure the accuracy of the model prediction. Non-maximum suppression is used to filter out redundant predicted lane lines.

[0061] In order to prove the effectiveness of this method, a comparative experiment was conducted on the CULane dataset with the existing methods, and the F1 score and predicted frames per second FPS were used as evaluation indicators. The results are shown in Table 1:

[0062] Table 1

[0063]

[0064] Among them, F1@50 means that the prediction is correct when the intersection-merge ratio is greater than 0.5, and F1@75 means that the prediction is correct when the intersection-merge ratio is greater than 0.75. FPS means frames per second. It can be seen that this method is higher than the commonly used models in the prior art in terms of the F1@50 evaluation index, and leads other models by more than 5 percentage points in terms of the F1@75 evaluation index. This shows that this method is more accurate in predicting lane lines and has potential application value in the direction of high-precision lane line prediction. At the same time, this method predicts 70 frames per second, which performs well in real-time performance.

[0065] In order to analyze the effectiveness of this method, an ablation experiment was conducted. On the basis of the single-stage lane line detection method, learnable line anchors, segmented intersection-over-union and KAN were added in turn to construct the detection head. The results are shown in Table 2:

[0066] Table 2

[0067]

[0068] According to the results in Table 2, adding learnable line anchors improves the F1 index from 77.45 to 79.27, proving that multi-stage line anchor adjustment and cross-stage attention can significantly improve the performance of the model. On the basis of including learnable line anchors, the loss function is modified using segmented intersection-over-union, which improves the F1 index from 79.27 to 79.74, proving that segmented intersection-over-union significantly and effectively improves the expressive power of the loss function. On the basis of including learnable line anchors and segmented intersection-over-union, the detection head is modified to the classification detection head constructed by KAN, which improves the F1 index from 79.74 to 79.91, proving that the classification detection head constructed by KAN can further improve the performance of the model.

[0069] In summary, this method improves the traditional fixed line anchor into a learnable line anchor, and combines KAN to construct the detection head and segmented intersection-union ratio, surpassing the control model in the F1 evaluation index and achieving the best effect. It has practical value in the field of high-precision lane line detection and autonomous driving.

Claims

1. A lane line detection method based on learnable line anchors and KAN network detection head, characterized by: The specific steps include: Step 1: Collect lane line images, perform data enhancement operations, mark lane line positions, and form a training set; Step 2: Input the training set data into the backbone image feature extraction model, perform multi-stage feature extraction, and obtain feature maps of different scales. , p=1,2,…P, P represents the feature extraction stage; Step 3: Define N rays as predefined line anchors, where the starting point of the i-th line anchor is , angle is , through the line anchor pooling method, from the feature map Extracting line anchor pooling features ; Step 4: Pool features for line anchors through cross-stage attention Add global information: ； in, It is the global feature output of cross-stage attention; for After convolution and resizing, the feature map, C is the number of channels of the pooled feature; Step 5: The global features obtained in step 4 Input the classification detection head and position detection head based on the KAN network respectively, and output the probability that the current prediction contains the lane line and the predicted lane line position respectively; the predicted lane line position includes the position and length of the predicted line anchor , and line anchor sampling points exist Offset in the axis direction; Step 6: Update the line anchor according to the line anchor position obtained by the position detection head; return to step 3 and use the updated line anchor to align the next scale. Perform line anchor pooling to obtain pooling features , and the pooling features of the previous scale Fusion is performed, and then global features are added through cross-stage attention: ； =conv( ); conv() represents the convolution operation, cat() represents the concatenation in the channel dimension, and then the classification detection head and position detection head based on the KAN network are used to output the predicted position and length of the line anchor, as well as the line anchor sampling point. Offset in the axis direction; Step 7: Repeat step 6 until the maximum scale feature map is completed. After pooling and adding global information, the classification detection head and position detection head of the KAN network are used to predict the probability and position of the lane line.

2. The lane line detection method based on learnable line anchors and KAN network detection head as claimed in claim 1, characterized in that: The data augmentation operations include randomly shifting the image in the horizontal and vertical directions, flipping the image in the horizontal direction, rotating the image at a set angle, and applying Gaussian blur to the image with a set probability.

3. The lane line detection method based on learnable line anchors and KAN network detection head as claimed in claim 1, characterized in that: by And feature pyramid as the backbone network for feature extraction; Said It is divided into four stages according to the resolution, and each stage feeds the output of the aggregation node into the backbone as the input of the next stage; The outputs of the last three stages are selected as the input of the feature pyramid, and feature maps of different scales from small to large are obtained through top-down fusion. , , .

4. The lane line detection method based on learnable line anchors and KAN network detection head as claimed in claim 1, characterized in that: The line anchor pooling method is as follows: for the feature map output by the backbone image feature extraction model, firstly divide it into two equal parts in the vertical direction through k straight lines parallel to the x-axis. Count the intersection points of k horizontal lines and each line anchor as the sampling points of the line anchor ; Use grid sampling technology to extract feature information of sampling point coordinates from feature maps and obtain pooled features of line anchors.

5. The lane line detection method based on learnable line anchors and KAN network detection head as claimed in claim 4, characterized in that: ,in is the downsampling scale of the backbone network, is the ordinate of the k horizontal lines.

6. The lane line detection method based on learnable line anchors and KAN network detection head as claimed in claim 1, characterized in that: Pair the lane line predictions obtained by the detection head with the annotated lane line positions to guide model learning; First, calculate the segmented intersection and union ratio as the intersection and union ratio : ； ； ； The segmented intersection and union ratio is segmented by k horizontal straight lines; among them, and represents the predicted half-width of the jth lane line and the half-width of the true lane line; when the lane line is vertical, the half-width of the lane line w= Pixels, when the lane line is tilted, , is the angle between the lane line and the horizontal direction, , For the upper and lower adjacent points Coordinate difference, For the upper and lower adjacent points Coordinate difference; is the jth predicted lane line coordinate, is the jth true value lane line coordinate, is the prediction interval of the current segment, is the true value interval of the current segment; min() and max() represent the minimum and maximum values ​​respectively; Then filter out the one with the largest intersection-combination ratio with the current true value predictions, The maximum number of predictions actually assigned to the current true value is obtained by adding the intersection and union of the predictions and rounding them up. ; Calculate the matching cost matrix between the predicted lane line and the true value lane line, and select the one with the highest matching degree in the matching cost matrix The prediction is used as a match for the true value, while ensuring that each prediction is assigned to at most one true value lane line; each position in the matching cost matrix represents the matching degree between a prediction and a true value; the matching degree of the matching cost matrix Includes similarity costs and classification costs: ； in, Represents the distance cost between the predicted lane line and the true value, and represents the starting point and angle cost of the line anchor, represents classified costs; and are the weights of similarity cost and classification cost respectively; Setting the total loss function for: ； in, , , They represent classification loss, line anchor loss, and intersection-over-union loss respectively. The calculation method of intersection-over-union loss is the same as that of segment intersection-over-union loss. The calculations are consistent.

7. The lane line detection method based on learnable line anchors and KAN network detection head as claimed in claim 6, characterized in that: set up , , , , .

8. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 7.

Images