Lane boundary detection method based on trainable line anchors and KAN network detection heads and computer-readable storage medium

Abstract

A lane boundary detection method based on learnable line anchors and KAN (Kolmogorov-Arnold Network) network detection head is presented. The method uses a backbone image feature extraction model to perform multi-stage feature extraction on an input lane boundary image, focusing on learning and adjusting line anchors, performs line anchor pooling on the feature map output from the backbone image feature extraction model using predefined line anchors, uses a KAN network-based detection head to generate initial lane boundary prediction results, adjusts the line anchor positions for line anchor pooling in the next stage of feature map, and combines the pooling results from the previous stage into global information, and after repeating this process multiple times, performs line anchor pooling on the final stage of feature map using the adjusted line anchors.

Description

[Technical Field]

[0001] The present invention belongs to the technical field of machine vision, and relates to lane boundary detection in autonomous driving, and more particularly to a lane boundary detection method based on trainable line anchors and KAN network detection heads. [Background technology]

[0002] Lane boundary detection is a key technology in autonomous driving and is the basis for road signs and vehicle navigation. With the rapid development of deep learning, lane boundary detection technology has made significant progress, improving both detection accuracy and real-time performance. Deep learning-based lane boundary detection methods can be divided into three types: segmentation-based methods, anchor-based methods, and parametric representation-based methods.

[0003] Segmentation-based methods detect lane boundary regions by separating them from the background in an image. Due to the complex image processing involved, this method has a slow detection speed and is difficult to meet real-time requirements. Parametric representation-based methods typically use mathematical models, such as polynomial functions, to fit the shape of lane boundary lines. Because the assumptions regarding the shape of lane boundary lines are overly simplified, the detection accuracy is low under complex road conditions. Anchor-based methods use a set of predefined anchors to represent potential positions of lane boundary lines, which are then combined with other features to predict the final lane boundary lines. This method offers a good balance between accuracy and real-time requirements, demonstrating a clear advantage over the other two methods.

[0004] Although anchor-based methods have demonstrated excellent performance in lane boundary detection, they still face several challenges. First, such methods rely heavily on predefined anchors, and the quality of the anchor definition affects detection accuracy. Therefore, such methods are limited in terms of flexibility. Furthermore, densely packed predefined anchors also affect detection speed. Second, anchor-based methods suffer from poor detection accuracy under complex lane boundary conditions (e.g., partially obstructed lane boundaries, dim lighting, strong lighting, etc.). Summary of the Invention [Problem to be solved by the invention]

[0005] To address the shortcomings of the prior art, the present invention proposes a lane boundary detection method based on trainable line anchors and a KAN network detection head. This method uses a KAN network as the detection head, and performs multiple line anchor pooling operations on feature maps of different stages output from the feature extraction network according to the detection results to optimize the trainable line anchors, thereby obtaining line anchors that are more closely matched with actual lane boundary lines and improving detection accuracy and efficiency. [Means for solving the problem]

[0006] In the lane boundary detection method based on trainable line anchors and KAN network detection head, the specific steps include the following steps:

[0007] In step 1, images of lane markings are collected, and a data enrichment operation is performed to label the positions of lane markings to form a training set.

[0008] In step 2, the training set data is applied to the backbone image feature extraction model. The input is then used to perform multi-stage feature extraction, and feature maps X' at different scales are generated. p where p=1,2,…P, and P represents the feature extraction stage.

[0009] In step 3, we define N lines as predefined line anchors. The i-th line anchor is the one whose starting point is (x i ,y i ) and the angle is θ i is.

[0010] The minimum-scale feature map X'1 output from the backbone image feature extraction model is first divided into k+1 equal parts vertically by k lines parallel to the x-axis. The intersections of the k horizontal lines with each predefined line anchor are collected to obtain the sampling points {(a j ,b j )|1≦j≦k}. Grid sampling technique is used to extract feature information of sampling point coordinates from feature map X'1, and pooling feature I1 of line anchors is obtained.

[0011] In step 4, we add global information to the pooling feature I1 of the line anchors by cross-stage attention as follows:

number

[0012] In step 5, the global feature XATT1 obtained in step 4 is input to the classification detection head and the position detection head based on the KAN (Kolmogorov-Arnold Network), and the predicted probability of inclusion of lane boundary lines and the predicted lane boundary line positions are output. The predicted lane boundary line positions are calculated based on the position and length (x1, y1, θ1, len1) of the predicted line anchors and the line anchor sampling points (a j ,b j ) and the offset amount in the x-axis direction.

[0013] In step 6, the line anchor is updated based on the line anchor position obtained by the position detection head. Return to step 3 and use the updated line anchor to calculate X' of the next scale. p Line anchor pooling is performed on pooling feature I p Then, we obtain the pooled features of the previous scale, I, as follows: p-1 ...Fuse it with I1 and add global features again using cross-stage attention.

number

[0014] The classification detection head and position detection head based on the KAN network again output the position and length of the predicted line anchor and the offset amount in the x-axis direction of the line anchor sampling point.

[0015] In step 7, the maximum scale feature map X' P Repeat step 6 until pooling and global information addition for is completed, and use the classification detection head and position detection head of the KAN network to predict the probability and position of lane boundary lines. [Effects of the Invention]

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] 1. The use of a learnable line anchor mechanism reduces the dependency on the quality of predefined line anchors, improving the generalization performance and flexibility of the method.

[0018] 2. The adoption of cross-stage attention and a detection head based on the KAN network not only enhances the feature extraction effect, but also improves the accuracy of the final prediction. [Brief explanation of the drawings]

[0019] [Figure 1]1 is a flowchart of a lane boundary detection method based on trainable line anchors and KAN network detection heads. [Figure 2] Schematic diagram of the model structure of DLA (Deep Layer Aggregation). [Figure 3] FIG. 2 is a schematic diagram of a lane boundary detection model according to the embodiment. [Figure 4] FIG. 1 is a schematic diagram of the structure of cross-stage attention in an embodiment. [Figure 5] 1 is a schematic diagram of a Kolmogorov-Arnold Network (KAN) network structure. DETAILED DESCRIPTION OF THE INVENTION

[0020] In order to make the technical solutions and effects of the present invention clearer and more obvious, the following describes the technical solutions of the present invention in detail with reference to the accompanying drawings.

[0021] As shown in Figure 1, a lane boundary detection method based on trainable line anchors and a KAN network detection head can improve detection accuracy by using high-quality line anchors to better match actual lane boundary lines. Furthermore, the detection head is the key component that generates the final prediction, and its performance directly affects the overall accuracy of the model. At the same time, a well-designed loss function can more effectively guide the model to learn key features. Therefore, this application makes improvements in these three directions to improve the accuracy and efficiency of lane boundary line prediction. The specific steps are as follows:

[0022] In step 1, this embodiment uses the public lane line image dataset CULane to train and test the model. The CULane dataset covers images of lane lines in a variety of road environments, including urban roads, highways, and rural roads. At the same time, the dataset includes images of lane lines in various scenarios, such as lane lines partially obstructed by vehicles, curved lane lines, lane lines at night with poor lighting conditions, and lane lines in shadow. The dataset sufficiently covers a variety of situations that may be encountered in practical applications.

[0023] The upper boundaries of images in the dataset typically do not contain lane markings. This content is not only useless for the model's lane marking prediction, but can also interfere with the prediction of lane marking positions. Therefore, we crop a fixed height portion of the image at the top and simultaneously adjust the true lane marking positions. The cropped images undergo a series of data enhancement operations, including randomly shifting the image horizontally and vertically, flipping the image horizontally, rotating the image by a set angle, and applying Gaussian blur with a set probability. Finally, the resulting images are rescaled to match the size of the original images, and randomly divided into training and test sets.

[0024] In step 2, the training set data is input into the backbone image feature extraction model to perform multi-stage feature extraction.

[0025] In this embodiment, DLA (deep layer aggregation) and feature pyramid are used as the backbone. Feature extraction is performed as a network of the above. As shown in Figure 2, the DLA includes four stages, each of which has a tree structure. The output of the aggregation node is fed back to the backbone as the input for the next stage, and all previous blocks are aggregated at each stage. DLA uses iterative deep aggregation (IDA) and hierarchical deep aggregation (HDA) to aggregate information at different stage and block levels. In DLA, the network is divided into different stages according to resolution, and hierarchical deep aggregation is used at each stage. Iterative deep aggregation propagates aggregated features from shallower stages to deeper stages.

[0026] As shown in Figure 3, the outputs of the latter three stages are selected as the inputs of the feature pyramid, and feature maps X'1, X'2, and X'3 with different scales from small to large are obtained by top-down fusion.

[0027] In step 3, the 172 lines are aligned with the predefined line anchors {x 0 ,y 0 ,θ 0}. The i-th line anchor is defined as the line whose starting point is (x i ,y i ) and the angle is θ i First, the minimum-scale feature map X'1 output from the backbone image feature extraction model is divided vertically into k+1 equal parts by k lines parallel to the x-axis. The intersections of the k horizontal lines with each predefined line anchor are collected to obtain the sampling points {(a j ,b j )|1≦j≦k}.

number

[0028] In step 4, to further improve the quality of the learnable line anchors, especially under complex conditions such as partially occluded lane lines, we use cross-stage attention to add rich global information to the pooled features, so that the pooled features of each line anchor interact with the entire feature map, enhancing the feature representation of the line anchors.

number

[0029] In step 5, as shown in Figure 5, the KAN network has a nested structure that combines nonlinear and linear activation. Each box in the figure represents a nonlinear activation function, and the input is added through multiple nonlinear activation functions and then becomes one output of the current layer. In this embodiment, a two-layer KAN network is used as the detection head, and the scan The global feature XATT1 obtained in step 4 is input to the classification detection head and the position detection head, respectively. The classification detection head calculates the predicted probability of inclusion of lane boundary lines (neg 1 ,pos 1 ) The position detection head outputs the position and length (x 1 ,y 1 ,θ 1 ,len1 ) and the line anchor sampling point (a j ,b j ) x-axis offset {d1,d2,…d k} and outputs the predicted lane boundary line position.

[0030] In step 6, the line anchor {x 0 ,y 0 ,θ 0} to {x 1 ,y 1 ,θ 1 Return to step 3 and use the updated line anchors to perform line anchor pooling on the next scale X'2, then fuse it with the pooled feature I1 from the previous scale as follows, and add global features again by cross-stage attention:

number

[0031] In step 7, repeat step 6 and change the line anchor to {x 2 ,y 2 ,θ 2}, and perform line anchor pooling and global information addition on the maximum-scale feature map X'3, and use the classification detection head and position detection head of the KAN network to predict the probability and position of lane boundary lines.

[0032] During the training phase, the lane boundary line predictions obtained by the detection head must be paired with true lane boundary lines to guide model learning. Because pairing quality affects the effectiveness of model learning, we propose using a dynamic top-t allocation method for matching positive samples during the training phase, setting a non-fixed number of predicted lane boundary lines for each true lane boundary line. This prevents predictions with poor matching accuracy from being assigned to the true value, which could affect the training effectiveness. The dynamic top-t allocation method is as follows.

[0033] The overlap of the segments (Intersection over Union) is calculated as follows to obtain the overlap IoU.

number

[0034] The division method for segment overlap is the same as that for line anchor pooling, and is performed using k horizontal lines. j and tw j represents the half-width of the predicted lane boundary line at the jth stage and the half-width of the true lane boundary line. If the lane boundary line is straight, the half-width of the lane boundary line is w = 15 pixels. If the lane boundary line is inclined,

number

number

[0035] The steps for calculating the overlap degree of segments are as follows: The smaller of the right boundary of the true value interval and the right boundary of the prediction interval is selected as the right boundary of the intersection, and the larger of the left boundary of the true value interval and the left boundary of the prediction interval is selected as the left boundary of the intersection. If the right boundary of the intersection is equal to or greater than the left boundary of the intersection, the intersection of the segments at that time is determined; otherwise, the intersection of the segments at that time is 0. The larger of the right boundary of the true value interval and the right boundary of the prediction interval is selected as the right boundary of the union, and the smaller of the left boundary of the true value interval and the left boundary of the prediction interval is selected as the left boundary of the union, thereby determining the union of the segments at that time. The sum of the intersections of all segments divided by the sum of the unions of all segments is determined as the overlap degree between all true lane boundary lines and predicted lane boundary lines.

[0036] The t predictions with the highest degree of overlap with the true value at that time are selected, and the degrees of overlap of the t predictions are summed and rounded to obtain the maximum number of predictions ct that can actually be assigned to the true value at that time. A matching cost matrix between the predicted lane boundary lines and the true lane boundary lines is calculated, and the ct predictions with the highest degree of matching in the matching cost matrix are selected as a single true value match, while ensuring that each prediction is assigned to at most one true lane boundary line. Each position in the matching cost matrix represents the degree of matching between one prediction and one true value. The matching degree Cost in the matching cost matrix includes similarity cost and classification cost as follows:

number

[0037] The total loss function loss is set as follows:

number

[0038] During the testing phase, the lane boundary line predictions obtained by the detection head may contain multiple predictions for the same actual lane boundary line. These redundant predictions must be filtered to ensure the accuracy of the model predictions. Non-Maximum Suppression (NMS) filters out redundant predicted lane boundaries.

[0039] To prove the effectiveness of our method, we conducted a comparative experiment with conventional methods on the CULane dataset. We used the F1 score and the number of predicted frames per second (FPS) as evaluation metrics. The results are shown in Table 1.

[0040] [Table 1] In the table, F1@50 indicates that the prediction is correct when the overlap rate is greater than 0.5, and F1@75 indicates that the prediction is correct when the overlap rate is greater than 0.75. FPS stands for frames per second. As can be seen, the F1@50 evaluation metric of this method is higher than that of commonly used models in the prior art, and the F1@75 evaluation metric leads other models by more than 5 points. This demonstrates that this method is more accurate in predicting lane boundary lines and has potential application value for high-precision lane boundary line prediction. At the same time, this method predicts 70 frames per second, demonstrating good performance in terms of real-time performance.

[0041] To analyze the effectiveness of this method, an ablation analysis was conducted. Based on a single-stage lane boundary detection method, trainable line anchors, segment overlap, and KAN construction detection heads were added in sequence. The results are shown in Table 2.

[0042] [Table 2]

[0043] As can be seen from the results in Table 2, adding learnable line anchors increased the F1 metric from 77.45 to 79.27, demonstrating that multi-stage line anchor adjustment and cross-stage attention can significantly improve model performance. Incorporating learnable line anchors and modifying the loss function using segment overlap significantly and effectively increased the F1 metric from 79.27 to 79.74, demonstrating that segment overlap significantly and effectively improves the expressive power of the loss function. Incorporating learnable line anchors and segment overlap and modifying the detection head to a KAN-based classification detection head increased the F1 metric from 79.74 to 79.91, demonstrating that a KAN-based classification detection head can further improve model performance.

[0044] In summary, this method improves the traditional fixed line anchors to trainable line anchors, and combines them with the detection head and segment overlap of KAN construction. The F1 evaluation index outperforms the control model, achieving the best results. Therefore, it is useful in the fields of high-precision lane boundary detection and autonomous driving.

Claims

1. A lane boundary detection method based on trainable line anchors and KAN network detection heads, comprising the following steps: Step 1: collecting images of lane markings, performing data enrichment operations, and labeling the locations of lane markings to form a training set; The training set data is input to the backbone image feature extraction model, and multi-stage feature extraction is performed to obtain feature maps X' at different scales. ｐ Step 2, where p=1, 2, ..., P, and P represents the feature extraction stage; Define N lines as predefined line anchors, and the i-th line anchor has a starting point (x ｉ , y ｉ ) and the angle is θ ｉ and the feature map X' is obtained by the line anchor pooling method. １ Pooling feature I of line anchors from １ Step 3: Extract By using cross-stage attention, we obtain the pooling feature I of the line anchors as follows: １ Step 4 of adding overall information to [Equation 1] In the formula, XATT １ is the global feature output from the cross-stage attention, and X １ is X' １ Step 4, where σ is the feature map after convolution and resizing, and C is the number of channels of the pooled features; The overall feature XATT obtained in step 4 １ Step 5 is to input the results of the classification detection head and the position detection head based on the KAN network, respectively, and output the probability of inclusion of the lane boundary line and the predicted lane boundary line position, respectively. The predicted lane boundary line position is determined by the position and length (x １ , y １ , θ １ ,len １ ) and the line anchor sampling point (a ｊ , b ｊ Step 5 includes an offset amount in the x-axis direction of the The line anchor is updated based on the line anchor position obtained by the position detection head, and the process returns to step 3 to calculate X' of the next scale using the updated line anchor. ｐ Line anchor pooling is performed on the pooled feature I ｐ and then pooling the previous scale features I as follows: ｐ－１ …I １ Then, by using cross-stage attention, we add global features again. [Equation 2] Step 6, where conv() represents a convolution operation and cat() represents a combination in the channel dimension, and the classification detection head and position detection head based on the KAN network again output the position and length of the predicted line anchor and the offset amount of the line anchor sampling point in the x-axis direction; Maximum scale feature map X' Ｐ Step 7: repeating step 6 until pooling and global information addition for , and predicting the probability and location of lane boundary lines using the classification detection head and location detection head of the KAN network; A lane boundary detection method based on trainable line anchors and a KAN network detection head, comprising:

2. The data enhancement operations include randomly shifting the image horizontally and vertically, flipping the image horizontally, rotating the image by a set angle, and applying a Gaussian to the image with a set probability.

2. The method for detecting lane lines based on trainable line anchors and a KAN network detection head according to claim 1, further comprising applying blurring.

3. Feature extraction is performed using DLA and feature pyramid as the backbone network. The DLA is divided into four stages according to the resolution. In each stage, the output of the aggregation node is fed back to the backbone as the input of the next stage. The outputs of the last three stages are selected as the input of the feature pyramid. By top-down fusion, feature maps X′ of different scales from small to large are generated. １ , X' ２ , X' ３ The lane boundary detection method based on trainable line anchors and KAN network detection heads according to claim 1, characterized in that:

4. In the line anchor pooling method, the feature map output from the backbone image feature extraction model is first divided into k+1 equal parts in the vertical direction by k lines parallel to the x-axis, and the intersections between the k horizontal lines and each line anchor are collected to obtain the sampling points {(a ｊ , b ｊ ) | 1 ≦ j ≦ k}, and using grid sampling technology to extract feature information of sampling point coordinates from the feature map to obtain pooling features of line anchors. [Request Item 5] [Number 3] where scale is the downsampling scale of the backbone network, and b ｊ 5. The lane boundary detection method based on trainable line anchors and KAN network detection heads according to claim 4, wherein k is the ordinate of the k horizontal lines.

6. The lane boundary predictions obtained by the detection head are paired with the labeled lane boundaries to guide model training; First, the overlapping degree of the segments is calculated as follows to be the overlapping degree IoU: [Equation 4] Regarding the overlapping degree of the segments, the division is performed using k horizontal lines, and pw ｊ and tw ｊ represents the half width of the predicted lane boundary line at the jth stage and the half width of the true lane boundary line. When the lane boundary line is vertical, the half width of the lane boundary line is w=15 pixels. When the lane boundary line is inclined, [Equation 5] θ is the angle between the lane boundary and the horizontal direction, [Equation 6] where dx is the difference in x-coordinates between adjacent points above and below, and dy is the difference in y-coordinates between adjacent points above and below. and px ｊ is the x-coordinate of the jth predicted lane boundary line, and tx ｊ is the x-coordinate of the jth true value of the lane boundary line, [px ｊ -pw ｊ ,px ｊ +pw ｊ ] is the prediction interval of the segment at that time, and [tx ｊ -tw ｊ , tx ｊ +tw ｊ ] is the true value interval of the segment at that time, and min() and max() represent the minimum and maximum values, respectively. Then, select the t predictions with the highest degree of overlap with the true value at that time, add up the degrees of overlap of the t predictions and then round them off to obtain the maximum number of predictions ct that can actually be assigned to the true value at that time, calculate a matching cost matrix between the predicted lane lines and the true lane lines, and select the ct predictions with the highest degree of matching in the matching cost matrix as a single true value match, while ensuring that each prediction is assigned to at most one true lane line at a time. Each position in the matching cost matrix represents the degree of matching between one prediction and one true value, and the matching degree Cost of the matching cost matrix includes similarity cost and classification cost as follows: [Equation 7] In the formula, C ｄｉｓｔ represents the distance cost between the predicted lane boundary line and the true value, and C ｘｙ and C ｔｈｅｔａ represents the cost of the start point and angle of the line anchor, and C ｃｌｓ represents the classification cost, and W ｓａｍｅ and W ｃｌｓ are the weights of similarity cost and classification cost, respectively, The total loss function loss is set as follows: [Equation 8] In the formula, loss ｃｌｓ , loss ｘｙｔｌ , loss ｉｏｕ The lane boundary detection method based on trainable line anchors and a KAN network detection head according to claim 1, wherein the above loss represents the classification loss, the line anchor loss, and the overlap loss, respectively, and the overlap loss calculation method is consistent with the calculation of the segment overlap IoU.

7. W ｓａｍｅ = 3, W ｃｌｓ = 1, w ｃｌｓ = 2, w ｘｙｔｌ = 0.2, w ｉｏｕ 7. The lane boundary detection method based on trainable line anchors and KAN network detection heads according to claim 6, wherein:

8. A computer-readable storage medium having a computer program stored thereon, the computer program causing the computer to carry out the method according to any one of claims 1 to 7 when the computer program is executed on the computer.

Images