A lane detection method based on multi-dimensional collaborative attention and feature aggregation
By adopting multi-dimensional coordinated attention and feature aggregation method in lane line detection, the problem of reduced lane line detection accuracy and speed in complex environments is solved, and a more efficient lane line detection effect is achieved.
Patent Information
- Application Number
- CN202411017247.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-07-29
AI Technical Summary
The existing lane line detection technology has reduced detection accuracy and speed in complex environments, making it difficult to meet high-performance needs, especially in the absence of lane line parts, shading, road area water and reflection.
The lane line detection method based on multi-dimensional synergistic attention and feature aggregation is adopted. By constructing a detection model including feature extraction network, auxiliary segmentation network, feature information transmission module and existence positioning module, the lane line is extracted using ResNet network and multi-dimensional synergistic attention module. Features, combining feature aggregation module and MLP classifiers, enhance the perception and detection speed of feature information.
It improves the accuracy and speed of lane line detection, especially in complex environments, performs well, meets real-time requirements, and enhances the perception and detection effect of the model.
Smart Images

Figure CN118552936B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of lane line detection, and specifically to a lane line detection method based on multi-dimensional collaborative attention and feature aggregation. Background Art
[0002] With the rise of artificial intelligence, autonomous driving has gradually become one of the hot research areas. Lane detection is a crucial part of autonomous driving. It not only helps vehicles follow the trajectory of lane lines and relieves driver fatigue, but also greatly improves driving safety. Although the current lane detection technology has made remarkable achievements, it still has shortcomings in complex environments. For example, the partial loss of lane lines, obstruction by vehicles ahead, water on the road and reflections may affect the field of view, resulting in reduced detection accuracy and speed. Faced with these problems, the existing lane detection algorithms have not yet fully met the requirements of high performance in complex environments.
[0003] Lane detection methods are currently divided into two categories: traditional lane detection methods and deep learning-based lane detection methods. Traditional lane detection methods can be further divided into feature-based lane detection and model-based lane detection. Feature-based lane detection methods usually use the specific shape, color, texture or geometric features of lanes in the image for detection. These feature information is compared with environmental features to distinguish lanes from the background. This type of method has high requirements on the integrity of lanes and is difficult to deal with situations where lanes are blocked or severely worn. Model-based lane detection methods mainly rely on image processing technology and geometric models. The specific steps usually include image preprocessing, edge detection, line detection, and lane fitting. For example, the basic principle of Hough transform for line detection is to convert points in image space to parameter space, find the point with the most cumulative votes in parameter space through a voting mechanism, and thus determine the straight line in the image. Model-based detection methods are more robust in dealing with factors such as noise and illumination changes, and can reduce the interference of noise and outliers. However, this type of method requires a large amount of data and model training, has a large amount of calculation, and is sensitive to changes in road conditions. In general, traditional lane detection methods not only have slow detection speeds but also poor accuracy.
[0004] The lane detection method based on deep learning mainly extracts lane feature information from the image through a convolutional neural network, and then uses these features for algorithm classification and reasoning to finally obtain a lane image. For example, the method based on semantic segmentation achieves accurate detection of lanes by classifying each pixel. However, the real-time performance of this method in some complex scenarios is still insufficient. Another method is based on row classification, which decomposes the image into several rows (or regions), and then classifies the pixels in each row to determine the location of the lane. Although this method has a relatively fast calculation speed, it has limited ability to handle curves and weak global information perception. Therefore, the method based on deep learning can currently perform good detection in most good situations, but it still has shortcomings in complex environments, such as blurred lanes, reflections, and occlusions. Summary of the invention
[0005] In view of the shortcomings of the prior art, the present invention provides a lane line detection method based on multi-dimensional collaborative attention and feature aggregation, which aims to solve the problems in the background technology.
[0006] To achieve the above object, the present invention provides the following technical solution: a lane detection method based on multi-dimensional collaborative attention and feature aggregation, comprising the following steps:
[0007] Step S1: construct a data set, which includes several lane images;
[0008] Step S2: constructing a lane detection model; the lane detection model includes a feature extraction network module, an auxiliary segmentation network module, a feature information transfer module and an existence positioning module; the feature extraction network module is composed of a ResNet network and a multi-dimensional collaborative attention module; the auxiliary segmentation network module is composed of an auxiliary layer and a combination layer; the feature information transfer module is composed of a feature aggregation FAM module and an MLP classifier; the existence positioning module is composed of an existence branch module and a positioning branch module; the feature extraction network module is connected to the feature information transfer module, and the feature information transfer module is connected to the existence positioning module; the auxiliary segmentation network module interacts with the feature extraction network module and is in a parallel structure with the feature information transfer module;
[0009] Step S3: Input the lane image into the feature extraction network module, and obtain the lane line feature image in the lane image through the ResNet network. The ResNet network is divided into 5 layers, and the MCA multi-dimensional collaborative attention module is added to the last three layers of the ResNet network; each layer of the ResNet network performs convolution, batch normalization, ReLu activation and maximum pooling operations on the input lane image to obtain the lane line feature image of each layer;
[0010] Step S4: inputting the obtained lane line feature images of the last three levels of the ResNet network into the auxiliary segmentation network module, fusing the lane line feature images of the last three levels into an auxiliary feature map through the auxiliary segmentation network module for output, and obtaining the auxiliary loss function by calculating the auxiliary feature map;
[0011] Step S5: Input the obtained lane line feature image of the last layer of the ResNet network into the feature aggregation FAM module in the feature information transmission module, aggregate the information of each direction of the lane line feature image through K iterations, and obtain the final lane line feature image, input the final lane line feature image into the MLP classifier, and the MLP classifier inputs the final lane line feature image into the existence positioning module;
[0012] Step S6: the existence branch module in the localization module calculates the existence prediction result of the lane line in the final lane line feature image X, obtains the existence loss function through the existence prediction result, judges whether the lane line in X exists, and the localization branch module calculates the coordinate prediction result of each lane line in the final lane line feature image, obtains the localization loss function through the coordinate prediction result, combines the coordinate prediction result with the number of lane line categories to obtain the specific coordinates of the lane line, obtains the expected loss function through the specific coordinate calculation, and finally classifies and locates each lane line in X; the existence branch module and the localization branch module interact, filter out the lane lines that do not exist in X, and use the auxiliary loss function, the existence loss function, the localization loss function and the expected loss function to calculate the overall loss of the lane line detection model;
[0013] Step S7: Use the lane line detection model after calculating the overall loss to process the input lane image and output the specific position of the lane line in the lane image.
[0014] Furthermore, the ResNet network consists of 1 convolutional layer and 4 residual blocks, which are encoded as L1 layer, L2 layer, L3 layer, L4 layer and L5 layer in sequence, wherein each residual block contains two residual units BasicBlock, each residual unit contains two 3×3 convolutional layers, each 3×3 convolutional layer is connected to a batch normalization layer and a ReLU activation function, and the MCA multi-dimensional collaborative attention module is added after the L3 layer, L4 layer and L5 layer; the auxiliary layer is composed of multi-layer convolutional layers, batch normalization layers and activation functions ReLU; the positioning branch module and the existence branch module are both composed of multi-layer convolutions, ReLU functions and fully connected layers.
[0015] Furthermore, the specific process of step S3 is as follows: the lane image is input into the L1 layer of the ResNet network. First, the lane image is convolved with a 7×7 convolution with a step size of 2 to obtain image information, and the image information is batch normalized to obtain a feature map. , feature map The activated feature map is obtained by ReLU activation function , and finally the activated feature map Perform convolution pooling to obtain the pooled feature map , that is, the feature information output by the L1 layer , expressed as:
[0016] (1);
[0017] In the formula, It is a 7×7 convolution operation; is batch normalization; is the activation function; It is a 3×3 convolution pooling; is the convolution kernel weight;
[0018] Will As the input of L2 layer, First, the first residual unit of the L2 layer is entered. After two convolution operations, two batch normalization operations and ReLU activation operations, the output feature of the first residual unit is obtained. , then As the input of the second residual unit of the L2 layer, the feature information of the L2 layer output is obtained after two convolution operations, two batch normalization operations and ReLU activation operations. , expressed as:
[0019] (2);
[0020] (3);
[0021] The structures of L2, L3, L4 and L5 are consistent. The feature information output by L3 is calculated according to equations (2) and (3): , feature information output by L4 layer And the feature information output by L5 layer ;
[0022] The MCA multi-dimensional collaborative attention module is added after the L3, L4 and L5 layers. Before inputting to the L4 layer, After being input into the L3 layer, the MCA multi-dimensional collaborative attention module connected to the L3 layer performs spatial dimension operation of attention to obtain the feature image of the third lane line. ,Will Input to L4 layer to get feature information ,Will After being input into the L4 layer, the MCA multi-dimensional collaborative attention module connected to the L4 layer performs spatial dimension operation of attention to obtain the feature image of the fourth lane line. ,Will As the input of L5 layer, output feature information , the feature information After being input into the L5 layer, the MCA multi-dimensional collaborative attention module connected to the L5 layer performs spatial dimension operation of attention to obtain the fifth lane line feature image .
[0023] Furthermore, the spatial dimension of attention is operated as follows: the MCA multi-dimensional collaborative attention module contains three branches: top branch, middle branch and bottom branch;
[0024] The operation of the top branch of the MCA multi-dimensional collaborative attention module is: As the input of the top branch, , represents a set of real numbers, C, H, and W are the number of channels, height, and width respectively; specifically, the top branch first converts the input Rotate 90° counterclockwise along the height H to get the feature map ,Will Perform a squeeze transformation to obtain a feature map ,Will Perform excitation transformation to obtain the feature map along the width direction , the feature map along the width direction The attention weight Aw is generated using the sigmoid function, and then the attention weight Aw is multiplied by element Multiply to get the enhanced feature map , and finally enhance the feature map Rotate 90° clockwise along height H to obtain the final output feature map of the top branch , the calculation formula is as follows:
[0025] (4);
[0026] (5);
[0027] (6);
[0028] (7);
[0029] (8);
[0030] (9);
[0031] In the formula, It means operating 90° counterclockwise along H; for Characteristic map along H 90° counterclockwise; and They are compression transform and excitation transform; is the feature map after compression transformation; is the feature map after excitation transformation; is the attention weight; yes and Enhanced feature map obtained by element-wise multiplication; It means operating 90° clockwise along H; yes The final output feature map obtained by rotating 90° clockwise along H; is the sigmoid function; is the dot product operation.
[0032] Furthermore, the operation of the branch in the MCA multi-dimensional collaborative attention module is as follows: First, the input Rotate 90° counterclockwise along width W to get the feature map ,Will Perform a squeeze transformation to obtain a feature map ,Will Perform excitation transformation to obtain the feature map along the width direction ;Will Generate attention weights using sigmoid function , and then the attention weights are multiplied by element and Multiply to get the enhanced feature map , and finally Rotate 90° clockwise along the width W to obtain the final output feature map of the middle branch , the calculation formula is as follows:
[0033] (10);
[0034] (11);
[0035] (12);
[0036] (13);
[0037] (14);
[0038] (15);
[0039] In the formula, It is operated to rotate 90° counterclockwise along the width W.
[0040] Furthermore, the operation of the bottom branch of the MCA multi-dimensional collaborative attention module is: first, the feature map Generate the same feature map as itself through the mapping function , then Perform compression transformation and excitation transformation to obtain the aggregated feature map , and then the feature map is given by the sigmoid function Generating Attention Weights , then and Multiply by element multiplication to generate enhanced feature map , and finally enhance the feature map Generate the final output feature map of the bottom branch through the mapping function , expressed as:
[0041] (16);
[0042] (17);
[0043] In the formula, represents a mapping function;
[0044] The top branch finally outputs the feature map , the final output feature map of the middle branch And the bottom branch finally outputs the feature map Fusion is performed to obtain the third lane line feature image , expressed as:
[0045] (18);
[0046] The output of layer L4 The fourth lane feature image is obtained by inputting it into the MCA multi-dimensional collaborative attention module connected to it for spatial dimension operation of attention. ;
[0047] The output of layer L5 The fifth lane feature image is obtained by inputting it into the MCA multi-dimensional collaborative attention module connected to it for spatial dimension operation of attention. .
[0048] Furthermore, the specific process of step S4 is: the third lane line feature image , the fourth lane feature image And the fifth lane feature image They are input into the auxiliary layer of the auxiliary segmentation network module respectively. , and After upsampling through multiple convolutional layers, batch normalization layers, and activation function ReLU, the third auxiliary feature image is obtained. , the fourth auxiliary feature image and the fifth auxiliary feature image , the calculation formula is as follows:
[0049] (19);
[0050] In the formula, represents upsampling; Represents the convolution operation; Represents a batch normalization operation; Represents the activation function ReLU; Represents the input third lane line feature image , the fourth lane feature image And the fifth lane feature image ; The third auxiliary feature image representing the output , the fourth auxiliary feature image and the fifth auxiliary feature image ;
[0051] The third auxiliary feature image , the fourth auxiliary feature image and the fifth auxiliary feature image Input to the combination layer for feature splicing and fusion to obtain the auxiliary feature map , expressed as:
[0052] F aux = Concat ([ F 3- uax , F 4- uax , F 5- uax ]) (20);
[0053] In the formula, Indicates feature concatenation and fusion;
[0054] pass Function Compute Auxiliary Feature Map , get the auxiliary feature map The probability value of the category is obtained to obtain the auxiliary loss function, and the calculation formula is as follows:
[0055] (twenty one);
[0056] L seg =-[φ log ( φ ̂ )+(1 - φ) log (1- φ ̂ )] (twenty two);
[0057] In the formula, for The probability value of the category; is the actual category label; is the auxiliary loss function.
[0058] Furthermore, the specific process of step S5 is: the fifth lane line feature image Input to the feature aggregation FAM module, using circular convolution Perform K iterations of aggregation in the row direction to obtain Feature information in the row direction , also using circular convolution Perform K iterations of aggregation in the column direction to obtain Feature information in column direction , using a nonlinear function Row direction and The dimension information in the column direction is and Perform residual connection and finally output the aggregated feature map F FAM , the calculation formula is as follows:
[0059] (twenty three);
[0060] (twenty four);
[0061] (25);
[0062] (26);
[0063] In the formula, Represents modular operation; and Represents height and width respectively; express The value at the kth iteration, where c, x, y represent the channel, row, and column respectively; is the step size of k iterations; represents W and H; D is the predefined convolution kernel; Indicates traversing all input channels; Represents the spatial range of the convolution kernel; is the value of the feature map at the kth iteration; is a nonlinear function;
[0064] Will and F FAM Perform fusion and splicing to obtain the feature map ,Will As the input of the MLP classifier, it is expressed as:
[0065] F MF = Concat ([ F 5 - mca , F FAM ]) (27);
[0066] The MLP classifier consists of a normalization layer, a fully connected layer, and an activation function. The MLP classifier first converts Perform normalization to obtain the standardized input feature vector , and then through the full connection layer and ReLU activation function Add nonlinear processing to get the output feature map Finally, Reshape into feature information, that is, the final lane feature image X output by the feature information transfer module, expressed as:
[0067] (28);
[0068] (29);
[0069] (30);
[0070] In the formula, and Respectively The mean and standard deviation of Indicates bias; Indicates reshaping.
[0071] Furthermore, in the final lane line feature image X, the lane line is formed by a number of anchor points. The part of the final lane line feature image X containing the lane line is divided into rows or columns and converted into a gridded format. The gridded structure in the final lane line feature image X is defined as an anchor point. The anchor point generated by row division is called a row anchor, and each grid corresponding to the row anchor is called a row anchor point. The anchor point generated by column division is called a column anchor, and each grid corresponding to the column anchor is called a column anchor point.
[0072] Furthermore, the specific process of step S6 is as follows: the final lane feature image X obtained in step S5 is input into the existing branch module, and after passing through multiple layers of convolution, ReLU function and full connection layer, the lane feature image X is transformed into a lane feature image. The function obtains the existence prediction results of row anchors and column anchors in the final lane feature image X respectively. The calculation formula is as follows:
[0073] (31);
[0074] (32);
[0075] In the formula, represents the existence prediction result of the j-th row anchor point in the i-th lane; represents the existence prediction result of the j-th column anchor point in the i-th lane; FC represents the fully connected layer;
[0076] use and The existence loss function is calculated, and the existence of row anchors and column anchors is judged by the existence loss function, and the existence of lane lines is inferred. The calculation process is:
[0077] (33);
[0078] In the formula, and are the number of lane lines and row anchor points assigned to row anchor points, respectively; and are the number of lane lines and column anchor points assigned to column anchor points respectively; onehot is a single hot encoding function; is the cross entropy loss function; There is a label for the jth row anchor point of the i-th lane line; The jth column anchor point of the i-th lane line has a label; Indicates that there is a loss function;
[0079] The final lane feature image X is input into the positioning branch module to identify the exact position of each lane in X. The row anchor points and column anchor points are output accordingly for the position of the lane. The row anchor points or column anchor points belonging to the same grid are then classified into the same lane to obtain the current lane position and the surrounding layout information.
[0080] The positioning branch module calculates the coordinate probability distribution of the lane line at each anchor point in X through the softmax function, which is expressed as:
[0081] (34);
[0082] (35);
[0083] In the formula, and Respectively represent the row anchor point and column anchor point coordinate prediction results;
[0084] The row and column anchor coordinate prediction results are combined with the number of row and column anchor categories to obtain the specific positions of the row and column anchors. The calculation formula is as follows:
[0085] Ex p i,j r = ∑ y =1 N dim r P i,j r [ y ]⋅ y (36);
[0086] Ex p i , j c = ∑ l=1 N dim c P i , j c [l]⋅l (37);
[0087] In the formula, and They are the row anchor point and column anchor point position coordinates respectively; Indicates the number of categories of row anchors; Indicates the number of categories of column anchors; and are the probabilities of the categories corresponding to the row anchor points and column anchor points respectively;
[0088] Calculate the position coordinates of the anchor point to get the expected loss function , expressed as:
[0089] (38);
[0090] In the formula, smooth is the loss function; represents the expected loss function; The lane number indicating the row anchor point; Indicates the category label of the row anchor; Category labels representing column anchors;
[0091] The lane detection model is learned using the positioning loss function. The calculation formula is as follows:
[0092] (39);
[0093] In the formula, represents the positioning loss function;
[0094] The overall loss of the lane detection model is calculated using the auxiliary loss function, existence loss function, positioning loss function, and expected loss function, expressed as:
[0095] (40);
[0096] In the formula, , and is the loss coefficient; Represents the overall loss function of the lane detection model.
[0097] Compared with the existing technology, the present invention has the following beneficial effects: the present invention integrates a multi-dimensional collaborative attention mechanism into the feature extraction network, focuses on the information such as the lane line position and texture in the direction and longitudinal direction, thereby enhancing the feature information of the lane line and improving the feature extraction capability; the present invention adopts a feature aggregation FAM module so that each pixel can obtain global information, strengthens the correlation between feature information through a multi-branch structure, improves the model's ability to perceive global information, and improves the effect of lane line detection in complex environments; by adopting a global feature anchor point classification method to detect lane lines, the lane lines are represented by sparse anchor point coordinates, which speeds up the reasoning speed, makes the detection speed faster, and meets the real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS
[0098] Figure 1 Schematic diagram of the lane line detection model structure of the present invention. DETAILED DESCRIPTION
[0099] like Figure 1 As shown, the present invention provides a technical solution: a lane line detection method based on multi-dimensional collaborative attention and feature aggregation, comprising the following steps:
[0100] Step S1: construct a data set, which contains several lane images, and divides the data set into a training set, a validation set, and a test set in proportion; the data set uses the labeled CULane data set, which contains several lane images. The lane images include various driving environment images, such as urban road images, highway images, suburban road images, etc., covering different lighting conditions during the day and at night, and also include a variety of typical scene images, such as congestion images, curve images, laneless images, shadow images, etc. These scenes are highly practical and challenging; before use, the labeled CULane data set is preprocessed, including data enhancement (rotation and flipping, etc.) of the lane images, and the resolution of the lane images is adjusted to 288×800.
[0101] Step S2: Construct a lane line detection model; the lane line detection model includes a feature extraction network module, an auxiliary segmentation network module, a feature information transfer module and an existence positioning module; the feature extraction network module is composed of a ResNet network and a multi-dimensional collaborative attention module; the auxiliary segmentation network module is composed of an auxiliary layer and a combination layer; the feature information transfer module is composed of a feature aggregation FAM module and an MLP classifier; the existence positioning module is composed of an existence branch module and a positioning branch module; the feature extraction network module is connected to the feature information transfer module, and the feature information transfer module is connected to the existence positioning module; the auxiliary segmentation network module interacts with the feature extraction network module and is in a parallel structure with the feature information transfer module.
[0102] Step S3: The lane image is input into the feature extraction network module, and the lane line feature image in the lane image is obtained through the ResNet network. The ResNet network is divided into 5 layers, and the MCA multi-dimensional collaborative attention module is added to the last three layers of the ResNet network; each layer of the ResNet network performs convolution, batch normalization, ReLu activation and maximum pooling operations on the input lane image to obtain the lane line feature image of each layer.
[0103] Among them, the ResNet network can solve problems such as gradient disappearance and has high efficiency and excellent performance; the ResNet network consists of 1 convolutional layer and 4 residual blocks, which are encoded as L1 layer, L2 layer, L3 layer, L4 layer and L5 layer in sequence, where each residual block contains two residual units BasicBlock, each residual unit contains two 3×3 convolutional layers, and each 3×3 convolutional layer is connected to a batch normalization layer and a ReLU activation function. In order to enhance the significant features of the lane lines in the lane image, the MCA multi-dimensional collaborative attention module is added behind the L3 layer, L4 layer and L5 layer; the MCA multi-dimensional collaborative attention module can focus on the lane line position and texture and other feature information in the lane image in the row and column directions, and then associate the lane line position and texture and other feature information in the lane image in the row and column directions, and finally improve the detection accuracy of the lane lines.
[0104] The specific process of step S3 is as follows: the lane image is input into the L1 layer of the ResNet network. First, the lane image is convolved with a 7×7 convolution with a step size of 2 to obtain image information, and the image information is batch normalized to obtain a feature map. , batch normalization operation can accelerate the training of ResNet network and increase the generalization ability of the entire lane detection model; feature map The activated feature map is obtained by ReLU activation function , the ReLU activation function can be used to increase the nonlinear characteristics of the ResNet network, which is particularly helpful in avoiding and correcting the gradient vanishing problem; finally, the activated feature map Perform convolution pooling to obtain the pooled feature map , that is, the feature information output by the L1 layer ; Convolutional pooling can further reduce the spatial size of the feature map, thereby reducing the number of parameters and the amount of calculation, and also helps to extract rotation and position invariant features; can be expressed as:
[0105] (1);
[0106] In the formula, It is a 7×7 convolution operation; is batch normalization; is the activation function; It is a 3×3 convolution pooling; is the convolution kernel weight.
[0107] Similarly, Input to the L2 layer, use the output of the L2 layer as the input of the L3 layer, use the output of the L3 layer as the input of the L4 layer, and use the output of the L4 layer as the input of the L5 layer; Among them, the residual blocks of the L2 layer, L3 layer, L4 layer and L5 layer all contain two BasicBlock residual units, each residual unit includes two convolution operations, two batch normalization operations and ReLU activation operations; As the input of L2 layer, First, the first residual unit of the L2 layer is entered, and after two convolution operations, two batch normalization operations and ReLU activation operations, the output features of the first residual unit are output. , then As the input of the second residual unit of the L2 layer, the feature information of the L2 layer output is obtained after two convolution operations, two batch normalization operations and ReLU activation operations. , which can be expressed as:
[0108] (2);
[0109] (3);
[0110] The structures of L2, L3, L4 and L5 are consistent, so the feature information of L3 output is calculated according to equations (2) and (3): , feature information output by L4 layer And the feature information output by L5 layer .
[0111] Since the MCA multi-dimensional collaborative attention module is added after the L3 layer, L4 layer and L5 layer, Before inputting to the L4 layer, After being input into the L3 layer, the MCA multi-dimensional collaborative attention module connected to the L3 layer performs spatial dimension operation of attention to obtain the feature image of the third lane line. ,Will Input to L4 layer to get feature information ,Will After being input into the L4 layer, the MCA multi-dimensional collaborative attention module connected to the L4 layer performs spatial dimension operation of attention to obtain the feature image of the fourth lane line. ,Will As the input of L5 layer, output feature information , the feature information After being input into the L5 layer, the MCA multi-dimensional collaborative attention module connected to the L5 layer performs spatial dimension operation of attention to obtain the fifth lane line feature image .
[0112] Among them, the specific process of the spatial dimension operation of attention is as follows: the MCA multi-dimensional collaborative attention module contains three branches: the top branch, the middle branch and the bottom branch. The top branch is used to capture the interaction between the row dimension information in the space, the middle branch is used to capture the interaction between the longitudinal dimension information in the space, and the bottom branch is responsible for capturing the interaction between channels. In the top branch and the middle branch, the permutation operation is used to capture the long-range dependency between the channel dimension and any spatial dimension. Finally, the outputs of the top branch, the middle branch and the bottom branch are simply averaged in the integration stage, and the averaged outputs are fused.
[0113] Specifically, the operation of the top branch of the MCA multi-dimensional collaborative attention module is: As the input of the top branch, , represents a set of real numbers, C, H, W are the number of channels, height and width respectively; the top branch first converts the input Rotate 90° counterclockwise along the height H to get the feature map , The dimension is W×H×C; in order to model the long-term dependency between the number of channels of the feature map and the high H, Perform a squeeze transformation to obtain a feature map , The dimension of is W×1×1; Perform an excitation transformation to capture the interaction between features in the row direction of the spatial dimension. The output dimension is still W×1×1, and the feature map along the width direction is obtained. , the feature map along the width direction Use the sigmoid function to generate a specific attention weight Aw, and then multiply the attention weight Aw with Multiply to get the enhanced feature map , the dimension is W×H×C, and finally the enhanced feature map Rotate 90° clockwise along height H to obtain the final output feature map of the top branch , the dimension returns to C×H×W, and the calculation formula is as follows:
[0114] (4);
[0115] (5);
[0116] (6);
[0117] (7);
[0118] (8);
[0119] (9);
[0120] In the formula, It means operating 90° counterclockwise along H; for Characteristic map along H 90° counterclockwise; and They are compression transform and excitation transform; is the feature map after compression transformation; is the feature map after excitation transformation; is the attention weight; yes and Enhanced feature map obtained by element-wise multiplication; It means operating 90° clockwise along H; yes The final output feature map obtained by rotating 90° clockwise along H; is the sigmoid function; is the dot product operation.
[0121] The operation of the branch in the MCA multi-dimensional collaborative attention module is as follows: First, the input Rotate 90° counterclockwise along the width W to obtain a feature map with dimensions H×C×W ,In order to model the long-term dependency between the number of channels and the width, Perform a squeeze transformation to obtain a feature map , with dimensions of H×1×1, Perform an excitation transformation to capture the interaction between features in the column direction of the spatial dimension. The output dimension is still H×1×1, and the feature map along the width direction is obtained. ,Will Use the sigmoid function to generate specific attention weights , and then the attention weights are multiplied by element and Multiply to get the enhanced feature map , the dimension is H×C×W, and finally Rotate 90° clockwise along the width W to obtain the final output feature map of the middle branch , the dimension returns to C×H×W, and the calculation formula is as follows:
[0122] (10);
[0123] (11);
[0124] (12);
[0125] (13);
[0126] (14);
[0127] (15);
[0128] In the formula, It is operated to rotate 90° counterclockwise along the width W.
[0129] The operation of the bottom branch of the MCA multi-dimensional collaborative attention module is: first, the feature map Generate the same feature map as itself through the mapping function , then Perform compression transformation and excitation transformation to obtain the aggregated feature map , and then the feature map is given by the sigmoid function Generate specific attention weights , then and Multiply by element multiplication to generate enhanced feature map , and finally enhance the feature map Generate the final output feature map of the bottom branch through the mapping function Compared with the top and middle branches, the bottom branches and It is calculated using a mapping function instead of rotating around H and W, so here we only list the calculation formulas that are different from the top branch and the middle branch, expressed as:
[0130] (16);
[0131] (17);
[0132] In the formula, Represents a mapping function.
[0133] The outputs of the top branch, the middle branch, and the bottom branch are integrated to obtain the third lane feature image. , which can be expressed as:
[0134] (18).
[0135] Similarly, the output of layer L4 The fourth lane feature image is obtained by inputting it into the MCA multi-dimensional collaborative attention module connected to it for spatial dimension operation of attention. .
[0136] The output of layer L5 The fifth lane feature image is obtained by inputting it into the MCA multi-dimensional collaborative attention module connected to it for spatial dimension operation of attention. .
[0137] Step S4: The third lane line feature image , the fourth lane feature image And the fifth lane feature image The auxiliary layers are respectively input into the auxiliary layer of the auxiliary segmentation network module. The auxiliary layer consists of a multi-layer convolution layer, a batch normalization layer and an activation function (ReLU). The auxiliary layer is used to adjust the spatial resolution of the feature information so that the feature information can be fused. Finally, the three layers of feature information output by the auxiliary layer are input into the combination layer for fusion to obtain the auxiliary feature map output by the entire auxiliary segmentation network module. ; Since it is an auxiliary segmentation network module, it mainly plays an auxiliary role, that is, when the entire lane line detection model is trained, an auxiliary segmentation task branch is added. This branch is mainly used to obtain auxiliary feature maps , and then better model local features and provide additional information assistance for the main task learning process of the entire lane detection model, improving the ability to recognize local details; therefore, the auxiliary segmentation network module is only used during training and will be removed during the testing phase, which can improve the training effect and generalization ability of the model without sacrificing reasoning performance.
[0138] Specifically, the third lane line feature image , the fourth lane feature image And the fifth lane feature image Input into the auxiliary layer, these three layers of feature information are upsampled (adjusted to the same spatial resolution for subsequent fusion) after passing through multiple layers of convolutional layers, batch normalization layers and activation functions (ReLU) to obtain the third auxiliary feature image , the fourth auxiliary feature image and the fifth auxiliary feature image , the calculation formula is as follows:
[0139] (19);
[0140] In the formula, represents upsampling; Represents the convolution operation; Represents a batch normalization operation; represents the activation function (ReLU); Represents the input third lane line feature image , the fourth lane feature image And the fifth lane feature image ; The third auxiliary feature image representing the output , the fourth auxiliary feature image and the fifth auxiliary feature image .
[0141] The third auxiliary feature image , the fourth auxiliary feature image and the fifth auxiliary feature image Input to the combination layer for feature splicing and fusion to obtain the auxiliary feature map , which can be expressed as:
[0142] F aux = Concat ([ F 3- uax , F 4- uax , F 5- uax ]) (20);
[0143] In the formula, Indicates feature concatenation and fusion.
[0144] pass Function Compute Auxiliary Feature Map , get the auxiliary feature map The probability value of the category is obtained to obtain the auxiliary loss function, and the calculation formula is as follows:
[0145] (twenty one);
[0146] L seg =-[φ log ( φ ̂ )+(1 - φ) log (1- φ ̂ )] (twenty two);
[0147] In the formula, for The probability value of the category; is the actual category label; is the auxiliary loss function.
[0148] Step S5: Input the lane line feature image of the last level of the ResNet network into the feature aggregation FAM module in the feature information transmission module, aggregate the information of each direction of the lane line feature image through K iterations, and obtain the final lane line feature image. Input the final lane line feature image into the MLP classifier, and the MLP classifier inputs the final lane line feature image into the existence positioning module.
[0149] Specifically, the fifth lane feature image Input to the feature aggregation FAM module, using circular convolution Perform K iterations of aggregation in the row direction to obtain Feature information in the row direction , also using circular convolution Perform K iterations of aggregation in the column direction to obtain Feature information in column direction , using a nonlinear function Row direction and The dimension information in the column direction is and Perform residual connection and finally output the aggregated feature map F FAM , the calculation formula is as follows:
[0150] (twenty three);
[0151] (twenty four);
[0152] (25);
[0153] (26);
[0154] In the formula, Represents modular operation; and Represents height and width respectively; express The value at the kth iteration, where c, x, y represent the channel, row, and column respectively; is the step size of k iterations; represents W and H; D is the predefined convolution kernel; Indicates traversing all input channels; Represents the spatial range of the convolution kernel; is the value of the feature map at the kth iteration; is a non-linear function.
[0155] Will and F FAM Perform fusion and splicing to obtain the feature map As the input of the MLP classifier, it is expressed as:
[0156] F MF = Concat ([ F 5 - mca , F FAM ]) (27).
[0157] Among them, the MLP classifier consists of a normalization layer, a fully connected layer, and an activation function. The MLP classifier first Perform normalization to obtain the standardized input feature vector , which helps to improve the stability of the training process, and then through the fully connected layer and ReLU activation function Add nonlinear processing to get the output feature map Finally, The feature information reshaped into a specific shape, that is, the final lane feature image X, is adapted to multiple output tasks (such as positioning and existence prediction) of the lane detection model, so that each part can be directly used for subsequent task processing, expressed as:
[0158] (28);
[0159] (29);
[0160] (30);
[0161] In the formula, and Respectively The mean and standard deviation of Indicates bias; Indicates reshaping.
[0162] Step S6: The existence positioning module is composed of a positioning branch module and an existence branch module. Both the positioning branch module and the existence branch module are composed of multiple layers of convolution, ReLU function and fully connected layer. The lane line detection of the present invention is implemented based on anchor point classification. Specifically, the part containing the lane line in the final lane line feature image is divided into rows or columns, and converted into a gridded format to form a learnable structured representation; in the final lane line feature image, these gridded structures are called anchor points, and the anchor points generated by row division are called row anchors, and each corresponding grid is called a row anchor point; the anchor points generated by column division are called column anchors, and each corresponding grid is called a column anchor point. The lane line is formed by these large number of anchor points. The final lane line feature image X is input into the existence branch module in the existence positioning module. After multiple layers of convolution, ReLU function and fully connected layer, it is used The function obtains the existence prediction results of row anchors and column anchors in the final lane feature image X respectively. The calculation formula is as follows:
[0163] (31);
[0164] (32);
[0165] In the formula, represents the existence prediction result of the j-th row anchor point in the i-th lane; represents the existence prediction result of the j-th column anchor point in the i-th lane; FC represents the fully connected layer.
[0166] Prediction results using row and column anchors and The existence loss function is calculated. The loss function can determine whether the row anchor and column anchor exist, thereby inferring the existence of the lane line. The calculation process is:
[0167] (33);
[0168] In the formula, and are the number of lane lines and row anchor points assigned to row anchor points, respectively; and are the number of lane lines and column anchor points assigned to column anchor points respectively; onehot is a single hot encoding function; is the cross entropy loss function; There is a label for the jth row anchor point of the i-th lane line; The jth column anchor point of the i-th lane line has a label; Indicates that there is a loss function.
[0169] The final lane feature image X is input into the positioning branch module to identify the exact position of each lane in X. The row anchor point and column anchor point are output accordingly to the position of the lane. The anchor points belonging to the same grid are then classified into the same lane, so as to accurately understand the current lane position of the vehicle and the surrounding layout information.
[0170] The positioning branch module calculates the specific coordinate probability distribution of the lane line at each anchor point in X through the softmax function, which can be expressed as:
[0171] (34);
[0172] (35);
[0173] In the formula, and Respectively represent the row anchor point and column anchor point coordinate prediction results.
[0174] The row and column anchor coordinate prediction results are combined with the number of row and column anchor categories to obtain the specific positions of the row and column anchors. The calculation formula is as follows:
[0175] Ex p i,j r = ∑ y =1 N dim r P i,j r [ y ]⋅ y (36);
[0176] Ex p i , j c = ∑ l=1 N dim c P i , j c [l]⋅l (37);
[0177] In the formula, and They are the row anchor point and column anchor point position coordinates respectively; Indicates the number of categories of row anchors; Indicates the number of categories of column anchors; and are the probabilities of the categories corresponding to the row anchor points and column anchor points respectively.
[0178] Calculate the position coordinates of the anchor point to get the expected loss function , the expected loss function can enhance the positioning of the lane line, and the calculation formula is as follows:
[0179] (38);
[0180] In the formula, smooth is the loss function; represents the expected loss function; The lane number indicating the row anchor point; Indicates the category label of the row anchor; Indicates the category label of the column anchor.
[0181] The positioning loss function is used to help the lane detection model learn how to correspond the pixel positions in the feature map with the actual lane coordinates, so as to achieve accurate lane positioning. The calculation formula is as follows:
[0182] (39);
[0183] In the formula, represents the positioning loss function.
[0184] Finally, the overall loss of the lane detection model is calculated using the auxiliary loss function, existence loss function, positioning loss function, and expected loss function, which can be expressed as:
[0185] (40);
[0186] In the formula, , and is the loss coefficient; Represents the overall loss function of the lane detection model.
[0187] Step S7: Use the lane line detection model after calculating the overall loss to process the input lane image and output the specific position of the lane line in the lane image.
[0188] The lane detection model is trained using the training set for 100 epochs with a learning rate of 0.05. The lane detection model is optimized using the stochastic gradient descent algorithm (SGD). After all training is completed, the optimal lane detection model parameters are saved and finally verified in the validation set and evaluated in the test set.
[0189] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A lane detection method based on multi-dimensional collaborative attention and feature aggregation, It is characterized in that The steps include: Step S1: construct a data set, which includes several lane images; Step S2: constructing a lane detection model; the lane detection model includes a feature extraction network module, an auxiliary segmentation network module, a feature information transfer module and an existence positioning module; the feature extraction network module is composed of a ResNet network and a multi-dimensional collaborative attention module; the auxiliary segmentation network module is composed of an auxiliary layer and a combination layer; the feature information transfer module is composed of a feature aggregation FAM module and an MLP classifier; the existence positioning module is composed of an existence branch module and a positioning branch module; the feature extraction network module is connected to the feature information transfer module, and the feature information transfer module is connected to the existence positioning module; the auxiliary segmentation network module interacts with the feature extraction network module and is in a parallel structure with the feature information transfer module; Step S3: Input the lane image into the feature extraction network module, and obtain the lane line feature image in the lane image through the ResNet network. The ResNet network is divided into 5 layers, and the MCA multi-dimensional collaborative attention module is added to the last three layers of the ResNet network; Each layer of the ResNet network performs convolution, batch normalization, ReLu activation, and maximum pooling operations on the input lane image to obtain the lane line feature image of each layer; Step S4: inputting the obtained lane line feature images of the last three levels of the ResNet network into the auxiliary segmentation network module, fusing the lane line feature images of the last three levels into an auxiliary feature map through the auxiliary segmentation network module for output, and obtaining the auxiliary loss function by calculating the auxiliary feature map; Step S5: Input the obtained lane line feature image of the last layer of the ResNet network into the feature aggregation FAM module in the feature information transmission module, aggregate the information of each direction of the lane line feature image through K iterations, and obtain the final lane line feature image, input the final lane line feature image into the MLP classifier, and the MLP classifier inputs the final lane line feature image into the existence positioning module; Step S6: the existence branch module in the localization module calculates the existence prediction result of the lane line in the final lane line feature image X, obtains the existence loss function through the existence prediction result, judges whether the lane line in X exists, and the localization branch module calculates the coordinate prediction result of each lane line in the final lane line feature image, obtains the localization loss function through the coordinate prediction result, combines the coordinate prediction result with the number of lane line categories to obtain the specific coordinates of the lane line, obtains the expected loss function through the specific coordinate calculation, and finally classifies and locates each lane line in X; the existence branch module and the localization branch module interact, filter out the lane lines that do not exist in X, and use the auxiliary loss function, the existence loss function, the localization loss function and the expected loss function to calculate the overall loss of the lane line detection model; Step S7: using the lane line detection model after calculating the overall loss to process the input lane image, and output the specific position of the lane line in the lane image; The ResNet network consists of 1 convolutional layer and 4 residual blocks, which are encoded into L1 layer, L2 layer, L3 layer, L4 layer and L5 layer in sequence. Each residual block contains two residual units BasicBlock, each residual unit contains two 3×3 convolutional layers, each 3×3 convolutional layer is connected to a batch normalization layer and ReLU activation function, and the MCA multi-dimensional collaborative attention module is added to the back of L3 layer, L4 layer and L5 layer; the auxiliary layer is composed of multi-layer convolutional layers, batch normalization layers and activation function ReLU; the positioning branch module and the existence branch module are both composed of multi-layer convolution, ReLU function and fully connected layers; The specific process of step S3 is as follows: the lane image is input into the L1 layer of the ResNet network. First, the lane image is convolved with a 7×7 convolution with a step size of 2 to obtain image information, and the image information is batch normalized to obtain a feature map. , feature map The activated feature map is obtained by ReLU activation function , and finally the activated feature map Perform convolution pooling to obtain the pooled feature map , that is, the feature information output by the L1 layer , expressed as: (1); In the formula, It is a 7×7 convolution operation; is batch normalization; is the activation function; It is a 3×3 convolution pooling; is the convolution kernel weight; Will As the input of L2 layer, First, the first residual unit of the L2 layer is entered. After two convolution operations, two batch normalization operations and ReLU activation operations, the output feature of the first residual unit is obtained. , then As the input of the second residual unit of the L2 layer, the feature information of the L2 layer output is obtained after two convolution operations, two batch normalization operations and ReLU activation operations. , expressed as: (2); (3); The structures of L2, L3, L4 and L5 are consistent. The feature information output by L3 is calculated according to equations (2) and (3). , feature information output by L4 layer And the feature information output by L5 layer ; The MCA multi-dimensional collaborative attention module is added after the L3, L4 and L5 layers. Before inputting to the L4 layer, After being input into the L3 layer, the MCA multi-dimensional collaborative attention module connected to the L3 layer performs spatial dimension operation of attention to obtain the feature image of the third lane line. ,Will Input to L4 layer to get feature information ,Will After being input into the L4 layer, the MCA multi-dimensional collaborative attention module connected to the L4 layer performs spatial dimension operation of attention to obtain the feature image of the fourth lane line. ,Will As the input of L5 layer, output feature information , the feature information After being input into the L5 layer, the MCA multi-dimensional collaborative attention module connected to the L5 layer performs spatial dimension operation of attention to obtain the fifth lane line feature image ; The spatial dimension of attention is operated as follows: the MCA multidimensional collaborative attention module contains three branches: top branch, middle branch and bottom branch; The operation of the top branch of the MCA multi-dimensional collaborative attention module is: As the input of the top branch, , represents a set of real numbers, C, H, and W are the number of channels, height, and width respectively; specifically, the top branch first converts the input Rotate 90° counterclockwise along the height H to get the feature map ,Will Perform a squeeze transformation to obtain a feature map ,Will Perform excitation transformation to obtain the feature map along the width direction , the feature map along the width direction The attention weight Aw is generated using the sigmoid function, and then the attention weight Aw is multiplied by element Multiply to get the enhanced feature map , and finally enhance the feature map Rotate 90° clockwise along height H to obtain the final output feature map of the top branch , the calculation formula is as follows: (4); (5); (6); (7); (8); (9); In the formula, It means operating 90° counterclockwise along H; for Characteristic map along H 90° counterclockwise; and They are compression transform and excitation transform; is the feature map after compression transformation; is the feature map after excitation transformation; is the attention weight; yes and Enhanced feature map obtained by element-wise multiplication; It means operating 90° clockwise along H; yes The final output feature map obtained by rotating 90° clockwise along H; is the sigmoid function; is the dot product operation; The operation of the branch in the MCA multi-dimensional collaborative attention module is as follows: First, the input Rotate 90° counterclockwise along width W to get the feature map ,Will Perform a squeeze transformation to obtain a feature map ,Will Perform excitation transformation to obtain the feature map along the width direction ;Will Generate attention weights using sigmoid function , and then the attention weights are multiplied by element and Multiply to get the enhanced feature map , and finally Rotate 90° clockwise along the width W to obtain the final output feature map of the middle branch , the calculation formula is as follows: (10); (11); (12); (13); (14); (15); In the formula, To rotate 90° counterclockwise along the width W; The operation of the bottom branch of the MCA multi-dimensional collaborative attention module is: first, the feature map Generate the same feature map as itself through the mapping function , then Perform compression transformation and excitation transformation to obtain the aggregated feature map , and then the feature map is given by the sigmoid function Generating Attention Weights , then and Multiply by element multiplication to generate enhanced feature map , and finally enhance the feature map Generate the final output feature map of the bottom branch through the mapping function , expressed as: (16); (17); In the formula, represents a mapping function; The top branch finally outputs the feature map , the final output feature map of the middle branch And the bottom branch finally outputs the feature map Fusion is performed to obtain the third lane line feature image , expressed as: (18); The output of layer L4 The fourth lane feature image is obtained by inputting it into the MCA multi-dimensional collaborative attention module connected to it for spatial dimension operation of attention. ; The output of layer L5 The fifth lane feature image is obtained by inputting it into the MCA multi-dimensional collaborative attention module connected to it for spatial dimension operation of attention. ; The specific process of step S4 is: the third lane line feature image , the fourth lane feature image And the fifth lane feature image They are input into the auxiliary layer of the auxiliary segmentation network module respectively. , and After upsampling through multiple convolutional layers, batch normalization layers, and activation function ReLU, the third auxiliary feature image is obtained. , the fourth auxiliary feature image and the fifth auxiliary feature image , the calculation formula is as follows: (19); In the formula, represents upsampling; Represents the convolution operation; Represents a batch normalization operation; Represents the activation function ReLU; Represents the input third lane line feature image , the fourth lane feature image And the fifth lane feature image ; The third auxiliary feature image representing the output , the fourth auxiliary feature image and the fifth auxiliary feature image ; The third auxiliary feature image , the fourth auxiliary feature image and the fifth auxiliary feature image Input to the combination layer for feature splicing and fusion to obtain the auxiliary feature map , expressed as: (20); In the formula, Indicates feature concatenation and fusion; pass Function Compute Auxiliary Feature Map , and obtain the auxiliary feature map The probability value of the category is obtained to obtain the auxiliary loss function, and the calculation formula is as follows: (21); (22); In the formula, for The probability value of the category; is the actual category label; is the auxiliary loss function; The specific process of step S5 is: the fifth lane line feature image Input to the feature aggregation FAM module, using circular convolution Perform K iterations of aggregation in the row direction to obtain Feature information in the row direction , also using circular convolution Perform K iterations of aggregation in the column direction to obtain Feature information in column direction , using a nonlinear function Row direction and The dimension information in the column direction is and Perform residual connection and finally output the aggregated feature map F FAM , the calculation formula is as follows: (23); (24); (25); (26); In the formula, Represents modular operation; and Represents height and width respectively; express The value at the kth iteration, where c, x, y represent the channel, row, and column respectively; is the step size of k iterations; represents W and H; D is the predefined convolution kernel; Indicates traversing all input channels; Represents the spatial range of the convolution kernel; is the value of the feature map at the kth iteration; is a nonlinear function; Will and F FAM Perform fusion and splicing to obtain the feature map ,Will As the input of the MLP classifier, it is expressed as: (27); The MLP classifier consists of a normalization layer, a fully connected layer, and an activation function. The MLP classifier first Perform normalization to obtain the standardized input feature vector , and then through the full connection layer and ReLU activation function Add nonlinear processing to get the output feature map Finally, Reshape into feature information, that is, the final lane feature image X output by the feature information transfer module, expressed as: (28); (29); (30); In the formula, and Respectively The mean and standard deviation of Indicates bias; Indicates reshaping.
2. The lane detection method based on multi-dimensional collaborative attention and feature aggregation according to claim 1, Features: In the final lane line feature image X, the lane line is formed by a number of anchor points. The part containing the lane line in the final lane line feature image X is divided into rows or columns and converted into a gridded format. The gridded structure in the final lane line feature image X is defined as an anchor point. The anchor point generated by row division is called a row anchor, and each grid corresponding to the row anchor is called a row anchor point. The anchor point generated by column division is called a column anchor, and each grid corresponding to the column anchor is called a column anchor point.
3. The lane detection method based on multi-dimensional collaborative attention and feature aggregation according to claim 2, Features: The specific process of step S6 is as follows: the final lane feature image X obtained in step S5 is input into the existing branch module, and after passing through multiple layers of convolution, ReLU function and full connection layer, the lane feature image X is transformed into a lane feature image using The function obtains the existence prediction results of row anchors and column anchors in the final lane feature image X respectively. The calculation formula is as follows: (31); (32); In the formula, represents the existence prediction result of the j-th row anchor point in the i-th lane; represents the existence prediction result of the j-th column anchor point in the i-th lane; FC represents the fully connected layer; use and The existence loss function is calculated, and the existence of row anchors and column anchors is judged by the existence loss function, and the existence of lane lines is inferred. The calculation process is: (33); In the formula, and are the number of lane lines and row anchor points assigned to row anchor points, respectively; and are the number of lane lines and column anchor points assigned to the column anchor points, respectively; Onehot is a single hot encoding function; is the cross entropy loss function; There is a label for the jth row anchor point of the i-th lane line; The jth column anchor point of the i-th lane line has a label; Indicates that there is a loss function; The final lane feature image X is input into the positioning branch module to identify the exact position of each lane in X. The row anchor points and column anchor points are output accordingly for the position of the lane. The row anchor points or column anchor points belonging to the same grid are then classified into the same lane to obtain the current lane position and the surrounding layout information. The positioning branch module calculates the coordinate probability distribution of the lane line at each anchor point in X through the softmax function, which is expressed as: (34); (35); In the formula, and Respectively represent the row anchor point and column anchor point coordinate prediction results; The row and column anchor coordinate prediction results are combined with the number of row and column anchor categories to obtain the specific positions of the row and column anchors. The calculation formula is as follows: (36); (37); In the formula, and They are the row anchor point and column anchor point position coordinates respectively; Indicates the number of categories of row anchors; Indicates the number of categories of column anchors; and are the probabilities of the categories corresponding to the row anchor points and column anchor points respectively; Calculate the position coordinates of the anchor point to get the expected loss function , expressed as: (38); In the formula, smooth is the loss function; represents the expected loss function; The lane number indicating the row anchor point; Indicates the category label of the row anchor; Category labels representing column anchors; The lane detection model is learned using the positioning loss function. The calculation formula is as follows: (39); In the formula, represents the positioning loss function; The overall loss of the lane detection model is calculated using the auxiliary loss function, existence loss function, positioning loss function, and expected loss function, expressed as: (40); In the formula, , and is the loss coefficient; Represents the overall loss function of the lane detection model.
