Lane line detection method for multi-fork complex road scene

Abstract

The invention discloses a lane line detection method for a multi-fork complex road scene. The method comprises the following steps: 1, acquiring a multi-fork scene data sample and preprocessing the multi-fork scene data sample; 2, constructing a lane line detection network of the multi-fork complex road scene, wherein the lane line detection network comprises an image feature extraction network based on an attention mechanism, a point cloud body column feature extraction network, a lane line generative adversarial network, a cross attention module, a visual angle conversion module and a detection head; 3, performing feature extraction on the data sample by using an attention mechanism-based image feature extraction network and a point cloud body column feature extraction network; 4, combining the lane line generative adversarial network and the cross attention module to generate fusion features; and 5, converting the fusion features into BEV features by using a visual angle conversion module, and transmitting the BEV features into a detection head to generate a lane line coordinate set. According to the invention, multi-modal information can be fully extracted, and the problems of long-distance dependence and depth jump of the lane line are effectively solved, so that the lane line detection effect in a special scene can be improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving perception, and more specifically to a lane line detection method, which is especially applicable to complex road scenes with multiple intersections. Background Art

[0002] In the perception of road structure elements, lane line detection is an important part of high-level autonomous driving environment perception, which requires the perception of lane lines in various road environments. Autonomous vehicles can autonomously select the drivable area based on the results of lane line detection, providing a basis for downstream decision-making and planning. However, the actual road conditions are complex, and there are many types of lane lines. Changes in lighting conditions, object shadows, wear and tear, lack of visual clues and other issues are all challenges faced by lane line detection.

[0003] Lane detection methods are generally divided into two types: traditional methods and deep learning. In traditional methods, the lane detection method based on color features converts the common RGB channel image into color features such as HSV, and then manually sets the threshold to detect the lane position; the lane detection method based on edge detection and Hough transform first converts the color image into a grayscale image, and then performs blur processing, edge detection and Hough transform. Traditional methods can usually only detect lane lines in simple scenes and may not be robust enough for complex situations. Compared with traditional detection algorithms, deep learning methods are divided into segmentation-based solutions, anchor-based methods, and parameter-based methods. The segmentation-based method models lane detection as a pixel-by-pixel classification problem. Each pixel is divided into a lane area or background, achieving image classification at the pixel level. However, in complex scenes with multiple intersections, the model is prone to misjudgment due to the long-distance dependence and large depth variation of lanes. The anchor-based method usually adopts a top-down approach to predict lanes. It uses the prior knowledge that lanes extend from near to far in the driving perspective to construct lane instances, design line type anchors, and regress the offsets between sampling points and predefined anchors. Non-maximum suppression (NMS) is applied to select the lane with the highest confidence. However, in complex scenes with multiple intersections, this extension trend of lanes is not obvious, especially when the perspective changes or the road conditions are complex, which makes this method prone to losing lane information. The parameter-based method uses a pre-defined parameter curve, generally through polynomial curve regression, to output a polynomial representing each lane, thereby detecting the shape of the lane. However, when the shape of the lane is irregular (such as sharp turns, forks, etc.), the fitting result will be greatly different from the actual situation, reducing the accuracy and practicality of the detection. Summary of the invention

[0004] The present invention is to avoid the deficiencies of the above-mentioned prior art, effectively solve the problems of long-distance dependence and depth jump of lane lines in complex road scenes with multiple forks, and provide a lane line detection method for complex road scenes with multiple forks, so as to meet the requirements of lane line detection accuracy, recall rate and information integrity on complex roads with multiple forks, thereby improving the lane line detection effect in special scenarios.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0006] The lane line detection method for a complex road scene with multiple intersections of the present invention is characterized in that it comprises the following steps:

[0007] Step 1: Obtain data of complex road scenes with multiple intersections and perform preprocessing;

[0008] Step 1.1: Construct a lane line dataset with lane line category labels in a complex road scene with multiple intersections , N is the total number of data samples, is the nth data sample, the nth data sample The category label of the dth lane line is recorded as , represents the nth complex road image, C, H and W represent the number of channels, height and width of the complex road image respectively; represents the nth point cloud data, and ,in, express The real position information of the middle lane line , express The three-dimensional coordinates of the d-th lane line in , represents the reflection intensity of the dth lane line, Indicates the maximum number of sampling points for the lane line;

[0009] Step 1.2: For the nth complex road image Perform data enhancement processing to obtain the nth enhanced image ;

[0010] Step 1.3: For the nth point cloud information Perform data enhancement processing to obtain the nth enhanced point cloud information ;

[0011] Step 2: Construct a lane detection network for complex road scenes with multiple intersections, including: an image feature extraction network based on the attention mechanism, a point cloud column feature extraction network, a lane generative adversarial network, a cross attention module, a perspective conversion module, and a detection head;

[0012] Step 2.1: Image feature extraction network based on attention mechanism Processing is performed to obtain the image features of the nth multi-scale lane line ;

[0013] Step 2.2: Point cloud volume column feature extraction network Processing is performed to obtain the pseudo image feature of the nth lane line ;

[0014] Step 2.3: Lane line generation adversarial network for nth lane line pseudo image features Processing is performed to obtain the nth multi-scale lane line point cloud feature And the judgment results The probability of being a real point cloud feature ; Weighted loss for building lane line generation adversarial networks , thereby using the gradient descent method to train the lane line generation adversarial network and calculate the weighted loss of the lane line generation adversarial network To update the network parameters until Until convergence, the trained lane line generation adversarial network is obtained;

[0015] Step 2.4: The cross attention module generates the nth optimal multi-scale lane line point cloud feature output by the trained lane line adversarial network and the image features of the n-th multi-scale lane line Process and obtain the nth fusion feature ;

[0016] Step 2.5: Viewpoint conversion module Processing is performed to obtain the nth lane line feature map ,in, is the number of channels of the lane feature map, , is the height and width of the lane feature map;

[0017] Step 2.6: The data is passed into the detection head to generate the nth data sample in a complex road scene with multiple intersections. Predicted lane line position set And the predicted lane line category ,in, represents the ith three-dimensional coordinate point of the predicted dth lane line in the nth data sample in the BEV space, Indicates the maximum number of three-dimensional coordinate points of the predicted lane line, Represents the predicted d-th lane line category.

[0018] The lane line detection method for complex road scenes with multiple intersections described in the present invention is also characterized in that the image feature extraction network of the attention mechanism in step 2.1 is composed of parallel sub-convolutional layers, A global max pooling layer, Global average pooling, It consists of a feature concatenation layer and a coordinate attention mechanism layer;

[0019] Step 2.1.1: Input into the feature extraction network based on the attention mechanism, first pass through After processing by parallel sub-convolutional layers, they are input separately A global max pooling layer and After processing by the global average pooling layer, the nth global average feature map is obtained accordingly and the i-th global maximum feature map ;

[0020] Step 2.1.2 and Passing together The nth lane line splicing feature is obtained by splicing and upsampling the feature splicing layers. ;

[0021] Step 2.1.3: The coordinate attention mechanism layer uses formula (1.1) and formula (1.2) to respectively The height and width directions of the pooling are respectively pooled, and the nth highly aggregated feature is obtained accordingly. Aggregate features with nth width ;

[0022] (1.1)

[0023] (1.2)

[0024] In formula (1.1) and formula (1.2), express The coordinate point features of the hth row and ith column in ; express The coordinate point features of the j-th row and w-th column in ;

[0025] Step 2.1.4: After transposition and After concatenation and dimensionality reduction, the concatenation is input into the ReLu function for processing to obtain the nth lane line coding feature. ;

[0026] Step 2.1.5: The coordinate attention mechanism layer After segmenting the image along the height and width directions, the width lane line coding features are obtained With height lane line coding features , and thus use equations (1.3) and (1.4) to obtain the attention weights of the nth lane line detection along the width direction and the attention weight along the height direction ;

[0027] (1.3)

[0028] (1.4)

[0029] In formula (1.3) and formula (1.4), , are the convolution operations along the height and width of the image, respectively. is the Sigmoid function;

[0030] Step 2.1.6: Use formula (1.5) to obtain the image features of the nth multi-scale lane line ;

[0031] (1.5)

[0032] Furthermore, the point cloud body column feature network in step 2.2 is composed of a voxel partitioning layer, an MLP layer, a convolutional layer, and a global maximum pooling layer;

[0033] Step 2.2.1, the voxel division layer will Divide into vertical columnar areas with a grid number of P, and select features in non-empty vertical columnar areas ,in, express The feature of the i-th non-empty vertical columnar region in , M is the maximum number of non-empty vertical columnar regions;

[0034] Step 2.2.2 After being processed by the MLP layer, convolution layer, and global maximum pooling layer, the nth lane line pseudo image feature is output. .

[0035] Furthermore, the lane line generative adversarial network in step 2.3 is composed of a generator G and a discriminator D, and both the generator and the discriminator are composed of a multi-layer convolution module, a deep convolution layer and a residual block;

[0036] Step 2.3.1. Input into the generator G, and after being processed by the multi-layer convolution module, the deep convolution layer and the residual block in turn, the nth generation layer feature is obtained , After the full connection layer and normalization processing, the predicted n-th multi-scale point cloud feature is output , Voxel information representing the predicted lane line position;

[0037] Step 2.3.2 and Input into the discriminator D, and processed by multi-layer convolution modules, deep convolution layers and residual blocks in turn to obtain the nth discriminant layer feature , After being processed by the Sigmoid activation function, the final output is The probability of being judged as a real point cloud feature ;

[0038] Step 2.3.3: Use formula (1.6) to construct the point cloud inversion loss of the generator G ;

[0039] (1.6)

[0040] In formula (1.6), express The expected value of represents L1 loss, The weight hyperparameter representing the loss;

[0041] Step 2.3.4: Use formula (1.7) to construct the point cloud discrimination loss of the discriminator D ;

[0042] (1.7)

[0043] In formula (1.7), express Expected value;

[0044] Step 2.3.5: Use formula (1.8) to construct the weighted loss of the lane line generation adversarial network ;

[0045] (1.8)

[0046] In formula (1.8), and There are two weight parameters for balancing and impact.

[0047] Further, the step 2.4 includes:

[0048] Step 2.4.1: Multi-scale lane line image features , optimal multi-scale point cloud features Projected into the same dimensional space, the query matrix is ​​obtained using equations (1.9), (1.10), and (1.11): , key matrix , value matrix :

[0049] (1.9)

[0050] (1.10)

[0051] (1.11)

[0052] In formula (1.9), formula (1.10), and formula (1.11), , , are three projection matrices;

[0053] Step 2.4.2: Generate the attention weights between multimodal lane line features using formula (1.12) :

[0054] (1.12)

[0055] In formula (1.12), represents the dimensions of the key matrix and query matrix, represents the activation function;

[0056] Step 2.4.3: and After point multiplication, the nth multimodal intermediate feature is obtained ;

[0057] Step 2.4.4: and After splicing, we get the nth fusion feature .

[0058] Further, the step 2.5 includes:

[0059] Step 2.5.1. Setup ,in, , , are the step lengths in the x direction, y direction, and z direction in the BEV space coordinate system, respectively, and a cell is constructed with the step lengths, thereby obtaining a BEV space composed of a plurality of cells;

[0060] Step 2.5.2: The perspective conversion module fuses the features After the convolution layer, normalization and sonftmax activation function, the depth probability distribution D of the nth lane line is obtained. n , D n and After splicing, the nth deep fusion feature is obtained ;

[0061] Step 2.5.3: Project it into the BEV space to obtain the three-dimensional point cloud coordinates of the lane line, assign the three-dimensional coordinate points of the lane line to the cells, and obtain the lane line features of the i-th cell in the BEV space ;

[0062] After being processed by the convolution layer, pooling layer, normalization, and ReLu function, the lane line feature map of the nth data sample is output. .

[0063] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the lane line detection method, and the processor is configured to execute the program stored in the memory.

[0064] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the lane line detection method when executed by a processor.

[0065] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0066] 1. The present invention designs a multimodal feature fusion module, and uses a cross-attention module to fuse the lane line point cloud features with the lane line image features to obtain rich feature space and depth information of the lane line in multi-intersection scenarios, overcoming the problem that the prior art can only detect lane lines in simple scenarios, thereby improving the robustness, adaptability and accuracy of lane line detection.

[0067] 2. The present invention uses the coordinate attention module (CA) to extract image features and enhance the lane line position weight. Taking into account the long and slender characteristics of the lane line, it overcomes the depth jump problem of the lane line in complex scenes with multiple intersections, and enhances the effect of lane line feature extraction without increasing the computational complexity of the detection network. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a schematic diagram of the complete lane line detection process;

[0069] Figure 2 This is the structural diagram of the coordinate attention mechanism;

[0070] Figure 3 Schematic diagram of lane line detection results. DETAILED DESCRIPTION

[0071] In this embodiment, Figure 1 As shown, a process of a lane line detection method for a complex road scene with multiple intersections includes the following steps:

[0072] Step 1: Obtain data of complex road scenes with multiple intersections and perform preprocessing;

[0073] Step 1.1: Construct category labels with lane lines in complex road scenes with multiple intersections Lane line dataset , N is the total number of data samples, is the nth data sample, the nth data sample The category label of the dth lane line is recorded as , represents the nth complex road image, C, H and W represent the number of channels, height and width of the complex road image respectively. In this example, C is 3, H and W are both 512; represents the nth point cloud data, and ,in, express The actual position information of the middle lane line , express The three-dimensional coordinates of the d-th lane line in , represents the reflection intensity of the dth lane line, Indicates the maximum number of sampling points for lane lines.

[0074] Step 1.2: For the nth complex road image Perform data enhancement processing for the nth complex road image input After scaling by a factor of 0.8, the image is adjusted back to the standard input size using bilinear interpolation, and then flipped by a ratio of 0.5; 20% of the total area is randomly masked by a rectangular mask, the image order is flipped, and finally the image data is Gaussian blurred to obtain the nth enhanced image. ; In this example, the enhanced image and Keep the format consistent.

[0075] Step 1.3: For the nth point cloud information Perform data enhancement processing and input the nth point cloud information Randomly flip the coordinate axis, randomly rotate the point cloud, and add random noise to enhance the point cloud data to obtain the nth enhanced point cloud information .

[0076] Step 2: Construct a lane detection network for complex road scenes with multiple intersections, including: an image feature extraction network based on the attention mechanism, a point cloud column feature extraction network, a lane generative adversarial network, a cross attention module, a perspective conversion module, and a detection head;

[0077] Step 2.1: Image feature extraction network based on attention mechanism Processing is performed to obtain the image features of the nth multi-scale lane line ;

[0078] The image feature extraction network of the attention mechanism in step 2.1 is composed of parallel sub-convolutional layers, A global max pooling layer, Global average pooling, feature concatenation layer and coordinate attention mechanism layer; in this embodiment, Take 2, the parallel convolution layer output channel is 3, Take 1, Take 1.

[0079] Step 2.1.1: Input into the feature extraction network based on the attention mechanism, first pass through After processing by parallel sub-convolutional layers, they are input separately A global max pooling layer and After processing by the global average pooling layer, the nth global average feature map is obtained accordingly and the i-th global maximum feature map ;

[0080] Step 2.1.2 and Passing together The nth lane line splicing feature is obtained by splicing and upsampling the feature splicing layers. ;

[0081] Step 2.1.3, the coordinate attention mechanism layer structure is as follows Figure 2 As shown, using formula (1.1) and formula (1.2) respectively Using (H, 1) and (1, W) pooling kernels to pool along the height and width directions, the nth highly aggregated feature is obtained accordingly. Aggregate features with nth width , overcome the long-distance dependency problem;

[0082] (1.1)

[0083] (1.2)

[0084] In formula (1.1) and formula (1.2), express The coordinate point features of the hth row and ith column in ; express The coordinate point features of the j-th row and w-th column in .

[0085] Step 2.1.4: After transposition and After concatenation and dimensionality reduction, the concatenation is input into the ReLu function for processing to obtain the nth lane line coding feature. ;

[0086] Step 2.1.5, Coordinate Attention Mechanism Layer Pair After segmenting the image along the height and width directions, the width lane line coding features are obtained With height lane line coding features , and thus use equations (1.3) and (1.4) to obtain the attention weights of the nth lane line detection along the width direction and the attention weight along the height direction ;

[0087] (1.3)

[0088] (1.4)

[0089] In formula (1.3) and formula (1.4), , They are convolution operations along the height and width directions of the image, respectively. In this embodiment, a 1*1 convolution kernel is used for convolution. is the Sigmoid function.

[0090] Step 2.1.6: Use formula (1.5) to obtain the image features of the nth multi-scale lane line ;

[0091] (1.5)

[0092] In this embodiment, the enhanced feature map with a size of 3×512×512 Input the image feature extraction network based on the attention mechanism, first use parallel convolutional layers, global average pooling and global maximum pooling to After processing, we get the global average feature map with size of 3×1×1. and the global maximum feature map , and then by The feature splicing layers are spliced ​​and upsampled to obtain a lane line splicing feature with a size of 3×512×512. , using (H, 1) and (1, W) pooling kernels to pool along the height and width directions to obtain 3×1×512 highly aggregated features Aggregate features with 3×512×1 width ,Will After transposition and After concatenation and dimensionality reduction, the concatenation is processed into the ReLu function to obtain a lane line coding feature with a size of 3×1×1024. , and finally encode the lane line features Use 1*1 convolution kernel to perform dimensionality increase operation, and then combine with sigmoid activation function to obtain the attention weight in the width direction. and the attention weight along the height direction ,Will Multiplying with the obtained attention weights to obtain the image features of multi-scale lane lines .

[0093] Step 2.2: Point cloud volume column feature extraction network Processing is performed to obtain the pseudo image feature of the nth lane line ; Among them, the point cloud volume column feature network is composed of a voxel division layer, an MLP layer, a convolution layer and a global maximum pooling layer; the point cloud volume column feature network is composed of a voxel division layer, 2 MLP layers, 3 convolution layers and 1 global maximum pooling layer;

[0094] Step 2.2.1, voxel division layer Divide into vertical columnar areas with a grid number of P, and select features in non-empty vertical columnar areas ,in, express The feature of the i-th non-empty vertical columnar region in , M is the maximum number of non-empty vertical columnar regions;

[0095] Step 2.2.2 After being processed by the MLP layer, convolution layer, and global maximum pooling layer, the nth lane line pseudo image feature is output. ;In this example, initialize P to 150;

[0096] Step 2.3: Lane line generation adversarial network for nth lane line pseudo image features Processing is performed to obtain the nth multi-scale lane line point cloud feature And the judgment results The probability of being a real point cloud feature , where the lane line generative adversarial network consists of a generator G and a discriminator D, and both the generator and the discriminator consist of R multi-layer convolution modules, I deep convolution layers and U residual blocks. In this example, R is 3, I is 2, and U is 6;

[0097] Step 2.3.1. Input into the generator G, and after being processed by the multi-layer convolution module, the deep convolution layer and the residual block in turn, the nth generation layer feature is obtained , After the full connection layer and normalization processing, the predicted n-th multi-scale point cloud feature is output , Voxel information representing the predicted lane line position;

[0098] Step 2.3.2 and Input into the discriminator D, and processed by multi-layer convolution modules, deep convolution layers and residual blocks in turn to obtain the nth discriminant layer feature , After being processed by the Sigmoid activation function, the final output is The probability of being judged as a real point cloud feature ;

[0099] Step 2.3.3: Use formula (1.6) to construct the point cloud inversion loss of generator G ,The point cloud inversion loss function is used to ensure that the generated point cloud position information is consistent with the actual position;

[0100] (1.6)

[0101] In formula (1.6), Represents the predicted multi-scale lane line point cloud features The expected value of represents L1 loss, The weight hyperparameter representing the loss.

[0102] Step 2.3.4: Use formula (1.7) to construct the point cloud discrimination loss of the discriminator D ;

[0103] (1.7)

[0104] In formula (1.7), express Expected value;

[0105] Step 2.3.5: Use formula (1.8) to construct the weighted loss of the lane line generation adversarial network ;

[0106] (1.8)

[0107] In formula (1.8), and There are two weight parameters for balancing and In this embodiment, and Take 0.3 and 0.7 respectively.

[0108] Step 2.3.6: Use the gradient descent method to train the lane line generation adversarial network and calculate the weighted loss of the lane line generation adversarial network To update the network parameters until Until convergence, the trained lane line generation adversarial network is obtained.

[0109] Step 2.4: The cross-attention module generates the nth optimal multi-scale lane line point cloud feature output by the adversarial network after the training. and the image features of the n-th multi-scale lane line Process and obtain the nth fusion feature ;

[0110] Step 2.4.1: Multi-scale lane line image features , optimal multi-scale point cloud features Projected into the same dimensional space, the query matrix is ​​obtained using equations (1.9), (1.10), and (1.11): , key matrix , value matrix :

[0111] (1.9)

[0112] (1.10)

[0113] (1.11)

[0114] In formula (1.9), formula (1.10), and formula (1.11), , , are three projection matrices.

[0115] Step 2.4.2: Generate the attention weights between multimodal lane line features using formula (1.12) :

[0116] (1.12)

[0117] In formula (1.12), represents the dimensions of the key matrix and query matrix, represents the activation function;

[0118] Step 2.4.3: and After point multiplication, the nth multimodal intermediate feature is obtained ;

[0119] Step 2.4.4: Multimodal intermediate features and After splicing, we get the nth fusion feature .

[0120] Step 2.5: Viewpoint conversion module Processing is performed to obtain the nth lane line feature map ,in, is the number of channels of the lane feature map, , is the height and width of the lane feature map;

[0121] Step 2.5.1. Setup ,in, , , are the step sizes in the x, y, and z directions in the BEV space coordinate system. In this example, the step size is 0.25, and cells are constructed with the step size, so that a BEV space consisting of 360×360×1 cells is obtained;

[0122] Step 2.5.2: The perspective conversion module fuses the features The first 118 features of are processed by convolution layer, normalization and sonftmax activation function in turn to obtain the nth lane line depth probability distribution D n , D n and After splicing, the nth deep fusion feature is obtained .

[0123] Step 2.5.3: Project it into the BEV space, and generate representations for pixels with lane line weights greater than 0.85 in the fusion features to obtain the 3D point cloud coordinates of the lane line, complete the transformation from the 2D coordinate system to the 3D coordinate system shared by all cameras, assign the 3D coordinate points of the lane line to the cells, and obtain the lane line features of the i-th cell in the BEV space , After being processed by the convolution layer, pooling layer, normalization, and ReLu function, the lane line feature map of the nth data sample is output. ;

[0124] Step 2.6: The data is passed into the detection head to generate the nth data sample in a complex road scene with multiple intersections. Predicted lane line position set And the predicted lane line category ,in, represents the ith three-dimensional coordinate point of the predicted dth lane line in the nth data sample in the BEV space, i=1,2,3..., , Indicates the maximum number of three-dimensional coordinate points of the predicted d-th lane line, represents the category of the predicted d-th lane line, and is the set of lane line positions The corresponding classification label is used to identify the type of the lane line. In this embodiment, the lane line type is set to solid line, dashed line, dashed line-special (special line type, such as variable guide lane line, deceleration marking, etc., the main line in the center is dashed line), solid line-special (special line type, such as variable guide lane line, deceleration marking, etc., the main line in the center is solid line), and curb.

[0125] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0126] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.

Claims

1. A lane line detection method for complex road scenes with multiple intersections, characterized in that: The steps include: Step 1: Obtain data of complex road scenes with multiple intersections and perform preprocessing; Step 1.1: Construct a lane line dataset with lane line category labels in a complex road scene with multiple intersections , N is the total number of data samples, is the nth data sample, the nth data sample The category label of the dth lane line is recorded as , represents the nth complex road image, C, H and W represent the number of channels, height and width of the complex road image respectively; represents the nth point cloud data, and ,in, express The real position information of the middle lane line , express The three-dimensional coordinates of the d-th lane line in , represents the reflection intensity of the dth lane line, Indicates the maximum number of sampling points for the lane line; Step 1.2: For the nth complex road image Perform data enhancement processing to obtain the nth enhanced image ; Step 1.3: For the nth point cloud information Perform data enhancement processing to obtain the nth enhanced point cloud information ; Step 2: Construct a lane detection network for complex road scenes with multiple intersections, including: an image feature extraction network based on the attention mechanism, a point cloud column feature extraction network, a lane generative adversarial network, a cross attention module, a perspective conversion module, and a detection head; Step 2.1: Image feature extraction network based on attention mechanism Processing is performed to obtain the image features of the nth multi-scale lane line ; Step 2.2: Point cloud volume column feature extraction network Processing is performed to obtain the pseudo image feature of the nth lane line ; Step 2.3: Lane line generation adversarial network for nth lane line pseudo image features Processing is performed to obtain the nth multi-scale lane line point cloud feature And the judgment results The probability of being a real point cloud feature ; Weighted loss for building lane line generation adversarial networks , thereby using the gradient descent method to train the lane line generation adversarial network and calculate the weighted loss of the lane line generation adversarial network To update the network parameters until Until convergence, the trained lane line generation adversarial network is obtained; Step 2.4: The cross attention module generates the nth optimal multi-scale lane line point cloud feature output by the trained lane line adversarial network and the image features of the n-th multi-scale lane line Process and obtain the nth fusion feature ; Step 2.5: Viewpoint conversion module Processing is performed to obtain the nth lane line feature map ,in, is the number of channels of the lane feature map, , is the height and width of the lane feature map; Step 2.6: The data is passed into the detection head to generate the nth data sample in a complex road scene with multiple intersections. Predicted lane line position set And the predicted lane line category ,in, represents the ith three-dimensional coordinate point of the predicted dth lane line in the nth data sample in the BEV space, Indicates the maximum number of three-dimensional coordinate points of the predicted lane line, Represents the predicted d-th lane line category.

2. The lane line detection method for complex road scenes with multiple intersections according to claim 1 is characterized in that: The image feature extraction network of the attention mechanism in step 2.1 is composed of parallel sub-convolutional layers, A global max pooling layer, Global average pooling, It consists of a feature concatenation layer and a coordinate attention mechanism layer; Step 2.1.1: Input into the feature extraction network based on the attention mechanism, first pass through After processing by parallel sub-convolutional layers, they are input separately A global max pooling layer and After processing by the global average pooling layer, the nth global average feature map is obtained accordingly and the i-th global maximum feature map ; Step 2.1.2 and Passing together The nth lane line splicing feature is obtained by splicing and upsampling the feature splicing layers. ; Step 2.1.3: The coordinate attention mechanism layer uses formula (1.1) and formula (1.2) to respectively The height and width directions of the pooling are respectively pooled, and the nth highly aggregated feature is obtained accordingly. Aggregate features with nth width ; (1.1) (1.2) In formula (1.1) and formula (1.2), express The coordinate point features of the hth row and ith column in ; express The coordinate point features of the j-th row and w-th column in ; Step 2.1.4: After transposition and After concatenation and dimensionality reduction, the concatenation is input into the ReLu function for processing to obtain the nth lane line coding feature. ; Step 2.1.5: The coordinate attention mechanism layer After segmenting the image along the height and width directions, the width lane line coding features are obtained With height lane line coding features , and thus use equations (1.3) and (1.4) to obtain the attention weights of the nth lane line detection along the width direction and the attention weight along the height direction ; (1.3) (1.4) In formula (1.3) and formula (1.4), , are the convolution operations along the height and width of the image, respectively. is the Sigmoid function; Step 2.1.6: Use formula (1.5) to obtain the image features of the nth multi-scale lane line ; (1.5)。 3. The lane line detection method for complex road scenes with multiple intersections according to claim 2 is characterized in that: The point cloud body column feature network in step 2.2 is composed of a voxel partitioning layer, an MLP layer, a convolutional layer, and a global maximum pooling layer; Step 2.2.1, the voxel division layer will Divide into vertical columnar areas with a grid number of P, and select features in non-empty vertical columnar areas ,in, express The feature of the i-th non-empty vertical columnar region in , M is the maximum number of non-empty vertical columnar regions; Step 2.2.2 After being processed by the MLP layer, convolution layer, and global maximum pooling layer, the nth lane line pseudo image feature is output. .

4. The lane line detection method for complex road scenes with multiple intersections according to claim 3 is characterized in that: The lane line generative adversarial network in step 2.3 is composed of a generator G and a discriminator D, and both the generator and the discriminator are composed of multi-layer convolution modules, deep convolution layers and residual blocks; Step 2.3.

1. Input into the generator G, and after being processed by the multi-layer convolution module, the deep convolution layer and the residual block in turn, the nth generation layer feature is obtained , After the full connection layer and normalization processing, the predicted n-th multi-scale point cloud feature is output , Voxel information representing the predicted lane line position; Step 2.3.2 and Input into the discriminator D, and processed by multi-layer convolution modules, deep convolution layers and residual blocks in turn to obtain the nth discriminant layer feature , After being processed by the Sigmoid activation function, the final output is The probability of being judged as a real point cloud feature ; Step 2.3.3: Use formula (1.6) to construct the point cloud inversion loss of the generator G ; (1.6) In formula (1.6), express The expected value of represents L1 loss, The weight hyperparameter representing the loss; Step 2.3.4: Use formula (1.7) to construct the point cloud discrimination loss of the discriminator D ; (1.7) In formula (1.7), express Expected value; Step 2.3.5: Use formula (1.8) to construct the weighted loss of the lane line generation adversarial network ; (1.8) In formula (1.8), and There are two weight parameters for balancing and impact.

5. The lane line detection method for complex road scenes with multiple intersections according to claim 4 is characterized in that: The step 2.4 comprises: Step 2.4.1: Multi-scale lane line image features , optimal multi-scale point cloud features Projected into the same dimensional space, the query matrix is ​​obtained using equations (1.9), (1.10), and (1.11): , key matrix , value matrix : (1.9) (1.10) (1.11) In formula (1.9), formula (1.10), and formula (1.11), , , are three projection matrices; Step 2.4.2: Generate the attention weights between multimodal lane features using formula (1.12) : (1.12) In formula (1.12), represents the dimensions of the key matrix and query matrix, represents the activation function; Step 2.4.3: and After point multiplication, the nth multimodal intermediate feature is obtained ; Step 2.4.4: and After splicing, we get the nth fusion feature .

6. The lane line detection method for complex road scenes with multiple intersections according to claim 5 is characterized in that: The step 2.5 comprises: Step 2.5.

1. Setup ,in, , , are the step lengths in the x direction, y direction, and z direction in the BEV space coordinate system, respectively, and a cell is constructed with the step lengths, thereby obtaining a BEV space composed of a plurality of cells; Step 2.5.2: The perspective conversion module fuses the features After the convolution layer, normalization and sonftmax activation function, the depth probability distribution D of the nth lane line is obtained. n , D n and After splicing, the nth deep fusion feature is obtained ; Step 2.5.3: Project it into the BEV space to obtain the three-dimensional point cloud coordinates of the lane line, assign the three-dimensional coordinate points of the lane line to the cells, and obtain the lane line features of the i-th cell in the BEV space ; After being processed by the convolution layer, pooling layer, normalization, and ReLu function, the lane line feature map of the nth data sample is output. .

7. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the lane line detection method described in any one of claims 1-6, and the processor is configured to execute the program stored in the memory.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the lane line detection method described in any one of claims 1 to 6 are executed.

Images