3D lane line detection method for screening local voxel features based on three-dimensional space

Abstract

The invention discloses a 3D lane line detection method for screening local voxel features based on a three-dimensional space, and belongs to the field of automatic driving. According to an existing method, foresight camera features are summarized into a grid to construct BEV (bird's-eye view) features. After image features are converted into voxel features, when a three-dimensional space is flattened into BEV features along a height dimension, lane line features of different heights cannot be highlighted. According to the method, depth information of pixels is predicted by using LSS (Lift, Spat, Shoot), and image features are converted into three-dimensional space voxel features. And carrying out hierarchical processing on the three-dimensional voxel features based on the height, respectively carrying out global pooling and local pooling, extracting global voxel features and local voxel features, carrying out fusion, and obtaining the fused BEV features. Aiming at the characteristic that a lane line is slender in an image, a strip-shaped convolution attention module based on a feature pyramid is provided in a feature extraction stage, the receptive field is obviously expanded, and the capture capability of lane line features is improved. According to the invention, high-precision 3D lane line detection is realized.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving and proposes a 3D lane line detection method based on screening local voxel features in three-dimensional space. Background Art

[0002] Lane detection is a core technology in the field of autonomous driving. It identifies lane information by processing images collected by the front-view camera. Visual lane detection is mainly divided into 2D lane detection and 3D lane detection. 3D lane detection methods based on deep learning usually include convolutional neural network-based, Transformer-based, and hybrid methods.

[0003] Traditional 2D lane detectors generate detection results on the camera plane, and combine the camera's internal and external parameters to map the results to three-dimensional space through IPM (inverse perspective projection). However, this method is easily affected by slopes and uneven lane lines, thereby reducing the accuracy and reliability of downstream tasks (such as autonomous driving path planning). To solve this problem, 3D lane line detection has gradually become a research focus. 3D detection can directly extract road structure and traffic information from three-dimensional space, significantly improving the safety, rationality and comfort of autonomous driving vehicles in path planning and motion control.

[0004] In real-time reasoning scenarios, 3D lane detection methods based on BEV features with higher computational efficiency have gradually attracted attention. This type of method predicts the pixel depth distribution through the depth branch in the network, and uses the predicted depth information and camera parameters to convert the image features into three-dimensional space and flatten them into BEV features. However, the existing method of flattening the BEV space along the height dimension usually simply aggregates the features in the same BEV grid, and cannot effectively distinguish and utilize information features at different heights. To solve this problem, the present invention proposes a feature screening module based on height information, combined with a strip convolution attention mechanism, which effectively enhances the ability to capture voxel features at different heights and overcomes the shortcomings of the prior art. Summary of the invention

[0005] Aiming at the shortcomings of existing 3D lane line detection methods, which flatten the BEV space along the height dimension and fail to highlight the voxel features of different heights, and the information features of lane lines that are slender and continuous in images, this paper proposes a 3D lane line detection method (HLaneNet) based on three-dimensional space screening of local voxel features.

[0006] In order to evaluate the height distribution of lane lines in the BEV three-dimensional space, the present invention first conducted a statistical analysis of the data set and found that the lane lines are mainly concentrated between 1m and 2.2m in the coordinate system. In the feature extraction stage, a strip convolution attention module integrated on the feature pyramid network is introduced to significantly expand the receptive field and improve the ability to capture lane line features. At the same time, a feature screening module based on height information is proposed to divide the height of the three-dimensional space into multiple intervals, and extract global height voxel features and local height voxel features through global pooling and local pooling, and fuse them. The global features are intended to cover a larger height range, while the local features are intended to emphasize the information-rich local height range. Thereby significantly improving the accuracy of 3D lane line detection.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0008] Step 1: A feature extraction network is designed to extract lane features. The feature extraction network consists of a backbone network ResNet50 model, a strip convolution attention module, and a neck network feature pyramid FPN. The strip convolution attention module consists of three parts: First, depthwise separable convolution is used to aggregate local information. Secondly, depthwise separable strip convolutions with kernel sizes of 7, 11, and 21 are used to capture multi-scale contextual information. Depthwise separable strip convolutions use strip convolutions to help extract strip features and effectively identify lanes. Finally, through a linear layer, the output of the linear layer is directly used as the attention weight. The weight coefficient acts on the input features after the linear layer, a residual structure, and outputs features through the feedforward layer.

[0009] Step 2: Depth prediction based on LSS (Lift, Splat, Shoot) depth estimation, converting 2D image features into 3D voxel features. First, obtain the depth features from the front view image, and output the front view image features F after passing through the feature extraction network. f , perform image depth encoding. The number of channels of the input feature map is adjusted through a convolution transformation dimension with a convolution kernel size of 1×1 and filled with 0 for subsequent depth probability calculation and feature vector extraction. Secondly, the Softmax probability is calculated for the depth dimension of the input tensor to obtain the probability distribution of each pixel at different depths. The eigenvalues ​​of each pixel in different depth intervals are converted into a probability distribution range of [0, 1]. Based on the image and depth range, a 3D frustum grid containing the coordinates of each pixel point (x, y, z) is generated. Among them, D and H v and W v They represent the depth range of the voxel feature, the height of the voxel feature, and the width of the voxel feature respectively.

[0010] Step 3: Propose a feature screening method based on height information. The structure is divided into two parts: local feature processing and global feature processing. Local feature slicing constructs local slicing features from the overall range [-5, 3], divides the overall range [-5, 3] into two partitions, including [-5, 1] ​​and [1, 3], and performs local pooling to obtain the bird's-eye view feature BEV. (-5,1) and BEV (1,3) . The channel attention mechanism is used to reweight the local slice features to effectively summarize the information at different heights. Local feature processing creates a SE channel attention mechanism with the number of input channels. Local features are extracted through two layers of convolution with a kernel size of 3 and a padding of 1, each followed by a BN and ReLU activation function. Global feature processing is similar to the local feature part, summarizing the global height information of [-5, 3]. Then the global feature is obtained and local features Among them, H g and W g Respectively represent the height and width of the global BEV feature map, H l and W l Represent the height and width of the local BEV feature map respectively. Local and global features are processed separately, and each part uses convolution, BN, ReLU activation function, channel attention and residual connection to extract richer features and avoid the gradient disappearance problem.

[0011] Step 4: The 3D lane detection head consists of a confidence head, a visibility head, a category head, a row index head, an X-axis offset head, and a Z-axis offset head. Each task head contains two convolutional layers, and the input of the detection head is the feature of the BEV feature encoder. Among them, B, C, H h and W h They represent the batch size, the number of channels, the height of the BEV feature map of the input detection head, and the width of the BEV feature map of the input detection head respectively. The total loss of the proposed method is the sum of the loss functions of all task heads.

[0012] Compared with the prior art, the present invention has the following advantages:

[0013] (1) In view of the shortcomings of the existing 3D lane detection method that the BEV space is flattened along the height dimension and the voxel features at different heights cannot be highlighted, the present invention proposes a feature screening module based on height information. By establishing global and local BEV slice features, the traditional flattening process is avoided and the information features at different heights are fully utilized.

[0014] (2) In view of the fact that lane lines are slender and continuous in images, existing methods are generally limited by small convolution kernel sizes. This paper integrates a strip convolution attention module into the feature pyramid neck network, significantly expanding the receptive field, thereby more accurately capturing the complete lane line features and improving detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flowchart of the steps of the 3D lane line detection method in an embodiment of the present invention;

[0016] Figure 2 is a schematic diagram of a network structure for implementing a 3D lane line detection method in an embodiment of the present invention;

[0017] Figure 3 is a schematic diagram of the structure of a strip convolution attention module in an embodiment of the present invention;

[0018] Figure 4 is a schematic diagram of the network structure of a multi-scale separable convolution module in an embodiment of the present invention;

[0019] Figure 5 is a lane slope and lane line height statistical data graph in an embodiment of the present invention;

[0020] Figure 6 It is a schematic diagram of the network structure of the feature screening module based on height information in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The present invention is described in detail below in conjunction with a specific implementation case, and the implementation case is named Implementation of a 3D lane line detection method based on screening local voxel features in three-dimensional space. The following implementation case is for technicians to understand the specific process of the present invention, but does not limit the present invention in any form. In addition, changes and improvements can be made on the basis of the present invention without departing from the concept of the present invention.

[0022] 3D lane detection is a key perception task in autonomous driving scenarios. Accurate lane positioning is crucial for obstacle avoidance, path planning, and lane keeping, which helps ensure that vehicles drive safely within the drivable area. 3D lane detection methods based on bird's-eye view (BEV) features generate bird's-eye view features by transforming 2D image features into real-world space. A common method for this transformation is to use explicit or implicit depth estimation for view transformation. There are two main strategies for generating BEV features: one is to project features onto the BEV plane, and the other is to sample features from the camera plane. For the first method, the network divides the BEV plane into regular grids and maps features from the camera plane into these grids. To this end, the network needs to establish a depth branch to predict the depth information of all pixels. By combining depth prediction and camera parameters, image features can be accurately projected into 3D space and spliced ​​into BEV features. The second method initializes multiple query points in the BEV space, each corresponding to a 3D coordinate, and projects it back to the image plane according to the camera parameters to obtain the corresponding pixel features. By interacting with the features of these query points, the pixel features are aggregated to construct the final BEV features.

[0023] The proposed HLaneNet method adopts the first strategy. When projecting the front view features to the ground to generate dense BEV features, deviations may occur during the spatial transformation process. Since this method assumes that all objects are located on a horizontal plane with a height of zero, objects such as lane lines with different height information may produce projection errors during the transformation process, causing the true 3D position of the lane lines to be distorted. Especially in 3D lane line detection, lane lines are usually presented as continuous and narrow lines in the front view, which affects the accuracy of existing detection algorithms. To solve this problem, a 3D lane line detection method HLaneNet based on three-dimensional space screening of local voxel features is proposed.

[0024] The overall idea of ​​3D lane line detection is as follows Figure 1 ,It mainly includes the following steps: feature extraction stage, view conversion, feature screening based on height information, and 3D lane line detection head.

[0025] Part 1: Feature Extraction

[0026] Between the backbone network and the neck network, a strip convolution attention module integrated with the feature pyramid network FPN is designed to enhance feature representation for lane feature extraction. The feature backbone network uses the ResNet50 model, and the front view image of the vehicle Input to the backbone network to extract multi-level feature maps, where B, H, and W represent the batch size, image height, and image width, respectively. The neck network uses the feature pyramid network FPN, which outputs multi-scale feature maps and then fuses them into a single feature map. Among them, B, C, H f and W f Represent the batch size, number of channels, image feature map height and image feature map width respectively.

[0027] from Figure 2 As can be seen in Figure 2, the strip convolution attention module is placed after the horizontal layer of FPN to minimize the computational cost. Instead of using a self-attention mechanism to generate similarity scores between query and value outputs, a novel multi-scale separable convolution module is designed to quantify the correlation between input tokens. The multi-scale separable convolution module contains three parts: a depthwise separable convolution for aggregating local information; a multi-branch depthwise separable strip convolution for capturing multi-scale contextual information; and a linear layer for modeling the relationship between different channels. The output of the linear layer is directly used as the attention weight. When the convolution kernel size exceeds 7×7, the performance of the convolution network is limited, which limits the downstream tasks that require a large receptive field. The strip convolution attention module selects large convolution kernels. The size of the convolution kernel is selected through ablation experiments. Three convolution kernel sizes of 7×7, 11×11, and 21×21 are selected as the convolution kernels of the multi-scale separable convolution module.

[0028] The following are shown Figure 3 Mathematical expression of the strip convolution attention module method:

[0029]

[0030] Y1=Att⊙(W2X)+X (0.2)

[0031] Y=FFN(Y1)+Y1 (0.3)

[0032] Among them, Att represents the output of the multi-scale separable convolution module as the attention weight coefficient, DConv 5×5 Indicates a 5×5 depthwise separable convolution, X represents the input feature, and W1 and W2 are two different linear layers. ⊙ is the element-wise matrix multiplication operation, Att is used as the weight coefficient, and the linearly projected feature W2X is adjusted by dot multiplication operation, and then added to the input feature to obtain the output feature Y1. FFN represents a forward layer, and the output feature Y. MutilScale i express Figure 4The i-th branch in . There are four branches in total, namely one depthwise separable convolution and three depthwise separable strip convolutions with different kernel sizes. In the depthwise separable strip convolution branch, two depthwise strip convolutions are used to approximate the standard depthwise convolution with a large kernel. There are two reasons for choosing depthwise strip convolution. On the one hand, strip convolution is lighter and can further reduce the amount of computation. To simulate the standard convolution with a kernel size of 7×7, only a pair of 7×1 and 1×7 convolutions is required, which effectively reduces the amount of parameters generated as the kernel size increases without any performance degradation. On the other hand, for similar strip-shaped targets such as lane lines, strip convolution can serve as a supplement to grid convolution to help extract strip features and more effectively identify lane lines.

[0033] Part 2: View Transformation

[0034] View conversion is an important step in converting image features into three-dimensional space features in autonomous driving tasks. The feature F extracted from the image is converted into f Input, perform view transformation on the image to construct the view cone, predict the depth of the image, based on the predicted depth, camera intrinsics and extrinsics, and then generate spatial voxel features based on the predicted image depth and image features Among them, D and H v and W v They represent the depth range of the voxel feature, the height of the voxel feature, and the width of the voxel feature respectively.

[0035] First, we obtain the depth features from the front view image. The tensor dimension of an image is After the feature extraction network, the image depth is encoded. The number of channels of the input feature map is adjusted from 512 to D+C through a convolution transformation dimension with a convolution kernel size of 1×1 and filled with 0, which means that it only acts on the channel dimension and does not change the spatial resolution. The main purpose is to redistribute the channels of the feature map for subsequent depth probability calculation and feature vector extraction. Secondly, the Softmax probability is calculated for the depth dimension of the input tensor to obtain the probability distribution of each pixel at different depths. The eigenvalues ​​of each pixel in different depth intervals are converted into a probability distribution range of [0, 1] to ensure that the sum of the depth probabilities of each pixel is 1. The shape of the resulting Depth is B×D×H d ×W d , H d and W d is the feature map height and feature map width. The Softmax calculation formula is as follows:

[0036]

[0037] In the above formula, the input of the Softmax function is a vector containing j elements, where x i and xj is one of the elements. The number of output channels D+C is 105, D is 41, and C is 64. Here, the first 41 channels represent the depth range from 4 meters to 45 meters, and the last 64 channels store the features of the current point. The channel splitting and inner product are used to obtain a 64×41 matrix, which represents the eigenvalues ​​of the feature point at each depth of 4 to 45 meters. The combination of depth and features will expand the depth tensor by one dimension on the depth channel, making its shape become B×1×D×H d ×W d Expand the feature map S by one dimension in the depth dimension, so that its shape becomes B×C×1×H d ×W d Through the broadcast mechanism, the depth probability distribution depth is multiplied by the feature channel to obtain a new feature tensor new S. The shape of the new feature tensor new S is B×C×D×H d ×W d , that is, the depth feature of each pixel is combined with its corresponding depth probability distribution to represent the features in each depth interval. This operation combines the depth information with the feature information to produce a deep encoding feature for each pixel, which contains the feature information of each pixel in each depth interval.

[0038] Get the depth feature matrix of a single 2D image. These features are mapped to the world coordinate system and associated with the grid of the BEV perspective. Create a frustum point cloud of the coordinate position of the feature map in the world coordinate system, which is the position of each point of the depth feature in the camera coordinate system. Create a 3D frustum grid for each pixel in the image. This grid links each pixel to its corresponding depth value z-axis, defining its (x, y, z) coordinates in 3D space.

[0039] Generate a depth grid: First define a depth interval range [4, 45], which means the depth ranges from 4 to 45. Then generate an arithmetic progression to represent the discrete values ​​of depth. Generate depth values ​​from 4 to 45, and the number of generated depth values ​​is 41. Reshape the generated depth values ​​to (D, 1, 1), where D is the number of depth intervals 41. Expand the depth values ​​to the same dimension as the feature map size (D, H d , W d ), H d and W d is the feature map height and feature map width, which makes each depth value correspond to each pixel of the feature map.

[0040] Generate coordinates on the image plane: First, generate the coordinates of the image in the horizontal direction. Generate W d Uniformly distributed values ​​represent the horizontal coordinates of each pixel in the image. Adjust its shape to (1, 1, W d), indicating that it is a row vector. Expand this vector to (D, H d , W d ), which means that each depth value and each image pixel share the same horizontal coordinate. Generate the image coordinates in the vertical direction. Generate H d Uniformly distributed values ​​represent the vertical coordinates of each pixel in the image. Adjust its shape to (1, H d , 1), indicating that it is a column vector. Expand this vector to (D, H d , W d ), which means that each depth value and each image pixel share the same vertical coordinate. Finally, the coordinates and depth values ​​are stacked.

[0041] Based on the image and depth range, a 3D frustum grid containing the (x, y, z) coordinates of each pixel is generated. This grid represents the position of each pixel in the image in 3D space, where: x and y are the coordinates in the image plane. z is the depth value of each pixel, indicating its depth position in 3D space.

[0042] Part 3: Feature screening based on height information

[0043] Figure 5 The distribution of lane slope and lane line height statistics are shown. This shows that the lane lines in the dataset all have slopes. The lane lines are mainly concentrated between 1m-2.2m in the coordinate system, and their positions in the spatial height [-5, 3] always belong to the range [1, 3]. Existing methods aggregate features that fall in the same flat BEV grid, and these methods cannot utilize different heights. A feature screening method based on height information is proposed. The structure is divided into two parts: local feature processing and global feature processing, and the gradient disappearance problem is alleviated through jump connections.

[0044] Slope=(y2-y1) / (z2-z1)(0.5)

[0045] Among them, Slope represents the slope of the lane, (x1, y1, z1) and (x2, y2, z2) are the starting point and end point of the lane respectively. The camera coordinate system is located at the center of the camera, with the X-axis pointing to the right, the Y-axis pointing downward, and the Z-axis pointing forward. The purpose of local feature slicing is to divide the BEV height space according to different heights and emphasize the information height range. By constructing local slicing features from the overall range [-5, 3], the lane lines are concentrated near the height of space [1, 3]. The overall range [-5, 3] is divided into 2 partitions, including [-5, 1] ​​and [1, 3], and local pooling is performed to obtain the bird's-eye view feature BEV (-5,1) and BEV (1,3)The channel attention mechanism is used to re-weight local slice features to effectively aggregate information at different heights.

[0046] Local feature processing, a channel attention mechanism SE is used here to enhance the representation ability of local features. A SE with the number of input channels is created. Local features are extracted through two layers of convolution, with a kernel size of 3 and a padding of 1, and each layer is followed by a BN and ReLU activation function. The skip connection is implemented through 1×1 convolution and BN. Its function is to skip the residual part and directly map the input feature to the output channel to help alleviate the gradient vanishing problem.

[0047] The global feature processing is similar to the local feature part, summarizing the global height information of [-5, 3]. Then the global feature is obtained and local features Perform tandem fusion to obtain BEV fusion features Among them, H g and W g Respectively represent the height and width of the global BEV feature map, H l and W l Respectively represent the height and width of the local BEV feature map, H c and W c They represent the height and width of the BEV feature map of the serial fusion, respectively. By processing local and global features (local depth features and global depth features) separately, each part uses convolution, BN, ReLU activation function, channel attention, and residual connection to extract richer features and avoid the gradient disappearance problem.

[0048] F l =SE(BEV (-5,1) ,BEV (1,3) ) (0.6)

[0049] F c =[F l ,F g ] (0.7)

[0050] Among them, SE represents the channel attention module, [,] represents the concatenation operation along the channel dimension, and BEV (-5,1) and BEV (1,3) Represent the bird's-eye view features of voxel feature transformation in the range of [-5, 1] ​​and [1, 3] respectively.

[0051] Part 4: 3D Lane Detection Head

[0052] The 3D lane line detection head consists of a confidence head, a visibility head, a category head, a row index head, an X-axis offset head, and a Z-axis offset head. The input of the detection head is the feature of the BEV feature encoder. Each task head contains two convolutional layers, B, C, H h and W h They represent the batch size, the number of channels, the height of the BEV feature map of the input detection head, and the width of the BEV feature map of the input detection head respectively. The total loss of the proposed method is the sum of the loss functions of all task heads.

[0053] To verify the effectiveness of the HLaneDet method in lane detection tasks, the present invention conducted detailed experiments on two different datasets, including the OpenLane dataset and the ONCE 3DLanes dataset. The OpenLane dataset is built based on the WaymoOpen dataset. It is a large-scale real-world three-dimensional lane detection dataset, containing more than 200,000 frames of images and more than 880,000 annotated lane information. Each frame of the image is accompanied by detailed camera internal and external parameter information, and all lane lines are accurately annotated. Even in the absence of the middle roadside line, detailed information about the reverse lane is also covered. The ONCE3DLanes dataset is another large-scale 3D lane detection dataset based on the real world, covering millions of different scene instances. The dataset includes more than 211,000 images with precise 3D lane point annotations, showing the lane conditions at different time periods and various geographical features, with a high degree of diversity and authenticity.

[0054] The performance of different methods on the OpenLane dataset is compared, and the results are shown in Table 1. The performance of different methods on the ONCE-3DLanes dataset is compared, and the results are shown in Table 2. The results show that the method HLaneNet proposed in this invention has reached the advanced level, proving the superiority of the method. Table 1

[0055]

[0056] Table 2

[0057]

[0058] The above description only describes the specific embodiments of the present invention, but the protection scope of the present application is not limited thereto. Any changes or substitutions made by a person skilled in the art within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the scope of the attached claims.

Claims

1. A 3D lane detection method based on three-dimensional space screening of local voxel features, characterized in that Here are the steps: Step 1: Design a feature extraction network to extract lane features; the feature extraction network consists of a backbone network ResNet50 model, a strip convolution attention module and a neck network feature pyramid FPN; the strip convolution attention module consists of three parts: first, use depthwise separable convolution to aggregate local information; second, use depthwise separable strip convolution with kernel sizes of 7, 11 and 21 to capture multi-scale contextual information, and use strip convolution to help extract strip features and effectively identify lanes; finally, pass through a linear layer, and the output of the linear layer is directly used as the attention weight; The weight coefficient acts on the input features after the linear layer, a residual structure, and output features through the feedforward layer; Step 2: Depth prediction based on the LSS (Lift, Splat, Shoot) depth estimation method, converting the two-dimensional image features into three-dimensional voxel features; first obtain the depth features from the front view image, and output the front view image features after passing through the feature extraction network Perform image depth encoding, where B, C, H f and W f Represent the batch size, number of channels, image feature map height and image feature map width respectively; perform dimensional transformation through a convolution with a convolution kernel size of 1×1 and padding of 0 to adjust the number of channels of the input feature map for subsequent depth probability calculation and feature vector extraction; secondly, calculate the Softmax probability of the depth dimension of the input tensor to obtain the probability distribution of each pixel at different depths; convert the eigenvalue of each pixel at different depth intervals into a probability distribution range of [0, 1]; generate a 3D frustum grid containing the coordinates of each pixel point (x, y, z) according to the image and depth range Among them, D and H v and W v They represent the depth range of the voxel feature, the height of the voxel feature, and the width of the voxel feature respectively; Step 3: A feature screening method based on height information is proposed. The structure is divided into two parts: local feature processing and global feature processing. Local feature slicing constructs local slicing features from the overall range [-5, 3], divides the overall range [-5, 3] into two partitions, including [-5, 1] ​​and [1, 3], and performs local pooling to obtain the bird's-eye view feature BEV. (-5,1) and BEV (1,3) ; Use the channel attention mechanism to reweight the local slice features; local feature processing, create a SE channel attention mechanism with the number of input channels; extract local features through two layers of convolution, with a kernel size of 3 and a padding of 1, and each layer is followed by BN and ReLU activation functions; global feature processing, similar to the local feature part, summarizes the global height information of [-5, 3]; pooling into global features and local features Among them, H g and W g Respectively represent the height and width of the global BEV feature map, H l and W l Respectively represent the height and width of the local BEV feature map; Step 4: The 3D lane detection head consists of a confidence head, a visibility head, a category head, a row index head, an X-axis offset head, and a Z-axis offset head; the input of the detection head is the feature of the BEV feature encoder. Each task head contains two convolutional layers, B, C, H h and W h They represent the batch size, number of channels, BEV feature map height of the input detection head, and BEV feature map width of the input detection head respectively.

2. The method according to claim 1, characterized in that Here are the steps: Part 1: Feature Extraction Between the backbone network and the neck network, a strip convolution attention module integrated with the feature pyramid network FPN is designed to enhance feature representation for lane line feature extraction; the feature backbone network uses the ResNet50 model, and the front view image of the vehicle Input to the backbone network to extract multi-level feature maps, where B, H, and W represent the batch size, image height, and image width respectively; the neck network uses the feature pyramid network FPN, which outputs multi-scale feature maps and then fuses them into a single feature map Among them, B, C, H f and W f Respectively represent the batch size, number of channels, image feature map height and image feature map width; The strip convolution attention module is placed after the horizontal layer of FPN to minimize the computational cost. Instead of using a self-attention mechanism to generate similarity scores between query and value outputs, a novel multi-scale separable convolution module is designed to quantify the correlation between input tokens. The multi-scale separable convolution module consists of three parts: a depthwise separable convolution for aggregating local information; a multi-branch depthwise separable strip convolution for capturing multi-scale contextual information; and a linear layer for modeling the relationship between different channels. The output of the linear layer is directly used as the attention weight. When the convolution kernel size exceeds 7×7, the performance of the convolution network will be limited, thus limiting the downstream tasks that require a large receptive field; the strip convolution attention module selects a large convolution kernel, and the size of the convolution kernel is selected through ablation experiments. Three convolution kernel sizes of 7×7, 11×11 and 21×21 are selected as the convolution kernels of the multi-scale separable convolution module; The following is the mathematical expression of the strip convolution attention module: Y1=Att⊙(W2X)+X(0.2) Y=FFN(Y1)+Y1(0.3) Among them, Att represents the output of the multi-scale separable convolution module as the attention weight coefficient, DConv 5×5 Indicates that the convolution kernel size is 5×5 depth-separable convolution, MutilScale i represents the i-th branch in the multi-scale separable convolution module. There are four branches in total, namely a depthwise separable convolution and three depthwise separable strip convolutions with different kernel sizes. X represents the input feature, W1 and W2 are two different linear layers. ⊙ is the element-wise matrix multiplication operation. Att is used as the weight coefficient. After the dot multiplication operation is performed with the linearly projected feature W2X, the output feature Y1 is obtained by adding the input feature. FFN represents a forward layer, and the output feature Y is obtained. Part 2: View Transformation View conversion is an important step in the autonomous driving task to convert image features into three-dimensional space features. The feature F extracted from the image is converted into f Input, perform view transformation on the image to construct the view cone, predict the depth of the image, based on the predicted depth, camera intrinsics and extrinsics, and then generate spatial voxel features based on the predicted image depth and image features Among them, D and H v and W v They represent the depth range of voxel features, the height of voxel features, and the width of voxel features respectively; First, we obtain the depth features from the front view image. The tensor dimension of an image is After the feature extraction network, the image depth is encoded; the number of channels of the input feature map is adjusted from 512 to D+C through a convolution transformation dimension with a convolution kernel size of 1×1 and filled with 0, which only acts on the channel dimension and does not change the spatial resolution; the channels of the feature map are reallocated for subsequent depth probability calculation and feature vector extraction; secondly, the Softmax probability is calculated for the depth dimension of the input tensor to obtain the probability distribution of each pixel at different depths; the eigenvalues ​​of each pixel at different depth intervals are converted into a probability distribution range of [0, 1] to ensure that the sum of the depth probabilities of each pixel is 1; the resulting Depth shape is B×D×H d ×W d , H d and W d is the feature map height and feature map width; the Softmax calculation formula is as follows: In the above formula, the input of the Softmax function is a vector containing j elements, where x i and x j is one of the elements; the number of output channels D+C is 105, D is 41, and C is 64; here the first 41 channels represent the depth range from 4 meters to 45 meters, and the last 64 channels store the features of the current point; the channel splits and performs the inner product to obtain a 64×41 matrix, which represents the eigenvalues ​​of the feature point at each depth of 4 to 45 meters; the combination of depth and features will expand the depth tensor by one dimension on the depth channel, making its shape become B×1×D×H d ×W d ; Expand the feature map S by one dimension in the depth dimension so that its shape becomes B×C×1×H d ×W d ; Through the broadcast mechanism, the depth probability distribution depth is multiplied by the feature channel to obtain a new feature tensor new S; the shape of the new feature tensor new S is B×C×D×H d ×W d , that is, the depth feature of each pixel is combined with its corresponding depth probability distribution to represent the features in each depth interval; this operation combines the depth information with the feature information to produce a deep encoding feature for each pixel, which contains the feature information of each pixel in each depth interval; Get the depth feature matrix of a single 2D image; map these features to the world coordinate system and associate them with the grid of the BEV perspective; create a view cone point cloud of the coordinate position of the feature map in the world coordinate system, which is the position of each point of the depth feature in the camera coordinate system; create a 3D view cone grid for each pixel in the image; this grid associates each pixel with its corresponding depth value z-axis, defining its (x, y, z) coordinates in 3D space; Generate a depth grid: First define a depth interval, a range [4, 45], indicating that the depth ranges from 4 to 45; then generate an arithmetic progression to represent the discrete values ​​of the depth; generate depth values ​​from the range of 4 to 45, and the number of generated depth values ​​is 41; adjust the shape of the generated depth values ​​to (D, 1, 1), where D is the number of depth intervals 41; expand the depth values ​​to the same dimension as the feature map size (D, H d , W d ), H d and W d is the feature map height and feature map width, which makes each depth value correspond to each pixel of the feature map; Generate coordinates on the image plane: First, generate the coordinates of the image in the horizontal direction; generate W d A uniformly distributed value represents the horizontal coordinate of each pixel in the image; adjust its shape to (1, 1, W d ), indicating that it is a row vector; expand this vector to (D, H d , W d ), which means that each depth value and each image pixel share the same horizontal coordinate; generate the coordinates of the image in the vertical direction; generate H d uniformly distributed values, representing the vertical coordinates of each pixel in the image; adjust its shape to (1, H d , 1), indicating that it is a column vector; expand this vector to (D, H d , W d ), which means that each depth value and each image pixel share the same vertical coordinate; finally, the coordinates and depth values ​​are stacked; Based on the image and depth range, a 3D frustum grid containing the (x, y, z) coordinates of each pixel point is generated; this grid represents the position of each pixel in the image in 3D space, where: x and y are the coordinates in the image plane; z is the depth value of each pixel, indicating its depth position in 3D space; Part 3: Feature screening based on height information The distribution of lane slope and lane line height are statistically analyzed. It shows that all lane lines in the dataset have slopes. Lane lines are mainly concentrated between 1m and 2.2m in the coordinate system, and their positions in the spatial height [-5, 3] always fall within the range of [1, 3]. A feature screening method based on height information is proposed. The structure is divided into two parts: local feature processing and global feature processing. The gradient vanishing problem is alleviated through jump connections. Slope=(y2-y1) / (z2-z1)(0.5) Among them, Slope is the slope of the lane, (x1, y1, z1) and (x2, y2, z2) are the starting point and end point of the lane respectively; the camera coordinate system is located at the center of the camera, with the X axis pointing to the right, the Y axis pointing downward, and the Z axis pointing forward; the purpose of local feature slicing is to divide the BEV height space according to different heights and emphasize the information height range; by constructing local slicing features from the overall range [-5, 3], the lane lines are concentrated near the height of space [1, 3]; the overall range [-5, 3] is divided into 2 partitions, including [-5, 1] ​​and [1, 3], and local pooling is performed to obtain the bird's-eye view feature BEV (-5,1) and BEV (1,3) ; Use the channel attention mechanism to re-weight local slice features and aggregate information at different heights; Local feature processing, a channel attention mechanism SE is used here to enhance the representation ability of local features; a SE with the number of input channels is created; two layers of convolution are used, with a kernel size of 3 and a padding of 1, and each layer is followed by a BN and ReLU activation function to extract local features; the skip connection is implemented through 1×1 convolution and BN, skipping the residual part and directly mapping the input features to the output channel to alleviate the gradient vanishing problem; The global feature processing is similar to the local feature part, summarizing the global height information of [-5, 3]; then the global feature and local features Perform tandem fusion to obtain BEV fusion features Among them, H g and W g Respectively represent the height and width of the global BEV feature map, H l and W l Respectively represent the height and width of the local BEV feature map, H c and W c They represent the height of the BEV feature map fused in series and the width of the BEV feature map fused in series respectively; By processing local and global features separately, each part uses convolution, BN, ReLU activation function, channel attention and residual connection; F l =SE(BEV (-5,1) ,BEV (1,3) )(0.6) F c =[F l ,F g ](0.7) Among them, SE represents the channel attention module, [,] represents the concatenation operation along the channel dimension, and BEV (-5,1) and BEV (1,3) Represent the bird's-eye view features of voxel feature conversion in the range of [-5, 1] ​​and [1, 3] respectively; Part 4: 3D Lane Detection Head The 3D lane detection head consists of a confidence head, a visibility head, a category head, a row index head, an X-axis offset head, and a Z-axis offset head; each task head contains two convolutional layers, and the input of the detection head is the feature of the BEV feature encoder. Among them, B, C, H h and W h They represent the batch size, number of channels, BEV feature map height of the input detection head, and BEV feature map width of the input detection head respectively.

Images