Automatic driving 3D target detection method based on local feature coding

Abstract

The application relates to an automatic driving 3D target detection method based on local feature coding, and belongs to the technical field of automatic driving road vehicle detection. The method comprises the following steps: S1, inputting road point cloud data into an Input network module for pretreatment; S2, performing three-dimensional feature extraction on the pretreated data through a three-dimensional backbone network module; the three-dimensional backbone network module comprises a deep layer downsampling convolution network, a self-attention encoding module and a cross-attention decoding module; S3, inputting the three-dimensional features into a two-dimensional backbone network module, projecting the three-dimensional features to a BEV perspective to obtain multi-scale two-dimensional features; the two-dimensional backbone network module comprises a self-attention position encoding module based on a Swin-Transformer and an enhanced feature fusion module; and S4, feeding the three-dimensional features and the two-dimensional features into a Head network module to fuse and obtain classification and regression results, and outputting detection information of vehicles, pedestrians and cyclists in various categories.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving road vehicle detection, and relates to an autonomous driving 3D target detection method based on local feature encoding and decoding. Background Art

[0002] As the number of cars increases, vehicle-related traffic problems are becoming increasingly serious, such as road congestion and frequent accidents, causing immeasurable losses to personal safety and the economy.

[0003] In recent years, deep learning algorithms have continued to advance. Leveraging the superior feature extraction capabilities of convolutional neural networks, deep learning methods have taken the field of object detection by storm. Research on deep learning-based object detection for autonomous driving has become a hot topic. Autonomous driving requires making immediate decisions based on environmental information. Typically, various sensors (such as cameras and lidar) are used to first perceive the surrounding environment, including roads, vehicles, pedestrians, and traffic signs. Three-dimensional (3D) object detection methods are then employed to generate a 3D digital representation of the vehicle's surroundings (including object shapes, positions, and categories) through data processing and algorithmic analysis.

[0004] Although deep learning-based detection algorithms have achieved promising results, they still face challenges: limited receptive fields lead to suboptimal detection, foreground points are not given more attention than background points, and features are poorly correlated, making feature extraction difficult and resulting in poor detection performance for some categories. Therefore, a new 3D object detection method for autonomous driving is urgently needed to address these issues. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a 3D target detection method for autonomous driving based on local feature encoding and decoding. To address the two problems of multi-level feature extraction and multi-scale feature context dependency in 3D target detection in autonomous driving, a point-voxel detection framework is adopted to propose a local feature encoding and decoding network LFED-RCNN (Local Feature Encode-Decode RCNN) that integrates multiple technologies. This method can accurately identify the information of vehicles, pedestrians, and cyclists on the road in the collected point cloud data, achieving fast and accurate detection results.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A 3D object detection method for autonomous driving based on local feature encoding and decoding includes the following steps:

[0008] S1: Input the road point cloud data into the Input network module for preprocessing;

[0009] S2: The pre-processed road point cloud data is subjected to 3D feature extraction through a 3D backbone network module to obtain corresponding 3D features; the 3D backbone network module includes a deep downsampling convolutional network (EDSNet) and a Transformer-based local feature encoding and decoding module (LEDNet); the Transformer-based local feature encoding and decoding module (LEDNet) includes a self-attention encoding module (SA Encoder) and a cross-attention decoding module (CA Decoder);

[0010] S3: Input the 3D features into a 2D backbone network module and project them to the BEV (Birds Eye View) perspective to obtain multi-scale 2D features; the 2D backbone network module includes a Swin-Transformer-based self-attention position encoding module (Position Encode, PE) and an enhanced feature fusion module;

[0011] S4: The 3D and 2D features are fed into the Head network module, which fuses and obtains the classification and regression results, and outputs the detection information of each category of vehicles, pedestrians, and cyclists.

[0012] Furthermore, in step S1, the preprocessing is to perform data enhancement on the road point cloud data, specifically: performing random scaling, random cropping, random arrangement and random flipping processing on the road point cloud data.

[0013] Furthermore, in step S2, the receptive field size calculation formula of the deep downsampling convolutional network (EDSNet) is:

[0014]

[0015] Among them, f k-1 is the size of the receptive field corresponding to the k-1th layer, l k is the convolution kernel size of the kth layer, s i is the step size of the i-th layer; it can be seen from the above formula that the receptive field of the convolutional network increases layer by layer, showing an iterative relationship. Therefore, the receptive field can be expanded by increasing the depth of the convolutional network to obtain deeper features and fuse multi-level 3D features.

[0016] Furthermore, in step S2, the self-attention encoding module (SA Encoder) is designed based on the multi-head self-attention (MSA) mechanism, which is composed of the multi-head self-attention (W hw _MSA) and D-window-based multi-head self-attention (Wd _MSA) and connected in a residual manner; the HW window focuses on the relationship between HW dimensions, such as a tensor of (-1, H*W, C), while the D window focuses on the relationship on the D dimension, such as a tensor of (-1, D, C);

[0017] The implementation details are as follows: the input 3D convolutional feature map is divided into small blocks of equal size along the HW dimension, and encoded in the HW direction using a sliding window; then, the output features are reconstructed to restore the shape of the initial features; along the D dimension, it is divided into small blocks again, and encoded using a sliding window in the D direction, and finally, the encoded feature map is reconstructed to the initial shape and size; at the same time, to prevent overfitting, a Dropout layer is added, and then the initial features and the encoded features are spliced ​​using a residual connection method; the local feature encoding process can be expressed as follows:

[0018] F l =W d _MSA(W hw _MSA{F S})

[0019] Among them, F S is the input feature, F l is the encoded feature, W d _MSA is the multi-head attention mechanism of D window, W hw _MSA is the multi-head attention mechanism for HW windows.

[0020] In general, SA Encoder divides the input features into small blocks and encodes the information in the small blocks in sequence along different dimensions, thereby maintaining the continuity of 3D local area features and helping to provide more detailed information for foreground targets.

[0021] Furthermore, in step S2, the cross attention decoding module (CA Decoder) specifically includes: the input feature map is the encoded multi-scale feature F l With F l `; First, perform a 3D convolution operation on the input features and reconstruct the multi-scale 3D feature maps into the same size to generate key-value pairs. The specific calculation is as follows:

[0022] Conv3d(x)=σ(w*x+b)

[0023] Among them, Conv3d(x) is a 3D convolution operation, x is the input feature map size, w is the convolution kernel size, b is the bias term, and σ is the activation function; after obtaining the same two features, first perform a 1*1*1 convolution to generate the initial query (Q), key (K), and value (V) respectively; reconstruct the space of Q, K, and V into R HW*D*C, where HW represents the size of the HW window, D represents the size of the D window, and C is the number of channels. The HW dimension cross attention weight is calculated by matrix multiplication; after that, the output feature is fed into the 1*1*1 convolution again to produce a new K and a new V; the initial Q, new K and new V are combined into a new query pair, and the space is reshaped into R D *HW*C , the D-dimensional cross attention weight is calculated by matrix multiplication. Through cross-query, the two input features are combined, which greatly improves the representation ability of the output features.

[0024] Furthermore, in step S3, the specific implementation details of the position encoding module are as follows: the input features are first normalized through a LayerNorm layer, and then passed to the WindowAttention layer for self-attention weight calculation. The input tensor is linearly transformed (qkv) to obtain the query, key, and value, and then the attention weight is calculated through the softmax function, and they are multiplied by the value to obtain the representation of the self-attention size; after the attention weight size is calculated, the original input and the output of the attention mechanism are added through a residual connection, and finally processed through the LayerNorm and MLP modules and the residual connection is used again to fuse the features. Among them, LayerNorm is a technology for standardizing input data, which helps to accelerate training and improve the generalization ability of the model. After the above operations, adding position association and rich context information to the 2D features helps to improve the quality of 2D features.

[0025] The beneficial effects of the present invention are as follows: the present invention adopts a point-voxel detection framework and constructs a local feature encoding and decoding network (LFED-RCNN) that integrates multiple technologies, which can accurately identify the information of vehicles, pedestrians, and cyclists on the road in the collected point cloud data, and achieve fast and accurate detection effects. Specifically, in the three-dimensional feature extraction stage, a three-dimensional CED backbone combining the convolutional network and the Transformer encoding and decoding structure is proposed, in which the deep downsampling convolutional network (EDSNet) extracts multi-level three-dimensional features, and the local encoding and decoding network (LEDNet) models the correlation between features and fuses deep and shallow features, thereby improving the model's ability to acquire features of foreground targets under complex backgrounds; secondly, a position encoding module is designed to perform position encoding on the two-dimensional features from the BEV perspective, establish long-term dependencies, and improve detection accuracy.

[0026] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0028] Figure 1 This is a schematic diagram of the structure of the autonomous driving 3D object detection model based on local feature encoding and decoding of the present invention;

[0029] Figure 2 Schematic diagram of the deep downsampling convolutional network (EDSNet);

[0030] Figure 3 Schematic diagram of the local feature encoding structure (SA Encoder);

[0031] Figure 4 Schematic diagram of the local feature cross decoding structure (CA Decoder);

[0032] Figure 5 It is the position encoding module in the two-dimensional backbone from the BEV perspective;

[0033] Figure 6 This is a performance comparison chart from the perspective of BEV;

[0034] Figure 7 A comparison chart of the recall rates of various methods. DETAILED DESCRIPTION

[0035] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0036] See also Figures 1 to 7 The present invention provides a 3D target detection method for autonomous driving based on local feature encoding and decoding, and implements a 3D target detection model for autonomous driving based on local feature encoding and decoding. The target detection model includes an input network module, a three-dimensional backbone network module, a two-dimensional backbone network module, and a head network module.

[0037] The three-dimensional backbone network module includes a deep downsampling convolutional network (EDSNet) and a Transformer-based local feature encoding and decoding module (LEDNet).

[0038] The formula for calculating the receptive field size of the deep downsampling convolutional network (EDSNet) is:

[0039]

[0040] Among them, f k-1 is the size of the receptive field corresponding to the k-1th layer, l k is the convolution kernel size of the kth layer, s i is the step size of the i-th layer. From the above formula, it can be seen that the receptive field of the convolutional network increases layer by layer, showing an iterative relationship. Therefore, the receptive field can be expanded to obtain deeper features by increasing the depth of the convolutional network. To this end, the present invention increases the depth of the downsampling convolutional network to obtain higher-level features and fuse multi-level 3D features, such as Figure 2 In the network design, the 16× and 32× downsampling layers produce feature maps of size 100*88*3 and 50*44*3, respectively. The choice of only adding 16 and 32 downsampling layers in the network design was based on several considerations. First, the purpose of adding downsampling layers is to enable efficient feature extraction at higher network levels and increase the receptive field, which helps improve the model's abstraction capabilities and ability to handle complex patterns.

[0041] In the Transformer-based local feature encoding and decoding module (LEDNet), the self-attention encoding module (SAEncoder) is designed based on the multi-head self-attention (MSA) mechanism, such as Figure 3 As shown. The self-attention encoding module (SAEncoder) is composed of the HW window-based (W hw _MSA) and D window based (W d _MSA) and are connected using residuals. The HW window focuses on the relationship between the HW dimensions, such as a tensor of (-1, H*W, C). The D window focuses on the relationship along the D dimension, such as a tensor of (-1, D, C).

[0042] The implementation details are as follows: the input 3D convolutional feature map is divided into small blocks of equal size along the HW dimension and encoded using a sliding window in the HW direction. Then, the output features are reconstructed to restore the shape of the original features; they are divided into small blocks along the D dimension again, encoded using a sliding window in the D direction, and finally the encoded feature map is reconstructed to the original shape and size. At the same time, to prevent overfitting, a dropout layer is added, and then the initial features and the encoded features are spliced ​​using a residual connection. The local feature encoding process can be expressed as follows:

[0043] F l =W d _MSA(W hw _MSA{F S})

[0044] Among them, F S is the input feature, F l is the encoded feature.

[0045] In general, SA Encoder divides the input features into small blocks and encodes the information in the small blocks in sequence along different dimensions, thereby maintaining the continuity of 3D local area features and helping to provide more detailed information for foreground targets.

[0046] In the Transformer-based local feature encoding and decoding module (LEDNet), the cross attention decoding module (CADecoder) is as follows Figure 4 As shown, the input feature map is the encoded multi-scale feature F l With F l First, perform a 3D convolution operation on the input features and reconstruct the multi-scale 3D feature maps into the same size to generate key-value pairs. The specific calculation is shown in formula (3):

[0047] Conv3d(x)=σ(w*x+b)

[0048] Where x is the input feature map size, w is the convolution kernel size, b is the bias term, and σ is the activation function. After obtaining the same two features, we first perform a 1*1*1 convolution to generate the initial query (Q), key (K), and value (V) respectively. The space of Q, K, and V is reconstructed into R HW*D*C , calculate the HW dimension cross attention weight by matrix multiplication. After that, the output feature is fed into the 1*1*1 convolution again to produce a new K and a new V. The initial Q, new K and new V are combined into a new query pair, and the space is reshaped into R D *HW*C , the D-dimensional cross attention weight is calculated by matrix multiplication. Through cross-query, the two input features are combined, which greatly improves the representation ability of the output features.

[0049] The two-dimensional backbone strengthens feature fusion based on the Swin-Transformer self-attention mechanism and outputs the layer to be predicted to the Head network module.

[0050] The complexity calculation formulas for Transformer and Swin-Transformer are as follows:

[0051] Ω T =4hwC 2 +2(hw) 2 C

[0052] Ω ST =4hwC 2 +2(M)2 HkDJ

[0053] In the above formula, the first is the complexity of Transformer, the second is the complexity of Swin-Transformer, hw represents the size of the input feature map, and M is the size of the sliding window. According to the above formula, it can be seen that the complexity of Swin-Transformer is only linearly related to the size of the feature map. When the sequence length is long, it is significantly lower than the quadratic linear relationship of Transformer. The position encoding module (PE) from the perspective of BEV is as follows Figure 5 As shown. The specific implementation details are as follows: the input features are first normalized through a LayerNorm layer and then passed to the WindowAttention layer for self-attention weight calculation. The input tensor is linearly transformed (qkv) to obtain the query, key, and value. The attention weights are then calculated through the softmax function and multiplied by the value to obtain a representation of the self-attention size. After the attention weight size is calculated, the original input and the output of the attention mechanism are added through a residual connection. Finally, the features are processed through the LayerNorm and MLP modules and the residual connection is used again to fuse the features. LayerNorm is a technology that standardizes the input data, which helps accelerate training and improve the generalization ability of the model. After the above operations, position association and rich contextual information are added to the 2D features, which helps improve the quality of 2D features.

[0054] The head network module classifies and regresses the prediction layer to obtain detection information of vehicles, pedestrians, and cyclists on the road.

[0055] Verification experiment:

[0056] 1) Calculation and standards of indicators

[0057] Calculation of IoU:

[0058]

[0059] We mainly examine three categories of difficulty. For cars, the IoU is 0.7, 0.7, 0.7, and for pedestrians and cyclists, the IoU is 0.5, 0.5, 0.5.

[0060] Calculation of precision and recall:

[0061]

[0062] When evaluating proposals in the network, all proposals are first sorted by confidence score. The proposal with the highest score is then compared with the ground-truth bounding box using the intersection-over-union (IoU) ratio. Proposals with a score greater than the threshold are marked as true positives (TP), while those with lower scores are marked as false positives (FP). The precision of the detection results is calculated using the first formula. Undetected foreground objects are marked as false negatives (FN), while true negatives (TN) are not counted. The recall of all detection results is calculated using the second formula.

[0063] Calculation of average precision:

[0064]

[0065] During the evaluation phase, different IoU ratios yield different precision and recall results. As recall increases, precision gradually decreases. Plotting a PR curve with recall on the horizontal axis and precision on the vertical axis, if the horizontal axis calculation uses equally spaced points [0, 0.025, ..., 0.975, 1] ​​and the vertical axis averages the result, the equation above shows.

[0066] 2) Experimental results

[0067] (1) Experimental data and settings

[0068] The experiment used two NVIDIA GeForce RTX 3090 graphics cards and PyTorch version 1.10.0. Distributed training was used with a batch size of 8 and 80 training epochs. Data augmentation was performed using rotation and cropping. The number of channels in the sparse convolutional backbone was 16, 32, 64, 64, 64, and 64, respectively. The point set abstraction radius was (0.4m, 0.8m), and training was performed with a learning rate of 0.01.

[0069] In order to evaluate the present method and verify its feasibility, experiments were conducted on the KITTI dataset and the ONCE dataset following the most advanced detection methods. The KITTI dataset was jointly created by the Karlsruhe Institute of Technology in Germany and Toyota Research Institute of America. It is used to evaluate the performance of computer vision technologies such as stereo, optical flow, visual odometry, 3D object detection, and 3D tracking in a vehicle environment. The dataset covers multiple real scenes such as urban areas, rural areas, and highways. Each image contains up to 15 vehicles and 30 pedestrians, with varying degrees of occlusion and truncation. It is currently one of the most widely used open source datasets in the field of 3D object detection. The dataset covers different weather, lighting, and traffic conditions, provides a variety of training and testing scenarios, and can effectively test the performance of the model in different environments. It contains 3712 training samples and 3769 verification samples. To further demonstrate the generalization of the model, the method of the present invention was also experimented on the ONCE dataset. The ONCE dataset covers a wide variety of driving scenarios, including different time periods (daytime / nighttime), weather conditions (sunny / cloudy / rainy), and locations (urban areas / highways / tunnels). The ONCE dataset contains one million frames of 3D point clouds, 15,000 of which are annotated with detection boxes. This method uses training samples to train the model and validation samples to verify the model's performance.

[0070] (2) Results and analysis

[0071] The results on the KITTI dataset and the comparison with other methods are shown in Table 1, the results on the ONCE dataset are shown in Table 2, and the ablation experiment is shown in Table 3.

[0072] Table 1 Experimental results on the KITTI dataset

[0073]

[0074]

[0075] Table 2 Experimental results on the ONCE dataset

[0076]

[0077] Table 3 Ablation experiments of each module

[0078]

[0079] From Table 1 and Table 2, it can be seen that the method of the present invention has good effects in all categories and has good generalization. From Table 3, it can be concluded that the effectiveness of each module proposed by the method of the present invention is good.

[0080] In summary, to address the issue of low detection accuracy in autonomous driving, this paper proposes a 3D object detection method for autonomous driving based on local feature encoding and decoding. By leveraging the robust feature extraction capabilities of both the 3D and 2D backbones, higher-quality features are obtained and effectively fused, enhancing feature representation and improving detection accuracy. This method is particularly advantageous for difficult-to-detect categories and in complex road conditions.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A 3D object detection method for autonomous driving based on local feature encoding and decoding, characterized in that: The method specifically comprises the following steps: S1: Input the road point cloud data into the Input network module for preprocessing; S2: The pre-processed road point cloud data is subjected to 3D feature extraction through a 3D backbone network module to obtain corresponding 3D features; the 3D backbone network module includes a deep downsampling convolutional network and a Transformer-based local feature encoding and decoding module; the Transformer-based local feature encoding and decoding module includes a self-attention encoding module and a cross-attention decoding module; The self-attention encoding module is designed based on a multi-head self-attention mechanism, consisting of a multi-head self-attention based on an HW window and a multi-head self-attention based on a D window, connected in a residual manner; the HW window focuses on the correlation between HW dimensions, while the D window focuses on the correlation on the D dimension; The implementation details are as follows: the input 3D convolutional feature map is divided into small blocks of equal size along the HW dimension, and encoded in the HW direction using a sliding window; then, the output features are reconstructed to restore the shape of the initial features; along the D dimension, it is divided into small blocks again, and encoded using a sliding window in the D direction, and finally, the encoded feature map is reconstructed to the initial shape and size; at the same time, to prevent overfitting, a Dropout layer is added, and then the initial features and the encoded features are spliced ​​using a residual connection method; the local feature encoding process is expressed as follows: in, is the input feature, is the encoded feature, It is the multi-head attention mechanism of D window, Multi-head attention mechanism for HW window; The cross attention decoding module specifically includes: the input feature map is the encoded multi-scale feature and First, perform a 3D convolution operation on the input features to reconstruct the multi-scale 3D feature maps into the same size to generate key-value pairs. The specific calculation is as follows: in, is a 3D convolution operation, is the input feature map size, is the convolution kernel size, b is the bias term, is the activation function; after obtaining the same two features, first perform 1*1*1 convolution to generate the initial query Q, key K and value V respectively; reconstruct the space of Q, K and V into , where HW represents the size of the HW window, D represents the size of the D window, and C is the number of channels. The HW dimension cross attention weight is calculated by matrix multiplication; after that, the output feature is fed into the 1*1*1 convolution again to produce a new K and a new V; the initial Q, new K and new V are combined into a new query pair, and the space is reshaped into , calculate the D-dimensional cross attention weights through matrix multiplication; S3: Input the 3D features into a 2D backbone network module and project them to the BEV perspective to obtain multi-scale 2D features; the 2D backbone network module includes a Swin-Transformer-based self-attention position encoding module and an enhanced feature fusion module; The specific implementation details of the positional encoding module are as follows: the input features are first normalized by a LayerNorm layer and then passed to the WindowAttention layer for self-attention weight calculation. The input tensor is linearly transformed to obtain the query, key, and value. The attention weights are then calculated using the softmax function and multiplied by the value to obtain the self-attention size. After the attention weights are calculated, the original input and the output of the attention mechanism are added together through a residual connection. Finally, the features are processed through the LayerNorm and MLP modules and the residual connection is used again to fuse the features. S4: The 3D and 2D features are fed into the Head network module, which fuses and obtains the classification and regression results, and outputs the detection information of each category of vehicles, pedestrians, and cyclists.

2. The autonomous driving 3D target detection method according to claim 1, characterized in that: In step S1, the preprocessing is to perform data enhancement on the road point cloud data, specifically: performing random scaling, random cropping, random arrangement and random flipping processing on the road point cloud data.

3. The autonomous driving 3D target detection method according to claim 1, characterized in that: In step S2, the receptive field size calculation formula of the deep downsampling convolutional network is: in, It is k -1 layer corresponds to the size of the receptive field, For the k The convolution kernel size of the layer, For the i The step size of the layer; From the above formula, we can see that the receptive field of the convolutional network increases layer by layer, showing an iterative relationship. Therefore, by increasing the depth of the convolutional network, the receptive field is expanded to obtain deep features and fuse multi-level 3D features.

Images