High-robustness three-dimensional target detection method based on Leiyu fusion

Abstract

The invention relates to a high-robustness three-dimensional target detection method based on Leiyu fusion, and belongs to the field of target detection. The method comprises the steps that laser point cloud data and image data are collected and preprocessed; extracting features of the preprocessed point cloud data and image data; converting the extracted point cloud features and image features into BEV features, and fusing the BEV features of the two features; and inputting the fusion features into a trained three-dimensional detection network to generate a high-robustness 3D target detection frame. According to the method, the complementary characteristics of two modes of the laser radar and the camera are combined, the synergistic effect of accurate and comprehensive three-dimensional object information provided by the laser radar and rich semantic information provided by the camera is fully played, better interference resistance is achieved on a noisy data set, the robustness of the system in a variable scene is remarkably improved, and the robustness of the system is improved. And it is ensured that the three-dimensional target detection task can still be efficiently and stably completed under the complex conditions of low light, severe weather, shielding and the like.

Description

Technical Field

[0001] The present invention belongs to the field of target detection and relates to a highly robust three-dimensional target detection method based on radar-vision fusion. Background Art

[0002] With the rapid development of autonomous driving technology, the environmental perception technology based on multi-sensor fusion has become an important means to achieve high-precision target detection. The fusion of radar and camera can make full use of the high-precision distance measurement and all-weather working ability of radar, as well as the high-resolution imaging and target classification ability of camera. However, due to the differences in data dimensions, resolutions and signal characteristics between the two sensors, how to efficiently fuse multi-modal data to achieve robust target detection remains a key issue in current research.

[0003] The existing radar-vision fusion target detection methods face multiple challenges. First, environmental interferences such as rain, fog, strong light, etc. will affect the image quality of the camera, while the radar signal may be affected by multi-path effects and occlusion, resulting in unstable detection results. Second, during the multi-modal data alignment and feature extraction process, information loss or noise may be introduced due to the time delay or spatial error of the sensor, thus reducing the detection accuracy and reliability. Therefore, how to improve the robustness of the system in complex environments is crucial.

[0004] To this end, the research and development of highly robust radar-vision fusion target detection technology aims to improve the accuracy and adaptability of target detection by improving the feature extraction and fusion algorithms, as well as optimizing the alignment and processing strategies of sensor data. This technology can not only effectively cope with the differences in sensor characteristics and environmental interferences, but also provide a strong guarantee for the safety and stability of the autonomous driving system in complex scenarios. To solve the above problems, multi-modal fusion technology has been widely applied to 3D target detection tasks. Multi-modal fusion methods generally include two core steps: feature extraction and feature fusion. In the feature extraction stage, three-dimensional spatial information and two-dimensional texture information are respectively extracted from the lidar and the camera. In particular, the camera has significant advantages in extracting semantic information of vehicles (such as type, size, shape, etc.); in the feature fusion stage, different modal features are effectively combined through various strategies (such as feature splicing, weighted fusion and deep fusion based on attention mechanism), so that the system forms a complementarity in spatial information and visual information, enhancing the perception ability of complex scenarios. Through the supplement of semantic information by the camera, the system can achieve more accurate vehicle type recognition and provide more reliable three-dimensional target detection results in complex traffic scenarios. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a highly robust 3D object detection method based on radar-vision fusion. By combining the complementary characteristics of lidar and camera modalities, the robustness of the system in variable scenarios is enhanced, ensuring that the 3D object detection task can be efficiently and stably completed under complex conditions such as low light, bad weather, and occlusion.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A highly robust 3D object detection method based on radar-vision fusion, which includes:

[0008] For the same scene, point cloud data and multi-modal image data of the scene are respectively collected, and the point cloud data and image data are respectively preprocessed; among them, the preprocessing of the point cloud data includes multi-level denoising and dynamic filtering; the preprocessing of the image data includes brightness compensation, deblurring, and anti-interference enhancement.

[0009] The preprocessed point cloud data is subjected to feature extraction through a lidar feature extraction network, and a point cloud BEV feature sequence is obtained based on the extracted point cloud features;

[0010] Multiple first image features of the image data are extracted through a deep neural network; the lidar provides projection depth for the camera; the KD-Tree algorithm is used to obtain the adjacent indices of the projected pixels; the adjacent depth is obtained according to the projection depth and the adjacent indices, and the projection depth and the adjacent depth are input into a double transformation module for encoding to generate a first depth feature; the camera BEV feature sequence is obtained according to the first image features and the first depth features;

[0011] The point cloud BEV feature sequence and the camera BEV feature sequence obtain a first fusion feature through a cross-attention mechanism, and the first fusion feature undergoes a self-attention mechanism to implement global feature operations to obtain a second fusion feature;

[0012] The second fusion feature is input into a detection network for 3D object detection to obtain a 3D object detection box.

[0013] Further, obtaining the point cloud BEV feature sequence based on the extracted point cloud features includes: first, inputting the extracted multiple point cloud features into a multi-layer perceptron network to expand the feature dimension of each point in each point cloud feature; then inputting the point cloud features with expanded dimensions into a max pooling layer to extract the global maximum feature value, obtaining a global feature vector corresponding to the number of the point cloud features, realizing compression of the number of feature points; and then forming a series of Query sets in sequence by each global feature vector, which is used to represent the feature query in the BEV view.

[0014] Further, obtaining the camera BEV feature sequence based on the image feature and the depth feature includes, for each image feature, performing the following processing: concatenating the first image feature and the first depth feature and inputting them into a depth network; in the depth network, processing the concatenation result of the first image feature and the first depth feature through three groups of CBR modules to generate a depth-aware camera feature, and dividing the depth-aware camera feature into a second depth feature F DN and a second image feature F CN ; wherein, the second depth feature F DN is multiplied by the second image feature after softmax operation F CN to obtain a third image feature F with depth information NDC ; combining the pre-generated three-dimensional space coordinates and the third image feature F NDC , and obtaining the camera BEV feature through a pooling operation.

[0015] Among them, combining the pre-generated three-dimensional space coordinates and the third image feature F NDC , and obtaining the camera BEV feature through a pooling operation includes, first, lifting the third image feature F NDC to a three-dimensional feature

[0016]

[0017] In the formula, D pred represents the depth distribution;

[0018] Then, project the three-dimensional feature onto the BEV plane, and aggregate the three-dimensional feature to the BEV feature map through efficient voxel pooling to obtain a highly robust camera BEV feature:

[0019]

[0020] In the formula, x, y, and z are three-dimensional coordinates respectively.

[0021] Further, obtaining the first fusion feature from the point cloud BEV feature sequence and the camera BEV feature sequence through a cross-attention mechanism includes:

[0022] The camera BEV feature sequence is linearly transformed through learnable weights, and the linearly transformed feature sequences are respectively denoted as k l and v l ; the point cloud BEV feature sequence is linearly transformed through learnable weights and then position-encoded, and the position-encoded feature is denoted as q l ; calculate the feature in q l and k lThe correlations of the features in [[]] are obtained to get multiple correlation matrices, and then softmax is used to normalize each correlation matrix to obtain normalized weights; each normalized weight is weighted and summed with the feature sequence v l to obtain the first fusion feature by weighted summation.

[0023] Further, the global feature operation of the first fusion feature is realized through the self-attention mechanism to obtain the second fusion feature, including: inputting the first fusion feature into three independent linear transformation layers respectively to generate query, key, and value feature sequences; transposing the key feature sequence, and then performing a dot product operation on the query and the transposed key feature sequence to obtain a similarity matrix; using softmax to normalize the similarity matrix to obtain a normalized weight distribution; and performing weighted summation on the normalized weight distribution and the value feature sequence to obtain the second fusion feature.

[0024] The beneficial effects of the present invention are as follows: By combining the complementary characteristics of lidar and camera modalities, the present invention gives full play to the synergistic effect of the accurate and comprehensive three-dimensional object information provided by lidar and the rich semantic information provided by the camera, has better anti-interference ability on the noisy data set, significantly improves the robustness of the system in variable scenarios, and ensures that the three-dimensional target detection task can be efficiently and stably completed under complex conditions such as low light, bad weather, and occlusion.

[0025] Other advantages, objectives, and features of the present invention will be described to some extent in the subsequent specification, and to some extent, will be obvious to those skilled in the art based on the study of the following text, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be described in detail preferably with reference to the accompanying drawings, where:

[0027] Figure 1 is a structural block diagram of a three-dimensional target detection method proposed in an embodiment of the present invention;

[0028] Figure 2 is a schematic flow chart for generating camera BEV features;

[0029] Figure 3 is a schematic flow chart for generating the first fusion feature and the second fusion feature. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] The following uses specific specific examples to illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0031] As Figure 1 shown, it is a high-robustness three-dimensional target detection method based on radar-vision fusion provided by an embodiment of the present invention. The method specifically includes the following steps:

[0032] Step 1: Collect lidar point cloud data and image data through a lidar and a camera sensor respectively, and perform noise reduction preprocessing on the collected point cloud data to improve the quality and reliability of the point cloud data and enhance the robustness of detection from the source

[0033] 1) Use a lidar sensor to obtain high-precision point cloud data of the surrounding environment, and at the same time capture multi-modal image data of the same scene through a camera to lay a high-quality foundation for subsequent fusion.

[0034] 2) Perform multi-level denoising and dynamic filtering on the lidar point cloud data to cope with environmental noise (such as rain, snow or multipath interference) and ensure the stability and reliability of the point cloud data in complex scenes.

[0035] 3) Perform standardization preprocessing on the image data, including operations such as brightness compensation, deblurring, and anti-interference enhancement, to improve the usability of the image data under extreme conditions such as strong light and weak light.

[0036] Step 2: Extract features from the preprocessed point cloud data through a lidar feature extraction network, and obtain a point cloud BEV feature based on the extracted point cloud features

[0037] 1) Input the optimized point cloud data into a high-robustness lidar feature extraction network to extract multiple point cloud features with three-dimensional spatial relationships and ensure accuracy under occlusion and sparse data conditions.

[0038] 2) Input the extracted multiple point cloud features into a multi-layer perceptron network to enhance the feature dimension of each point. Assume that the original point cloud data contains n points, and the initial dimension of each point is 4. The 4 dimensions are the x, y, and z coordinates of the point in space and the reflectivity value of the point. To enhance the robustness of the point cloud features, the point cloud data can be normalized before input to reduce the influence caused by sensor noise and different data distributions.

[0039] 3) After passing through the multi-layer perceptron, the feature dimension of each point will be expanded from 4 to 512. This dimension elevation operation can make the representation of points more abundant. At the same time, regularization techniques (such as Dropout or Batch Normalization) are used to improve the generalization ability in the network, so as to better retain the detailed information helpful for subsequent detection tasks.

[0040] 4) Then, the point cloud features with elevated dimensions are input into a max pooling layer to compress the number of points n. After compression, each point cloud feature obtains a 1×512-dimensional feature vector q. i The operation of the max pooling layer can extract the global maximum eigenvalue in the point cloud and obtain a global feature vector q with a fixed dimension. i This feature vector condenses the information of all points in the point cloud features and is a more compact representation of the point cloud, which is used to summarize the features of the entire point cloud structure.

[0041] 5) The sequentially processed point cloud features are used as the basis for constructing the BEV input.

[0042] 6) The processed point cloud feature vectors are organized into a series of Query sets q 1 , q 2 , …, q i , which are used to represent feature queries from the BEV perspective. Each Query set contains deep features from the point cloud and can support the model to detect and understand target objects more accurately from the BEV perspective.

[0043] Step 3: As Figure 2 shown, the image features of the image data are extracted through a deep neural network. With the depth information of the lidar points, graph matching and adjacent depths are used to enhance the image features, so that it has better anti-interference ability on the noisy data set.

[0044] 1) The preprocessed image data is input into a deep convolutional neural network (such as ResNet or VGG) to extract multiple image features to adapt to the diversity of target shapes and environmental changes and ensure the stability of detection. Through multi-scale feature processing, feature representations including target types (such as vehicles, pedestrians), shape contours, textures, and edge information are generated to enhance the system's ability to capture details.

[0045] 2) The lidar provides projection depth (projection depth can also be called depth information) for the camera and is defined as where B S is the batch size, and N CLet \(n\) be the number of multi - perspectives, \(H\) and \(W\) be the height and width of the image respectively. Project the lidar onto the camera to map the 3D point cloud onto the image plane, from which the index of the projected pixel can be obtained and is defined as where \(N\) P is the number of points projected onto the pixel, and 2 represents the pixel coordinates \((u, v)\). The conversion formula between the three - dimensional coordinates of the point cloud and the two - dimensional coordinates of the planar image is as follows:

[0046]

[0047] where \(P\) x , \(P\) y , \(P\) z represent the three - dimensional positions of the lidar points, \((u, v)\) represents the corresponding two - dimensional positions, \(Z\) C represents its projected depth on the image plane, \(K\) represents the camera intrinsic depth, \(R\) and \(T\) represent the rotation and translation of the lidar relative to the camera reference frame, and \(h\) represents the scale factor due to down - sampling.

[0048] 3) Use the KD - Tree algorithm to obtain the neighboring indices of the projected pixels, defined as where \(K\) graph represents the number of neighbors for each projected pixel;

[0049] 4) Obtain the neighboring depth \(D_{n}\) according to \(D\) S and Then, input \(D\) and \(D\) S into the dual - transformation module for depth feature encoding, denoted as \(D\) K SK , which is composed of simple components such as convolutional layers, batch normalization, and RELU activation. SK

[0050] 5) For each extracted image feature the following operations are performed:

[0051] ① Concatenate \(F\) Cam and \(D\) SK , and then in the Depth Net, process them sequentially through three groups of CBR modules to generate a new depth - aware camera feature, denoted as \(F\) DC Divide \(F\) along the \(C\) DC dimension into two new features, a new depth feature \(F\) DN , and a new image context feature \(F\) CN . Among them, \(C\) DC =\(C\) C +\(C\) D , \(C\) C is different depth information, \(C\) DFor different image feature parts. Subsequently, F DN After passing through the softmax operation, it is multiplied by F CN to obtain a new image feature with depth information

[0052] ②Finally, use LSS and BEVDepth, combine the pre-generated three-dimensional space coordinates and F NDC , and output highly robust camera BEV features through pooling operations. Specifically, first, the image feature F NDC combined with depth information is promoted to a three-dimensional feature The feature of each pixel is projected into the corresponding depth range in the three-dimensional space:

[0053]

[0054] where D pred is the depth distribution.

[0055] 6) Project the promoted three-dimensional feature onto the BEV plane, and aggregate the three-dimensional feature into the BEV feature map through efficient voxel pooling to obtain highly robust camera BEV features:

[0056]

[0057] Step Four. As shown in Figure 3 , the point cloud BEV feature and the camera BEV feature obtain the first fusion feature through the cross-attention mechanism

[0058] 1) For the input point cloud BEV feature sequence and camera BEV feature sequence, first introduce a set of learnable weights W 1 , W 2 to perform a linear transformation on the input matrix. Then, perform positional encoding on the processed point cloud BEV feature sequence and set it as q l . The linearly transformed camera BEV feature sequence is then set as k l and v l . Then use q i to retrieve the correlation of the feature sequence k j to obtain the correlation matrix:

[0059]

[0060] 2) Introduce a normalization factor during the calculation process to limit the numerical range. The correspondence between the lidar feature and the camera feature can be determined by the maximum position of the correlation degree b i , and then use the softmax operation to normalize it:

[0061] n i = softmax(b i ), n i ∈R H*W*H*W (5)

[0062] 3) The normalized weight n i is weighted and summed with v i to obtain the first deep fusion feature c i . In this step, the query update of the BEV feature is completed, and c i contains the modality information of lidar and camera. This modality-agnostic feature fusion method can adapt to different sensor combinations and can flexibly adjust the number of queries of the cross-attention mechanism and the dimension of the output features according to the resolution of the image features and task requirements.

[0063] Step Five: As Figure 3 shown, the first fusion feature c i is further processed, and global feature operations are implemented through the BEV self-attention mechanism to obtain the global fusion feature, and finally it is input into a general 3D detection head for object detection

[0064] 1) To achieve the interactive operation of global features in the BEV perspective, the input first fusion feature will be processed by the self-attention mechanism; specifically, the original first fusion feature is respectively passed through three independent linear transformation layers to generate the corresponding query, key, and value feature sequences. Each feature sequence is used for querying information, representing the similarity between features, and the weighted synthesis of the final features. This operation enables the first fusion feature to have the ability to interact with global features in the self-attention mechanism.

[0065] 2) Calculate the similarity between the query and the key to determine the importance of each feature to other features. Specifically, the query performs a dot product operation with the transposed key matrix to obtain the similarity matrix between features. Each element in the similarity matrix represents the relationship and correlation between different positions in the first fusion feature sequence.

[0066] 3) Use the softmax operation on this similarity matrix to generate a normalized weight distribution. The normalized weight distribution represents the relative importance of each position to other positions. This step enables the model to focus on more important features while paying attention to global information.

[0067] 4) After the weight normalization is completed, these weights are weighted and summed with the corresponding value features to obtain the feature representation after global fusion. The weighted summation operation in this step can effectively aggregate the information of all BEV feature points, thereby generating the second fusion feature containing global context information.

[0068] The above operation process can be expressed by the following formula:

[0069]

[0070] Among them, q represents the query feature sequence, k represents the key feature sequence, v represents the value feature sequence, represents the normalization factor.

[0071] Deep feature fusion will be performed through the self-attention mechanism, and these enhanced features will be input into the single-shot multi-box detector structure to complete 3D object detection. This method utilizes the columnar point cloud features efficiently extracted by PointPillars and combines the self-attention mechanism to make the feature expression richer and more accurate. Finally, this architecture can improve the recognition and localization accuracy of objects in 3D space and obtain more reliable 3D object detection results.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the purpose and scope of the present technical solution, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A highly robust three-dimensional target detection method based on radar and vision fusion, characterized in that: The method includes, for the same scene, respectively collecting point cloud data and multimodal image data of the scene, and respectively preprocessing the point cloud data and the image data; The preprocessed point cloud data is subjected to feature extraction through the LiDAR feature extraction network, and a point cloud BEV feature sequence is obtained based on the extracted point cloud features; Extract multiple first image features of image data through deep neural network; provide projection depth for camera through laser radar; use KD-Tree algorithm to obtain neighboring index of projection pixel; obtain neighboring depth according to projection depth and neighboring index, input projection depth and neighboring depth into double transformation module for encoding to generate first depth feature; obtain camera BEV feature sequence according to first image feature and first depth feature; The point cloud BEV feature sequence and the camera BEV feature sequence are used to obtain a first fused feature through a cross attention mechanism, and the first fused feature is used to obtain a second fused feature through a self-attention mechanism to implement a global feature operation; The second fused feature is input into the detection network to perform 3D object detection to obtain a 3D object detection frame.

2. The method according to claim 1, characterized in that The method of obtaining a point cloud BEV feature sequence based on the extracted point cloud features includes: first, inputting the extracted multiple point cloud features into a multi-layer perceptron network, and expanding the feature dimension of each point in each point cloud feature; then inputting each point cloud feature after the expanded dimension into a maximum pooling layer to extract the global maximum eigenvalue, and obtaining a global feature vector corresponding to the number of point cloud features, thereby compressing the number of feature points; and then sequentially organizing each global feature vector into a series of Query sets for representing feature queries from a BEV perspective.

3. The method according to claim 1, characterized in that The step of obtaining a camera BEV feature sequence according to the image feature and the depth feature comprises: for each of the first image features, performing the following processing: The first image feature and the first depth feature are spliced ​​and input into a deep network; in the deep network, the splicing result of the first image feature and the first depth feature is processed by three groups of CBR modules to generate a depth perception camera feature, and the depth perception camera feature is divided into a second depth feature F DN and the second image feature F CN ; Wherein, the second depth feature F DN After the softmax operation, it is multiplied by the second image feature F CN , get the third image feature F with depth information NDC ; Combine the pre-generated three-dimensional space coordinates and the third image feature F NDC , the camera BEV features are obtained through pooling operation.

4. The method according to claim 3, characterized in that: The combination of the pre-generated three-dimensional space coordinates and the third image feature F NDC , the camera BEV feature is obtained by pooling operation, firstly, the third image feature F NDC Promoted to 3D features Where D pred represents the depth distribution; Then the three-dimensional features Projected onto the BEV plane, the 3D features are aggregated into the BEV feature map through efficient voxel pooling to obtain highly robust camera BEV features: Where x, y, and z are three-dimensional coordinates.

5. The method according to claim 1, characterized in that: The point cloud BEV feature sequence and the camera BEV feature sequence obtain a first fusion feature through a cross attention mechanism, including: the camera BEV feature sequence is linearly transformed through a learnable weight, and the feature sequences after the linear transformation are respectively recorded as k l and v l The point cloud BEV feature sequence is linearly transformed by the learnable weights and then position-encoded. The feature sequence after position encoding is recorded as q l ; Calculate q l The characteristics and k l The correlation of the features in is obtained to obtain multiple correlation matrices, and then the correlation matrices are normalized using softmax to obtain normalized weights; each of the normalized weights is compared with the feature sequence v l Perform weighted summation to obtain the first fusion feature.

6. The method according to claim 1, characterized in that The first fused feature is subjected to a self-attention mechanism to implement a global feature operation to obtain a second fused feature, including: inputting the first fused feature into three independent linear transformation layers respectively to generate query, key and value feature sequences; transposing the key feature sequence, and then performing a dot product operation on the query and the transposed key feature sequence to obtain a similarity matrix; using softmax to normalize the similarity matrix to obtain a normalized weight distribution; and performing a weighted summation of the normalized weight distribution and the value feature sequence to obtain the second fused feature.

7. The method according to claim 1, characterized in that The preprocessing of the point cloud data includes multi-level denoising and dynamic filtering; the preprocessing of the image data includes brightness compensation, deblurring and anti-interference enhancement.

Images