A 4D millimeter wave radar and camera fusion three-dimensional target detection method based on surface parameter estimation guidance

Abstract

The application belongs to the technical field of automatic driving, and particularly relates to a 4D millimeter wave radar and camera fusion three-dimensional target detection method based on surface parameter estimation guidance. The method effectively combines the information of images and point clouds through an explicit target surface depth fitting model, fits the target surface depth, and then realizes accurate estimation of the three-dimensional space position of the target. Furthermore, based on the dense three-dimensional space information obtained through surface depth estimation, the method guides the perspective conversion of image features and the feature extraction of pseudo-point cloud space position information in the subsequent module, and then realizes a substantial performance improvement of the target detection algorithm.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology, and in particular relates to a 3D target detection method based on surface parameter estimation-guided 4D millimeter-wave radar and camera fusion. Background Technology

[0002] 3D target detection, a crucial component of autonomous driving systems, provides reliable and accurate perception information for subsequent modules such as prediction, decision-making, planning, and control. The new generation of 4D millimeter-wave radar incorporates target elevation information and improves radar resolution, significantly increasing the density of millimeter-wave radar point clouds. Furthermore, its speed measurement capabilities, cost-effectiveness, and all-weather operation make it a promising candidate for future target detection applications. To enhance the environmental perception capabilities of target detection algorithms, multimodal fusion technology is widely employed to leverage the complementary strengths of data from different sensors. By fusing visual and 4D millimeter-wave radar features to enhance multimodal data interaction, the accuracy and robustness of target detection algorithms can be improved, providing more comprehensive and stable environmental information for autonomous driving systems.

[0003] Currently, mainstream multimodal fusion target detection algorithms typically perform information interaction at the feature level, extract features from each modality in different branches of the model, then project the features of different modalities onto a unified space for fusion, and finally determine the output result based on the fused features. The performance of multimodal target detection algorithms based on 4D millimeter-wave radar and camera fusion is mainly constrained by the following factors: (1) 4D millimeter-wave radar point clouds are sparse, making it difficult to directly express the surface shape and contour features of the target. The existing backbone network's extraction effect on sparse point clouds is insufficient to meet the requirements of target detection algorithms; (2) Since the image data of multimodal feature fusion does not contain depth information, it is necessary to estimate the depth of the image features during the projection of image features onto three-dimensional space. The accuracy of depth estimation will greatly affect the effect of multimodal feature fusion, and thus affect the accuracy of the target detection algorithm.

[0004] Therefore, there is an urgent need for a 3D target detection method based on the fusion of 4D millimeter-wave radar and camera guided by surface parameter estimation. This method enhances the fusion interaction between image semantic features and 4D millimeter-wave radar point cloud features by estimating the target surface depth, thereby improving the accuracy of multi-source fusion target detection algorithms. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a 4D millimeter-wave radar and camera fusion method for 3D target detection based on surface parameter estimation guidance, which can significantly improve the detection capability of 3D targets.

[0006] The technical solution for implementing the present invention is as follows:

[0007] A 4D millimeter-wave radar and camera fusion method for 3D target detection based on surface parameter estimation is described below:

[0008] Image branching: The accurate spatial information of 4D millimeter-wave radar point cloud is used to guide the transformation of camera image features. Combined with the semantic and texture features of the camera image, the target surface depth is estimated at the instance level based on the target surface depth fitting model, and finally the BEV features corresponding to the image branch are generated.

[0009] Radar branch processing: Radar cross section and velocity information are embedded as additional dimensions into the 4D millimeter-wave radar point cloud features. The point cloud features are extracted using a cylindrical point cloud division method to generate radar point cloud BEV features.

[0010] Surface pseudo-point cloud branch processing: Project the dense target surface depth information generated by the target surface depth fitting model onto three-dimensional space to generate target surface depth pseudo-point cloud and perform feature extraction to generate pseudo-point cloud BEV features.

[0011] Feature fusion and target prediction: The BEV features generated from the image branch, radar branch and surface pseudo point cloud branch are stitched together, and further feature extraction is used to generate fused BEV features and predict 3D detection results.

[0012] Optionally, the specific process of image branching processing described in this invention is as follows:

[0013] S11. Extract features from the camera image to obtain effective semantic and texture features in the image;

[0014] S12. Aggregate the image features to generate multi-scale features F. I ;

[0015] S13. Fit the target depth based on the target surface depth fitting model, merge the multi-scale features, the depth features output by the target surface depth fitting model and the millimeter-wave radar point cloud projection features, and transfer the merged features from the two-dimensional PV perspective to the three-dimensional space, and compress the height dimension in the three-dimensional space to finally generate the BEV features corresponding to the image branch.

[0016] Optionally, the specific process of step S13 of the present invention is as follows:

[0017] S131, Mask Image M DGeneration: Using 4D millimeter-wave radar point clouds and image instance segmentation maps as input, the radar point clouds in 3D space are projected onto the 2D image instance segmentation map, and points falling within the foreground mask of the 2D image instance segmentation are retained as reference points. The average depth value of all reference points in each mask is used as the depth value of that mask. This value is used to supplement the depth values ​​of all pixels in the mask, generating a mask map M with depth. D ;

[0018] S132, Image Features I S Generation: The multi-scale features F of the compressed image I Further feature extraction is performed using two-dimensional convolution, transforming it into feature I suitable for depth fitting of the target surface. S ;

[0019] S133, Feature F ID Generation: Transfer feature I S A mask image M with the same size after processing with two-dimensional convolution. D The features F are spliced ​​together to generate features with depth and texture information. ID ;

[0020] S134, 2D bounding box generation: For boxes with N... B A foreground mask for segmenting two-dimensional image instances of each target is used to determine the corresponding non-discriminatory 2D bounding box based on the maximum and minimum values ​​of each target on the x and y axes in the image coordinate system.

[0021] S135, Depth Calculation: Guided by 2D bounding boxes, the RoI Align module is used to determine the depth of each target in feature F. ID The corresponding features in the model are used, and a lightweight multilayer perceptron network combined with a quadratic surface equation is employed to calculate the target surface depth, generating instance-level target surface depth estimation features F. D ;

[0022] S136, BEV Feature Generation: This involves generating multi-scale features F from the original image. I Target surface depth estimation feature F D The point cloud features of the 4D millimeter-wave radar point cloud projected to the PV view are stitched together at the same size, and the viewpoint transformation of the image features is completed to generate the BEV features of the image.

[0023] Optionally, the specific process of S135 in this invention is as follows:

[0024] S1351. Obtain feature F using the RoI Align module. ID Multi-channel feature data of the bounding boxes corresponding to each target in the model are obtained and their dimensions are standardized and compressed.

[0025] S1352. The numerical values ​​of the compression features are used as the coefficients of the quadratic surface equation, and the calculation process of the coefficients of the quadratic surface equation is supervised and trained.

[0026] S1353, for N B For each target, the corresponding quadratic surface equation is used to predict the depth corresponding to all pixel positions inside the target. This completes the depth supplementation of the instance segmentation mask for each target, generating fine-grained instance-level target surface depth estimation features F. D .

[0027] Optionally, the process of supervising the calculation of the coefficients of the quadratic surface equation described in this invention is as follows:

[0028] S13521. For a frame of data, calculate the loss value of all targets contained in the frame of data according to the loss function, and add all the losses together as the target surface depth fitting loss output for that frame.

[0029] S13522. Based on the target surface depth fitting loss under multiple sets of data, the process of calculating the quadratic surface coefficients involves training the neural network parameters. Through multiple iterations of optimization, the final target surface depth fitting network parameters are obtained when the loss converges.

[0030] Optionally, the loss function for training the S13522 neural network described in this invention is:

[0031]

[0032] Where, N B N represents the number of bounding boxes in the current frame. R The number of reference points for each target. This represents the depth value of the radar point cloud projected onto the target. The point cloud depth value is predicted using a matching equation at the location in the projected radar point cloud corresponding to the image position. The depth average of the projected radar point cloud. The average depth is predicted by the matching equation at all pixel locations in the instance mask.

[0033] Optionally, the specific process of S136 in this invention is as follows:

[0034] S1361, The multi-scale features F obtained from the original image I Target surface depth estimation feature F D The point cloud features of the 4D millimeter-wave radar point cloud projected onto the PV view are stitched together at the same size to generate the fused feature F. v ;

[0035] S1362. Predicting fusion features F using a deep estimation network. v Depth distribution P in three-dimensional space D By fusing feature F v With depth distribution P D Perform dot product to project the fused features into 3D space;

[0036] S1363. Compress the fused features in three-dimensional space in the height dimension to generate image BEV features.

[0037] Optionally, the specific process of radar branch processing described in this invention is as follows:

[0038] S21, Discretize the three-dimensional point cloud of the input 4D millimeter-wave radar from the BEV perspective, divide the point cloud into different cylinders according to the projection coordinates of the point cloud on the two-dimensional plane, and expand the features of each point cloud according to the cylinder features.

[0039] S22, set the number of point clouds in each cylinder to N, and use random sampling or zero-padding to reduce the number of point clouds in all P non-empty cylinders to N; further extract features from the point clouds, and change the feature size to (C r (P, N); Perform max pooling operation on the dimension containing N, further transforming the feature size into (C, P, N); r ,P);

[0040] S23. Project the cylindrical feature back to its original position to generate a dimension of (C). r BEV features of radar point cloud (H,W).

[0041] Optionally, the extension described in this invention is as follows:

[0042] The point cloud features are expanded to the D dimension based on the position, velocity, reflection intensity information of the 4D millimeter-wave radar point cloud, as well as the deviation of the point cloud from the geometric center and cluster center in the cylinder.

[0043] Optionally, the specific process of surface pseudo-point cloud branching processing described in this invention is as follows:

[0044] S31. The instance-level target surface depth estimation feature F generated in step S1363 is used to... D Based on the calibration parameters between the camera and the 4D millimeter-wave radar, the target surface depth pseudo point cloud in three-dimensional space is projected into three-dimensional space to obtain a dense target surface depth pseudo point cloud in three-dimensional space.

[0045] S32. Extract features from the pseudo-point cloud of the target surface depth, generating a value of (C). p BEV features of pseudo point cloud (H,W).

[0046] Optionally, the specific process of feature fusion and target prediction described in this invention is as follows:

[0047] S41. The generated image BEV features, radar point cloud BEV features, and pseudo point cloud BEV features are stitched together, and fusion feature extraction is performed.

[0048] S42. Use a general anchor-based detection head to predict the 3D detection box and obtain the target's position, size, orientation, and category information.

[0049] Beneficial effects:

[0050] This invention's target detection method effectively combines image and point cloud information through an explicit target surface depth fitting model to fit the target surface depth, thereby achieving accurate estimation of the target's three-dimensional spatial position. Furthermore, in subsequent modules, the dense three-dimensional spatial information obtained from the surface depth estimation guides the viewpoint transformation of image features and the feature extraction of pseudo-point cloud spatial position information, thus significantly improving the performance of the target detection algorithm. Testing on publicly available datasets shows that this method achieves improved accuracy and robustness compared to existing multimodal detection methods, effectively promoting subsequent research in the field of autonomous driving. Attached Figure Description

[0051] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a diagram illustrating the overall architecture of the method proposed in this invention.

[0053] Figure 2 The present invention provides a flowchart of the target surface depth fitting model framework.

[0054] Figure 3 The visualization results of this invention on the public dataset VOD are shown. Each scene includes the visualization results of images, 3D point clouds, and BEV point clouds. In the visualization results of 3D point clouds and BEV point clouds, the light-colored boxes are the ground truth values, the dark-colored boxes are the bounding boxes predicted by the method of this invention, and the direction pointed by the triangle is the direction of vehicle movement.

[0055] Figure 4 The visualization results of this invention are shown on the public dataset TJ4DRadSet. Each scene includes the visualization results of images, 3D point clouds, and BEV point clouds. In the visualization results of 3D point clouds and BEV point clouds, the light-colored boxes are the ground truth values, the dark-colored boxes are the bounding boxes predicted by the method of this invention, and the direction pointed by the triangle is the direction of vehicle movement. Detailed Implementation

[0056] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0057] It should be noted that, in the absence of conflict, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0058] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0059] The design concept of this invention is as follows: using 4D millimeter-wave radar point clouds and optical camera images as input, the predicted depth of each foreground target is determined through an explicit target surface depth fitting model, obtaining fine-grained target depth information. This depth information is then used to enhance the image feature perspective transformation and 4D millimeter-wave radar point cloud feature extraction process. Finally, the 4D millimeter-wave radar features and image features are fused under a bird's-eye view, and the detection results are output using a general two-dimensional backbone network and detection head. This method achieves complementary advantages between the position and velocity information of the point cloud and the semantic and color information of the image through efficient fusion of multimodal data, further improving the accuracy and stability of the target detection algorithm. This multimodal data fusion detection method can provide more comprehensive, reliable, and stable environmental information for autonomous driving systems, thereby effectively supporting the continuous development of intelligent driving technology.

[0060] The overall architecture of the 4D millimeter-wave radar and camera fusion 3D target detection method based on surface parameter estimation proposed in this invention is as follows: Figure 1 As shown, it mainly consists of four parts:

[0061] (1) Image branch: The image feature perspective conversion is guided by the target surface depth fitting model. Specifically, the accurate spatial information of the 4D millimeter-wave radar point cloud is used to guide the conversion of camera image features. Combined with the semantic and texture features of the camera image, the target surface depth is estimated at the instance level based on the target surface depth fitting model, and finally the BEV feature corresponding to the image branch is generated.

[0062] (2) Radar branch: Realize the multi-dimensional information feature extraction of position and velocity of 4D millimeter-wave radar point cloud. Specifically, the radar cross section and velocity information are embedded as additional dimensions into the 4D millimeter-wave radar point cloud features. The point cloud features are extracted by using the columnar point cloud division method to generate radar point cloud BEV features.

[0063] (3) Surface pseudo-point cloud branch: Three-dimensional spatial feature extraction based on fine-grained depth information of target surface; specifically, the dense target surface depth information generated by the target surface depth fitting model is projected onto three-dimensional space to generate target surface depth pseudo-point cloud and perform feature extraction to generate pseudo-point cloud BEV features.

[0064] (4) Feature Fusion and Target Prediction. Since the image and radar point cloud use different coordinate systems, direct spatial alignment of multimodal information is not possible. Therefore, we choose to achieve multimodal feature interaction and stitching from the bird's-eye view (BEV) perspective. Finally, the generated fused BEV features can be effectively regressed into 3D bounding boxes with category attributes using a general detection head.

[0065] Since images acquired by a monocular camera in the image branch do not contain pixel depth information, accurate depth estimation is required when image features are converted from perspective (PV) to perspective (BEV). To address this issue, this invention fully utilizes the accurate spatial information of 4D millimeter-wave radar point clouds to guide the conversion of image features. Considering the sparsity of point clouds, it combines the semantic information of the image to achieve instance-level target surface depth estimation, thereby improving the accuracy of depth estimation in the process of increasing the spatial geometric dimension of image features.

[0066] In the radar branch, 4D millimeter-wave radar can provide not only three-dimensional spatial information of point clouds, but also radar cross section (RCS) and Doppler velocity corresponding to the point clouds. In order to make full use of the multidimensional information of radar point clouds, this invention processes point clouds with reference to the idea of ​​Point Pillars, and embeds radar cross section and velocity information as additional dimensions into the point cloud features to enhance the recognition ability of point clouds under different target types and motion states, thereby improving the detection performance of the radar branch.

[0067] In the surface pseudo-point cloud branch, this invention projects the dense target surface depth information generated by the target surface depth fitting model onto three-dimensional space to generate a target surface depth pseudo-point cloud, and extracts its features by referring to the point cloud data processing method in the radar branch, thereby making up for the defects of 4D millimeter-wave radar in terms of point cloud density.

[0068] In the feature fusion and detection result output module, this invention concatenates the BEV features generated by the image branch, radar branch, and surface pseudo-point cloud branch in the same dimension, and uses a general two-dimensional backbone network for feature extraction to generate fused BEV features. Then, a general detection head is used to effectively regress the fused BEV features into a three-dimensional bounding box with category attributes to obtain the final target detection result.

[0069] This application provides a 4D millimeter-wave radar and camera fusion 3D target detection method based on surface parameter estimation, the specific process of which is as follows:

[0070] S1 Image Branch Processing: Extract semantic and texture features from camera images, project the features from a two-dimensional plane to a three-dimensional space, and compress them in the height dimension to generate the bird's-eye view (BEV) features corresponding to the camera images.

[0071] The specific process for this step is as follows:

[0072] S11. Use the Swin-T network as the backbone network to extract image features and obtain effective semantic and texture features in the image.

[0073] S12. Using FPN as the neck network, the image features obtained by the backbone network are aggregated to generate multi-scale feature maps, reducing the feature size from the original 256*704 to 1 / 8 of its size.

[0074] S13. This step uses the accurate spatial information of 4D millimeter-wave radar point cloud to guide the transformation of camera image features. Specifically, the depth of the target is fitted based on the target surface depth fitting model to output depth features. The multi-scale features generated by the neck network, the depth features output by the target surface depth fitting model, and the projection features of the millimeter-wave radar point cloud are merged. The merged features are transferred from the two-dimensional PV perspective to the three-dimensional space, and the height dimension is compressed in the three-dimensional space to finally generate the BEV features corresponding to the image branch.

[0075] The framework flowchart for perspective transformation is as follows: Figure 1 As shown in the image branch.

[0076] like Figure 2 The flowchart shown below illustrates the framework of the target surface depth fitting model. The specific process is as follows:

[0077] S131. Perform data preprocessing: Given a 4D millimeter-wave radar point cloud P = (x i ,y i ,z i ) NPThe radar point cloud in 3D space is projected onto the 2D image instance segmentation map as input, along with an image instance segmentation map (obtained by processing the original camera image using a general image instance segmentation algorithm). Points falling within the 2D image instance segmentation foreground mask (the foreground mask is a result of the image instance segmentation map, indicating whether a pixel is a target and, if so, what type) are retained. These points are preserved as reference points to provide depth, and they are represented as... Where u i and v i It is a pixel coordinate, d i It is true depth;

[0078] The average depth value of all reference points in each mask is used as the depth value of that mask. This depth value is then used to supplement the depth values ​​of all pixels in the mask. This supplementation involves directly assigning the obtained depth values ​​to all pixels in the mask, generating a mask image M with depth information. D This allows the depth information of the point cloud to be incorporated into the image features;

[0079] S132. The multi-scale features of the image extracted from the image branch by the backbone network and neck network model are further extracted through a two-dimensional convolutional module (Conv+BN+ReLU) to convert them into features suitable for target surface depth fitting. S ;

[0080] S133, Feature I S A mask image M with the same size after processing with two-dimensional convolution. D The features are stitched together to generate a feature map F containing depth and texture information. ID And use it as input for subsequent target surface depth fitting;

[0081] S134, For those with N B A 2D image instance segmentation foreground mask is used for each target instance. The corresponding classless 2D bounding box is determined based on the maximum and minimum values ​​of each target on the x and y axes in the image coordinate system. Used for feature localization in the process of depth fitting of target surface;

[0082] S135. A neural network is used to estimate the surface parameters of each target to obtain more accurate depth estimation features. Specifically: guided by a 2D bounding box, the RoI Align module is used to determine the depth estimation features of each target. ID The corresponding features in the model are used, and a lightweight multilayer perceptron network combined with a quadratic surface equation is employed to calculate the target surface depth, generating instance-level target surface depth estimation features F. D The specific process for this step is as follows:

[0083] S1351. Obtain feature map F using the RoI Align module. ID The multi-channel feature data of the two-dimensional bounding boxes corresponding to each target in the data are converted to a uniform size of 192×7×7. The two-dimensional planar features are converted into one-dimensional features through a planarization operation, and the data size of the one-dimensional features is changed from 1×9408 to 1×6 through a multilayer perceptron network.

[0084] S1352. Use the values ​​of the 1×6 features obtained by the multilayer sensing network as the coefficients of the quadratic surface equation to complete the acquisition of the parameters of the surface depth fitting equation.

[0085] The input is the coordinates (u) of all pixels in the instance mask in the image. s ,v s The output is the depth value d of all corresponding pixels in space. s The matching equation adopts the equation of a quadratic surface in three-dimensional space, and its specific form is as follows:

[0086] d s =a c ·u 2 +b c ·v 2 +c c ·u·v+d c ·u+e c ·v+f c (1)

[0087] Among them, a c ,b c ,c c ,d c ,e c ,f c These are the coefficients of the matching equations;

[0088] The calculation process of the quadratic surface coefficients in this part is supervised to improve the accuracy of the target surface depth fitting. Sparse point cloud depth values ​​and the average point cloud depth are used as ground truth for supervision, and a matching equation is used to calculate the predicted depth corresponding to all pixel positions inside the mask.

[0089] S13521. For a frame of data, calculate the loss value of all targets contained in the frame according to the loss function, and sum all the losses as the target surface depth fitting loss output for that frame. The formula for calculating the loss value SFLouss corresponding to the example is shown below:

[0090]

[0091] Where, N B N represents the number of bounding boxes in the current frame.R The number of reference points for each target. This represents the depth value of the radar point cloud projected onto the target. The point cloud depth value is predicted using a matching equation at the location in the projected radar point cloud corresponding to the image position. This represents the average depth of the projected radar point cloud. The average depth is predicted by matching the equation at all pixel locations in the instance mask;

[0092] S13522. The process of calculating quadratic surface coefficients based on the target surface depth fitting loss under multiple sets of data involves training neural network parameters. Through multiple iterations of optimization, the final target surface depth fitting network parameters are obtained when the loss converges.

[0093] S1353, N B Each target employs its corresponding quadratic surface equation to predict the depth of all pixel locations within the target, completing the depth supplementation of the instance segmentation mask for each target, and generating fine-grained instance-level target surface depth estimation features F. D .

[0094] S136, The multi-scale features F obtained from the original image I Target surface depth estimation features F D The point cloud features of the 4D millimeter-wave radar point cloud projected onto the PV view are stitched together at the same size, and the viewpoint transformation of the image features is completed to generate a size of (C). i The BEV features of the image (H,W). The specific process of this step is as follows:

[0095] S1361, The multi-scale features F obtained from the original image I Target surface depth estimation features F D The point cloud features of the 4D millimeter-wave radar point cloud projected onto the PV view are stitched together at the same size to generate the fused feature F. v ;

[0096] S1362. Predicting fusion features F using a deep estimation network. v Depth distribution P in three-dimensional space D By fusing feature F v With depth distribution P D Perform dot product to project the fused features into 3D space;

[0097] S1363. Compress the fused features in the three-dimensional space in the height dimension to generate a size of (C). i BEV features of H,W) images.

[0098] S2. Radar Point Cloud Branching Processing: Using 4D millimeter-wave radar point clouds as input, the point cloud features are extracted using a cylindrical point cloud segmentation method. The specific process of this step is as follows:

[0099] S21. Discretize the three-dimensional point cloud of the input 4D millimeter-wave radar from the BEV perspective, and divide the point cloud into different cylinders according to the projection coordinates of the point cloud on the two-dimensional plane.

[0100] The features of each point cloud are expanded based on the columnar features, as shown in equation (3):

[0101]

[0102] The parameters in the above formula include the spatial coordinates [x,y,z] in the radar coordinate system, the intensity information s, the velocity information v, and the target absolute radial velocity v after compensation for the vehicle's own velocity provided by VoD. comp , the radial distance r from the radar center, and the longitudinal and lateral angles [α,β] detected by the radar. Where footnote c represents the distance to the arithmetic mean of all points in the cylinder, and footnote p represents the deviation from the geometric center (x,y) of the cylinder;

[0103] By combining the position, velocity, and reflection intensity information of 4D millimeter-wave radar point cloud, as well as the deviation of the point cloud from the cluster center and geometric center in the cylinder, the point cloud features are expanded to the D dimension (D is 11 in the VoD dataset and 13 in the TJ4D dataset).

[0104] S22. Set the number of point clouds N in each cylinder, and use random sampling or zero-padding to reduce the number of point clouds in all P non-empty cylinders to N, to obtain a dense tensor of size (D, P, N); further extract features from the point clouds using a lightweight network containing linear layers, normalization, and activation functions, to change the feature size to (C). r (P, N); Perform max pooling operation on the dimension containing N, further transforming the feature size into (C, P, N); r ,P);

[0105] S23. Finally, project the cylinder feature back to its original position to generate a size of (C). r BEV features of radar point cloud (H,W).

[0106] S3. Surface pseudo-point cloud branching processing:

[0107] Based on the transfer matrix of the 4D millimeter-wave radar and camera, the target depth estimated by the target surface depth fitting model is projected onto a 3D space in the radar coordinate system, generating a dense pseudo-point cloud. The same point cloud processing method as the radar branch is then used to extract features from the dense pseudo-point cloud, generating pseudo-point cloud features from the BEV perspective, thus supplementing the sparse point cloud features in the original 4D millimeter-wave radar branch. The process provided by the bureau for this step is as follows:

[0108] S31. The instance-level target surface depth estimation feature F generated in step S1365 is used to... D Based on the calibration parameters between the camera and the 4D millimeter-wave radar, the target surface depth pseudo point cloud in three-dimensional space is projected into three-dimensional space to obtain a dense target surface depth pseudo point cloud in three-dimensional space.

[0109] S32. Using the same radar point cloud processing method as in step S2, feature extraction is performed on the target surface depth pseudo-point cloud to generate a size (C). p BEV features of pseudo-point clouds (H,W);

[0110] S4, Fusion and Prediction

[0111] The BEV features generated from the image branch, radar branch, and surface pseudo-point cloud branch are stitched together, and further feature extraction is used to generate fused BEV features and predict 3D detection results.

[0112] S41. The image BEV features, radar point cloud BEV features, and pseudo point cloud BEV features generated in steps S1, S2, and S3 are stitched together, and a general two-dimensional backbone network and neck network are used for fusion feature extraction.

[0113] S42. Use a general anchor-based detection head to predict the 3D detection box and obtain the target's position, size, orientation, and category information.

[0114] This method uses 4D millimeter-wave radar point clouds and optical camera images as input. It determines the predicted depth of each foreground target through an explicit target surface depth fitting model, obtaining fine-grained target depth information. This depth information is then used to enhance image feature perspective transformation and 4D millimeter-wave radar point cloud feature extraction. Finally, the 4D millimeter-wave radar features and image features are fused under a bird's-eye view. This method achieves complementary advantages between the position and velocity information of the point cloud and the semantic and color information of the image through efficient fusion of multimodal data, further improving the accuracy and stability of the target detection algorithm. This multimodal data fusion detection method can provide more comprehensive, reliable, and stable environmental information for autonomous driving systems, thereby effectively supporting the continuous development of intelligent driving technology.

[0115] like Figure 3-4As shown, tested on publicly available datasets, this method achieves improved accuracy and robustness compared to existing multimodal detection methods. The detection results are visualized as follows: Figures 3-4 .

[0116] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A 4D millimeter-wave radar and camera fusion three-dimensional target detection method based on surface parameter estimation guidance, characterized in that, The specific process is as follows: Image branching: The accurate spatial information of 4D millimeter-wave radar point cloud is used to guide the transformation of camera image features. Combined with the semantic and texture features of the camera image, the target surface depth is estimated at the instance level based on the target surface depth fitting model, and finally the BEV features corresponding to the image branch are generated. Radar branch processing: Radar cross section and velocity information are embedded as additional dimensions into the 4D millimeter-wave radar point cloud features. The point cloud features are extracted using a cylindrical point cloud division method to generate radar point cloud BEV features. Surface pseudo-point cloud branch processing: Project the dense target surface depth information generated by the target surface depth fitting model onto three-dimensional space to generate target surface depth pseudo-point cloud and perform feature extraction to generate pseudo-point cloud BEV features. Feature fusion and target prediction: The BEV features generated from the image branch, radar branch, and surface pseudo-point cloud branch are concatenated, and further feature extraction is performed to generate fused BEV features and predict 3D detection results; the specific process of the image branch processing is as follows: S11. Extract features from the camera image to obtain effective semantic and texture features in the image; S12. Aggregate the image features to generate multi-scale features. ; S13. Fit the target depth based on the target surface depth fitting model, merge the multi-scale features, the depth features output by the target surface depth fitting model, and the millimeter-wave radar point cloud projection features, and transfer the merged features from the two-dimensional PV perspective to three-dimensional space, and compress the height dimension in three-dimensional space to finally generate the BEV features corresponding to the image branch; the specific process of step S13 is as follows: S131, Mask Image Generation: Using 4D millimeter-wave radar point clouds and image instance segmentation maps as input, the radar point clouds in 3D space are projected onto the 2D image instance segmentation map, and points falling within the foreground mask of the 2D image instance segmentation are retained as reference points. The average depth value of all reference points in each mask is used as the depth value of that mask. This value is used to supplement the depth values ​​of all pixels in the mask, generating a mask map with depth. ; S132, Image Features Generation: Multi-scale features of the compressed image Further feature extraction is performed using two-dimensional convolution, transforming the features into those suitable for depth fitting of the target surface. ; S133, Features Generation: Converting features A mask image with the same size as after processing with two-dimensional convolution. The features are stitched together to generate features with depth and texture information. ; S134, 2D bounding box generation: For those with A foreground mask for segmenting two-dimensional image instances of each target is used, based on the image coordinate system of each target. and The maximum and minimum values ​​on the axis determine the corresponding category-indiscriminate 2D bounding boxes; S135, Depth Calculation: Guided by 2D bounding boxes, the RoI Align module is used to determine the feature depth of each target. The corresponding features in the model are used, and a lightweight multilayer perceptron network combined with a quadratic surface equation is employed to calculate the target surface depth, generating instance-level target surface depth estimation features. ; S136, BEV Feature Generation: This involves generating multi-scale features from the original image. Target surface depth estimation features The point cloud features of the 4D millimeter-wave radar point cloud projected to the PV view are stitched together at the same size, and the viewpoint transformation of the image features is completed to generate the BEV features of the image.

2. The 4D millimeter-wave radar and camera fusion three-dimensional target detection method based on surface parameter estimation guided by claim 1, characterized in that, The specific process of S135 is as follows: S1351. Obtaining features using the RoI Align module. Multi-channel feature data of the bounding boxes corresponding to each target in the model are processed and their dimensions are standardized and compressed. S1352. The numerical values ​​of the compression features are used as the coefficients of the quadratic surface equation, and the calculation process of the coefficients of the quadratic surface equation is supervised and trained. S1353, to Each target employs its corresponding quadratic surface equation to predict the depth corresponding to all pixel positions within the target, thus completing the depth supplementation of the instance segmentation mask for each target and generating fine-grained instance-level target surface depth estimation features. .

3. The 4D millimeter-wave radar and camera fusion three-dimensional target detection method based on surface parameter estimation guided by claim 2, characterized in that, The supervised training process for calculating the coefficients of the quadratic surface equation is as follows: S13521. For a frame of data, calculate the loss value of all targets contained in the frame of data according to the loss function, and add all the losses together as the target surface depth fitting loss output for that frame. S13522. Based on the target surface depth fitting loss under multiple sets of data, the process of calculating the quadratic surface coefficients involves training the neural network parameters. Through multiple iterations of optimization, the final target surface depth fitting network parameters are obtained when the loss converges.

4. The 4D millimeter-wave radar and camera fusion three-dimensional target detection method based on surface parameter estimation guided by claim 1, characterized in that, The specific process of S136 is as follows: S1361. Obtain multi-scale features from the original image Target surface depth estimation features The point cloud features, 4D millimeter-wave radar point cloud projected onto the PV view, and other point cloud features are stitched together at the same size to generate fused features. ; S1362. Predicting fusion features using deep estimation networks. Depth distribution in three-dimensional space By fusing features With depth distribution Perform dot product to project the fused features into 3D space; S1363. Compress the fused features in the three-dimensional space in the height dimension to generate image BEV features.

5. The 4D millimeter-wave radar and camera fusion three-dimensional target detection method based on surface parameter estimation guided by claim 1, characterized in that, The specific process of radar branch processing is as follows: S21, Discretize the three-dimensional point cloud of the input 4D millimeter-wave radar from the BEV perspective, divide the point cloud into different cylinders according to the projection coordinates of the point cloud on the two-dimensional plane, and expand the features of each point cloud according to the cylinder features. S22, Set the number of point clouds in each column. And use random sampling or zero-padding to all The number of point clouds within each non-empty cylinder is processed as follows: One; further feature extraction is performed on the point cloud features, and the feature size is transformed. ;exist Perform max pooling on the dimension in which the feature size is located, further reducing the feature size to... ; S23. Project the cylinder feature back to its original position to generate a dimension of... The radar point cloud BEV characteristics.

6. The 4D millimeter-wave radar and camera fusion three-dimensional target detection method based on surface parameter estimation guided by claim 5, characterized in that, The expansion is as follows: Based on the position, velocity, and reflection intensity information of the point cloud from the 4D millimeter-wave radar, as well as the deviation of the point cloud from the geometric center and cluster center within the cylinder, the point cloud features are expanded to... Dimension.

7. The 4D millimeter-wave radar and camera fusion three-dimensional target detection method based on surface parameter estimation guided by claim 1, characterized in that, The specific process of the surface pseudo-point cloud branching process is as follows: S31. Estimate the surface depth features of the instance-level target. Based on the calibration parameters between the camera and the 4D millimeter-wave radar, the target surface depth pseudo point cloud in three-dimensional space is projected into three-dimensional space to obtain a dense target surface depth pseudo point cloud in three-dimensional space. S32. Extract features from the pseudo-point cloud of the target surface depth to generate pseudo-point cloud BEV features.

8. The 4D millimeter-wave radar and camera fusion three-dimensional target detection method based on surface parameter estimation guided by claim 1, characterized in that, The specific process of feature fusion and target prediction is as follows: S41. The generated image BEV features, radar point cloud BEV features, and pseudo point cloud BEV features are stitched together, and fusion feature extraction is performed. S42. Use a general anchor-based detection head to predict the 3D detection box and obtain the target's position, size, orientation, and category information.

Images