Surface parameter estimation guidance-based 4D millimeter wave radar and camera fusion three-dimensional target detection method

Abstract

The invention belongs to the technical field of automatic driving, and particularly relates to a 4D millimeter wave radar and camera fusion three-dimensional target detection method based on surface parameter estimation guidance. According to the method, information of an image and point cloud is effectively combined through an explicit target surface depth fitting model, the target surface depth is fitted, then accurate estimation of the three-dimensional space position of the target is achieved, and dense three-dimensional space information is obtained based on surface depth estimation in a subsequent module, so that accurate estimation of the three-dimensional space position of the target is achieved. And visual angle conversion of image features and feature extraction of false point cloud spatial position information are guided, so that the performance of a target detection algorithm is greatly improved.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving technology, and in particular relates to a three-dimensional target detection method based on 4D millimeter-wave radar and camera fusion guided by surface parameter estimation. Background Art

[0002] Three-dimensional object detection, a key component of autonomous driving systems, provides reliable and accurate perception information for subsequent prediction, decision-making, planning, and control modules. The new generation of 4D millimeter-wave radars integrates target elevation information with traditional millimeter-wave radars, while also improving radar resolution and significantly increasing the density of millimeter-wave radar point clouds. Furthermore, their speed measurement capabilities, cost-effectiveness, and all-weather operation make them promising for future applications in object detection technology. To enhance the environmental perception capabilities of object detection algorithms, multimodal fusion technology is currently widely used to complement the strengths of different sensor data. By fusing visual and 4D millimeter-wave radar features to achieve multimodal data interaction enhancement, the accuracy and robustness of object detection algorithms can be improved, providing more comprehensive and stable environmental information for autonomous driving systems.

[0003] At present, mainstream multimodal fusion target detection algorithms usually perform information interaction at the feature level, extract features from each modal data separately in different branches of the model, then project the features of different modalities into a unified space for fusion, and finally determine the output result based on the fusion features. The performance of multimodal target detection algorithms based on 4D millimeter-wave radar and camera fusion is mainly restricted by the following factors: (1) 4D millimeter-wave radar point clouds are sparse and difficult to directly express the surface shape and contour features of the target. The extraction effect of existing backbone networks for sparse point clouds is difficult to meet the requirements of target detection algorithms; (2) Since the image data of multimodal feature fusion does not contain depth information, the depth of image features needs to be estimated during the projection of image features into three-dimensional space. The accuracy of depth estimation will greatly affect the effect of multimodal feature fusion, and thus affect the accuracy of target detection algorithms.

[0004] Therefore, there is an urgent need for a three-dimensional target detection method based on 4D millimeter-wave radar and camera fusion guided by surface parameter estimation, which can enhance the fusion interaction of image semantic features and 4D millimeter-wave radar point cloud features by estimating the target surface depth, thereby improving the accuracy of the multi-source fusion target detection algorithm. Summary of the Invention

[0005] To address the above problems, the present invention provides a three-dimensional target detection method based on 4D millimeter-wave radar and camera fusion guided by surface parameter estimation, which can greatly improve the detection capability of three-dimensional targets.

[0006] The technical solutions for implementing the present invention are as follows:

[0007] A 3D target detection method based on 4D millimeter-wave radar and camera fusion guided by surface parameter estimation. The specific process is as follows:

[0008] Image branch processing: Utilizes the accurate spatial information of the 4D millimeter-wave radar point cloud to guide the conversion of camera image features. Combining the semantic and texture features of the camera image, instance-level target surface depth estimation is achieved based on the target surface depth fitting model, ultimately generating the BEV features corresponding to the image branch.

[0009] Radar branch processing: The radar cross section and velocity information are embedded as additional dimensions into the 4D millimeter-wave radar point cloud features. The point cloud features are extracted using the cylindrical point cloud segmentation method to generate radar point cloud BEV features.

[0010] Surface pseudo point cloud branch processing: Project the dense target surface depth information generated by the target surface depth fitting model into three-dimensional space to generate a target surface depth pseudo point cloud and perform feature extraction to generate a pseudo point cloud BEV feature;

[0011] Feature fusion and target prediction: The BEV features generated by the image branch, radar branch, and surface pseudo point cloud branch are spliced ​​together, and further feature extraction is performed to generate fused BEV features and predict 3D detection results.

[0012] Optionally, the specific process of the image branch processing of the present invention is:

[0013] S11, extracting features from the camera image to obtain effective semantic and texture features in the image;

[0014] S12, aggregating the image features to generate multi-scale features F I ;

[0015] S13. Fit the target depth based on the target surface depth fitting model, merge the multi-scale features, the depth features output by the target surface depth fitting model, and the millimeter-wave radar point cloud projection features, and transfer the merged features from the two-dimensional PV perspective to the three-dimensional space. Finally, compress the height dimension in the three-dimensional space to generate the BEV features corresponding to the image branch.

[0016] Optionally, the specific process of step S13 of the present invention is:

[0017] S131, mask pattern M DGeneration: Using 4D millimeter-wave radar point cloud and image instance segmentation map as data input, the radar point cloud in 3D space is projected onto the 2D image instance segmentation map, and the points falling within the 2D image instance segmentation foreground mask are retained as reference points; the average depth value of all reference points in each mask is used as the depth value of the mask, and this value is used to supplement the depth values ​​corresponding to all pixels in the mask to generate a mask map M with depth. D ;

[0018] S132, Image Features I S Generation: The compressed image multi-scale features F I Further feature extraction is performed through two-dimensional convolution to convert it into feature I suitable for target surface depth fitting. S ;

[0019] S133, Feature F ID Generate: transform feature I S The mask image M with the same size as the one after two-dimensional convolution D Splice and generate feature F with depth information and texture information ID ;

[0020] S134, 2D bounding box generation: For N B The foreground mask of the two-dimensional image instance segmentation of each target is determined according to the maximum and minimum values ​​of each target on the x and y axes in the image coordinate system, which does not distinguish the corresponding category.

[0021] S135, Depth Calculation: Using the 2D bounding box as a guide, the RoI Align module is used to determine the depth of each target in the feature F. ID The corresponding features in the target surface are calculated using a lightweight multi-layer perceptron network with a quadratic surface equation to generate an instance-level target surface depth estimation feature F. D ;

[0022] S136, BEV feature generation: multi-scale features F obtained from the original image I , target surface depth estimation feature F D The three are spliced ​​at the same size, and the perspective conversion of the image features is completed to generate the image BEV features.

[0023] Optionally, the specific process of S135 of the present invention is:

[0024] S1351, use RoI Align module to obtain feature F ID The multi-channel feature data of the bounding box corresponding to each target in the image are resized and compressed;

[0025] S1352: Using the values ​​of the compressed features as coefficients of the quadratic surface equation, and performing supervised training on the calculation process of the coefficients of the quadratic surface equation;

[0026] S1353, N B The quadratic surface equation corresponding to the target is used to predict the depth of all pixel positions inside the target, and the depth of each target corresponding to the instance segmentation mask is supplemented to generate a fine-grained instance-level target surface depth estimation feature F. D .

[0027] Optionally, the present invention performs supervised training on the process of calculating the coefficients of the quadratic surface equation, and the specific process is:

[0028] S13521. For a frame of data, calculate the loss values ​​of all targets contained in the frame of data according to the loss function, and add up all the losses as the target surface depth fitting loss output of the frame;

[0029] S13522. Based on the target surface depth fitting loss under multiple sets of data, the quadratic surface coefficient calculation process involves the training of neural network parameters. Through multiple iterative tuning, the final target surface depth fitting network parameters are obtained when the loss converges.

[0030] Optionally, the loss function of the S13522 neural network training described in the present invention is:

[0031]

[0032] Among them, N B is the number of bounding boxes in the current frame, N R is the number of reference points in each target, is the depth value of the radar point cloud projected onto the target, is the point cloud depth value predicted by the matching equation at the corresponding image position of the projected radar point cloud, is the average depth of the projected radar point cloud, The average of the depths predicted by the matching equation at all pixel locations in the instance mask.

[0033] Optionally, the specific process of S136 of the present invention is:

[0034] S1361, multi-scale features F obtained from the original image I , target surface depth estimation feature F D , 4D millimeter wave radar point cloud projected to the PV perspective point cloud features. These three are spliced ​​at the same size to generate the fusion feature F v ;

[0035] S1362. Use the depth estimation network to predict the fusion feature F v Depth distribution P in three-dimensional space D , by fusion feature F v and depth distribution P D Perform dot multiplication and project the fused features into three-dimensional space;

[0036] S1363. Compress the fused features in the three-dimensional space in the height dimension to generate image BEV features.

[0037] Optionally, the specific process of the radar branch processing of the present invention is as follows:

[0038] S21 discretizes the 3D point cloud of the input 4D millimeter-wave radar from the BEV perspective and divides the point cloud into different cylinders according to the projection coordinates of the point cloud on the 2D plane; and expands the features of each point cloud based on the cylinder features;

[0039] S22, set the number of point clouds in each cylinder N, and use random sampling or zero padding operation to process the number of point clouds in all P non-empty cylinders into N; further extract the feature of the point cloud and change the feature size to (C r ,P,N); perform the maximum pooling operation on the dimension where N is located, and further transform the feature size into (C r ,P);

[0040] S23, project the cylindrical feature to the original position and generate a size of (C r ,H,W) radar point cloud BEV features.

[0041] Optionally, the expansion of the present invention is:

[0042] Based on the position, velocity, reflection intensity information of the 4D millimeter-wave radar point cloud, and the deviation of the point cloud from the geometric center and cluster center in the cylinder, the point cloud features are expanded to D dimensions.

[0043] Optionally, the specific process of the surface pseudo point cloud branch processing of the present invention is as follows:

[0044] S31, the instance-level target surface depth estimation feature F generated in step S1363 D , projecting the calibration parameters between the camera and the 4D millimeter-wave radar into the three-dimensional space to obtain a dense pseudo-point cloud of the target surface depth in the three-dimensional space;

[0045] S32, extract the feature of the target surface depth pseudo point cloud and generate a size of (C p ,H,W) pseudo point cloud BEV features.

[0046] Optionally, the specific process of feature fusion and target prediction in the present invention is:

[0047] S41, splicing the generated image BEV features, radar point cloud BEV features and pseudo point cloud BEV features, and performing fusion feature extraction;

[0048] S42. Use a general anchor-based detection head to predict a three-dimensional detection frame to obtain the target's position, size, orientation, and category information.

[0049] Beneficial effects:

[0050] This target detection method effectively combines image and point cloud information through an explicit target surface depth fitting model to fit the target surface depth, thereby accurately estimating the target's three-dimensional spatial position. In subsequent modules, the dense three-dimensional spatial information obtained based on the surface depth estimation guides the perspective conversion of image features and the feature extraction of pseudo-point cloud spatial position information, thereby significantly improving the performance of the target detection algorithm. Tests on public datasets have shown that this method achieves improved accuracy and robustness compared to existing multimodal detection methods, effectively promoting subsequent research in the field of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 This is a diagram of the overall architecture of the method proposed in the present invention.

[0053] Figure 2 A flow chart of the target surface depth fitting model framework is proposed for the present invention.

[0054] Figure 3 This is the visualization detection result of the present invention on the public dataset VOD. Each scene contains the visualization results of the image, 3D point cloud, and BEV point cloud. The light-colored boxes in the 3D point cloud and BEV point cloud visualization results are the true values, the dark-colored boxes are the predicted bounding boxes of the method of the present invention, and the direction pointed by the triangle is the vehicle's forward direction.

[0055] Figure 4 This is the visualization detection result of the present invention on the public dataset TJ4DRadSet. Each scene contains the visualization results of the image, 3D point cloud, and BEV point cloud. The light-colored boxes in the 3D point cloud and BEV point cloud visualization results are the true values, the dark-colored boxes are the predicted bounding boxes by the method of the present invention, and the direction pointed by the triangle is the vehicle's forward direction. DETAILED DESCRIPTION

[0056] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0057] It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments may be combined with each other; and, based on the embodiments in this disclosure, all other embodiments obtained by persons of ordinary skill in the art without creative work are within the scope of protection of this disclosure.

[0058] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0059] The design concept of the present invention is: using 4D millimeter-wave radar point cloud and optical camera image as input, the predicted depth of each foreground target is determined by an explicit target surface depth fitting model, and fine-grained target depth information is obtained. The depth information is used to enhance the image feature perspective conversion and 4D millimeter-wave radar point cloud feature extraction process, and finally the 4D millimeter-wave radar features and image features are fused from a bird's-eye view perspective, and a universal two-dimensional backbone network and detection head are used to output the detection results. This method realizes the complementary advantages of the position and velocity information of the point cloud and the semantic color information of the image through the efficient fusion of multimodal data, further improving the accuracy and stability of the target detection algorithm. This multimodal data fusion detection method can provide unmanned driving systems with more comprehensive, reliable and stable environmental information, thereby effectively supporting the continuous development of intelligent driving technology.

[0060] The overall architecture of the 4D millimeter wave radar and camera fusion 3D target detection method based on surface parameter estimation guidance proposed in this paper is as follows: Figure 1 As shown, it mainly consists of four parts:

[0061] (1) Image branch: The image feature perspective conversion is guided by the target surface depth fitting model. Specifically, the accurate spatial information of the 4D millimeter-wave radar point cloud is used to guide the conversion of camera image features. The semantic and texture features of the camera image are combined to achieve instance-level target surface depth estimation based on the target surface depth fitting model, and finally generate the BEV features corresponding to the image branch.

[0062] (2) Radar branch: This branch realizes the extraction of multi-dimensional information features of position and velocity of 4D millimeter-wave radar point cloud. Specifically, radar cross section and velocity information are embedded into 4D millimeter-wave radar point cloud features as additional dimensions. The point cloud features are extracted using the cylindrical point cloud segmentation method to generate radar point cloud BEV features.

[0063] (3) Surface pseudo point cloud branch: 3D spatial feature extraction based on fine-grained depth information of the target surface; specifically, the dense target surface depth information generated by the target surface depth fitting model is projected into 3D space to generate a target surface depth pseudo point cloud and perform feature extraction to generate a pseudo point cloud BEV feature;

[0064] (4) Feature fusion and target prediction. Since images and radar point clouds use different coordinate systems, direct spatial alignment of multimodal information is not possible. Therefore, we choose to implement multimodal feature interaction and splicing from a bird's-eye view (BEV) perspective. Finally, the generated fused BEV features can be effectively regressed into a 3D bounding box with category attributes using a universal detection head.

[0065] Since the images captured by the monocular camera in the image branch do not contain pixel depth information, accurate depth estimation is required when converting image features from perspective view (PV) to BEV view. To address this problem, the present invention fully utilizes the accurate spatial information of the 4D millimeter-wave radar point cloud to guide the conversion of image features. Taking into account the sparsity of the point cloud, the present invention combines the semantic information of the image to achieve instance-level target surface depth estimation, thereby improving the accuracy of depth estimation of image features in the process of increasing spatial geometric dimensions.

[0066] In the radar branch, 4D millimeter-wave radar can provide not only the three-dimensional spatial information of point clouds, but also the radar cross section (RCS) and Doppler velocity corresponding to the point clouds. In order to make full use of the multi-dimensional information of radar point clouds, the present invention processes point clouds with reference to the idea of ​​Point Pillars, and embeds radar cross section and velocity information as additional dimensions into point cloud features to enhance the recognition ability of point clouds under different target types and motion states, thereby improving the detection performance of the radar branch.

[0067] In the surface pseudo point cloud branch, the present invention projects the dense target surface depth information generated by the target surface depth fitting model into three-dimensional space to generate a target surface depth pseudo point cloud, and extracts its features with reference to the point cloud data processing method in the radar branch, thereby compensating for the shortcomings of the 4D millimeter-wave radar in point cloud density.

[0068] Because in the feature fusion and detection result output module, the present invention splices the BEV features generated by the image branch, radar branch and surface pseudo point cloud branch in the same dimensional features, and adopts a universal two-dimensional backbone network for feature extraction to generate fused BEV features, and then uses a universal detection head to effectively regress the fused BEV features into a three-dimensional bounding box with category attributes to obtain the final target detection result.

[0069] The present application embodiment provides a 3D target detection method based on 4D millimeter wave radar and camera fusion guided by surface parameter estimation. The specific process is as follows:

[0070] S1 image branch processing: extracts semantic and texture features from the camera image, projects the features from the two-dimensional plane to the three-dimensional space, and then compresses them in the height dimension to generate the bird's-eye view (BEV) features corresponding to the camera image.

[0071] The specific process of this step is:

[0072] S11. Use the Swin-T network as the backbone network to extract image features and obtain effective semantic and texture features in the image;

[0073] S12, using FPN as the neck network to aggregate the image features obtained by the backbone network, generating a multi-scale feature map, and reducing the feature size from the original 256*704 to 1 / 8 of its size;

[0074] S13. This step uses the accurate spatial information of the 4D millimeter-wave radar point cloud to guide the conversion of camera image features. Specifically, the target depth is fitted based on the target surface depth fitting model to output depth features, the multi-scale features generated by the neck network, the depth features output by the target surface depth fitting model, and the millimeter-wave radar point cloud projection features are merged, the merged features are transferred from the two-dimensional PV perspective to the three-dimensional space, and the height dimension is compressed in the three-dimensional space to finally generate the BEV features corresponding to the image branch.

[0075] The framework flow chart of perspective transformation is as follows Figure 1 The image branches are shown.

[0076] like Figure 2 As shown in the figure, the target surface depth fitting model framework flow chart, the specific process is:

[0077] S131, perform data preprocessing: Given a 4D millimeter wave radar point cloud P = (x i ,y i ,z i ) NPAnd the image instance segmentation map (the image instance segmentation map is obtained by processing the original camera image using a general image instance segmentation algorithm) as input, the radar point cloud in the three-dimensional space is projected onto the two-dimensional image instance segmentation map, and the points that fall within the two-dimensional image instance segmentation foreground mask (the foreground mask is the result of the image instance segmentation map, indicating whether a pixel is an object and what kind of object it is) are retained. These points are retained as reference points to provide depth, and they are represented as where u i and v i is the pixel coordinate, d i It is the true depth;

[0078] The average depth value of all reference points in each mask is used as the depth value of the mask, and the depth value corresponding to all pixels in the mask is supplemented by the depth value. The supplementation is to directly assign the obtained depth value to all pixels in the mask to generate a mask image M with depth. D , thereby introducing the depth information of the point cloud into the image features;

[0079] S132, further extract the multi-scale features of the image after the backbone network and the neck network model in the image branch through a two-dimensional convolution module (Conv+BN+ReLU), and convert it into a feature I suitable for target surface depth fitting S ;

[0080] S133, feature I S The mask image M with the same size as the one after two-dimensional convolution D Splice and generate a feature map F with depth information and texture information ID , and use it as the input for subsequent target surface depth fitting;

[0081] S134、For N B The foreground mask of the two-dimensional image instance segmentation of the target instance is used to determine the corresponding category-indistinguishable two-dimensional bounding box according to the maximum and minimum values ​​of each target on the x and y axes in the image coordinate system. Used for feature positioning in the target surface depth fitting process;

[0082] S135, use a neural network to estimate the surface parameters of each target to obtain more accurate depth estimation features. Specifically: use the 2D bounding box as a guide, and use the RoI Align module to determine the depth of each target in the feature F. ID The corresponding features in the target surface are calculated using a lightweight multi-layer perceptron network with a quadratic surface equation to generate an instance-level target surface depth estimation feature F. D The specific process of this step is as follows:

[0083] S1351, use RoI Align module to obtain feature map F ID The multi-channel feature data of the two-dimensional bounding box corresponding to each target is converted to a uniform size of 192×7×7; the two-dimensional plane features are converted to one-dimensional features through a flattening operation, and the data size of the one-dimensional features is changed from 1×9408 to 1×6 through a multi-layer perceptron network;

[0084] S1352. Use the values ​​of the 1×6 features obtained by the multi-layer perception network as the coefficients of the quadratic surface equation to complete the acquisition of the parameters of the surface depth fitting equation.

[0085] The input is the coordinates of all pixels in the instance mask in the image (u s ,v s ), the output is the depth value d of all corresponding pixels in space s , the matching equation adopts the quadratic surface equation in three-dimensional space, and its specific form is as follows:

[0086] d s =a c ·u 2 +b c ·v 2 +c c ·u·v+d c u+e c ·v+f c (1)

[0087] Among them, a c ,b c ,c c ,d c ,e c ,f c are the coefficients of the matching equation;

[0088] The quadratic surface coefficient calculation process in this part is supervised and trained to improve the accuracy of the target surface depth fitting. The sparse point cloud depth value and the average point cloud depth value are used as the true value for supervision, and the matching equation is used to calculate the corresponding predicted depth of all pixel positions inside the mask.

[0089] S13521. For a frame of data, calculate the loss value of all targets contained in the frame of data according to the loss function, and add up all the losses to output as the target surface depth fitting loss of the frame. The calculation formula of the corresponding loss value SFLoss is as follows:

[0090]

[0091] Among them, N B is the number of bounding boxes in the current frame, NR is the number of reference points in each target, is the depth value of the radar point cloud projected onto the target, is the point cloud depth value predicted by the matching equation at the corresponding image position of the projected radar point cloud, is the average depth of the projected radar point cloud, is the average of the depths predicted by the matching equations at all pixel locations in the instance mask;

[0092] S13522. The process of calculating the quadratic surface coefficient based on the target surface depth fitting loss under multiple sets of data involves the training of neural network parameters. Through multiple iterative tuning, the final target surface depth fitting network parameters are obtained when the loss converges.

[0093] S1353, N B The quadratic surface equation corresponding to the target is used to predict the depth of all pixel positions inside the target, complete the depth supplement of each target corresponding instance segmentation mask, and generate a fine-grained instance-level target surface depth estimation feature F D .

[0094] S136, multi-scale features F obtained from the original image I , target surface depth estimation feature F D , 4D millimeter wave radar point cloud projected to the PV perspective point cloud features, these three are spliced ​​at the same size, and the perspective conversion of the image features is completed to generate a size of (C i ,H,W) image BEV features. The specific process of this step is as follows:

[0095] S1361, multi-scale features F obtained from the original image I , target surface depth estimation feature F D , 4D millimeter wave radar point cloud projected to the PV perspective point cloud features. These three are spliced ​​at the same size to generate the fusion feature F v ;

[0096] S1362. Use the depth estimation network to predict the fusion feature F v Depth distribution P in three-dimensional space D , by fusion feature F v and depth distribution P D Perform dot multiplication and project the fused features into three-dimensional space;

[0097] S1363, compress the fusion features in the three-dimensional space in the height dimension to generate a size of (C i ,H,W) image BEV features.

[0098] S2, radar point cloud branch processing: using 4D millimeter wave radar point cloud as input, the point cloud features are extracted using the cylindrical point cloud segmentation method. The specific process of this step is as follows:

[0099] S21. Discretize the 3D point cloud of the input 4D millimeter-wave radar from the BEV perspective, and divide the point cloud into different cylinders according to the projection coordinates of the point cloud on the 2D plane;

[0100] The features of each point cloud are expanded according to the cylindrical features, as shown in formula (3):

[0101]

[0102] The parameters in the above formula include the spatial coordinates [x, y, z] in the radar coordinate system, the intensity information s, the velocity information v, and the target absolute radial velocity v after compensating the vehicle's own velocity provided by VoD. comp , the radial distance r from the radar center, and the longitudinal and lateral angles [α, β] of radar detection. The footnote c represents the distance from the arithmetic mean of all points in the cylinder, and the footnote p represents the deviation from the geometric center (x, y) of the cylinder.

[0103] By combining the position, velocity, and reflection intensity information of the 4D millimeter-wave radar point cloud, as well as the deviation of the point cloud from the cluster center and the geometric center in the cylinder, the point cloud features are expanded to D dimensions (D is 11 in the VoD dataset and 13 in the TJ4D dataset);

[0104] S22. Set the number of point clouds in each cylinder N, and use random sampling or zero padding to process the number of point clouds in all P non-empty cylinders into N, so as to obtain a dense tensor of size (D, P, N); further extract the point cloud features, and use a lightweight network containing linear layers, normalization and activation functions to reduce the feature size to (C r ,P,N); perform the maximum pooling operation on the dimension where N is located, and further transform the feature size into (C r ,P);

[0105] S23, finally project the cylindrical feature to the original position and generate a size of (C r ,H,W) radar point cloud BEV features.

[0106] S3. Surface pseudo point cloud branch processing:

[0107] Based on the transfer matrix of the 4D millimeter-wave radar and camera, the target depth estimated by the target surface depth fitting model is projected into the three-dimensional space of the radar coordinate system to generate a dense pseudo point cloud. The same point cloud processing method as the radar branch is used to extract features from the dense pseudo point cloud and generate pseudo point cloud features from the BEV perspective to complement the sparse point cloud features in the original 4D millimeter-wave radar branch. The process of this step is as follows:

[0108] S31, the instance-level target surface depth estimation feature F generated in step S1365 D , projecting the calibration parameters between the camera and the 4D millimeter-wave radar into the three-dimensional space to obtain a dense pseudo-point cloud of the target surface depth in the three-dimensional space;

[0109] S32, using the same radar point cloud processing method in step S2 to extract features from the target surface depth pseudo point cloud, generating a point cloud with a size of (C p ,H,W) pseudo point cloud BEV features;

[0110] S4. Fusion and prediction

[0111] The BEV features generated by the image branch, radar branch and surface pseudo point cloud branch are spliced, and further feature extraction is performed to generate fused BEV features and predict 3D detection results.

[0112] S41, splicing the image BEV features, radar point cloud BEV features and pseudo point cloud BEV features generated in steps S1, S2 and S3, and using a general two-dimensional backbone network and neck network to perform fusion feature extraction;

[0113] S42. Use a general anchor-based detection head to predict a three-dimensional detection frame to obtain the target's position, size, orientation, and category information.

[0114] This method uses a 4D millimeter-wave radar point cloud and an optical camera image as input. It determines the predicted depth of each foreground target through an explicit target surface depth fitting model, obtaining fine-grained target depth information. This depth information is then used to enhance the image feature perspective conversion and 4D millimeter-wave radar point cloud feature extraction process, ultimately fusing the 4D millimeter-wave radar features and image features from a bird's-eye view. Through the efficient fusion of multimodal data, this method achieves complementary advantages between the position and velocity information of the point cloud and the semantic and color information of the image, further improving the accuracy and stability of the target detection algorithm. This multimodal data fusion detection method can provide unmanned driving systems with more comprehensive, reliable, and stable environmental information, effectively supporting the continued development of intelligent driving technology.

[0115] like Figure 3-4As shown in the figure, after testing on public datasets, this method has achieved improved accuracy and robustness compared to existing multimodal detection methods. The detection results are visualized as follows Figure 3-Figure 4 .

[0116] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A 3D target detection method based on 4D millimeter-wave radar and camera fusion guided by surface parameter estimation, characterized in that: The specific process is: Image branch processing: Utilizes the accurate spatial information of the 4D millimeter-wave radar point cloud to guide the conversion of camera image features. Combining the semantic and texture features of the camera image, instance-level target surface depth estimation is achieved based on the target surface depth fitting model, ultimately generating the BEV features corresponding to the image branch. Radar branch processing: The radar cross section and velocity information are embedded as additional dimensions into the 4D millimeter-wave radar point cloud features. The point cloud features are extracted using the cylindrical point cloud segmentation method to generate radar point cloud BEV features. Surface pseudo point cloud branch processing: Project the dense target surface depth information generated by the target surface depth fitting model into three-dimensional space to generate a target surface depth pseudo point cloud and perform feature extraction to generate a pseudo point cloud BEV feature; Feature fusion and target prediction: The BEV features generated by the image branch, radar branch, and surface pseudo point cloud branch are spliced ​​together, and further feature extraction is performed to generate fused BEV features and predict 3D detection results.

2. The 3D target detection method based on 4D millimeter-wave radar and camera fusion guided by surface parameter estimation according to claim 1 is characterized in that: The specific process of the image branch processing is as follows: S11, extracting features from the camera image to obtain effective semantic and texture features in the image; S12, aggregating the image features to generate multi-scale features F I ; S13. Fit the target depth based on the target surface depth fitting model, merge the multi-scale features, the depth features output by the target surface depth fitting model, and the millimeter-wave radar point cloud projection features, and transfer the merged features from the two-dimensional PV perspective to the three-dimensional space. Finally, compress the height dimension in the three-dimensional space to generate the BEV features corresponding to the image branch.

3. The 3D target detection method based on 4D millimeter-wave radar and camera fusion guided by surface parameter estimation according to claim 2 is characterized in that: The specific process of step S13 is: S131, mask pattern M D Generation: Using 4D millimeter-wave radar point cloud and image instance segmentation map as data input, the radar point cloud in 3D space is projected onto the 2D image instance segmentation map, and the points falling within the 2D image instance segmentation foreground mask are retained as reference points; the average depth value of all reference points in each mask is used as the depth value of the mask, and this value is used to supplement the depth values ​​corresponding to all pixels in the mask to generate a mask map M with depth. D ; S132, Image Features I S Generation: The compressed image multi-scale features F I Further feature extraction is performed through two-dimensional convolution to convert it into feature I suitable for target surface depth fitting. S ; S133, Feature F ID Generate: Feature I S The mask image M with the same size as the one after two-dimensional convolution D Splice and generate feature F with depth information and texture information ID ; S134, 2D bounding box generation: For N B The foreground mask of the two-dimensional image instance segmentation of each target is determined according to the maximum and minimum values ​​of each target on the x and y axes in the image coordinate system, which does not distinguish the corresponding category. S135, Depth Calculation: Using the 2D bounding box as a guide, the RoI Align module is used to determine the depth of each target in the feature F. ID The corresponding features in the target surface are calculated using a lightweight multi-layer perceptron network with a quadratic surface equation to generate an instance-level target surface depth estimation feature F. D ; S136, BEV feature generation: multi-scale features F obtained from the original image I , target surface depth estimation feature F D The three are spliced ​​at the same size, and the perspective conversion of the image features is completed to generate the image BEV features.

4. The 3D target detection method based on 4D millimeter-wave radar and camera fusion guided by surface parameter estimation according to claim 3 is characterized in that: The specific process of S135 is as follows: S1351, use RoI Align module to obtain feature F ID The multi-channel feature data of the bounding box of each target is unified and compressed; S1352: Using the values ​​of the compressed features as coefficients of the quadratic surface equation, and performing supervised training on the calculation process of the coefficients of the quadratic surface equation; S1353, N B The quadratic surface equation corresponding to the target is used to predict the depth of all pixel positions inside the target, and the depth of each target corresponding to the instance segmentation mask is supplemented to generate a fine-grained instance-level target surface depth estimation feature F. D .

5. The 3D target detection method based on 4D millimeter-wave radar and camera fusion guided by surface parameter estimation according to claim 4 is characterized in that: The supervised training of the coefficient calculation process of the quadratic surface equation is as follows: S13521. For a frame of data, calculate the loss values ​​of all targets contained in the frame of data according to the loss function, and add up all the losses as the target surface depth fitting loss output of the frame; S13522. Based on the target surface depth fitting loss under multiple sets of data, the quadratic surface coefficient calculation process involves the training of neural network parameters. Through multiple iterative tuning, the final target surface depth fitting network parameters are obtained when the loss converges.

6. The 3D target detection method based on 4D millimeter-wave radar and camera fusion guided by surface parameter estimation according to claim 3 is characterized in that: The specific process of S136 is as follows: S1361, multi-scale features F obtained from the original image I , target surface depth estimation feature F D , 4D millimeter wave radar point cloud projected to the PV perspective point cloud features. These three are spliced ​​at the same size to generate the fusion feature F v ; S1362. Use the depth estimation network to predict the fusion feature F v Depth distribution P in three-dimensional space D , by fusion feature F v and depth distribution P D Perform dot multiplication and project the fused features into three-dimensional space; S1363. Compress the fused features in the three-dimensional space in the height dimension to generate image BEV features.

7. The 3D target detection method based on 4D millimeter wave radar and camera fusion guided by surface parameter estimation according to claim 1, characterized in that: The specific process of the radar branch processing is as follows: S21 discretizes the 3D point cloud of the input 4D millimeter-wave radar from the BEV perspective and divides the point cloud into different cylinders according to the projection coordinates of the point cloud on the 2D plane; and expands the features of each point cloud based on the cylinder features; S22, set the number of point clouds in each cylinder N, and use random sampling or zero padding operation to process the number of point clouds in all P non-empty cylinders into N; further extract the feature of the point cloud and change the feature size to (C r ,P,N); perform the maximum pooling operation on the dimension where N is located, and further transform the feature size into (C r ,P); S23, project the cylindrical feature to the original position and generate a size of (C r ,H,W) radar point cloud BEV features.

8. The 3D target detection method based on 4D millimeter wave radar and camera fusion guided by surface parameter estimation according to claim 7, characterized in that: The expansion is: Based on the position, velocity, reflection intensity information of the 4D millimeter-wave radar point cloud, and the deviation of the point cloud from the geometric center and cluster center in the cylinder, the point cloud features are expanded to D dimensions.

9. The 3D target detection method based on 4D millimeter-wave radar and camera fusion guided by surface parameter estimation according to claim 3, characterized in that: The specific process of the surface pseudo point cloud branch processing is as follows: S31, the instance level target surface depth estimation feature F D , projecting the calibration parameters between the camera and the 4D millimeter-wave radar into the three-dimensional space to obtain a dense pseudo-point cloud of the target surface depth in the three-dimensional space; S32. Extract features from the target surface depth pseudo point cloud to generate pseudo point cloud BEV features.

10. The 3D target detection method based on 4D millimeter wave radar and camera fusion guided by surface parameter estimation according to claim 1, characterized in that: The specific process of feature fusion and target prediction is as follows: S41, splicing the generated image BEV features, radar point cloud BEV features and pseudo point cloud BEV features, and performing fusion feature extraction; S42. Use a general anchor-based detection head to predict a three-dimensional detection frame to obtain the target's position, size, orientation, and category information.

Images