Multi-modal depth fusion 3D target detection method based on reliable depth estimation

Abstract

The invention discloses a multi-modal depth fusion 3D target detection method based on reliable depth estimation, and the method comprises the steps: firstly constructing an image feature extraction network and a millimeter-wave radar point cloud feature extraction network, and respectively extracting image features and millimeter-wave radar point cloud features; secondly, fusing the image and millimeter wave radar point cloud features to obtain depth probability distribution, multiplying the depth probability distribution by the image features to obtain image view cone features, making a depth probability distribution label by using the laser radar point cloud after depth completion, and performing supervision training based on the depth probability distribution; then combining the view cone features of the image and the point cloud features of the millimeter wave radar, and obtaining a feature FA by using a Splat network and a deformable attention mechanism; and finally, based on the feature FA, completing 3D target detection by using a 3D target detection head, and outputting a detection result. According to the method, depth estimation is optimized through the millimeter wave radar, so that the accuracy and robustness of 3D target detection are improved.

Description

Technical Field

[0001] The present invention relates to the field of 3D target detection, and in particular to a multi-modal depth fusion 3D target detection method based on reliable depth estimation. Background Art

[0002] Multimodal 3D object detection technology is a key technology in the field of autonomous driving and robotics. It plays a vital role in key links such as object detection and tracking, vehicle positioning, and path planning. This technology improves the understanding of the surrounding environment and the accuracy and reliability of the autonomous driving system under complex road conditions by integrating data from multiple sensors such as cameras and radars.

[0003] In 3D object detection, the integration of camera and radar data is particularly critical. Some advanced methods, such as RCBEVDet and Camera Radar Net (CRN), achieve excellent performance through innovative techniques for radar feature extraction and bird's-eye view (BEV) point scattering. However, these methods mainly focus on the fusion of BEV level, and less consider the impact of image depth estimation on the overall effect. Although radar can provide accurate distance measurement under any lighting conditions, its data lacks the rich semantic information in the image, which is crucial for distinguishing and identifying objects in complex environments.

[0004] Depth estimation, the conversion from 2D images to 3D scenes, is a core task in multimodal 3D object detection. Monocular depth estimation is widely adopted due to its cost-effectiveness and ease of operation, but it suffers from the problems of being unable to recover the absolute scale of the scene and inaccurate depth estimation under poor lighting or weather conditions. Combining millimeter-wave radar and monocular vision can provide accurate scene distance measurement in low visibility conditions, thereby improving the accuracy of depth estimation.

[0005] We propose a multimodal deep fusion 3D object detection method based on reliable depth estimation, which optimizes the use of radar in improving camera depth estimation. This method can use radar data to enhance the spatial accuracy and semantic richness of deep feature maps, achieving more reliable and refined 3D object detection. At the same time, by adopting techniques such as multimodal deformable criss-cross attention (MDCA), multimodal data is effectively fused to ensure a comprehensive and accurate perception of the environment. Summary of the invention

[0006] In view of the above problems, the present invention proposes a multi-modal depth fusion 3D target detection algorithm method based on reliable depth estimation, which includes the following steps:

[0007] S1: Use the residual network (Residual Net) and feature pyramid network (Feature Pyramid Networks) as the image feature extraction network, and use PointPillars as the millimeter wave radar point cloud feature extraction network to extract image features I and millimeter wave radar point cloud features R respectively: specifically including the following steps:

[0008] (1.1) Extracting image features

[0009] During the extraction process of image feature I, the input image passes through the residual network and the feature pyramid network successively, and the dimension is The image feature I, where Indicates the number of viewing angles of the image, Represents the number of features of the image, and Represent the height and width of the image features respectively.

[0010] (1.2) Extracting millimeter wave radar point cloud features

[0011] During the feature extraction process of millimeter-wave radar point cloud, the millimeter-wave radar point cloud passes through PointPillars and obtains a dimension of The millimeter wave radar point cloud feature R, where is the depth of the millimeter wave radar point cloud feature.

[0012] S2: Depth estimation stage: The image features and millimeter wave radar point cloud features are integrated in the depth feature space to obtain a more accurate depth probability distribution, and then the depth probability distribution is multiplied by the image features to obtain the image cone features. The depth probability distribution label is created using the depth-completed lidar point cloud, and supervised training is performed based on the depth probability distribution label, which specifically includes the following steps:

[0013] (2.1) Deep feature fusion and depth estimation

[0014] For image features and millimeter wave radar point cloud features, depth estimation and height estimation are performed through deep convolutional networks respectively, and their respective depth features are calculated, and the dimensions of image features are converted into Convert to The feature dimension of the millimeter wave radar point cloud Convert to

[0015] F I =Conv(I) (1)

[0016] F R =Conv(R) (2)

[0017] Among them, F IRepresents the deep features of the image, F R represents the deep features of the millimeter-wave radar point cloud, and Conv represents the convolution operation.

[0018] In getting F I and F R After two features are obtained, they are fused in the following way:

[0019] F D =softmax(F I +F R ) (3)

[0020] Among them, F D Represents the scene depth probability distribution, and its dimension is Softmax represents the normalized exponential function.

[0021] The image cone feature calculation process requires the image feature I and the depth probability distribution F D Multiply them together to get the image frustum feature V.

[0022] (2.2) LiDAR point cloud depth completion

[0023] It is proposed to use depth completion algorithm to optimize the lidar depth labels to increase the density of depth radar labels, thereby improving the model's estimation accuracy of depth.

[0024] (2.3) The depth-completed lidar depth labels are cloud reconstructed and filtered to obtain depth probability distribution labels.

[0025] The laser radar point cloud reconstruction process can be regarded as the inverse process of mapping the point cloud to the spherical surface:

[0026]

[0027]

[0028] Through the above formula, the points on the sphere can be mapped to the Cartesian coordinate system to reconstruct the point cloud. (x′, y′, z′) represents the coordinates of the reconstructed point cloud. Compared with the point cloud before depth completion, the point cloud after depth completion is denser and can cover the entire scene.

[0029] In order to enable the model to focus more on the target object, in the later stage of training, point clouds other than the target object will be filtered out to improve the model's depth estimation ability for specific objects. The main idea of ​​the filtering algorithm is to achieve it through the transformation of the coordinate system. Since the 3D target box has information such as the center point coordinates, rotation angle, and box size, it can be regarded as a coordinate system transformation. The coordinate system centered on the lidar is denoted as U, and the coordinate system with the center of the 3D target box as the origin is denoted as V:

[0030] p U =(x,y,z,1) (9)

[0031] t=(t x ,t y ,t z ) (10)

[0032]

[0033] p V =Tp U (12)

[0034] where p U represents the coordinates of the laser radar point cloud before transformation, p V represents the coordinates of the laser radar point cloud with the center of the 3D target box as the origin, t represents the displacement vector from the coordinate system U to V, G is the rotation matrix of the 3D target box, and T is a coordinate system transformation matrix obtained by combining the displacement vector and the rotation matrix; by converting p V The three-dimensional coordinates (x′, y′, z′) are compared with the length, width and height (l, w, h) of the 3D target box. If the following formula is satisfied:

[0035]

[0036] Then it means p V Inside the 3D target box. Through this filtering algorithm, the points inside the 3D target box can be filtered out, thereby obtaining a label with only the depth of the target object.

[0037] S3: The feature fusion stage combines the image cone feature V and the millimeter wave radar point cloud feature R to obtain a dimension of The characteristic is denoted by F VR , using the Splat network to sample F VR Multi-view feature compression Dimension, thus obtaining the bird's-eye view feature Then, the deformable attention mechanism is used to extract features from the bird's-eye view feature B, and the dimension is The characteristic is denoted by F A , used by the 3D object detection head to detect 3D objects. The fusion process of the model is implemented through the following sub-steps:

[0038] (3.1) Calculating bird’s-eye view features

[0039] The image cone feature V and the millimeter-wave radar point cloud feature R are fused by merging:

[0040] F VR =[V; R] (14)

[0041] Among them, F VR Represents the features after the image cone feature V and the millimeter wave radar point cloud feature R are combined.

[0042] The image cone feature V and the millimeter wave radar point cloud feature R are merged, and the Splat network is used to sample the merged features. Then, the sampled features are mapped to the bird's-eye view grid using the camera's internal and external parameters. In this process, if a grid maps multiple features, the summation and pooling method is used to calculate the feature value of the grid. Finally, the dimension is obtained. Bird's-eye view feature B.

[0043] (3.2) Using the deformable attention mechanism to fuse bird’s-eye view features and millimeter-wave radar point cloud features

[0044] The deformable attention mechanism can learn the offsets of each position and adjust the attention weights according to these offsets. In this way, the model can more flexibly focus on important areas in the image while avoiding a lot of calculations of the attention mechanism:

[0045]

[0046] Among them, F A represents the final attention feature; p q Indicates the reference point, Relative to the reference point p q Offset, The bird’s-eye view features and millimeter-wave radar point cloud features of the offset points around the reference point, A mqk represents the attention matrix, W m and W' m They all represent the parameters of the linear layer. Take a reference point p at feature B q , with reference point p q The characteristic values ​​of K offset points around the center sample Calculate the reference point p using the eigenvalues ​​of K offset points q Positional features. Calculate all points of the bird’s-eye view feature B to get the attention feature F. A .

[0047] S4: Based on feature F A , use the 3D object detection head to complete 3D object detection and output the detection results.

[0048] The role of a 3D object detection head in an autonomous driving system is to process the BEV (bird’s eye view) features from various sensors and accurately identify the position, orientation, and size of objects in the environment. Implementing a 3D detection head involves creating algorithms that can interpret sensor data (such as from LiDAR) to build a three-dimensional understanding of the surrounding environment. We use the feature fusion F A Regression prediction of the 3D target box, including the target category, size, center point, steering angle, and speed. Each feature is calculated by a convolutional neural network, which enables us to use the estimated parameters to accurately calculate the 3D target in the scene.

[0049] Beneficial effects of the present invention: The multimodal deep fusion 3D target detection method based on reliable depth estimation improves the accuracy and robustness of 3D target detection through deep feature fusion and label optimization strategies. The mean absolute error (MAE) and root mean square error (RMSE) of depth estimation are reduced by 18.78% and 12.8% respectively, and better robustness is shown under low visibility conditions, with MAE and RMSE reduced by 6.7% and 9.6% respectively. Compared with the baseline model, the present invention improves the 3D target detection accuracy by 0.8%. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a framework diagram of a multi-modal deep fusion 3D object detection algorithm based on reliable depth estimation of the present invention;

[0051] Figure 2 It is a multimodal deep fusion module diagram of the present invention;

[0052] Figure 3 It is the laser radar point cloud coordinate conversion diagram of the present invention;

[0053] Figure 4 This is the result of depth completion and filtering of the LiDAR point cloud;

[0054] Figure 5 It is a diagram of the multimodal deformable attention module of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be described in detail below according to the accompanying drawings and preferred embodiments, and the purpose and effect of the present invention will become more clear. The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0056] Millimeter-wave radar has the ability to provide scene depth information and is highly robust to complex lighting and weather conditions. Using millimeter-wave radar to assist vision in depth estimation can significantly improve the model's depth estimation performance. In this embodiment, a method of fusing visual and millimeter-wave radar depth features is proposed for depth estimation: the camera and millimeter-wave radar are fused in the depth feature space, and the depth information of the millimeter-wave radar point cloud is used to assist the image in estimating the depth probability distribution of the scene; the lidar point cloud is used to make depth labels, and the depth completion algorithm is used to optimize the depth labels, and the depth probability distribution is supervised and trained to improve the accuracy and reliability of depth estimation. In the common lidar point cloud depth completion algorithm, we use an interpolation algorithm to generate depth labels, which generates high-quality depth labels without training, improving the accuracy and robustness of 3D target detection.

[0057] like Figure 1 As shown in the figure, the basic process of the multimodal deep fusion 3D target detection algorithm based on reliable depth estimation is divided into 4 key nodes. 1. Feature extraction: Extract image features and millimeter-wave radar point cloud features from the image and millimeter-wave radar point cloud. 2. Depth estimation: First, the image features are fused with the spatial features of the millimeter-wave radar point cloud to estimate the depth probability distribution. Then, the depth probability distribution is supervised and trained using the lidar depth label optimized by the depth completion algorithm to improve the accuracy of the depth probability distribution. Finally, the image cone features are calculated using the depth probability distribution and image features. 3. Feature fusion: First, the image cone features are merged with the millimeter-wave radar point cloud features, and then the Splat network is used for sampling and calculation to obtain the bird's-eye view features. Finally, the deformable attention module is used to extract features related to the target object from the bird's-eye view features for 3D target detection; 4. 3D target detection: Use the 3D target detection head to detect targets in the scene. The algorithm specifically includes the following steps:

[0058] S1: Extract features of images and millimeter-wave radar point clouds: To ensure model accuracy and improve model inference speed, residual networks and feature pyramid networks are used as image feature extraction networks, and PointPillars is used as millimeter-wave radar point cloud feature extraction network. The specific steps include:

[0059] (1.1) Extracting image features

[0060] In the image feature extraction network, the residual network uses jump connections to solve the gradient vanishing problem, allowing the network to easily train deep networks without losing accuracy, thus having powerful representation capabilities. In the residual network series, the residual network-18 provides a balance between depth, complexity, and performance, allowing it to ensure performance while requiring less computing resources and time for training and reasoning. The feature pyramid network extracts and fuses image features at multiple scales, effectively improving the performance of the model at different scales while increasing the amount of computation to a very small extent.

[0061] During the image feature extraction process, the input image passes through the residual network-18 and the feature pyramid network, and the dimension is The characteristics of Indicates the number of viewing angles of the image, and Represent the height and width of the image features respectively.

[0062] (1.2) Extracting millimeter wave radar point cloud features

[0063] In the millimeter-wave radar point cloud feature extraction network, PointPillars is a 3D target detection algorithm based on radar point cloud, which converts point cloud data into a special two-dimensional representation: point pillars. Then, it is processed using a two-dimensional convolutional neural network. PointPillars uses this method to simplify the processing of point cloud data, improve detection accuracy and speed, and is easier to deploy. In this embodiment, PointPillars consists of three parts: PFN (Pillar FeatureNet), CNN, and Detection Head, where PFN is used to extract point cloud features. Therefore, this embodiment only uses the PFN part as the millimeter-wave radar point cloud feature extraction network.

[0064] During the feature extraction process of millimeter-wave radar point cloud, the millimeter-wave radar point cloud is processed by PFN to obtain a dimension of The millimeter wave radar point cloud feature R, where is the depth of the millimeter wave radar point cloud feature.

[0065] S2: In the depth estimation stage, the image features and the millimeter-wave radar point cloud features are fused in the depth feature space to obtain a more accurate depth probability distribution. The depth probability distribution is then multiplied by the image features to obtain the image cone features. In order to further improve the depth estimation capability of the model, the depth probability distribution label is created using the depth-completed lidar point cloud, and supervised training is performed based on the depth probability distribution label. Specifically, the following steps are included:

[0066] (2.1) Deep feature fusion and depth estimation

[0067] The process of deep feature fusion and depth estimation is as follows Figure 2 As shown in , for two different features: image features and millimeter wave radar point cloud features, depth estimation and height estimation are performed respectively to calculate their respective depth features. Figure 2 As shown, the dimension of image features is A depth estimation network is used to convert it into For millimeter wave radar point cloud features, its feature dimension is Use the height estimation network to convert it into

[0068] F I =Conv(I) (1)

[0069] F R =Conv(R) (2)

[0070] Among them, F I Represents the deep features of the image, F R represents the deep features of the millimeter-wave radar point cloud, and Conv represents the convolution operation.

[0071] In getting F I and F R After two features are obtained, they are fused in the following way:

[0072] F D =softmax(F I +F R ) (3)

[0073] Among them, F D Represents the scene depth probability distribution, and its dimension is Softmax represents the normalized exponential function; since F I and F R Both feature dimensions are In this embodiment, the fusion is performed by adding. I and F R The addition of two deep features can ensure accuracy without increasing the complexity of the model.

[0074] The image cone feature calculation process requires the image feature I depth probability distribution F D Multiply to get the image frustum features:

[0075]

[0076] in Indicates that the feature dimension is The image cone features are and Matrix multiplication.

[0077] (2.2) LiDAR point cloud depth completion

[0078] In obtaining the depth probability distribution F D After that, F can be queried by using the depth tag D Perform supervised training to more accurately estimate the depth probability distribution F of the scene D Therefore, we propose to use the depth completion algorithm to optimize the lidar depth label to increase the density of the depth radar label, thereby improving the model's estimation accuracy of the depth. The depth completion algorithm is performed with the lidar as the origin center. Figure 3 As shown, the process of the depth completion algorithm specifically includes the following steps:

[0079] (2.2.1) To create depth labels using the LiDAR point cloud collected synchronously with the image and millimeter wave radar point cloud, the LiDAR point cloud must first be converted from the Cartesian coordinate system to the polar coordinate system and then mapped onto the spherical surface:

[0080]

[0081] In the formula, (x, y, z) is used to represent the coordinates of the millimeter wave radar point cloud in the Cartesian coordinate system; r represents the distance from the origin in the Cartesian coordinate system, and θ represents the horizontal angle. Represents the pitch angle. The laser radar point cloud is converted from the Cartesian coordinate system to the polar coordinate system through trigonometric conversion. The laser radar takes itself as the origin and rotates one circle for scanning. Therefore, the value range of the point cloud horizontal angle θ is 0° to 360°. The value range is -30°~10°.

[0082] (2.2.2) Convert the polar coordinates of the lidar point cloud into spherical coordinates,

[0083]

[0084] In the formula, s represents the depth map of the laser radar point cloud, and the depth of each direction is stored in a two-dimensional matrix as r. n represents the horizontal division of the sphere into n parts, m represents the vertical division of the sphere into m parts, index i represents the i-th row of the matrix, and index j represents the j-th column of the matrix. The mapping relationship between index i, j and horizontal angle and pitch angle is shown in formula (9) and formula (10). After mapping the polar coordinates to the sphere and storing them in a two-dimensional matrix, the horizontal angle θ and pitch angle can be obtained. Query to get the depth of the corresponding angle (2.2.3) After converting the LiDAR point cloud into a depth map, since the LiDAR point cloud cannot fill the depth map and there are a large number of zero values ​​in the depth map, it is necessary to use a depth completion algorithm to interpolate the depth map s and fill the zero values ​​in the depth map to increase the density of the point cloud.

[0085] Assumptions i,j The depth value of is 0, and the depth completion process is as follows: 1. Get the values ​​of the surrounding circle whose depth is not 0, recorded as x; 2. Calculate the average value of x, recorded as If the depth of the surrounding circle is 0, then 3. i,j Set to 4. From s 1,1 Traverse to s n,m , repeat the above steps 1-3 for a total of n×m times; 5. Repeat the above steps 1-4 until all values ​​in the depth map are not 0.

[0086] The above algorithm can be calculated in parallel through convolution, which can speed up the interpolation process and improve computational efficiency and performance:

[0087]

[0088] In the formula, s' represents the greater than logic operation for each element of s one by one, and k represents a 3×3 convolution kernel. The process of using the convolution kernel for depth estimation is as follows: 1. Use k to convolve s' to calculate the sum of the surrounding elements of each element, which is recorded as s sum ; 2. Use k to convolve s', and you can calculate the number of non-zero depths around each element, denoted as s num ; 3. Assume that the horizontal angle and pitch angle of a point are θ and Its depth value is if If the value of is 0, Assigned to 4. Repeat steps (2.2.1)-(2.2.3) until all values ​​are not 0.

[0089] After the interpolation algorithm, the density of the lidar point cloud can be greatly increased.

[0090] (2.3) The depth labels of the depth-completed LiDAR are reconstructed and filtered. The results are as follows: Figure 4 As shown, the deep probability distribution label is obtained.

[0091] In the later stage of model training, we use a filtering algorithm to filter out depth values ​​outside the 3D target box, so that the model can extract target features more effectively, thereby having higher accuracy and robustness when processing depth information.

[0092] The laser radar point cloud reconstruction process can be regarded as the inverse process of mapping the point cloud to the spherical surface:

[0093]

[0094]

[0095] Through the above formula, the points on the sphere can be mapped to the Cartesian coordinate system to reconstruct the point cloud. (x′, y′, z′) represents the coordinates of the reconstructed point cloud. Compared with the point cloud before depth completion, the point cloud after depth completion is denser and can cover the entire scene.

[0096] In order to enable the model to focus more on the target object, in the later stage of training, this embodiment will filter out point clouds other than the target object to improve the model's depth estimation ability for specific objects. The main idea of ​​the filtering algorithm is to achieve it through the transformation of the coordinate system. Since the 3D target box has information such as the center point coordinates, rotation angle, and box size, it can be regarded as a coordinate system transformation. The coordinate system centered on the lidar is denoted as U, and the coordinate system with the center of the 3D target box as the origin is denoted as V:

[0097] p U =(x,y,z,1) (18)

[0098] t=(t x ,t y ,t z ) (19)

[0099]

[0100] p V =Tp U (twenty one)

[0101] where p U represents the coordinates of the laser radar point cloud before transformation, p V represents the coordinates of the laser radar point cloud with the center of the 3D target box as the origin, t represents the displacement vector from the coordinate system U to V, G is the rotation matrix of the 3D target box, and T is a coordinate system transformation matrix obtained by combining the displacement vector and the rotation matrix; by converting p V The three-dimensional coordinates (x′, y′, z′) are compared with the length, width and height (l, w, h) of the 3D target box. If the following formula is satisfied:

[0102]

[0103] Then it means p V Inside the 3D target box. Through this filtering algorithm, the points inside the 3D target box can be filtered out, thereby obtaining a label with only the depth of the target object.

[0104] S3: The feature fusion stage combines the image cone feature V and the millimeter wave radar point cloud feature R to obtain a dimension of The characteristic is denoted by F VR , using the Splat network to sample F VR Multi-view feature compression Dimension, thus obtaining the bird's-eye view feature Then, the deformable attention mechanism is used to extract features from the bird's-eye view feature B, and the dimension is The characteristic is denoted by F A , used by the 3D object detection head to detect 3D objects. The fusion process of the model is implemented through the following sub-steps:

[0105] (3.1) Calculating bird’s-eye view features

[0106] The image cone feature V and the millimeter wave radar point cloud feature R are fused by merging:

[0107] F VR =[V; R] (23)

[0108] Among them, F VR Represents the features after the image cone feature V and the millimeter wave radar point cloud feature R are combined.

[0109] Bird's-eye view features are a perspective of viewing an object or scene from above, just like a bird looking down at the ground in the air. Converting data into bird's-eye view features can better perform tasks such as target detection and path planning. Figure 1 As shown in the figure, the image cone feature V and the millimeter wave radar point cloud feature R are merged, and the Splat network is used to sample the merged features. Then, the sampled features are mapped to the bird's-eye view grid using the internal and external parameters of the camera. In this process, if a grid maps multiple features, the summation and pooling method will be used to calculate the feature value of the grid. Finally, the dimension is obtained Bird's-eye view feature B.

[0110] (3.2) Using the deformable attention mechanism to fuse bird’s-eye view features and millimeter-wave radar point cloud features

[0111] like Figure 5 As shown in the figure, the deformable attention mechanism can learn the offset of each position and adjust the attention weights according to these offsets. In this way, the model can pay more attention to the important areas in the image more flexibly while avoiding the large amount of calculation of the attention mechanism:

[0112]

[0113] Among them, FA represents the final attention feature; p q Indicates the reference point, Relative to the reference point p q Offset, The bird’s-eye view features and millimeter-wave radar point cloud features of the offset points around the reference point, A mpk represents the attention matrix, W m and W' m They all represent the parameters of the linear layer. Take a reference point p at feature B q , with reference point p q The characteristic values ​​of K offset points around the center sample Calculate the reference point p using the eigenvalues ​​of K offset points q Positional features. Calculate all points of the bird’s-eye view feature B to get the attention feature F. A .

[0114] S4: Based on feature F A ,In the 3D object detection stage, a 3D object detection head is used to complete the 3D object detection task and output the detection results.

[0115] Using the feature fusion F A The category, size, center point, direction and speed of the 3D target box are predicted by regression. Each feature is calculated using convolution, and the 3D target in the scene is calculated using the estimated parameters to complete the 3D target detection. The detection results are shown in Table 1.

[0116] Table 1 3D object detection results

[0117]

[0118]

[0119] The model of the present invention is based on CRN. The depth completion in the table represents the network model after using the depth completion algorithm, and the depth fusion represents the performance of the network model after using the deep fusion network. The present invention represents the network model after using depth completion and depth fusion. The models in the table are all camera and millimeter wave radar fusion models, where NDS (nuScenesDetectionScore) is a comprehensive indicator given by the nuScenes dataset for evaluating model performance, and the calculation method is given by formula (25). mAP (Mean Average Precision; average accuracy of all categories) represents the average accuracy under different thresholds, mATE represents the average translation error, mASE represents the average scale error, mAOE represents the average direction error, mAVE represents the average speed, and mAAE represents the average attribute error.

[0120]

[0121] This embodiment is optimized on the basis of CRN. After optimization by the depth completion algorithm, the NDS is improved by 0.3%; after using the deep fusion network to assist image depth prediction, the NDS is improved by 0.5%; finally, after combining the two optimization methods into the technical solution of the present invention, the NDS is improved by 0.8%.

[0122] The present invention optimizes the depth estimation process in multimodal 3D target detection, proposes a deep feature fusion model and a lidar depth label optimization algorithm, and improves the model's 3D target detection capability. The deep feature fusion model uses the depth information of the millimeter-wave radar point cloud to assist the image in depth estimation, so that the model can more accurately calculate the scene depth, thereby improving the model's scene representation capability. The lidar depth label optimization algorithm can increase the density of the lidar point cloud and obtain a more informative depth label. Compared with the prior art, the ideas of the present invention improve the accuracy and credibility of depth estimation in the process of multimodal 3D target detection.

[0123] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention shall be included in the protection scope of the invention.

Claims

1. A multi-modal deep fusion 3D object detection method based on reliable depth estimation, characterized in that: The following steps are involved: S1: Construct an image feature extraction network and a millimeter-wave radar point cloud feature extraction network to extract image features I and millimeter-wave radar point cloud features R respectively; S2: Fuse the image features and the millimeter-wave radar point cloud features in the depth feature space to obtain the depth probability distribution, multiply the depth probability distribution with the image features to obtain the image frustum features, use the depth-completed lidar point cloud to create the depth probability distribution label, and perform supervised training based on the depth probability distribution; S3: Combine the image cone features and the millimeter wave radar point cloud features, and then use the Splat network and deformable attention mechanism to obtain the feature F A ; S4: Based on feature F A , use the 3D object detection head to complete 3D object detection and output the detection results.

2. The multimodal depth fusion 3D target detection method based on reliable depth estimation according to claim 1, characterized in that: The specific process of extracting image features and millimeter wave radar point cloud features is as follows: S1.1, in the process of extracting image features I, the input image passes through the residual network and the feature pyramid network successively, and the dimension is The image feature I, where Indicates the number of viewing angles of the image, Represents the number of features of the image, and Represent the height and width of the image features respectively; S1.2, In the process of millimeter wave radar point cloud feature extraction, the millimeter wave radar point cloud passes through PointPillars and obtains a dimension of The millimeter wave radar point cloud feature R, where is the depth of the millimeter wave radar point cloud feature.

3. The multimodal depth fusion 3D target detection method based on reliable depth estimation according to claim 2, characterized in that: The specific process of step S2 is as follows: S2.

1. For the image feature I and the millimeter wave radar point cloud feature R, the depth estimation and height estimation are performed through the deep convolutional network respectively, and the respective depth features are calculated. Convert to The feature dimension of the millimeter wave radar point cloud Convert to F I =Conv(I) (1) F R =Con(R) (2) Among them, F I Represents the deep features of the image, F R Represents the depth features of the millimeter wave radar point cloud, and Conv represents the convolution operation; In getting F I and F R After two features are obtained, they are fused in the following way: F D =softmax(F I +F R ) (3) Among them, F D Represents the scene depth probability distribution, and its dimension is Softmax represents the normalized exponential function; Image feature I depth probability distribution F D Multiply them together to get the image cone feature V; S2.2, use the depth completion algorithm to optimize the lidar depth label and increase the density of the depth radar label; S2.3, perform cloud reconstruction and filtering on the depth-completed lidar depth labels to obtain depth probability distribution labels; The laser radar point cloud reconstruction process is regarded as the inverse process of mapping the point cloud to the sphere. The points on the sphere are mapped to the Cartesian coordinate system to re-form the point cloud. (x′, y′, z′) represents the coordinates of the reconstructed point cloud. By transforming the coordinate system, point clouds other than the target object are filtered out. The coordinate system centered on the lidar is recorded as U, and the coordinate system with the center of the 3D target frame as the origin is recorded as V: p U =(x,y,z,1) (4) t=(t x ,t y ,t z ) (5) p V =Tp U (7) where p U represents the coordinates of the laser radar point cloud before transformation, p V represents the coordinates of the laser radar point cloud with the center of the 3D target box as the origin, t represents the displacement vector from the coordinate system U to V, G is the rotation matrix of the 3D target box, and T is a coordinate system transformation matrix obtained by combining the displacement vector and the rotation matrix; by converting p V The three-dimensional coordinates (x′, y′, z′) are compared with the length, width and height (l, w, h) of the 3D target box. If the following formula is satisfied: Then it means p V Inside the 3D target box, filter out the points inside the 3D target box to obtain a label with only the depth of the target object.

4. The multi-modal depth fusion 3D target detection method based on reliable depth estimation according to claim 3, characterized in that: The depth completion algorithm is performed with the laser radar as the origin center.

5. The multi-modal depth fusion 3D object detection method based on reliable depth estimation according to claim 4, characterized in that: The specific implementation process of step S3 is as follows: S3.

1. Fusion of image cone features V and millimeter wave radar point cloud features R by merging: F VR =[V;R] (9) Among them, F VR Represents the features after the image cone feature V and the millimeter wave feature R are combined; Use the Splat network to sample the merged features, and then use the internal and external parameters of the camera to map the sampled features to the bird's-eye view grid, resulting in a dimension of Bird's-eye view feature B; S3.

2. Use deformable attention mechanism to fuse bird's-eye view features and millimeter-wave radar point cloud features: Among them, F A represents the final attention feature; p q Indicates the reference point, Relative to the reference point p q Offset, The bird’s-eye view features and millimeter-wave radar point cloud features of the offset points around the reference point, A mqk represents the attention matrix, W m and W' m Both represent the parameters of the linear layer; take a reference point p at feature B q , with reference point p q The characteristic values ​​of K offset points around the center sample Calculate the reference point p using the eigenvalues ​​of K offset points q Positional features, calculate all points of the bird's-eye view feature B to get the attention feature F A .

6. The multi-modal depth fusion 3D object detection method based on reliable depth estimation according to claim 5, characterized in that: In the process of mapping the sampled features onto the bird's-eye view grid using the internal and external parameters of the camera, if a grid maps multiple features, the feature value of the grid will be calculated by summing and pooling.

Images