Progressive deep fusion multi-modal 3D target detection method

Abstract

The invention relates to a multi-modal 3D target detection method based on progressive deep fusion. Comprising the following steps: acquiring point cloud data and image information, and processing to obtain structured point cloud voxel features and standardized image information; dividing the point cloud into regular cylinder grids, extracting in-column point-level features by using a multi-layer sensor to generate column-level features, mapping the column-level features to a bird's-eye view feature map, generating a three-dimensional anchor frame through a region candidate network to predict a target position and attitude offset, and outputting a spatial ROI (Region of Interest); based on a YOL3D single-stage target detection method, multi-scale features are extracted by adopting a shared backbone and multi-scale fusion processing, multi-head collaborative regression target three-dimensional parameters of depth detection are utilized, and external parameter transformation is performed to generate a spatial 3D-ROI unified with laser radar coordinates; cross-modal geometric alignment is achieved through TAM, feature complementary fusion is completed through RAM, finally, feature cubes are fused, and categories, three-dimensional positions and postures of multiple targets are output. According to the invention, the detection precision of small targets, shielded targets and long-distance targets is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of automatic driving environment perception, and particularly relates to a progressive deep fusion multi-modal 3D target detection method. BACKGROUND

[0002] An automatic driving car is a new intelligent traffic tool integrating perception, decision and control, and realizes autonomous perception and safe driving in a complex road environment by relying on advanced sensor systems, automatic driving algorithms and execution devices. Its core system includes environment perception, trajectory prediction, decision planning and motion control modules, among which the environment perception module as the "visual center" of the automatic driving system undertakes the real-time perception and target identification tasks of the surrounding environment, and provides key basis for decision and control. The accuracy of environment perception directly affects the safety and reliability of the vehicle in a dynamic scene. The current mainstream perception technology can be divided into 2D detection based on visual images, 3D detection based on laser radar point clouds and multi-modal detection based on image and point cloud fusion. The visual sensor performs outstandingly in texture and semantic recognition, but is limited by depth blur and light sensitivity; the laser radar can accurately provide three-dimensional geometric information, but lacks rich semantic features. Therefore, the multi-modal fusion technology based on the complementary characteristics of geometry and semantics of the two becomes an important direction to improve the perception performance of automatic driving.

[0003] In view of the problem that detection accuracy and real-time performance are difficult to be considered at the same time in a complex traffic scene, there are various multi-modal fusion strategies, mainly including three categories of data level fusion, feature level fusion and deep level fusion. The data level fusion directly splices the image and the point cloud in the perception front end, and the structure is simple but is easily affected by calibration error and synchronization delay; the feature level fusion matches and re-scores the independent detection results at the candidate frame level, and has a certain robustness but lacks deep semantic interaction; the deep level fusion method (such as BEVFusion, TransFusion, etc.) realizes high-level feature interaction through cross-modal attention and Transformer structure, and significantly improves the detection accuracy in a complex scene. However, this kind of method usually has large calculation overhead and high deployment complexity, which limits the practical application in the automatic driving system. Therefore, how to reduce the calculation complexity and resource consumption while maintaining high-precision fusion detection has become a key challenge in the current multi-modal three-dimensional target detection research. SUMMARY

[0004] The purpose of the application is to provide a progressive deep fusion multi-modal 3D target detection method, which realizes high-precision and high-real-time three-dimensional target detection in a complex road environment by designing a laser radar feature coding network, an image deep detection module based on a YOLO structure and a cross-modal attention fusion mechanism at a region of interest (ROI) level.

[0005] The technical solution for achieving the objective of this invention is: a progressive deep fusion multimodal 3D target detection method, comprising the following steps:

[0006] Step (1): Obtain point cloud data through LiDAR, acquire image information through binocular camera, perform ROI cropping and spatial normalization on point cloud data to obtain structured point cloud voxel features; perform color enhancement, CLAHE histogram equalization and stereo correction on binocular image information to align the epipolar lines of the left and right eye images to obtain standardized image information.

[0007] Step (2): For structured point cloud voxel features, the point cloud is divided into regular cylindrical grids and the point-level features within the cylinders are extracted using a multilayer perceptron to generate cylindrical features. The cylindrical features are then mapped to the bird's-eye view BEV feature map and a three-dimensional anchor box is generated through the region candidate network RPN to predict the target position and attitude offset, and the spatial ROI is output.

[0008] Step (3): Based on the standardized image information obtained in step (1), multi-scale features are extracted by using the YOLO3D single-stage target detection method with shared backbone and multi-scale fusion processing. The three-dimensional parameters of the target are regressed by multi-head collaborative regression of depth detection. Finally, external parameter transformation is performed to generate a spatial 3D-ROI that is unified with the coordinates of the lidar.

[0009] Step (4): Based on the spatial ROI and spatial 3D-ROI, cross-modal geometric alignment is achieved through TAM, feature complementarity fusion is completed through RAM, and finally the feature cube is fused to output the category, three-dimensional position and pose of multiple targets, thus realizing multimodal 3D target detection.

[0010] Furthermore, step (1) of "performing ROI extraction and spatial normalization on the point cloud data to obtain structured point cloud voxel features" specifically refers to:

[0011] The point cloud data set captured by LiDAR is used to define the Region of Interest (ROI) by setting spatial boundaries, thus obtaining the effective spatial boundary processing range. Based on the effective spatial boundary range, the region of interest of the vehicle is divided into column-level grids, and the position index of each column-level grid falling into the effective ROI is performed to obtain the structured point cloud voxel features; the expression is as follows:

[0012] P = {p i |p i =(x i ,y i ,z i ,I i ),=1,2,…,N}#(1)

[0013] ROI = {p i |=x min ≤xi ≤x max ,y min ≤y i ≤y max ,z min ≤z i ≤z max}#(2)

[0014] ROI=[x min ,x max ]×[y min ,y max ]×[z min ,z max ]#(3)

[0015]

[0016] where x min ,x max is the ROI on the longitudinal space range of the vehicle, y min ,y max is the ROI within the lateral perception range of the vehicle, z min ,z max is the height of the ROI in the vertical direction, and u, v are the pillar unit position indices.

[0017] Further, step (2) is specifically:

[0018] Step (21): Point cloud columnar encoding, MLP feature extraction:

[0019] On the basis of maintaining the columnar voxel expression ability, through point cloud columnar encoding, MLP feature extraction, the original point cloud is mapped to a high-dimensional feature space to obtain a fixed-length columnar feature g (u,v) that is aggregated by pooling, which is expressed as follows:

[0020]

[0021] where is the mean coordinate of all points in the column, μ, σ ∈ R 7 is the statistical mean and variance of the data set, is the intermediate hidden feature obtained by MLP, W, b, and c are the MLP weight matrix, bias term, and channel number, respectively; n (u,v) is the number of points in the column, and g (u,v) is the fixed-length columnar feature of the (u, v) grid aggregated by pooling.

[0022] Step (22): BEV mapping: the fixed-length columnar feature g (u,v)The BEV grid is filled by coordinate mapping and uniformly normalized, and finally a regular BEV feature tensor is formed; represented as follows:

[0023]

[0024] where F BEV is the dense BEV feature tensor, HxW is the discrete resolution of the BEV grid in the x and y directions, and C is the number of column-level feature channels; μ c represents the mean of the features on channel C, and σ c represents the standard deviation of the features on the channel.

[0025] Step (23): Predicting the generation of the spatial ROI:

[0026] Based on the BEV feature tensor obtained in step (22), a region candidate network is constructed, and an anchor box parameterization representation and a sharing mechanism are used to generate the weight parameters required for the potential target ROI in the three-dimensional space; the confidence and offset of the anchor box are predicted through a convolution sliding window to extract the spatial ROI, and the ROI region with the highest score and geometric information is obtained.

[0027] Further, step (23) is specifically as follows:

[0028] Anchor box parameterization representation and sharing mechanism:

[0029] First, the prediction box (x, y, z, l, w, h, θ) is encoded into a normalized offset with the anchor box center (x a ,y a ,z a ) and size (l a ,w a ,h a ) as the reference, realizing scale-invariant 3D detection box regression; at the same time, the RPN network sharing mechanism is used to obtain the classification branch weight parameter W cls and the regression branch weight parameter W reg ; represented as:

[0030]

[0031] where (Δx, Δy, Δz, Δl, Δw, Δh, Δθ) is the dimension offset, is the shared weight, W cls is the classification branch parameter for front / back system binary classification, and W reg is the regression branch parameter for predicting the anchor box offset.

[0032] Spatial ROI extraction:

[0033] The RPN network is used to generate anchor boxes and predict offset, the target candidate boxes are sorted according to the confidence, and the target boxes with high confidence are projected to three-dimensional space to form a spatial ROI, which is represented as:

[0034]

[0035] In the formula, S is a confidence threshold value, B i is a candidate box, t s is a score threshold, IoU(B i ,B j ) is a three-dimensional IoU, v i and v j respectively represent the voxel sets of the two detection boxes in the three-dimensional space, M is the final detection box set after NMS, Δx, Δl, Δθ represent the normalized offset output by the network, R L is the final 3D ROI obtained by the laser radar detection method, is the corresponding sampling window, is a bilinear interpolation weight, and F(x,y) is a channel vector of the feature map at the grid coordinates.

[0036] Further, step (3) is specifically:

[0037] After the shared backbone and multi-scale fusion processing, it is represented as:

[0038]

[0039] P l =Conv 3×3 (F l +Up(P l+1 ))#(19)

[0040] Where F is a set of original feature tensors output by different scales, P l is a multi-scale feature map fused by the FPN / PAN bidirectional path;

[0041] After the multi-head collaborative regression processing of the depth detection, it is represented as:

[0042]

[0043] Where is the expected disparity, is the pixel depth, x,y is the position of the center of the decoded three-dimensional bounding box in the x,y direction of the camera coordinate system, w,l,h is the real length, width and height of the decoded three-dimensional bounding box;

[0044] Three-dimensional parameter regression and spatial 3D-ROI generation, denoted as:

[0045]

[0046] where d is the pixel depth, (x c ,y c ,z c ) is the camera coordinate system, x I ,y I ,z I ,θ I ,(w,l,h) I is the three-dimensional detection frame parameter after reconstruction, p L is the camera coordinate system unified with the LiDAR point cloud after mapping, R I is the 3D ROI unified with the LiDAR frame coordinate of the image branch.

[0047] Further, step (4) is specifically:

[0048] Step (41): Cross-modal geometric alignment:

[0049] TAM maps the spatial ROI of LiDAR and the spatial 3D-ROI feature of the image into a unified spatial coordinate system; denoted as:

[0050] Convert the camera coordinate system to the laser radar coordinate system:

[0051]

[0052] Map both to a BEV grid of uniform resolution, denoted as:

[0053]

[0054] In the formula: p L , T C→L denotes the mapping of all ROI geometric elements from the camera system to the LiDAR system, denotes the unified modal feature;

[0055] Step (42): Feature complementary fusion:

[0056] RAM introduces the semantic texture information of the image branch in units of BEV pixels on the basis of the unified modal feature obtained in step (41), enhances the geometric feature expression of the laser radar point cloud information, and realizes the fusion and collaborative optimization of perception accuracy and semantic understanding; denoted as:

[0057] Linearly transform the input feature to generate three groups of representations of query, key and value:

[0058]

[0059] Compute attention weights:

[0060]

[0061] Fusion feature output is:

[0062]

[0063] Where W q , W k , W v is the matrix of linear mapping, A is the spatial attention matrix, F RAM is the fusion feature;

[0064] Step (43) fuses the feature cube and the detection head output, represented as:

[0065] The RAM output is concatenated with the original radar point cloud feature To form a fusion cube with uniform channel number:

[0066]

[0067] Flattening, completing target classification and 3D bounding box regression through a fully connected layer:

[0068]

[0069] Finally, through residual decoding, the bounding box is obtained, and target classification and 3D box regression are completed:

[0070]

[0071] Where F fs is the fusion feature cube, is the target classification, is the 3D bounding box regression, W cls , W reg is the fully connected classification / regression weight, b anchor represents the initial anchor frame generated by the candidate frame generation network, b pre is the final 3D detection result.

[0072] Compared with the prior art, the present application has the following advantages:

[0073] The present application aims at the problems of low fusion efficiency, high computational complexity and inaccurate cross-modal feature alignment existing in the existing multi-modal three-dimensional target detection method, and proposes a multi-modal 3D target detection method with progressive deep fusion; by adopting a laser radar feature perception framework based on Pillar structure and multi-layer perceptron (MLP) aggregation, the structured expression of point cloud is realized, and the spatial coding efficiency is significantly improved; in the fusion stage, the ROI-Attention mechanism is introduced, the geometric alignment of visual and laser radar features is realized by using the translation perception module (TAM), and the cross-modal coordinate error is eliminated; at the same time, the attention weighting strategy is introduced through the region perception module (RAM), and the complementary semantic and geometric features are adaptively selected, realizing feature-level deep fusion and optimization.

[0074] The method effectively improves the detection accuracy of small targets, occlusions and long-distance targets, and verifies its superior performance on the KITTI public dataset: the 3D detection accuracy is improved by 2.3%, 2.9% and 3.9% under simple, medium and difficult three difficulties respectively, and the inference time is only 51ms, realizing the efficient balance of detection accuracy and real-time performance. BRIEF DESCRIPTION OF DRAWINGS

[0075] Figure 1 The flow chart of the multi-modal 3D target detection method with progressive deep fusion of the present application.

[0076] Figure 2 The flow chart of the point column network target detection based on laser radar of the present application.

[0077] Figure 3 The flow chart of the target detection based on vision of the present application.

[0078] Figure 4 The comparison chart of the detection results of the method and the baseline method in the simulation scene.

[0079] Figure 5 The comparison chart of the BEV detection results of the method and the baseline method in the simulation scene under complex scene. DETAILED DESCRIPTION

[0080] The present application will be described in further detail below with reference to the accompanying drawings.

[0081] Referring to the accompanying Figure 1 The multi-modal 3D target detection method with progressive deep fusion of the present application comprises four steps of data input and preprocessing, laser radar point cloud feature extraction, image information three-dimensional detection and alignment, multi-modal ROI fusion and result output:

[0082] S1: The laser radar and binocular camera obtain point cloud data and image information respectively, the ROI is intercepted and the space is normalized, and the structured point cloud voxel features are obtained; the color enhancement, CLAHE histogram equalization and stereo correction are performed on the binocular image information to ensure that the left and right eye images are aligned with the epipolar line, and the standardized image information is obtained;

[0083] S2: For the structured point cloud features in S1, the point cloud is divided into a regular column grid, and the column-level features are extracted by using a multi-layer perceptron to generate a column (Pillar) level feature, and then the column-level feature is mapped to a bird's eye view (BEV) feature map, and a three-dimensional anchor box is generated by a region proposal network (RPN) to predict the target position and attitude offset, thereby outputting a spatial ROI;

[0084] S3: For the standardized image information in S1, a shared backbone and multi-scale fusion processing are used to extract multi-scale features based on the YOLO3D single-stage target detection method, a multi-head collaborative estimation of target depth is used to regress the target three-dimensional parameters, and finally the three-dimensional bounding box is converted to the laser radar coordinate according to the external parameter matrix, thereby obtaining a unified spatial 3D-ROI;

[0085] S4: A region of interest feature-attention fusion method (ROI-Attention) is designed, a translation perception module (TAM) is used to realize cross-modal geometric alignment, the camera features are mapped to the laser radar coordinate system and the resolution is unified, a region perception module (RAM) is used to select complementary features by using an attention mechanism, the joint optimization of geometric and semantic information is realized, and finally the target classification and three-dimensional frame regression are completed through a full connection layer, thereby outputting the class, spatial coordinates, size and attitude angle of each target, and realizing the 3D detection of multiple targets.

[0086] In step S1, the laser radar and binocular camera obtain point cloud data and image information respectively:

[0087] The original laser radar point cloud is a sparse unstructured three-dimensional scattering body, which is difficult to process efficiently, thereby limiting the application of convolution structure. Therefore, based on the point column network method, a strategy of dividing the radar point cloud information into a regular grid column (Pillar) is adopted, a multi-layer perceptron (MLP) is used to extract and aggregate the point cloud in each grid column, and a maximum pooling is used to generate a three-dimensional pixel level feature, thereby realizing the efficient conversion of three-dimensional point cloud to two-dimensional grid.

[0088] The point cloud set captured by the laser radar is defined by setting a spatial boundary to obtain an effective spatial boundary processing range. Based on the effective spatial boundary range, the vehicle region of interest is divided into a column-level grid, and each column-level grid falling within the effective ROI is indexed by position to obtain a structured point cloud voxel feature. The specific definition is as follows:

[0089] P = {p i | p i = (x i , y i , z i , I i ), = 1, 2, …, N}#(1)

[0090] ROI = {p i | = x min ≤ x i ≤ x max , y min ≤ y i ≤ y max , z min ≤ z i ≤ z max}#(2)

[0091] ROI = [x min , x max ] x [y min , y max ] x [z min , z max ]#(3)

[0092]

[0093] where x min , x max are the ROI on the longitudinal space of the vehicle, y min , y max are the ROI within the lateral perception range of the vehicle, z min , z max are the height of the ROI in the vertical direction, and u, v are the pillar unit position indices.

[0094] In step S2, based on the voxel features of the structured point cloud obtained in S1, the laser radar target detection method generates a spatial ROI through point cloud columnar coding, MLP feature extraction, BEV mapping and RPN prediction, realizes the three-dimensional position and attitude estimation of the target:

[0095] S21 point cloud columnar coding, MLP feature extraction:

[0096] On the basis of maintaining the column-level voxel expression ability, through point cloud columnar coding, MLP feature extraction, the original point cloud is mapped to a high-dimensional feature space to obtain a fixed-length column-level feature g (u,v) pooled and aggregated. It is expressed as follows:

[0097]

[0098] where is the mean coordinate of all points in the column, μ, σ ∈ R 7 is the statistical mean and variance of the data set, is the intermediate hidden feature obtained by MLP, W, b, c are the weight matrix, bias term and channel number of MLP respectively. n (u,v) is the number of points in the column, g (u,v) is the fixed-length column-level feature aggregated by pooling for the (u, v) grid, which can be directly scattered and filled into the BEV grid for feature construction.

[0099] S22 BEV mapping:

[0100] The fixed-length column-level feature g (u,v) obtained in S21 is filled into the BEV grid through coordinate mapping and uniform normalization, and finally a regular BEV feature tensor is formed. It is represented as follows:

[0101]

[0102] where F BEV is the dense BEV feature tensor, H × W is the discrete resolution of the BEV grid in the x and y directions, and C is the number of column-level feature channels. μ c represents the mean of the features on channel C, σ c represents the standard deviation of the features on the channel. This normalization step not only helps to alleviate the offset effect of the zero-filled area in the BEV feature map due to sparse point cloud on the overall distribution, but also effectively improves the training convergence speed and stability of the subsequent two-dimensional convolutional network, providing a more robust feature basis for downstream detection tasks.

[0103] S23 prediction generates spatial ROI:

[0104] Based on the BEV feature tensor obtained in S21, a region candidate network is constructed, which adopts anchor box parameterization representation and sharing mechanism to generate the weight parameters required for potential target ROI in three-dimensional space. Through convolution sliding window prediction of anchor box confidence and offset, spatial ROI extraction is performed to obtain the ROI region with the highest score and geometric information, which provides an anchoring basis for the subsequent cross-modal fusion of image and point cloud. Specifically as follows:

[0105] Anchor box parameterization representation and sharing mechanism:

[0106] First, the predicted box (x, y, z, l, w, h, θ) is encoded into the center (x a , y a , z a ) and size (l a , w a , h a) is the normalized offset for reference, realizing scale-invariant 3D bounding box regression. At the same time, the RPN network sharing mechanism is adopted to obtain the classification branch weight parameter W cls and the regression branch weight parameter W reg . It is expressed as:

[0107]

[0108] where (Δx, Δy, Δz, Δl, Δw, Δh, Δθ) is the dimension offset, is the shared weight, W cls is the classification branch parameter for front / back system binary classification, W reg is the regression branch parameter for predicting anchor box offset.

[0109] Spatial ROI extraction:

[0110] The RPN network is used to generate anchor boxes and predict offsets, and the target candidate boxes are sorted according to the confidence, and the target boxes with high confidence are selected to project into three-dimensional space to form spatial ROI. It is expressed as:

[0111]

[0112] In the formula: S is the candidate box set reserved after confidence threshold filtering, B i is the candidate box, t s is the score threshold, IoU(B i , B j ) is the three-dimensional IoU, and respectively represent the voxel sets of the two detection boxes in the three-dimensional space, M is the final detection box set reserved after NMS, Δx, Δl, Δθ represent the normalized offset output by the network, R L is the final 3D ROI obtained by the laser radar detection method, is the corresponding sampling window, is the bilinear interpolation weight, and F(x, y) is the channel vector of the feature map under the grid coordinates.

[0113] In step S3, based on the standardized image information obtained in S1, the YOLO3D single-stage target detection method is used to extract multi-scale features based on shared backbone and multi-scale fusion processing, and the three-dimensional parameters of the target are regressed by using the multi-head cooperative detection, and finally the spatial 3D-ROI unified with the laser radar coordinates is generated by performing external parameter transformation:

[0114] After shared backbone and multi-scale fusion processing, it is expressed as:

[0115]

[0116] P l =Conv 3×3 (F l +Up(P l+1 ))

[0117] where F is the original feature tensor set of different scales output, P l is the multi-scale feature map fused by FPN / PAN bidirectional path.

[0118] Multi-head collaborative regression processing after depth detection, denoted as:

[0119]

[0120] where is the expected disparity, is the pixel depth, x, y is the position of the center of the decoded three-dimensional bounding box in the x, y direction of the camera coordinate system, w, l, h is the real length, width and height of the decoded three-dimensional bounding box.

[0121] Three-dimensional parameter regression and spatial 3D-ROI generation, denoted as:

[0122]

[0123] where d is the pixel depth, (x c ,y c ,z c ) is the camera coordinate system, x I ,y I ,z I ,θ I ,(w,l,h) I is the three-dimensional detection frame parameter after reconstruction, p L is the camera coordinate system unified with LiDAR point cloud after mapping, R I is the 3D ROI unified with LiDAR frame coordinate of the image branch.

[0124] In step S4, based on the spatial ROI and spatial 3D-ROI obtained in S2, S3 respectively, cross-modal geometric alignment is realized by TAM, feature complementary fusion is completed by RAM, the final fusion feature cube is obtained, and the class, three-dimensional position and attitude of the output multi-target of the detection head are output, realizing 3D detection of multi-target:

[0125] S41 Cross-modal geometric alignment:

[0126] TAM maps the spatial ROI of LiDAR and the spatial 3D-ROI feature of image to a unified spatial coordinate system. Denoted as:

[0127] Transform camera coordinate system to LiDAR coordinate system:

[0128]

[0129] Map both to a unified resolution BEV grid, denoted as:

[0130]

[0131] where p L , T C→L denotes the mapping of all ROI geometric elements from camera system to LiDAR system, denotes the unified modal feature.

[0132] S42 Feature complementary fusion:

[0133] RAM on the basis of the unified modal feature obtained in S41, introduce the semantic texture information of image branch in BEV pixel unit, enhance the geometric feature expression of LiDAR point cloud information, realize the fusion and collaborative optimization of perception accuracy and semantic understanding. Denoted as: First, linear transformation is performed on the input feature

[0134] , generating three groups of representations of query, key and value:

[0135] Calculate attention weight:

[0136]

[0137] Fusion feature output is:

[0138]

[0139] where W q , W k , W v is the matrix of linear mapping, A is the spatial attention matrix, F RAM is the fusion feature.

[0140] S43 fusion feature cube and detection head output, denoted as:

[0141] Concatenate the RAM output with the original radar point cloud feature

[0142] , form a uniform channel number fusion cube:

[0143] Then flatten it, complete target classification and 3D bounding box regression through fully connected layer:

[0144] ​

[0145]

[0146]

[0147] Finally, the bounding box is obtained by residual decoding, and target classification and 3D box regression are completed:

[0148]

[0149] where F fs is the fused feature cube, is the target classification, is the 3D bounding box regression, W cls , W reg is the fully connected classification / regression weight, b anchor represents the initial anchor box generated by the candidate box generation network, b pre is the final 3D detection result.

[0150] Embodiments

[0151] The above progressive deep fusion multi-modal three-dimensional target detection method is verified by simulation experiments. To verify the superiority of the method, based on the OpenPCDet framework, model training and evaluation are completed on an Ubuntu 22.04 workstation equipped with an RTX 4090 and 128 GB of memory to ensure that the results can be reproduced and compared horizontally. The KITTI dataset is used, and the training set and validation set are divided by 3:1. The evaluation index is mAP under IoU=0.7. The Adam optimizer and dynamic learning rate adjustment are used for training. The batch size is 16, and a total of 150 Epochs are trained.

[0152] Comparative examples

[0153] The present application takes the SFD method as the baseline method, and compares a variety of mainstream detection methods. The detection task is divided into three categories of simple, medium and difficult according to the target size and occlusion degree on the KITTI dataset to verify the performance of the progressive deep fusion multi-modal 3D target detection method proposed in the present application. As shown in the following table, the average precision (AP) of the proposed method is 97.8%, 91.2% and 89.5% under three difficulty levels, respectively, which is about 2.3%, 2.9% and 3.9% higher than the SFD baseline. The inference time is 51 ms, which balances accuracy and real-time performance, and significantly enhances the detection ability of small targets, occlusions and distant targets.

[0154]

[0155] As Figure 4As shown, the present application compares and displays the input images, single-modal detection results and multi-modal fusion effects in three typical road scenes. Compared with YOLO3D and the baseline SFD, the method of the present application performs better in occlusion, small target and long-distance target detection, effectively reducing missed detection and false detection. Overall, the fusion strategy in this paper has significant advantages in structure alignment and feature complementation, significantly improving the detection accuracy and boundary fitting effect, verifying its practicality and robustness in actual autonomous driving scenarios. From the perspective of BEV, the proposed algorithm shows excellent detection stability and robustness. As shown in the three typical traffic scenes, compared with the baseline method, false detection, missed detection and boundary ambiguity often occur. The method performs more accurately in target dense, occlusion overlap and long-distance detection, and the prediction result is compact and clear, fully embodying its advantages in multi-modal fusion and feature alignment. Figure 5

[0156] The progressive deep fusion multi-modal 3D target detection method of the present application is for the automatic driving perception task in complex traffic environment. The method takes laser radar point cloud and binocular image as input, and constructs a layered detection framework of "encoding-alignment-fusion-decision", wherein the radar target detection method adopts point column network to realize efficient BEV feature encoding, and the image target detection method uses YOLO3D to generate a three-dimensional bounding box and aligns to a unified coordinate system; the fusion branch introduces a translation perception module (TAM) and a region perception module (RAM) in the ROI-Attention mechanism, realizes geometric alignment and feature complementation, and thus improves the detection accuracy of small targets, occlusions and long-distance targets. The experimental results based on the KITTI dataset show that the three-dimensional detection accuracy of the method under simple, medium and difficult difficulty is improved by 2.3%, 2.9% and 3.9% respectively compared with the baseline, and the inference time is 51ms, which balances the accuracy and real-time performance, provides a new idea for efficient three-dimensional target detection in multi-modal perception tasks, and lays a technical foundation for the practical application of intelligent driving systems.

[0157] The above is an exemplary embodiment of the present application, and the protection scope of the present application is defined by the claims and their equivalents. Those skilled in the art should understand that the present application is not limited by the above examples, and the above examples and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.​

Claims

1. A progressive deep fusion multimodal 3D target detection method, characterized in that, Includes the following steps: Step (1): Obtain point cloud data through LiDAR, acquire image information through binocular camera, perform ROI cropping and spatial normalization on point cloud data to obtain structured point cloud voxel features; perform color enhancement, CLAHE histogram equalization and stereo correction on binocular image information to align the epipolar lines of the left and right eye images to obtain standardized image information. Step (2): For structured point cloud voxel features, the point cloud is divided into regular cylindrical grids and the point-level features within the cylinders are extracted using a multilayer perceptron to generate cylindrical features. The cylindrical features are then mapped to the bird's-eye view BEV feature map and a three-dimensional anchor box is generated through the region candidate network RPN to predict the target position and attitude offset, and the spatial ROI is output. Step (3): Based on the standardized image information obtained in step (1), multi-scale features are extracted by using the YOLO3D single-stage target detection method with shared backbone and multi-scale fusion processing. The three-dimensional parameters of the target are regressed by multi-head collaborative regression of depth detection. Finally, external parameter transformation is performed to generate a spatial 3D-ROI that is unified with the coordinates of the lidar. Step (4): Based on the spatial ROI and spatial 3D-ROI, cross-modal geometric alignment is achieved through TAM, feature complementarity fusion is completed through RAM, and finally the feature cube is fused to output the category, three-dimensional position and pose of multiple targets, thus realizing multimodal 3D target detection.

2. The method according to claim 1, characterized in that, Step (1) "performs ROI extraction and spatial normalization on the point cloud data to obtain structured point cloud voxel features" specifically means: The point cloud data set captured by the lidar is used to define the ROI by setting spatial boundaries, thereby obtaining the effective spatial boundary processing range. Based on the effective spatial boundary range, the region of interest of the vehicle is divided into column-level grids, and the position index of each column-level grid falling into the effective ROI is performed to obtain the structured point cloud voxel features. expression as follows: P={p i |p i =(x i ,y i ,z i ,I i ),=1,2,…,N}#(1) ROI={p i |=x min ≤x i ≤x max ,and min ≤y i ≤y max ,z min ≤z i ≤z max }#(2) ROI=[x min ,x max ]×[y min ,and max ]×[z min ,z max ]#(3) Where x min ,x max For the ROI in the longitudinal space of the vehicle, y min ,y max For the ROI within the vehicle's lateral perception range, z min ,z max Let u and v be the vertical height of the ROI, and u and v be the position indices of the support unit.

3. The method according to claim 2, characterized in that, Step (2) specifically involves: Step (21): Point cloud cylindrical encoding, MLP feature extraction: While preserving the expressive power of columnar voxels, the original point cloud is mapped to a high-dimensional feature space through point cloud columnar encoding and MLP feature extraction, resulting in fixed-length columnar features g after pooling aggregation. (u,v) ; indicates the following: W∈R C×7 #(7) in Let μ,σ be the mean coordinates of all points within the column. 7 The statistical mean and variance of the dataset. These are the intermediate hidden features obtained through MLP, where W, b, and c are the MLP weight matrix, bias term, and number of channels, respectively; n (u,v) h is the number of points within the column. (u,v) For the (u,v)th cell, a column-level feature of fixed length after pooling aggregation; Step (22): BEV mapping: Map the fixed-length column-level features g obtained in step (21) (u,v) The BEV feature tensor is filled into the BEV mesh through coordinate mapping and then uniformly normalized to form a regular BEV feature tensor, as shown below: c=1,…,C#(10) Where F BEV For dense BEV feature tensors, H×W is the discrete resolution of the BEV grid in the x and y directions, and C is the number of cylinder-level feature channels; μ c σ represents the mean of the features on channel C. c The standard deviation represents the characteristics of a channel; Step (23): Predict the generated spatial ROI: Based on the BEV feature tensor obtained in step (22), a region candidate network is constructed. An anchor box parameterization and sharing mechanism is used to generate the weight parameters required for potential target ROIs in three-dimensional space. Through convolution... The confidence and offset of the predicted anchor frame are used to extract the spatial ROI, and the ROI region with the highest score and geometric information is obtained by using the sliding window prediction.

4. The method according to claim 3, characterized in that, Step (23) is as follows: Anchor box parameterization representation and sharing mechanism: First, the prediction box (x, y, z, l, w, h, θ) is encoded as a bounding box centered at (x, y, z, l, w, h, θ). a ,y a ,z a ) and size (l a ,w a ,h a Using a normalized offset as a reference, scale-invariant 3D bounding box regression is achieved; simultaneously, the classification branch weight parameter W is obtained using the RPN network sharing mechanism. cls and regression branch weight parameter W reg ; indicates as: Δθ=θ-θ a #(12) Where (Δx,Δy,Δz,Δl,Δw,Δh,Δθ) is the dimensional offset. It is a shared weight, W cls W is the classification branch parameter used for binary classification in front-end / back-end systems. reg These are the regression branch parameters used to predict anchor frame offset; Spatial ROI extraction: An anchor boxes are generated and offsets are predicted using an RPN network. Target candidate boxes are ranked according to confidence levels, and high-confidence boxes are projected into 3D space to form spatial ROIs; represented as: k=1,…,k#(17) In the formula: S is the set of candidate boxes retained after confidence threshold filtering, B i For candidate boxes, t s For the score threshold, IoU(B) i B j () represents the three-dimensional IoU. and Let represent the voxel sets of the two detection boxes in 3D space, M be the final set of detection boxes retained after NMS, and Δx, Δl, and Δθ represent the normalized offsets output by the network. R represents the absolute parameter of the finally reconstructed detection box. L The final 3D ROI obtained by the LiDAR detection method. For the corresponding sampling window, F(x,y) represents the bilinear interpolation weights, and F(x,y) represents the channel vector of the feature map in grid coordinates.

5. The method according to claim 4, characterized in that, Step (3) specifically involves: After sharing the backbone and multi-scale fusion processing, it is represented as: P l =Conv 3×3 (F l +Up(P l+1 ))#(19) Where F is the original feature tensor set output at different scales, and P l This is a multi-scale feature map after FPN / PAN bidirectional path fusion; Multi-head collaborative regression processing after depth testing is represented as: w=e Δw w a ,l=e Δl l a ,h=e Δh h a #(21) in It is expected parallax. is the pixel depth, x, y are the positions of the center of the decoded 3D bounding box in the x, y directions of the camera coordinate system, and w, l, h are the actual length, width, and height of the decoded 3D bounding box; Three-dimensional parametric regression and spatial 3D-ROI generation are represented as follows: z I =σ(t z )z max ,i I =θ0+t θ Where d is the pixel depth, (x c ,y c ,z c Let x be the camera coordinate system. I ,y I ,z I ,θ I ,(w,l,h) I p represents the parameters of the reconstructed 3D detection box. L To establish a unified camera coordinate system with the LiDAR point cloud after mapping, R I A 3D ROI with image branches and LiDAR bounding box coordinates unified.

6. The method according to claim 5, characterized in that, Step (4) is as follows: Step (41): Cross-modal geometry alignment: TAM maps the spatial ROI of LiDAR and the spatial 3D-ROI features of the image to a unified spatial coordinate system; represented as: Convert the camera coordinate system to the lidar coordinate system: Mapping both onto a BEV mesh of uniform resolution is represented as: In the formula: p L T C→L This represents the mapping of all ROI geometric elements from the camera system to the LiDAR system. Represents uniform modal characteristics; Step (42): Feature Complementary Fusion: The unified modal features obtained by RAM in step (41) Based on this, semantic texture information of image branches is introduced at the BEV pixel level to enhance the geometric feature representation of LiDAR point cloud information, achieving a fusion and collaborative optimization of perception accuracy and semantic understanding; represented as: For input features Perform a linear transformation to generate three sets of representations: query, key, and value. Calculate attention weights: The fused feature output is: Among them W q W k W v A is a linear mapping matrix, A is the spatial attention matrix, and F is the spatial attention matrix. RAM It is a feature of fusion; Step (43) fuses the feature cube and the detection head output, represented as: Compare RAM output with original radar point cloud features Cascaded to form a fusion cube with a uniform number of channels: Flattening the structure and using fully connected layers to perform object classification and 3D bounding box regression: Finally, the bounding box is obtained through residual decoding, completing the target classification and 3D bounding box regression: Where F fs To fuse feature cubes, Classify the target. For 3D bounding box regression, W cls W reg For fully connected classification / regression weights, b anchor b represents the initial anchor box generated by the candidate box generation network. pre This is the final 3D detection result.

Images