Multi-modal sparse fusion three-dimensional target detection method in structured environment

Abstract

The invention relates to a multi-modal sparse fusion three-dimensional target detection method in a structured environment, and the method comprises the steps: carrying out the cross-domain self-compensation of point cloud geometric features and image spectral features, and carrying out the fusion of multi-view sparse features; the perception capability and the recognition precision of a long-distance target, a complex shielding target and a small-scale target, and the detection performance and the real-time performance of a long-distance dynamic target are improved; a novel ParScaleNet image backbone network is designed, and a parallel multi-branch structure is introduced to enhance the multi-scale feature representation capability and the channel dependency relationship modeling capability; a staged progressive feature fusion strategy is adopted, hierarchical feature interaction is realized on a fine-grained spatial scale, and a cross-channel adaptive weighting mechanism is combined, so that the network can efficiently extract and focus key features, the feature expression ability of the model is improved, the stability and robustness of detection precision are ensured, and the detection accuracy is improved. And high-efficiency and high-precision target detection in a structured road scene is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of 3D target detection, and specifically relates to a multimodal sparse fusion three-dimensional target detection method in a structured environment. Background Art

[0002] In autonomous driving technology, 3D object detection, as one of the core tasks of the environmental perception system, plays a crucial role in identifying and localizing objects in three-dimensional space. Its accuracy directly impacts the vehicle's situational awareness of pedestrians, other vehicles, and obstacles. To overcome the inherent limitations of single sensors, the industry generally adopts multi-sensor fusion technology, integrating the data advantages of heterogeneous sensors such as cameras and radar (LiDAR) to build more robust perception systems. Current mainstream fusion methods can be divided into two categories based on their technical approaches: multi-sensor fusion based on dense detection and multi-sensor fusion based on sparse detection.

[0003] Multi-sensor fusion methods based on dense detection directly process raw dense data (such as full-resolution point clouds and pixel-level images) and fully exploit data details through deep fusion of multimodal features (such as pixel-level feature concatenation and channel attention weighting). This approach can fully preserve the target's geometric features and surface texture information, demonstrating significant advantages in detecting small-scale objects such as pedestrians and cyclists. In contrast, sparse detection methods use sparse representation techniques such as keypoint extraction and voxelization to compress data size and combine them with cross-sensor feature fusion to achieve efficient inference. These methods utilize computational optimization strategies such as sparse convolution and have significant application value in scenarios with stringent real-time requirements for in-vehicle embedded platforms.

[0004] However, existing methods still face a core challenge: sparse detection schemes reduce computational complexity through dimensionality reduction and sparse convolution, improving efficiency through data compression. However, they inevitably lose high-frequency geometric details (such as object edges and texture information) during feature extraction, resulting in reduced small object classification accuracy and degraded long-range detection performance. Breaking this trade-off between efficiency and accuracy within a sparse representation framework and establishing a real-time, high-precision 3D object detection system that meets both real-time and power requirements has become a bottleneck hindering the performance improvement of autonomous driving perception systems. Summary of the Invention

[0005] To solve the above problems, the present invention proposes a multimodal sparse fusion three-dimensional target detection method in a structured environment, designs a new ParScaleNet image backbone network, and enhances the multi-scale feature representation capability and channel dependency modeling capability by introducing a parallel multi-branch structure; adopts a staged progressive feature fusion strategy to realize hierarchical feature interaction at a fine-grained spatial scale, and combines it with a cross-channel adaptive weighting mechanism to enable the network to efficiently extract and focus on key features, which not only improves the feature expression ability of the model, but also ensures the stability and robustness of detection accuracy, realizing efficient and high-precision target detection in structured road scenes, and improving the perception ability and recognition accuracy of long-distance targets, complex occlusions and small-scale targets, as well as the detection performance and real-time performance of long-distance dynamic targets.

[0006] To achieve the above-mentioned object, the technical solution adopted by the present invention is: a multimodal sparse fusion three-dimensional detection method in a structured environment, comprising a first sensor information detection backbone network 1, a second sensor information detection backbone network 2, and a fusion detection backbone network 3, wherein the first sensor information detection backbone network 1 comprises S1 and S2; the second sensor information detection backbone network 3 comprises S3, S4, and S5; the fusion detection backbone network 3 comprises a link module, S6, and S7; and specifically comprises the following steps: S1 point cloud feature extraction, including original point cloud information, point cloud initial query and semantic transfer, and geometric migration; Original point cloud information: The cloud data collected by the first sensing unit such as radar is pre-processed and used as the original data input; Initial point cloud query: sparse voxelized point cloud data and structured features extracted through the VoxelNet backbone network; Semantic Transfer Module: This module integrates camera semantic information with LiDAR BEV features through point cloud-image projection, multi-scale feature extraction, and deformable attention enhancement, improving the semantic understanding of LiDAR point clouds. S2 radar candidate generation, including the first sensor information detection head and the first sensor information candidate: Radar detection head: extracts 3D features to generate bounding boxes and classifies objects; Radar Candidates: Use sparse convolution to extract potential target spatial candidate boxes; S3 raw image information input: The second sensor unit such as the camera provides 360° image information and point cloud geometric information transmitted through the geometric migration module; after preprocessing, it is used as input.

[0007] S4 image initial query: including ParScaleNet backbone network and FPN network; ParScaleNet backbone network: extracts image feature maps containing spatial and texture information, providing visual information for feature fusion; FPN network: constructs a feature pyramid and fuses multi-scale features to enhance the detection capability of objects of different scales; S5 image candidate generation, including the second sensor information detection head S11, BEV perspective transformation S8, second sensor information branch detection S9 and camera candidate S10: Camera detection head S11: extracts image modal object information from the second sensor information, locates it through 2D bounding box, and combines it with point cloud data for 3D target positioning; BEV perspective transformation S8: converting the image into BEV perspective and projecting the point cloud into the plane coordinates of the three-dimensional world; Camera branch detection S9: extracts 2D features from the image, generates candidate boxes and performs object classification; Camera candidate S10: potential target area or object position extracted from the image of the second sensor information; S6 sparse candidate fusion: fusion of sparse candidates from two modalities; S7 feature fusion and 3D object detection result generation, including the Transformer fusion layer and 3D object detection result module; Transformer network: uses the self-attention mechanism to fuse image and point cloud data, capture long-range dependencies and global information, and improve the accuracy of 3D object detection; 3D object detection results: Output 3D object detection results.

[0008] Further, the step S4 includes: S41: Extracting feature maps from image backbone networks ,in is a matrix, are the height, width and number of channels of the feature map respectively; S42: Input feature map Divide into multiple sub-feature maps, and extract multi-scale information through convolution operations of different scales to obtain enhanced feature maps ; S43: Add an FPN network after the ParScaleNet backbone network, and build a feature pyramid structure to fuse feature information at different levels, thereby enhancing the feature extraction capability of the convolutional neural network. S44: Aggregate global and local information through spatial pyramid to obtain channel descriptors. Then learn channel attention weights through channel-to-channel interaction. , the formula is: , in is the activation function, is the second learnable transformation function, is the first learnable transformation function, is the input feature map; S45: Channel attention weight Apply the Softmax operation to obtain the normalized attention weights : ; S46: Attention weight Applied to feature maps , get the weighted feature map , and finally splice to get the output feature map ;

[0009] Furthermore, the step S7 specifically includes: S71: Input features are processed through multiple encoder layers, each of which includes a self-attention mechanism and a feedforward neural network, and then passed through a decoder to generate the category and location of each target through object query; S72: After Transformer fusion, it passes through the detection head to obtain the final category prediction and 3D bounding box regression.

[0010] The present invention fully combines the geometric perception advantages of lidar and the texture parsing capabilities of the camera to achieve efficient and high-precision target detection in structured road scenes, and uses the initial query mechanism to extract sparse candidate areas of each modality. While reducing computational overhead and improving detection efficiency, it ensures high accuracy of detection results. A new ParScaleNet image backbone network based on the improved ResNet architecture is designed, which enhances the multi-scale feature representation capability and channel dependency modeling capability by introducing a parallel multi-branch structure; ParScaleNet adopts a staged progressive feature fusion strategy to realize hierarchical feature interaction at a fine-grained spatial scale, and combines a cross-channel adaptive weighting mechanism to enable the network to efficiently extract and focus on key features, thereby improving detection accuracy while ensuring computational efficiency. This invention not only improves the feature expression capability of the model, but also ensures the stability and robustness of detection accuracy while maintaining efficient reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings: Figure 1 It is an overall flow chart of the step execution operation of the present invention; Figure 2 is an architectural diagram of the ParScaleNet backbone network of the present invention; Figure 3 It is the image feature extraction flow chart of the present invention; Figure 4 The detection results of the present invention under different modal inputs are Figure 1 ; Figure 5 The detection results of the present invention under different modal inputs are Figure 2 ; Labels in the figure: first modality sensing information detection backbone network 1, second modality sensing information detection backbone network 2, fusion detection backbone network 3. DETAILED DESCRIPTION

[0012] In order to make the technical solutions and advantages of this patent clearer, more complete and easier to understand, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0013] A multimodal sparse fusion 3D target detection method in a structured environment, such as Figure 1 As shown, the specific steps include: S1: First, the raw point cloud data and its corresponding annotation information collected by the first sensor unit, such as a lidar, are passed as input to the point cloud detection backbone network for processing. This step includes preliminary preprocessing and formatting of the point cloud data to ensure the accuracy and completeness of the data, supporting subsequent feature extraction and target detection. The sparse point cloud data is then converted into a structured grid using a voxelization method. Specifically, the point cloud data is divided into multiple small voxel units. Assuming that the point cloud data contains points , divide the space into pieces of size Each point is mapped to the corresponding voxel according to its coordinates: in, Yes The index of the voxel cell to be mapped, Indicates the point cloud data The coordinates of the points, Represents voxels in Dimensions in direction; The voxelization size is , keep the point cloud in Fine-grained information about direction. Detection range: \left [ {-54m,54m} \right ]\times \left [ {-54m,54m} \right ]\times \left [ {-54m,3m} \right ] , covering the typical perception areas of autonomous driving scenarios; Each voxel contains multiple points , indicating that it belongs to a voxel All points of: in, represents a point in space; In the query area, different point cloud features can be extracted. The spatial coordinates of each point Based on the feature, the normal vector is usually calculated by the local neighborhood points, Neighborhood Select a subset of the point cloud , the normal vector Calculated by the least squares method: , in, Representing normal vectors and points The dot product of Finally, the VoxelNet backbone network is combined with the semantic transfer information provided by S5 to generate effective structured feature representation; Semantic transfer refers to transferring high-level semantic information (e.g., object category, scene context) from other modalities (e.g., images) into sparse point cloud features. Since the point cloud data collected by the first sensor unit is usually sparse, extracting semantic information directly from it is insufficient. Therefore, semantic features extracted by the image deep network can be used to supplement the point cloud data, so that each point or voxel has a more clear semantic description. S2, first regress the target position, usually by using heat map regression to predict the center position of each target. The heat map represents the probability distribution of the target center point. Assume is the output heat map, size is ( and is the spatial size of the heat map, is the number of target categories), where the value of each pixel represents the probability of whether there is a target at that location: , in, Is a weight matrix used to weight the features, represents the eigenvalue at position; Later, during the detection process, although the thermal map can calibrate the center point of the target, due to resolution issues, the predicted center point may deviate from the actual center point. Therefore, the radar detection head will perform center point offset regression to correct the predicted center point position. It can be regressed by the following formula: in, is a weight matrix used to weight the eigenvalues. is a eigenvector representing the eigenvalue of the input; Then, size regression is performed on the feature map to predict the 3D size of the target. However, due to the characteristics of the first sensor information data, the target is usually a three-dimensional object with rotation. Therefore, the radar detection head needs to regress the target's rotation angle, usually expressed as the yaw angle. The target position, size, and rotation information obtained through regression are combined with the position of the heat map to generate the radar candidate 3D bounding box of the target. Then, in order to select the box most likely to contain the target from the numerous candidate boxes, non-maximum suppression (NMS) is used to remove the candidate boxes with high overlap, retaining only the candidate boxes most relevant to the target. IoU (Intersection over Union) is usually used as the overlap metric; Based on the features within the candidate box, a confidence score is calculated for each candidate box (for example, the probability of the target center position obtained by heatmap regression). Boxes with higher confidence scores are more likely to contain the actual target; Finally, the remaining candidate boxes are further optimized and radar candidates are output. The output generated by radar candidates includes: the center position of each candidate box ; The size of each candidate box ; The rotation angle of each candidate box and the confidence score (likelihood) of each candidate box; S3, Raw Image Information: The raw image data collected by the second sensor unit (e.g., camera), the corresponding annotation information, and the geometric transfer information passed in S1 are passed as input to the image detection backbone network for processing. This step includes preliminary preprocessing and formatting of the image data, such as color normalization, rescaling, and data augmentation, to ensure data accuracy and consistency, providing reliable input for subsequent feature extraction and object detection. Geometric transfer refers to the process of aligning spatial information between different sensors (such as lidar and camera) through geometric transformation during cross-modal data fusion. Specifically, the extrinsic information between the camera and lidar is used to project point cloud data into the image coordinate system, or to map the spatial information in the image back to the point cloud space, thereby ensuring that the two modalities correspond to each other in the same coordinate system. S4, when designing the image branch backbone network, we took into account the following problems in the process of extracting features layer by layer in the deep convolutional network: when the convolution kernel slides across the entire image, since large-scale targets occupy more pixels, the convolution process tends to retain more information about large-scale targets and as the number of network layers deepens, the features of small-scale targets gradually decay, leading to network degradation and marginalization of small-scale targets. To solve these problems, this paper adopts a combination of ParScaleNet network, Feature Pyramid Network (FPN) and ELA attention mechanism module to extract image features. Figure 2 and 3 As shown, the process is as follows: S41: First, feature extraction is performed through a 7×7 initial convolution layer, and the step size is used. Downsampling: in, Represents the feature map after convolution operation, Indicates a convolution operation using a convolution kernel of size 1 and an output channel number of 64. is the input image or the feature map of the previous layer, represents the size of the output feature map, and are the height and width of the input image respectively; Then pass by Max Pooling , further reducing the spatial size to obtain the feature map : , in, is the input feature map, For batch normalization, is the activation function, It is the maximum pooling operation; Then, the input feature map is evenly divided into sub-feature maps according to the channel , and extract multi-scale spatial information through layered residual convolution, and finally splice to obtain the multi-scale enhanced feature map : ; Then, channel attention modeling is performed on the enhanced features of each scale. Channel descriptors are generated by spatial pyramid aggregation and then passed through two layers of point-by-point convolution and Activate the channel interaction module to obtain the channel attention weight , and later Normalization. The normalized attention weights Applied to the corresponding multi-scale features Finally, the weighted features of each scale are concatenated to obtain the output feature map. S42: Feature Pyramid Network (FPN) uses the ParScaleNet network to extract multi-scale feature maps (C2, C3, C4, C5). It then upsamples the deep feature map (C5) to restore the resolution and adds the upsampled features element-by-element with the low-level features to generate enhanced multi-scale feature maps (P2, P3, P4, P5). FPN outputs multi-scale feature maps suitable for tasks such as object detection and segmentation, effectively solving the multi-scale object detection problem. S5 first uses the object query mechanism to initialize the query features and then uses the self-attention mechanism to enhance the information interaction between different queries. , in, is the query vector (Query), and They are weight matrices, which are used to convert the input vector into query, key, and value. is the dimension of the key vector; After that, the query vector processed by the self-attention mechanism Will perform deformation attention interaction with the corresponding image features: , in, is the query vector processed by the deformable attention mechanism, is the image feature map; After the camera query is interactive, it passes through a feed-forward network for object classification and 3D bounding box regression; Subsequently, the BEV transformation of the camera modality is mainly used to convert the camera image features into the BEV space of the LiDAR coordinate system for multimodal fusion; The multi-view features extracted by the camera detection head and the camera internal and external parameters Internal Reference (camera projection matrix), external parameters (Rotation & Translation) for coordinate transformation. And because the camera image itself lacks 3D information, it is necessary to first estimate the depth D of each pixel. Using the depth information D and the camera's internal and external parameters, the 2D image pixels are back-projected into 3D space: in, is the 3D world coordinate, is the normalized two-dimensional image coordinate, is the depth value, is the inverse matrix of the camera intrinsic parameter matrix, is the rotation matrix, Is the translation vector. Then the projected 3D point cloud Divide into BEV grid to form BEV features ; Subsequently, N queries are selected as candidate instances and initialized. The query vector and image features are interacted at multiple levels through the cross-attention mechanism to extract the features of the target instance, capture the spatial information and semantic characteristics of the target instance, and thus achieve effective extraction of the target instance features. Finally, the object query result of the camera detection head is interacted with the BEV feature of the camera branch detection through cross-attention to obtain a new query feature. : , in, is the query feature after processing by the self-attention mechanism, It is a feature map from the BEV perspective, which enables the query feature to perceive the BEV semantic information and then predict the target category and the 3D bounding box in the camera coordinate system , respectively: , , in, The query feature is the one that has undergone perspective transformation. Project the 3D bounding box to the LiDAR coordinate system to obtain the camera candidate feature from the LiDAR perspective. ; S6, input the results of S2 and S5 respectively and At this point, all candidate features are located in a unified 3D space and can be fused. In order to ensure that the features of different modalities have the same feature distribution, the LiDAR candidate features and the camera candidate features are linearly transformed once and then feature fusion is performed; S7, design the feature fusion module of the fusion backbone network, namely the Transformer fusion layer: S71, in the Transformer fusion layer, queries of different modalities interact deeply through the attention mechanism: specifically, a fusion decoder layer is used to integrate multimodal information. This process starts with each query point paying attention to all query points of two or even more modalities at the same time. First, the similarity score matrix between the query points is calculated, and then the information of the query points of other modalities is weighted and aggregated based on the matrix to establish a cross-modal information flow channel. In order to enhance the semantic expression ability, the attention mechanism adopts a multi-head attention structure. By parallel calculation of the Q (Query), K (Key), and V (Value) attention distributions in different subspaces, the correlation between multimodal data in different semantic dimensions is captured respectively, and finally the fusion of cross-modal features is achieved; S72: Output the final 3D object detection result.

[0014] The visualization results of target detection are as follows Figure 4 As shown, it mainly includes detection boxes from six camera perspectives and point cloud views of primary sensor information, such as lidar. A detailed analysis is as follows: The left side (six image sub-images) shows the 2D object detection results from different cameras, such as primary sensor units. Each sub-image corresponds to a different camera perspective around the vehicle: Front Left, Front, Front Right, Back Left, Back, and Back Right. Each sub-image is annotated with a different colored detection box, representing a detected object (such as a vehicle, pedestrian, etc.). Different colors may represent different object categories, for example: pink boxes represent pedestrians, orange boxes represent vehicles, and cyan boxes represent large trucks. The right side (point cloud view) shows a top-down view of the lidar point cloud data, where the purple area represents the density of the lidar point cloud, with the vehicle at the center. Red and other colored boxes represent detected objects, and the box positions correspond to the object's projection in 3D space.

[0015] Figure 5 Shown is the visualization result of object detection in another frame.

[0016] Although the above embodiments have been described in detail for the purpose of illustrating the invention, persons familiar with the technical field may make various changes, additions, deletions or equivalent substitutions to the above embodiments without departing from the scope and spirit of the claims of the present invention. Therefore, the present invention is by no means limited to the above embodiments.

Claims

1. A multimodal sparse fusion three-dimensional object detection method in a structured environment, characterized by: The information obtained by the first sensing unit is processed by the first sensing information detection backbone network (1) to extract the first sensing information candidate features, and the information obtained by the second sensing unit is processed by the second sensing information detection backbone network (2) to extract the second sensing information candidate features. These features are linearly transformed by the linking and sparse candidate modules, and then feature fused by the fusion detection backbone network (3) including the Transformer fusion layer to output the three-dimensional target detection results.

2. The multimodal sparse fusion three-dimensional object detection method in a structured environment according to claim 1, characterized in that: The first sensing unit may be a radar, and the second sensing unit may be a camera.

3. The multimodal sparse fusion three-dimensional object detection method in a structured environment according to claim 1, characterized in that: The first sensor information detection backbone network (1) includes original point cloud information, point cloud initial query and semantic transfer and geometric migration, radar detection head, and radar candidate module; the second sensor information detection backbone network (2) includes original image information, image initial query composed of ParScaleNet and FPN network in sequence, camera detection head, BEV perspective transformation, camera branch detection, and camera candidate module.

4. The multimodal sparse fusion three-dimensional target detection method in a structured environment according to claim 3, characterized in that: The ParScaleNet program is as follows: input, Convolution, input splitting, multi-scale convolution processing, attention calculation, feature weighting, feature fusion, Convolution, output.

5. The multimodal sparse fusion three-dimensional object detection method in a structured environment according to claim 3, characterized in that: The original point cloud information module inputs the cloud data collected by the first sensing unit as original data after preprocessing; The point cloud initial query module extracts features from sparse voxelized point cloud data through the VoxelNet backbone network; The semantic transfer module integrates camera semantic information with radar BEV features through point cloud-image projection, multi-scale feature extraction, and deformable attention enhancement, improving the semantic understanding capability of radar point clouds. The radar detection head extracts 3D features to generate bounding boxes and classify objects; Radar candidates use sparse convolution to extract potential target spatial candidate boxes; The original image information is input into the image information provided by the second sensing unit and is used as input after preprocessing; Image initial query: including ParScaleNet and FPN networks; among them: ParScaleNet extracts image feature maps containing spatial and texture information, providing visual information for feature fusion; The FPN network constructs a feature pyramid and fuses multi-scale features to enhance the detection capability of objects of different scales; The camera detection head extracts the object information of the second sensor information modality, locates it through 2D bounding box, and combines it with point cloud data for 3D target positioning; BEV perspective transformation converts the image into BEV perspective and projects the point cloud into the plane coordinates of the three-dimensional world; Camera branch detection extracts 2D features from the image, generates candidate boxes and performs object classification; potential target regions or object locations extracted by the camera from the second sensor information image; Sparse candidate fusion combines the sparse candidate features of radar candidates and camera candidates; The geometric migration transfers the point cloud geometric information input from the original point cloud information to the original image information module; Semantic transfer transfers the original image information to the point cloud initial query.

6. The method for multimodal sparse fusion 3D object detection in a structured environment according to claim 3 or 5, characterized in that: The image initial query procedure is: Extracting feature maps from image backbone networks ,in is a matrix, are the height, width and number of channels of the feature map respectively; Input feature map Divide into multiple sub-feature maps, and extract multi-scale information through convolution operations of different scales to obtain enhanced feature maps ; The channel descriptor is obtained by aggregating global and local information through the spatial pyramid. Then the channel attention weight is learned through the interaction between channels. , the formula is: ， in is the activation function, is the second learnable transformation function, is the first learnable transformation function, is the input feature map; Channel attention weight Apply the Softmax operation to obtain the normalized attention weights : ， The attention weight Applied to feature maps , get the weighted feature map , and finally splice to get the output feature map .

Images