Multi-modal sparse fusion three-dimensional target detection method in unstructured environment

Abstract

The invention relates to a multi-modal sparse fusion three-dimensional target detection method in an unstructured environment, and the method is based on a complexity perception candidate region dynamic sampling strategy, and through the cross-domain self-compensation of point cloud geometric features and image spectral features, and the fusion of multi-view sparse features, the detection of a multi-modal sparse fusion three-dimensional target is realized. The perception capability and the recognition precision of the obstacle area, and the remote dynamic target detection performance and the real-time performance are improved; an anti-deformation convolution group and a terrain gradient perception attention module are introduced into a ParScaleNet image backbone network, so that the extraction of edge features of a complex scene is enhanced, and the cross-scale characterization capability of the features is improved; according to the method, a ContraSpaceOpt module with semantic maintenance capability is deployed behind a Transform, excessive smoothness of obstacle features is inhibited through class comparison loss functions and feature space anisotropy constraints, the space separability of obstacles is maintained, and the problems of point cloud fracture and texture misalignment caused by the obstacles are effectively solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of 3D target detection, and specifically relates to a multimodal sparse fusion three-dimensional target detection method in an unstructured environment. Background Art

[0002] In autonomous driving in unstructured environments, 3D object detection, as one of the core tasks of the environmental perception system, plays a crucial role in identifying and localizing objects in three-dimensional space. Its accuracy directly impacts the vehicle's situational awareness of obstacles such as billboards, auto-rickshaws, and animals. Multimodal 3D object detection overcomes the inherent limitations of a single sensor and builds a robust perception system by synergistically integrating the data advantages of heterogeneous sensors such as cameras and LiDAR. Current mainstream fusion methods can be divided into two categories based on their technical approaches: multi-sensor fusion based on dense detection and multi-sensor fusion based on sparse detection.

[0003] Multi-sensor fusion methods based on dense detection directly process raw dense data and fully exploit data details through deep fusion of multimodal features. This solution can fully preserve the geometric features and surface texture information of the target, and has shown significant advantages in obstacle detection tasks such as motorized rickshaws and animals. Its high-resolution features are particularly important for identifying hidden obstacles obscured by large targets. In contrast, sparse detection methods compress data size through sparse representation techniques such as key point extraction and terrain gridding, and achieve efficient inference through cross-sensor feature fusion. This type of method utilizes computational optimization strategies such as sparse convolution and has important application value in scenarios where vehicle embedded platforms process complex unstructured environmental data in real time.

[0004] However, existing methods still face core challenges: although sparse detection schemes reduce computational complexity through dimensionality reduction and sparse convolution and improve efficiency through data compression, they inevitably lose high-frequency geometric information details (such as billboard edge features, the outlines of auto-rickshaws on the road, and suddenly appearing animals) during the feature extraction process, resulting in reduced recognition accuracy in complex obstacle areas and attenuated performance in long-distance dynamic target detection. Summary of the Invention

[0005] To solve the above problems, the present invention proposes a multimodal sparse fusion three-dimensional target detection method in unstructured environments, which can improve the perception ability and recognition accuracy of obstacle features, and the performance and real-time performance of long-distance dynamic target detection; avoid the degradation of high-frequency features of obstacles in unstructured scenes, and solve the problems of point cloud fracture and texture inaccuracy caused by obstacles.

[0006] In order to achieve the above-mentioned object, the technical solution adopted by the present invention is: a multimodal sparse fusion three-dimensional target detection method in an unstructured environment, including a first sensor information detection backbone network 1, a second sensor information detection backbone network 2 and a fusion detection backbone network 3, wherein the first sensor information detection backbone network 1 includes S1 and S2; the second sensor information detection backbone network 3 includes S3, S4 and S5; the fusion detection backbone network 3 includes a link module, S6 and S7; specifically comprising the following steps:

[0007] S1 point cloud feature extraction, including original point cloud information, point cloud initial query and semantic transfer, and geometric migration; Original point cloud information: The cloud data collected by the first sensing unit such as radar is pre-processed and used as the original data input; Initial point cloud query: sparse voxelized point cloud data and structured features extracted through the VoxelNet backbone network; Semantic Transfer Module: This module integrates camera semantic information with LiDAR BEV features through point cloud-image projection, multi-scale feature extraction, and deformable attention enhancement, improving the semantic understanding of LiDAR point clouds. S2 radar candidate generation, including the first sensor information detection head and the first sensor information candidate: Radar detection head: extracts 3D features to generate bounding boxes and classifies objects; Radar Candidates: Use sparse convolution to extract potential target spatial candidate boxes; S3 raw image information input: The second sensor unit such as the camera provides 360° image information and point cloud geometric information transmitted through the geometric migration module; after preprocessing, it is used as input.

[0008] S4 image initial query: including ParScaleNet backbone network, FPN network and ELA self-attention mechanism network; ParScaleNet backbone network: extracts image feature maps containing spatial and texture information, providing visual information for feature fusion; ELA attention network: enhances the feature extraction capability of convolutional neural networks; FPN network: constructs a feature pyramid and fuses multi-scale features to enhance the detection capability of objects of different scales; S5 image candidate generation, including the second sensor information detection head S11, BEV perspective transformation S8, second sensor information branch detection S9 and camera candidate S10: Camera detection head S11: extracts image modal object information from the second sensor information, locates it through 2D bounding box, and combines it with point cloud data for 3D target positioning; BEV perspective transformation S8: convert the image into BEV perspective and project the point cloud into the plane coordinates of the three-dimensional world; Camera branch detection S9: extracts 2D features from the image, generates candidate boxes and performs object classification; Camera candidate S10: potential target area or object position extracted from the image of the second sensor information; S6 sparse candidate fusion: fusion of sparse candidates from two modalities; S7 feature fusion and 3D object detection result generation, including the feature fusion module composed of the Transformer network and the ContraSpaceOpt module, and the 3D object detection result module; Transformer network: uses the self-attention mechanism to fuse image and point cloud data, capture long-range dependencies and global information, and improve the accuracy of 3D object detection; ContraSpaceOpt module: optimizes over-smoothing and dimensionality collapse in Transformer; 3D object detection results: Output 3D object detection results.

[0009] Further, the step S4 includes: S41: Extracting feature maps from image backbone networks ,in is a matrix, are the height, width and number of channels of the feature map respectively; S42: Input feature map Divide into multiple sub-feature maps, and extract multi-scale information through convolution operations of different scales to obtain enhanced feature maps ; S43: An ELA attention mechanism module is added after the image backbone network to enhance the feature extraction capability of the convolutional neural network through convolutional layers, group normalization layers, activation functions, and forward propagation processes; S44: Add an FPN network after the ELA network to build a feature pyramid structure to fuse feature information at different levels and enhance the feature extraction capability of the convolutional neural network. S45: Aggregate global and local information through spatial pyramid to obtain channel descriptors. Then learn channel attention weights through channel-to-channel interaction. , the formula is: , in is the activation function, is the second learnable transformation function, is the first learnable transformation function, is the input feature map; S46: Channel attention weight Apply the Softmax operation to obtain the normalized attention weights : ; S47: Attention weight Applied to feature maps , get the weighted feature map , and finally splice to get the output feature map ;

[0010] Furthermore, the step S7 specifically includes: S71: Input features are processed through multiple encoder layers, each of which includes a self-attention mechanism and a feedforward neural network, and then passed through a decoder to generate the category and location of each target through object query; S72: ContraSpaceOpt via uniformity loss Ensure that the representation is more evenly distributed and optimize the over-smoothing problem in multi-layer networks. The specific formula is: in, and Node and nodes The eigenvector of is the temperature parameter, which adjusts the influence of the similarity score and affects the sharpness of the distribution; Improve the effective rank of representation by optimizing uniformity loss , making the representation more diverse. The update formula is: , in, is the basic feature matrix, is the ratio of the scaling factor to the temperature parameter, controlling the adjustment strength. Calculate the similarity between nodes; S73: After using ContraSpaceOpt to solve the over-smoothing and dimensionality collapse of Transformer, the final category prediction and 3D bounding box regression are obtained through the detection head.

[0011] This invention effectively addresses point cloud fragmentation and texture misalignment caused by billboard occlusion and other issues in unstructured scenes through a cross-domain self-compensation mechanism that integrates point cloud geometric features from LiDAR and other sources with image spectral features from cameras. A complexity-aware dynamic sampling strategy for candidate regions, combined with spatial entropy weighting of multi-view sparse features, improves the detection rate of unstructured obstacles (such as billboards, autorickshaws, and animals) while significantly reducing false alarm rates in complex environments. The adaptive ParScaleNet image backbone network introduces a deformation-resistant convolutional group and terrain gradient-aware attention module into the ResNet architecture. Through a dynamic channel reorganization mechanism within a multi-branch topology, it enhances the extraction of edge features in complex scenes, effectively improving the network's cross-scale representation capabilities for typical unstructured features such as muddy ruts, autorickshaws, and animals. To address the degradation of high-frequency features in unstructured scenes, the ContraSpaceOpt module, which preserves the semantics of obstacles, is proposed. This module deploys a two-stream adversarial optimization architecture after the Transformer encoding layer. Through the obstacle category comparison loss function and feature space anisotropy constraints, it suppresses feature over-smoothing while maintaining the spatial separability of complex obstacles, improving the feature identification ability in unstructured environments such as billboards, auto rickshaws, and animals. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings: Figure 1 It is an overall flow chart of the step execution operation of the present invention; Figure 2 is an architectural diagram of the ParScaleNet backbone network of the present invention; Figure 3 It is the image feature extraction flow chart of the present invention; Figure 4 This is a diagram of the ContraSpaceOpt module architecture of the present invention; Figure 5 The detection results of the present invention under different modal inputs are Figure 1 ; Figure 6 The detection results of the present invention under different modal inputs are Figure 2 ; Labels in the figure: first modality sensing information detection backbone network 1, second modality sensing information detection backbone network 2, fusion detection backbone network 3. DETAILED DESCRIPTION

[0013] In order to make the technical solutions and advantages of this patent clearer, more complete and easier to understand, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0014] A multimodal sparse fusion 3D target detection method in unstructured environments, such as Figure 1 As shown, the specific steps include: S1: First, the raw point cloud data and its corresponding annotation information collected by the first sensor unit, such as a lidar, are passed as input to the point cloud detection backbone network for processing. This step includes preliminary preprocessing and formatting of the point cloud data to ensure the accuracy and completeness of the data, supporting subsequent feature extraction and target detection. The sparse point cloud data is then converted into a structured grid using the voxelization method. Specifically, the point cloud data is divided into multiple small voxel units. Assuming that the point cloud data contains points , divide the space into Each point is mapped to the corresponding voxel according to its coordinates: in, Yes The index of the voxel cell to be mapped, Indicates the point cloud data The coordinates of the points, Represents voxels in Dimensions in direction; The voxelization size is , keep the point cloud in Fine-grained information about the direction. Detection range: \left [ {-54m,54m} \right ]\times \left [ {-54m,54m} \right ]\times \left [ {-54m,3m} \right ] , covering the typical perception areas of autonomous driving scenarios; Each voxel contains multiple points , indicating that it belongs to a voxel All points: in, represents a point in space; In the query area, different point cloud features can be extracted. The spatial coordinates of each point Based on the feature, the normal vector is usually calculated by the local neighborhood points, Neighborhood Select a subset of the point cloud , the normal vector Calculated by the least squares method: , in, Representing normal vectors and points The dot product of Finally, the VoxelNet backbone network is combined with the semantic transfer information provided by S5 to generate an effective structured feature representation; Semantic transfer refers to transferring high-level semantic information (e.g., object category, scene context) from other modalities (e.g., images) into sparse point cloud features. Since the point cloud data collected by the first sensor unit is usually sparse, extracting semantic information directly from it is insufficient. Therefore, semantic features extracted by the image deep network can be used to supplement the point cloud data, so that each point or voxel has a more clear semantic description. S2, first regress the target position, usually by using heat map regression to predict the center position of each target. The heat map represents the probability distribution of the target center point. Assume is the output heat map, the size is ( and is the spatial size of the heat map, is the number of target categories), where the value of each pixel represents the probability of whether there is a target at that location: , in, Is a weight matrix used to weight the features, represents the eigenvalue at position; Later, during the detection process, although the thermal map can calibrate the center point of the target, due to resolution issues, the predicted center point may deviate from the actual center point. Therefore, the radar detection head will perform center point offset regression to correct the predicted center point position. It can be regressed by the following formula: in, is a weight matrix used to weight the eigenvalues. is a eigenvector representing the eigenvalue of the input; Then, size regression is performed on the feature map to predict the 3D size of the target. However, due to the characteristics of the first sensor information data, the target is usually a three-dimensional object with rotation. Therefore, the radar detection head needs to regress the target's rotation angle, usually expressed as the yaw angle. The target position, size, and rotation information obtained through regression are combined with the position of the heat map to generate the radar candidate 3D bounding box of the target. Then, in order to select the box most likely to contain the target from the numerous candidate boxes, non-maximum suppression (NMS) is used to remove the candidate boxes with high overlap, retaining only the candidate boxes most relevant to the target. IoU (Intersection over Union) is usually used as the overlap metric; Based on the features within the candidate box, a confidence score is calculated for each candidate box (for example, the probability of the target center position obtained by heatmap regression). Boxes with higher confidence scores are more likely to contain the actual target; Finally, the remaining candidate boxes are further optimized and radar candidates are output. The output generated by radar candidates includes: the center position of each candidate box ; The size of each candidate box ; The rotation angle of each candidate box and the confidence score (likelihood) of each candidate box; S3, Raw Image Information: The raw image data collected by the second sensor unit (e.g., camera), the corresponding annotation information, and the geometric transfer information passed in S1 are passed as input to the image detection backbone network for processing. This step includes preliminary preprocessing and formatting of the image data, such as color normalization, rescaling, and data augmentation, to ensure data accuracy and consistency, providing reliable input for subsequent feature extraction and object detection. Geometric transfer refers to the process of aligning spatial information between different sensors (such as lidar and camera) through geometric transformation during cross-modal data fusion. Specifically, the extrinsic information between the camera and lidar is used to project point cloud data into the image coordinate system, or to map the spatial information in the image back to the point cloud space, thereby ensuring that the two modalities correspond to each other in the same coordinate system. S4, when designing the image branch backbone network, we took into account the following problems in the process of extracting features layer by layer in the deep convolutional network: when the convolution kernel slides across the entire image, since large-scale targets occupy more pixels, the convolution process tends to retain more information about large-scale targets and as the number of network layers deepens, the features of small-scale targets gradually decay, leading to network degradation and marginalization of small-scale targets. To solve these problems, this paper adopts a combination of ParScaleNet network, Feature Pyramid Network (FPN) and ELA attention mechanism module to extract image features. Figure 2 and Figure 3 As shown, the process is as follows: S41: First, feature extraction is performed through a 7×7 initial convolution layer, and the step size is used. Downsampling: in, Represents the feature map after convolution operation, Indicates a convolution operation using a convolution kernel of size 1 and an output channel number of 64. is the input image or the feature map of the previous layer, represents the size of the output feature map, and are the height and width of the input image respectively; Then pass by Max Pooling , further reducing the spatial size to obtain the feature map : , in, is the input feature map, For batch normalization, is the activation function, It is the maximum pooling operation; Then, the input feature map is evenly divided into sub-feature maps according to the channel , and extract multi-scale spatial information through layered residual convolution, and finally splice to obtain the multi-scale enhanced feature map : ; Then, channel attention modeling is performed on the enhanced features of each scale. Channel descriptors are generated by spatial pyramid aggregation and then passed through two layers of point-by-point convolution and Activate the channel interaction module to obtain the channel attention weight , and later Normalization. The normalized attention weights Applied to the corresponding multi-scale features Finally, the weighted features of each scale are concatenated to obtain the output feature map. S42: ELA uses strip pooling to extract information in the horizontal and vertical directions, and replaces 2D convolution with 1D convolution to reduce the amount of calculation and improve the ability to capture information. First, ELA uses strip pooling to obtain horizontal and vertical information. and vertical direction (1, W) global features, and then the extracted horizontal features and vertical features , ELA uses 1D convolution for feature enhancement: in, and is the processed output, and is the input feature map, and is the feature map after processing, Represents a normalization operation. The final attention weight is the product of the horizontal and vertical attentions. This avoids compression of the channel dimension, retains complete information while capturing the horizontal and vertical position information, and improves the expressive power of spatial attention. S43: To further alleviate the marginalization problem of small-scale objects in deep networks, the network adopts an encoder-decoder structure (feature pyramid network) to upsample and fuse the multi-scale features extracted by each module of the ParScaleNet network, effectively enhancing the modeling of multi-scale spatial information and long-range dependencies between channels, thereby improving overall image recognition performance. The FPN (Feature Pyramid Network) uses the ParScaleNet network to extract multi-scale feature maps (C2, C3, C4, C5), then begins upsampling to restore resolution using the deep feature map (C5). The upsampled features are then element-by-element added and fused with the lower-level features to generate enhanced multi-scale feature maps (P2, P3, P4, P5). The FPN outputs multi-scale feature maps suitable for tasks such as object detection and segmentation, effectively solving the problem of multi-scale object detection. S5 first uses the object query mechanism to initialize the query features and then uses the self-attention mechanism to enhance the information interaction between different queries. , in, is the query vector (Query), and They are weight matrices, which are used to convert the input vector into query, key, and value. is the dimension of the key vector; After that, the query vector processed by the self-attention mechanism Will perform deformation attention interaction with the corresponding image features: , in, is the query vector processed by the deformable attention mechanism, is the image feature map; After the camera query is interactive, it passes through a feed-forward network for object classification and 3D bounding box regression; Subsequently, the BEV transformation of the camera modality is mainly used to convert the camera image features into the BEV space of the LiDAR coordinate system for multimodal fusion; The multi-view features extracted by the camera detection head and the camera internal and external parameters Internal Reference (camera projection matrix), external parameters (Rotation & Translation) for coordinate transformation. And because the camera image itself lacks 3D information, it is necessary to first estimate the depth D of each pixel. Using the depth information D and the camera's internal and external parameters, the 2D image pixels are back-projected into 3D space: , , in, is the 3D world coordinate, is the normalized two-dimensional image coordinate, is the depth value, is the inverse matrix of the camera intrinsic parameter matrix, is the rotation matrix, Is the translation vector. Then the projected 3D point cloud Divide into BEV grid to form BEV features ; Subsequently, N queries are selected as candidate instances and initialized. The query vector and image features are interacted at multiple levels through the cross-attention mechanism to extract the features of the target instance, capture the spatial information and semantic characteristics of the target instance, and thus achieve effective extraction of the target instance features. Finally, the object query result of the camera detection head is interacted with the BEV feature of the camera branch detection through cross-attention to obtain a new query feature. : , in, is the query feature after processing by the self-attention mechanism, It is a feature map from the BEV perspective, which enables the query feature to perceive the BEV semantic information and then predict the target category and the 3D bounding box in the camera coordinate system , respectively: , , in, The query feature is the one that has undergone perspective transformation. Project the 3D bounding box to the LiDAR coordinate system to obtain the camera candidate feature from the LiDAR perspective. ; S6, input the results of S2 and S5 respectively and At this point, all candidate features are located in a unified 3D space and can be fused. In order to ensure that the features of different modalities have the same feature distribution, the LiDAR candidate features and the camera candidate features are linearly transformed once and then feature fusion is performed; S7, design the feature fusion module of the fusion backbone network, including the Transformer and ContraSpaceOpt modules. The specific process is as follows: S71, interactively fuse the coefficient candidates of the two modalities using the Transformer network; S72, such as Figure 4 As shown, first normalize the features and preprocess them. When the scale parameter is greater than 0, the input features are normalized. (B is the batch size, N is the sequence length, and d is the feature dimension) L2 normalization is performed. Specifically, the feature vector of each position of each sample is normalized independently. Maintaining the original feature direction but with unit length provides a stable basis for subsequent contrastive learning; Then the comparison relationship model is performed and the similarity matrix between features is calculated by batch matrix multiplication: , in, is a vector, is the dimension of the vector. Then batch matrix multiplication is performed; , in, is the adjusted negative sample vector, is the similarity score, is the original vector; This operation makes the feature at each position a weighted combination of other features, with the weight coefficient determined by the similarity, essentially constructing the "negative sample direction" of the current feature in the sample space; Finally, the original features are linearly combined with the negative sample directions and then normalized through the LayerNorm layer; S73: Output the final 3D object detection result.

[0015] The visualization results of target detection are as follows Figure 5As shown, the main background of the picture is the point cloud data collected by the laser radar (LiDAR), which appears as a white dot structure on a black background, reflecting the three-dimensional spatial information of the surrounding environment. The circular concentric lines usually represent the scanning range of the laser radar, and the center point represents the current position of the vehicle. Multi-view RGB camera images are embedded in the four corners of each sub-image, showing the real scene of the vehicle's surrounding environment. These images help to supplement visual information such as color and material that the laser radar cannot capture, and enhance the accuracy of target detection and recognition. There are cube boxes of different colors in the picture, representing the multiple types of objects detected. The green box represents a car or truck, the yellow box represents a large vehicle such as a bus or truck, the orange box represents a smaller target such as a pedestrian or cyclist, and the red box represents a more difficult or uncertain target.

[0016] Figure 6 Shown is the visualization result of object detection in another frame.

[0017] Although the above embodiments have been described in detail for the purpose of illustrating the invention, persons familiar with the technical field may make various changes, additions, deletions or equivalent substitutions to the above embodiments without departing from the scope and spirit of the claims of the present invention. Therefore, the present invention is by no means limited to the above embodiments.

Claims

1. A multimodal sparse fusion three-dimensional object detection method in an unstructured environment, characterized by: The information obtained by the first sensing unit is processed by the first sensing information detection backbone network (1) to extract the first sensing information candidate features, and the information obtained by the second sensing unit is processed by the second sensing information detection backbone network (2) to extract the second sensing information candidate features. The features are transformed by the linking and sparse candidate modules, and then the fusion detection backbone network (3) composed of Transformer and ContraSpaceOpt is used to perform feature fusion and output the 3D target detection results.

2. The multimodal sparse fusion three-dimensional target detection method in an unstructured environment according to claim 1, characterized in that: The first sensing unit may be a radar, and the second sensing unit may be a camera.

3. The multimodal sparse fusion three-dimensional target detection method in an unstructured environment according to claim 1, characterized in that: The ContraSpaceOpt program is as follows: input representation, calculate similarity matrix, calculate gradient update, update representation, normalize, and output final representation.

4. The multimodal sparse fusion 3D object detection method in an unstructured environment according to claim 1, characterized in that: The ContraSpaceOpt loss of uniformity Ensure that the representation is more evenly distributed and optimize the over-smoothing problem in multi-layer networks. The specific formula is: , in: and Node and nodes The eigenvector of is the temperature parameter, which adjusts the influence of the similarity score and affects the sharpness of the distribution; By optimizing the uniformity loss, the effective rank of the representation is improved, making the representation more diverse. The update formula is: , in: is the basic feature matrix, is the ratio of the scaling factor to the temperature parameter, controlling the adjustment strength. Calculate the similarity between nodes.

5. The multimodal sparse fusion three-dimensional target detection method in an unstructured environment according to claim 1, characterized in that: The first sensor information detection backbone network (1) includes original point cloud information, point cloud initial query and semantic transfer and geometric migration, radar detection head, and radar candidate module; the second sensor information detection backbone network (2) includes original image information, image initial query composed of ParScaleNet, ELA attention network and FPN network in sequence, camera detection head, BEV perspective transformation, camera branch detection, and camera candidate module.

6. The multimodal sparse fusion three-dimensional target detection method in an unstructured environment according to claim 5, characterized in that: The ParScaleNet program is as follows: input, Convolution, input splitting, multi-scale convolution processing, attention calculation, feature weighting, feature fusion, Convolution, output.

7. The multimodal sparse fusion 3D object detection method in an unstructured environment according to claim 5, characterized in that: The original point cloud information module inputs the cloud data collected by the first sensing unit as original data after preprocessing; The point cloud initial query module extracts features from sparse voxelized point cloud data through the VoxelNet backbone network; The semantic transfer module integrates camera semantic information with LiDAR BEV features through point cloud-image projection, multi-scale feature extraction, and deformable attention enhancement, improving the semantic understanding capability of LiDAR point clouds. The radar detection head extracts 3D features to generate bounding boxes and classify objects; Radar candidates use sparse convolution to extract potential target spatial candidate boxes; The original image information is input into the image information provided by the second sensing unit and is used as input after preprocessing; Image initial query: including ParScaleNet, ELA self-attention network and FPN network; among them: ParScaleNet extracts image feature maps containing spatial and texture information, providing visual information for feature fusion; The ELA attention network enhances the feature extraction capability of convolutional neural networks; The FPN network constructs a feature pyramid and fuses multi-scale features to enhance the detection capability of objects of different scales; The camera detection head extracts the object information of the second sensor information modality, locates it through 2D bounding box, and combines it with point cloud data for 3D target positioning; BEV perspective transformation converts the image into BEV perspective and projects the point cloud into the plane coordinates of the three-dimensional world; Camera branch detection extracts 2D features from the image, generates candidate boxes and performs object classification; potential target regions or object locations extracted by the camera from the second sensor information image; Sparse candidate fusion combines the sparse features of radar candidate and camera candidate modalities; The geometric migration transfers the point cloud geometric information input from the original point cloud information to the original image information module; Semantic transfer transfers the original image information to the point cloud initial query.

8. The multimodal sparse fusion three-dimensional object detection method in an unstructured environment according to claim 5 or 7, characterized in that: The image initial query procedure is: Extracting feature maps from image backbone networks ,in is a matrix, are the height, width and number of channels of the feature map respectively; Input feature map Divide into multiple sub-feature maps, and extract multi-scale information through convolution operations of different scales to obtain enhanced feature maps ; The channel descriptor is obtained by aggregating global and local information through the spatial pyramid. Then the channel attention weight is learned through the interaction between channels. , the formula is: ， in is the activation function, is the second learnable transformation function, is the first learnable transformation function, is the input feature map; Channel attention weight Apply the Softmax operation to obtain the normalized attention weights : ， The attention weight Applied to feature maps , get the weighted feature map , and finally splice to get the output feature map .

Images