Multi-modal three-dimensional object detection method, electronic device, and program product

Abstract

The application relates to the field of vehicle-mounted environment perception, in particular to a multi-modal three-dimensional target detection method, an electronic device and a program product. The method comprises the following steps: acquiring a laser radar bird's-eye view and a vehicle-mounted camera image; extracting features of the two types of images through a double-branch convolutional neural network to obtain two types of feature maps; projecting a three-dimensional anchor frame to the two types of feature maps respectively, and obtaining first fusion features after multi-modal feature fusion processing; performing regression screening processing on the first fusion features to obtain a plurality of three-dimensional candidate regions; projecting the three-dimensional candidate regions to the two types of feature maps respectively, and obtaining second fusion features after multi-modal feature fusion processing; and generating a three-dimensional target detection result based on the second fusion features. The method can improve the target detection accuracy and enhance the perception stability in complex driving scenes.

Description

Technical Field

[0001] This application relates to the field of vehicle-mounted environmental perception, and in particular to a multimodal three-dimensional target detection method, electronic device, and program product. Background Technology

[0002] Intelligent driving environmental perception typically uses sensors such as cameras and lidar for target detection, and improves the vehicle's ability to recognize the surrounding scene through multimodal fusion.

[0003] Existing multimodal detection solutions often employ feature stitching, post-processing fusion, or multi-view projection, which makes it difficult to fully coordinate the representational differences between lidar point clouds and image data, easily leading to insufficient feature complementarity. In scenarios with occlusion, low illumination, or severe weather, the loss of local information will further reduce detection accuracy. At the same time, the complex fusion process will increase the computational burden, making it difficult to meet the real-time requirements of the vehicle platform.

[0004] Existing technologies cannot effectively suppress redundant background features during the feature fusion process, resulting in a large amount of invalid interference information in the fused features, which directly reduces the localization accuracy of candidate regions and the accuracy of target classification. Summary of the Invention

[0005] This application provides a multimodal 3D target detection method, electronic device, and program product to improve the accuracy of 3D target detection in complex scenes.

[0006] In a first aspect, embodiments of this application provide a multimodal three-dimensional target detection method, including:

[0007] Acquire aerial views from LiDAR and images from vehicle-mounted cameras;

[0008] The first feature map and the second feature map are obtained by extracting features from the bird's-eye view of the LiDAR and the image of the vehicle camera through a dual-branch convolutional neural network.

[0009] The preset 3D anchor frame is projected onto the first feature map and the second feature map respectively, and the first fused feature is obtained after multimodal feature fusion processing;

[0010] The first fused feature is subjected to regression screening to obtain multiple three-dimensional candidate regions;

[0011] The three-dimensional candidate regions are projected onto the first feature map and the second feature map respectively, and the second fused feature is obtained after multimodal feature fusion processing;

[0012] Three-dimensional target detection results are generated based on the second fusion feature.

[0013] In one possible implementation, the multimodal feature fusion process includes:

[0014] The two local features output by pooling after projection are convolved to reduce dimensionality, resulting in two dimensionality-reduced features.

[0015] The two dimensionality-reduced features are averaged element-wise to obtain the fused features.

[0016] In one possible implementation, the regression screening process includes:

[0017] Boundary box regression and target calculation are performed on the first fused feature to obtain the boundary box offset and target score;

[0018] The preset 3D anchor frame is corrected based on the bounding box offset, and suppression filtering is performed in combination with the target score to obtain multiple 3D candidate regions.

[0019] In one possible implementation, generating a three-dimensional target detection result includes:

[0020] Based on the second fusion feature, the bounding box parameters, category information, and heading angle information are calculated.

[0021] The three-dimensional target detection results are generated based on the bounding box parameters, category information, and heading angle information.

[0022] In one possible implementation, it also includes:

[0023] A preset training dataset is input into a dual-branch convolutional neural network to obtain multi-stage fused features;

[0024] Generate a reverse attention heatmap based on multi-stage fusion features and complete feature modulation;

[0025] The loss is calculated based on the modulated fusion features, and the network weights are updated.

[0026] In one possible implementation, generating a reverse attention heatmap includes:

[0027] Gradient calculation is performed on the multi-stage fusion features based on gradient-weighted class activation mapping to obtain an initial attention heatmap;

[0028] The initial attention heatmap is inverted to obtain the reverse attention heatmap.

[0029] In one possible implementation, feature modulation is performed, including:

[0030] The reverse attention heatmap is thresholded to obtain a first heatmap;

[0031] The first heatmap is multiplied element-wise with the multi-stage fusion feature to complete feature modulation.

[0032] In one possible implementation, the loss is calculated based on the modulated fusion features, including:

[0033] Based on the modulated fusion features, the bounding box regression loss, classification loss, and heading angle loss are calculated respectively.

[0034] The total loss is obtained by weighted summation, and the network weights are updated based on the total loss.

[0035] Secondly, embodiments of this application provide a multimodal three-dimensional target detection device, comprising:

[0036] The acquisition module is used to acquire bird's-eye view images from the LiDAR and images from the vehicle-mounted camera;

[0037] The feature analysis module is used to extract features from the LiDAR bird's-eye view and the vehicle-mounted camera image through a dual-branch convolutional neural network to obtain a first feature map and a second feature map.

[0038] The processing module is used to project the preset three-dimensional anchor frame onto the first feature map and the second feature map respectively, and obtain the first fused feature after multimodal feature fusion processing;

[0039] The filtering module is used to perform regression filtering on the first fused feature to obtain multiple three-dimensional candidate regions.

[0040] The processing module is further configured to project the three-dimensional candidate region onto the first feature map and the second feature map respectively, and obtain the second fused feature after multimodal feature fusion processing;

[0041] The generation module is used to generate three-dimensional target detection results based on the second fusion feature.

[0042] In one possible implementation, the processing module is specifically used for:

[0043] The two local features output by pooling after projection are convolved to reduce dimensionality, resulting in two dimensionality-reduced features.

[0044] The two dimensionality-reduced features are averaged element-wise to obtain the fused features.

[0045] In one possible implementation, the filtering module is specifically used for:

[0046] Boundary box regression and target calculation are performed on the first fused feature to obtain the boundary box offset and target score;

[0047] The preset 3D anchor frame is corrected based on the bounding box offset, and suppression filtering is performed in combination with the target score to obtain multiple 3D candidate regions.

[0048] In one possible implementation, the generation module is specifically used for:

[0049] Based on the second fusion feature, the bounding box parameters, category information, and heading angle information are calculated.

[0050] The three-dimensional target detection results are generated based on the bounding box parameters, category information, and heading angle information.

[0051] In one possible implementation, it also includes:

[0052] A preset training dataset is input into a dual-branch convolutional neural network to obtain multi-stage fused features;

[0053] Generate a reverse attention heatmap based on multi-stage fusion features and complete feature modulation;

[0054] The loss is calculated based on the modulated fusion features, and the network weights are updated.

[0055] In one possible implementation, generating a reverse attention heatmap includes:

[0056] Gradient calculation is performed on the multi-stage fusion features based on gradient-weighted class activation mapping to obtain an initial attention heatmap;

[0057] The initial attention heatmap is inverted to obtain the reverse attention heatmap.

[0058] In one possible implementation, feature modulation is performed, including:

[0059] The reverse attention heatmap is thresholded to obtain a first heatmap;

[0060] The first heatmap is multiplied element-wise with the multi-stage fusion feature to complete feature modulation.

[0061] In one possible implementation, the loss is calculated based on the modulated fusion features, including:

[0062] Based on the modulated fusion features, the bounding box regression loss, classification loss, and heading angle loss are calculated respectively.

[0063] The total loss is obtained by weighted summation, and the network weights are updated based on the total loss.

[0064] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0065] The memory stores computer-executed instructions;

[0066] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0067] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0068] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0069] The multimodal 3D target detection method, electronic device, and program product provided in this application extract features from the LiDAR bird's-eye view and the vehicle-mounted camera image respectively. The detection box is projected onto the two feature maps in two stages to complete feature fusion. The method first roughly selects candidate target regions and then performs fine calculations on the candidate regions. It fully combines the complementary advantages of the two types of sensor data, and refines the position and category attributes of the target while ensuring detection efficiency. This achieves the effect of improving the accuracy of 3D target detection and enhancing the perception stability in complex driving scenarios. Attached Figure Description

[0070] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0071] Figure 1 A schematic diagram of a scene for multimodal 3D target detection provided in this application;

[0072] Figure 2 Flowchart of the multimodal three-dimensional target detection method provided in this application Figure 1 ;

[0073] Figure 3 Flowchart of the multimodal three-dimensional target detection method provided in this application Figure 2 ;

[0074] Figure 4 Flowchart of the multimodal three-dimensional target detection method provided in this application Figure 3 ;

[0075] Figure 5 Flowchart of the multimodal three-dimensional target detection method provided in this application Figure 4 ;

[0076] Figure 6 Flowchart of the multimodal three-dimensional target detection method provided in this application Figure 5 ;

[0077] Figure 7Flowchart of the multimodal three-dimensional target detection method provided in this application Figure 6 ;

[0078] Figure 8 Flowchart of the multimodal three-dimensional target detection method provided in this application Figure 7 ;

[0079] Figure 9 This is a schematic diagram of the structure of the multimodal three-dimensional target detection device provided in this application;

[0080] Figure 10 A schematic diagram of the structure of the electronic device provided in this application.

[0081] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0082] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0083] Intelligent driving environment perception technology is typically deployed in autonomous vehicles or advanced driver assistance systems. It relies on onboard cameras and sensors such as LiDAR to identify and locate vehicles, pedestrians, traffic facilities, and obstacles on the road. Related systems generally include sensor acquisition terminals, onboard computing platforms, and perception processing links for decision-making and planning, to meet the real-time perception needs of vehicles in urban roads, highways, and low-light or occluded scenarios.

[0084] Existing multimodal target detection schemes typically achieve environmental perception through joint processing of camera images and LiDAR point clouds. For example, one type of scheme first extracts image features and point cloud projection features separately, and then generates detection results through feature stitching or post-processing; another type of scheme converts the point cloud into a bird's-eye view or multi-view representation, and then matches it with the corresponding output of the image branch to complete candidate region selection and target recognition.

[0085] However, in the above-mentioned schemes, image data and point cloud data differ significantly in spatial representation and semantic representation. Simple stitching or post-fusion cannot achieve sufficient coordination, and the problem of insufficient complementarity between geometric and texture information is likely to occur. When the target is in a scene where vehicles obstruct it, pedestrians are crossing, it is under low light at night, or there is rain or fog interference, the lack of local information in a single modality will be further amplified, leading to unstable candidate regions, increased target localization deviation, and thus missed detections or false detections.

[0086] Complex fusion processes typically require numerous intermediate processing steps, increasing the computational burden and inference latency of the onboard platform, making it difficult to balance detection accuracy and real-time performance. If the perception results cannot stably output 3D target information, it will directly affect the reliability of subsequent path planning and obstacle avoidance control.

[0087] The multimodal 3D target detection method provided in this application extracts features from LiDAR bird's-eye view and camera image through a dual-branch network. In the candidate box generation stage, the 3D anchor boxes are projected onto the dual-modal feature map and fused by pooling. Candidate regions are obtained through regression screening. In the detection stage, the candidate regions are projected onto the dual-modal feature map for secondary pooling fusion. Finally, 3D detection results are generated based on the fused features, so that geometric information and texture information are gradually aligned and fully complementary in the spatial dimension. When information of one modality is missing in an occluded scene, the other modality can effectively compensate, thereby improving the accuracy and robustness of 3D target detection in complex traffic scenes. At the same time, the entire fusion process is implemented based on pooling and element-wise operations without adding extra network parameters, which is suitable for the real-time perception requirements of vehicle embedded platforms.

[0088] The multimodal 3D target detection method provided in this application is applicable to an environment perception architecture consisting of an onboard camera, LiDAR, and onboard computing platform. Through a two-stage fusion detection process centered around 3D anchor frames and 3D candidate regions, it achieves joint utilization of multimodal information, which helps to improve the detection accuracy of 3D targets in complex scenes and enhance the robustness and practicality of 3D target perception in complex driving scenarios.

[0089] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0090] Figure 1 This application provides a schematic diagram of the multimodal 3D target detection application scenario, such as... Figure 1 As shown, it includes: a data acquisition terminal 101 and an on-board processing terminal 102.

[0091] The data acquisition terminal 101 is used to simultaneously acquire lidar point cloud data and vehicle camera image data during vehicle operation, and send the acquired data to the vehicle processing terminal.

[0092] The vehicle-mounted processing unit 102 is used to receive LiDAR point cloud data and vehicle-mounted camera image data sent by the data acquisition unit, project the LiDAR point cloud data to generate a bird's-eye view, and perform a multimodal 3D target detection method on the bird's-eye view and vehicle-mounted camera image to generate 3D target detection results.

[0093] Figure 2 Flowchart of the multimodal three-dimensional target detection method provided in this application Figure 1 ,like Figure 2 As shown, the method includes:

[0094] S201. Acquire aerial views from the LiDAR and images from the vehicle-mounted camera.

[0095] Specifically, a vehicle-mounted LiDAR system collects 3D point cloud data, which includes the spatial coordinates and reflection intensity information of the target surface. A vehicle-mounted camera collects 2D image data containing color and texture information. The collected 3D point cloud data is discretized vertically, and the distribution density and height characteristics of the point cloud within each grid are statistically analyzed. The statistical results are then mapped to a horizontal 2D plane to generate a bird's-eye view. This bird's-eye view preserves the target's geometric outline and spatial distribution in the horizontal direction while reducing the processing dimensionality of the 3D point cloud.

[0096] LiDAR point cloud data and camera image data have different physical meanings and data structures at the raw data level, making direct fusion difficult. By projecting the 3D point cloud into a bird's-eye view, the point cloud data is transformed into a 2D mesh structure similar to the image data, establishing a unified data form for subsequent parallel processing by the dual-branch network. The bird's-eye view preserves the spatial location information of the point cloud while avoiding the computational burden of processing each point of the original point cloud point by point.

[0097] By achieving synchronous acquisition and format unification of multi-source heterogeneous sensor data, the dimensionality of LiDAR point cloud data can be reduced without losing key spatial location information. Through bird's-eye view generation and processing, the 3D point cloud data is effectively compressed into a 2D pseudo-image. While preserving the horizontal geometric contour and spatial distribution characteristics of the target, the data processing dimension of the subsequent network model is greatly reduced, thereby reducing the consumption of computing resources and facilitating real-time processing of the vehicle-mounted embedded platform.

[0098] S202. Feature extraction is performed on the LiDAR bird's-eye view and the vehicle-mounted camera image through a dual-branch convolutional neural network to obtain the first feature map and the second feature map.

[0099] Specifically, the dual-branch convolutional neural network (CNN) processes two types of inputs with different spatial representations. The first branch CNN inputs a bird's-eye view, which, after multiple convolutions and pooling operations, outputs a first feature map containing the target's geometric contours, spatial distribution, and depth information. The second branch CNN inputs an image from an onboard camera, which, after multiple convolutions and pooling operations, outputs a second feature map containing the target's texture details, edge features, and color information. Both branches have identical network structures and independent parameters, each extracting features from its respective input data, ultimately outputting two feature maps of the same size.

[0100] The dual-branch network structure, by configuring independent feature extraction channels for each modality, enables each branch to adjust its parameters according to the unique distribution of the input data, thereby extracting more modality-specific features. By extracting high-level semantic features from the original input data, low-level pixel or grid information is transformed into semantically expressive feature representations, while maintaining the consistency of the feature maps of the two modalities in spatial scale, allowing subsequent fusion operations to be performed based on the same spatial coordinate system.

[0101] By separating and processing dual-modal input data, the modality bias problem of a single network when processing heterogeneous graph data is effectively avoided. At the same time, information interference caused by direct mixing is avoided, so that the geometric information of the LiDAR and the appearance information of the camera can be fully represented, providing input for subsequent multimodal feature fusion.

[0102] S203. Project the preset three-dimensional anchor frame onto the first feature map and the second feature map respectively, and obtain the first fused feature after multimodal feature fusion processing.

[0103] Specifically, a preset 3D anchor frame is used as the initial template for the candidate detection region. It is predefined in the detection space. Each anchor frame includes spatial parameters to characterize the candidate region, so as to cover the spatial distribution of the target that may appear in the scene.

[0104] The preset 3D anchor frames are projected onto the corresponding spatial positions of the first feature map and the second feature map, respectively. Local feature regions corresponding to each anchor frame are cropped from the first feature map and the second feature map, and pooling operation is performed on each local feature region. The two pooled local features are input into the convolutional layer for dimensionality reduction processing, and then fusion is completed by element-wise averaging to obtain the first fused feature.

[0105] Preset anchor boxes cover different possible locations and scales of the target in the bird's-eye view space. By projecting the anchor boxes onto the bimodal feature map, a correspondence between geometric spatial location and semantic features is established. Pooling operations unify variable-sized local feature regions into fixed-size feature blocks, eliminating the inconsistency in feature dimensions caused by differences in target scale and anchor box size. Convolutional dimensionality reduction reduces the number of feature channels, lowering the computational complexity of subsequent processing. Element-wise average fusion performs pixel-level merging at corresponding spatial locations of the two modal features, jointly encoding geometric and appearance information in the same spatial coordinates.

[0106] By achieving the first fusion of bimodal features during the candidate box generation stage, each spatial location covered by the preset anchor box simultaneously obtains the geometric feature response from the LiDAR and the appearance feature response from the camera. Through anchor box projection and pooling processing, the bimodal features are precisely aligned in spatial location, avoiding the computational redundancy caused by dense fusion of the entire feature map.

[0107] S204. Perform regression screening on the first fusion feature to obtain multiple three-dimensional candidate regions.

[0108] Specifically, the first fused feature is input into the regression network. A fully connected layer calculates the positional and size offsets of each anchor box relative to the true target bounding box. Simultaneously, a binary classification network calculates the confidence score of each anchor box indicating that it contains the target. The anchor boxes are then corrected based on the positional and size offsets to obtain corrected 3D bounding boxes. The corrected bounding boxes are sorted from highest to lowest confidence score. The top-scoring bounding boxes are selected, and non-maximum suppression (NMS) is applied to this set to remove redundant bounding boxes with overlap exceeding a preset threshold. The retained bounding boxes are then output as 3D candidate regions.

[0109] The preset 3D anchor boxes have fixed positions and sizes, which deviate from the true target bounding boxes. A regression network learns the mapping relationship from the anchor boxes to the true bounding boxes, allowing for fine-tuning of the anchor box positions and sizes. The confidence score reflects the probability that the anchor box region contains the target, used to distinguish foreground and background regions. Non-maximum suppression removes duplicate candidate boxes based on overlap, retaining the candidate box with the highest confidence score to avoid multiple detection responses for the same target.

[0110] By filtering out high-quality candidate regions that may contain the target from densely distributed preset anchor boxes, the initial screening and position correction from anchor boxes to candidate regions are completed. This effectively filters out a large number of negative sample anchor boxes that do not contain the target, reducing the number of candidate boxes from thousands to hundreds and significantly reducing the computational load in subsequent detection stages.

[0111] S205. Project the three-dimensional candidate regions onto the first feature map and the second feature map respectively, and obtain the second fused feature after multimodal feature fusion processing.

[0112] Specifically, the three-dimensional candidate regions retained after screening are mapped to the corresponding positions of the first feature map and the second feature map in the bird's-eye view space and the image space, respectively. Local feature regions corresponding to each candidate region are cropped from the first feature map and the second feature map, and pooling operation is performed on each local feature region. The two pooled local features are input into the convolutional layer for dimensionality reduction processing, and then fusion is completed by element-wise averaging to obtain the second fused feature.

[0113] Candidate regions are rough estimates of the possible locations of the target, and their spatial extent is closer to the true target boundary than the initial anchor box. Projecting these candidate regions onto the bimodal feature map and extracting local features at the corresponding locations focuses the feature extraction range on the local region where the potential target is located, effectively eliminating interference from the background region. Compared to the anchor box projection pooling in S203, this stage uses fewer candidate regions with more precise locations, resulting in more spatially semantically focused feature blocks. Element-wise averaging fusion of the bimodal features is then performed again, further complementing and enhancing geometric and appearance information within a more precise spatial range.

[0114] By performing a second bimodal feature fusion based on the filtered candidate regions during the target detection stage, the spatial focus range of the fused features is compressed from dense anchor boxes to precise candidate regions, thus completing the joint encoding of geometric and appearance information at a finer spatial scale.

[0115] S206. Generate three-dimensional target detection results based on the second fusion feature.

[0116] Specifically, the second fused feature is input into the detection network, and bounding box regression, target classification, and heading angle estimation are performed through multiple parallel fully connected layers. The bounding box regression branch outputs the fine offset of the candidate region relative to the ground truth target bounding box, and performs secondary correction on the position and size of the candidate region. The target classification branch outputs the probability distribution of the target belonging to each preset category, and selects the category with the highest probability as the category label for the target. The heading angle estimation branch outputs the target's heading angle parameter. The corrected bounding box, category label, and heading angle parameter are combined to generate the 3D target detection result.

[0117] Based on the geometric and appearance information jointly encoded in the second fusion feature, the detection network simultaneously solves three sub-problems: target localization, classification, and orientation estimation through a multi-task learning architecture. Bounding box regression compensates for residual biases between candidate regions and the ground truth targets using fine-grained offsets, further improving localization accuracy. Target classification utilizes shape and texture information from the fusion feature for category discrimination. Heading angle estimation uses the spatial distribution symmetry of targets in the fusion feature for orientation prediction. The three tasks share the same fusion feature input and output corresponding prediction results through their respective network branches. During training, a multi-task loss function is used for joint optimization, enabling mutual constraints and promotion among the tasks.

[0118] By decoding the fused features into structured detection results that can be directly used by downstream modules, including the target's spatial location, 3D dimensions, category attributes, and orientation information, a complete transformation from raw sensor data to environmental semantic information is achieved. Through the simultaneous output of three tasks—bounding box offset correction, category classification, and heading angle estimation—the detection results possess complete spatial and semantic attributes, providing accurate and comprehensive environmental perception input for subsequent path planning and decision control modules.

[0119] The dual-branch network provided in this application extracts features from both the LiDAR bird's-eye view and the camera image. In the candidate box generation stage, a pooling fusion guided by a preset anchor box is introduced. After the candidate space is compressed by regression screening, pooling fusion guided by the candidate region is performed again in the detection stage. This allows the geometric features and appearance features to undergo two alignments and complementary enhancements in the spatial dimension, from coarse to fine. When information is missing in a single modality in an occluded scene, the other modality can effectively compensate, thereby improving the accuracy and robustness of 3D target detection in complex traffic scenes. At the same time, the entire fusion process is based on pooling and element-wise operations, without adding additional network parameters, and is adapted to the real-time perception requirements of vehicle-mounted embedded platforms.

[0120] Figure 3 Flowchart of the multimodal three-dimensional target detection method provided in this application Figure 2 ,like Figure 3 As shown, this embodiment, based on the above embodiments, provides a detailed description of the multimodal 3D target detection method, including a multimodal feature fusion processing procedure for generating the first fusion feature and the second fusion feature, comprising:

[0121] S301 performs convolutional dimensionality reduction on the two local features output by pooling after projection, resulting in two dimensionality-reduced features.

[0122] S302, perform element-wise averaging on the two dimensionality-reduced features to obtain the fused features.

[0123] Specifically, in generating the first fused feature: After projecting the preset 3D anchor boxes onto the first feature map and the second feature map respectively, the local feature regions corresponding to each anchor box are cropped from the first feature map and output as the first local feature through pooling; the local feature regions corresponding to each anchor box are cropped from the second feature map and output as the second local feature through pooling. The first and second local features are then input into a preset 1x1 convolutional layer. This convolutional layer performs channel dimension compression transformation on the input features, outputting the first and second dimensionality-reduced features, with the two dimensionality-reduced features having the same number of channels. The first and second dimensionality-reduced features are summed at each spatial location and divided by two. After traversing all spatial locations and all channels, they are recombined into a new feature tensor, which is the first fused feature output by element-wise averaging.

[0124] Preset 3D anchor frames are densely arranged to cover the bird's-eye view space, with numerous anchor frames evenly distributed in space. The size of the local feature regions corresponding to each anchor frame varies due to differences in anchor frame scale. Pooling operations unify these variable-sized local regions into feature blocks of fixed size, eliminating the inconsistency in feature dimensions caused by differences in anchor frame scale. One-to-one convolution linearly combines features along the channel dimension, compressing high-dimensional local features to a lower dimension, while learningable channel weights are used for feature selection. Element-wise averaging numerically merges the bimodal features at corresponding spatial locations, jointly encoding the geometric and apparent responses into a unified feature representation, ensuring that each location in the fused feature simultaneously reflects the degree of response of both modalities to that spatial location.

[0125] In the candidate box generation stage, densely distributed pre-defined anchor boxes are spatially correlated with bimodal feature maps. Through concatenated operations of pooling, dimensionality reduction, and element-wise averaging, the bimodal local features corresponding to a large number of anchor boxes are efficiently converted into a first fused feature in a unified format, providing a feature foundation for subsequent regression screening. The number of dense anchor boxes is typically in the thousands to tens of thousands. Pooling and dimensionality reduction operations compress the high-dimensional local features corresponding to each anchor box into low-dimensional feature vectors, significantly reducing the amount of data processed by the subsequent regression network. Element-wise averaging fusion completes modality merging in a parameter-free manner, ensuring that the first fused feature maintains the complementarity of geometric and appearance information without increasing network parameters, providing computational feasibility for large-scale parallel processing in the candidate box generation stage.

[0126] In generating the second fusion feature: the filtered and retained 3D candidate regions are projected onto the first feature map and the second feature map, respectively. Local feature regions corresponding to each candidate region are cropped from the first feature map and output as third-path local features after pooling. Similarly, local feature regions corresponding to each candidate region are cropped from the second feature map and output as fourth-path local features after pooling. The third and fourth local features are then input into a preset 1x1 convolutional layer, which uses the same number of output channels as the first stage, outputting the dimensionality-reduced third and fourth features. The third and fourth dimensionality-reduced features are summed at each spatial location and divided by two. After traversing all spatial locations and all channels, they are recombined into a new feature tensor, which is the second fusion feature output by element-wise averaging.

[0127] Candidate regions are sparse sets refined through regression correction and confidence screening. Their number is far less than the initial anchor boxes, and their spatial localization accuracy is higher. The local feature regions corresponding to the candidate regions are more spatially semantically focused, eliminating feature interference from a large number of background regions. Pooling operations unify the local features corresponding to the candidate regions to a fixed size, perform channel compression using a one-to-one convolution, and achieve spatial correspondence fusion of bimodal features through element-wise averaging. Compared to the first stage, the local features involved in the fusion in the second stage originate from more precise spatial locations, resulting in higher spatial alignment accuracy of their geometric and appearance information, and stronger semantic discriminative power in the fused features.

[0128] During the detection phase, a second bimodal feature fusion is performed based on candidate regions with more precise spatial locations. This compresses the spatial focus of the fused features from dense anchor boxes to sparse candidate regions, completing the joint encoding of geometric and appearance information at a finer spatial scale. This provides high-quality fused feature input for the final target classification and fine-grained bounding box regression.

[0129] The number of candidate regions is far less than the number of anchor boxes in the first stage, so the computational cost of the second stage is significantly lower than that of the first stage. Since the spatial localization accuracy of the candidate regions is higher than that of the initial anchor boxes, the second fused features are more focused and accurate in spatial semantics, avoiding feature interference from irrelevant background regions. This enables the subsequent detection network to perform fine-grained bounding box regression, class discrimination, and heading angle estimation based on high-quality fused features.

[0130] This invention performs multimodal feature fusion processing in two stages. In the candidate box generation stage, coarse-grained fusion is performed based on dense anchor boxes to cover all possible spatial locations of the target to ensure recall. In the detection stage, fine-grained fusion is performed based on sparse candidate regions to focus on high-confidence spatial regions to improve localization accuracy. The two stages of fusion share the same convolutional dimensionality reduction and element-wise averaging operation mechanism, achieving coarse-to-fine feature alignment and modal complementarity while maintaining computational lightweightness.

[0131] Figure 4 Flowchart of the multimodal three-dimensional target detection method provided in this application Figure 3 ,like Figure 4 As shown, this embodiment, based on the above embodiments, provides a detailed description of the multimodal three-dimensional target detection method, which includes:

[0132] S401, perform bounding box regression and target calculation on the first fused feature to obtain the bounding box offset and target score.

[0133] Specifically, the first fused feature is input into a regression branch network, which consists of multiple stacked fully connected layers. Its output dimension matches the parameter space dimension of the preset 3D anchor boxes. The first fused feature is calculated via forward propagation in the regression branch network, outputting the bounding box offsets for each set of anchor boxes. These offsets include the horizontal coordinate offset of the center point, the spatial size scaling, and the orientation angle offset. The same first fused feature is then input in parallel into a target classification branch network, which consists of multiple stacked fully connected layers. Its output dimension is two-dimensional, and after activation function mapping, it outputs the target score for each anchor box. The target score represents the confidence that the anchor box region contains a target.

[0134] Preset 3D anchor boxes are uniformly distributed within the bird's-eye view space, with systematic deviations in position and size compared to the true target bounding boxes. A regression branch network compensates for and corrects the spatial parameters of the anchor boxes by learning a mapping function from feature maps to offsets. A target classification branch network uses binary classification to determine whether the area covered by the anchor box belongs to the foreground or background; its output score reflects the probability that the region contains any type of target. Both branches share the same fused feature input, but their parameters are updated independently, optimizing for different task objectives.

[0135] The offset parameter used to correct the spatial position of the anchor frame and the confidence score used to analyze the effectiveness of the anchor frame are simultaneously decoded from the fused features. The regression branch and the classification branch share the fused feature representation. Through multi-task learning, the synchronous output of position correction and quality analysis is achieved, avoiding the redundant overhead of repeatedly calculating the same feature.

[0136] S402, the preset 3D anchor box is corrected according to the bounding box offset and the target score is combined to suppress and filter, resulting in multiple 3D candidate regions.

[0137] Specifically, the bounding box offsets output by the regression branch network are obtained. The horizontal coordinate offset, size scaling, and orientation angle offset from these offsets are then superimposed onto the corresponding parameters of each preset 3D anchor box to obtain the corrected 3D bounding boxes. The target score output by the target classification branch network is obtained, and the corrected 3D bounding boxes are sorted from highest to lowest target score. The top-ranked bounding boxes are selected as the initial screening results. The bounding boxes in the initial screening results are compared pairwise according to their spatial overlap, and the intersection-union ratio (IUR) is calculated. When the IUR exceeds a preset threshold, the bounding box with the lower target score is removed, and the bounding box with the higher score is retained. The final retained bounding boxes are output as 3D candidate regions.

[0138] The correction operation moves the anchor boxes from preset fixed positions to the actual target positions, making the bounding boxes more closely match the actual spatial range of the target. Sorting and selection retains the bounding boxes with the highest target-specific scores, filtering out most low-confidence background anchor boxes. Non-maximum suppression eliminates redundancy based on the spatial overlap between bounding boxes, retaining the highest-scoring prediction among multiple high-scoring bounding boxes densely distributed around the same target, thus avoiding multiple responses from a single target.

[0139] The densely distributed pre-defined anchor boxes are compressed into a sparse set of high-quality candidate regions, completing the transition from candidate box generation to candidate region selection, providing a concise and accurate input for the detection stage. Through a three-level selection process of regression correction, confidence ranking, and overlap suppression, the number of candidate regions is effectively reduced while maintaining a high target recall rate, thus reducing the computational burden of subsequent detection stages.

[0140] This invention embodiment processes the first fusion feature in parallel through regression and classification branches, and outputs bounding box offsets and target scores simultaneously. After correcting the anchor box spatial parameters using the offsets, it combines the target scores for sorting and overlap suppression filtering. While maintaining a high target recall rate, it compresses the number of candidate regions to a usable range, providing accurate and concise candidate region inputs for the detection stage.

[0141] Figure 5 Flowchart of the multimodal three-dimensional target detection method provided in this application Figure 4 ,like Figure 5 As shown, this embodiment, based on the above embodiments, provides a detailed description of the multimodal three-dimensional target detection method, which includes:

[0142] S501, based on the second fusion feature, the bounding box parameters, category information and heading angle information are calculated.

[0143] Specifically, the second fused feature is input into the detection network, which consists of multiple parallel branches. The first branch is the bounding box regression branch, which maps the second fused feature through a fully connected layer and outputs the fine offset of the candidate region relative to the ground truth target bounding box, including the horizontal coordinate offset of the center point, the spatial size scaling, and the orientation angle offset. The second branch is the classification branch, which maps the second fused feature through a fully connected layer and outputs the probability distribution of the target belonging to multiple preset categories. The third branch is the heading angle estimation branch, which maps the second fused feature through a fully connected layer and outputs the target's orientation angle parameter. The three branches share the same second fused feature input and independently calculate their respective output results.

[0144] The second fusion feature focuses on candidate regions in the spatial dimension, with more refined encoded geometric and appearance information. The bounding box regression branch compensates for residual biases based on the candidate regions, making the bounding boxes fit the true contours of the target even better. The classification branch uses the shape geometry and appearance texture from the fusion features to determine the category and outputs probability values ​​for each category. The heading angle estimation branch uses the symmetry features of the target's spatial distribution to predict the heading angle. The three tasks share the underlying feature representation, and feature sharing and inter-task constraints are achieved through a multi-task learning framework.

[0145] By decoding the second fusion feature into a structured intermediate detection result, including refined spatial location parameters, category probability distribution, and orientation angle parameters, the necessary information is provided for generating a complete detection result. The three parallel branches simultaneously output localization, classification, and orientation information, making full use of the multimodal complementary information encoded in the fusion feature, avoiding error accumulation caused by serial processing, and improving the accuracy of the final detection result.

[0146] S502 generates 3D target detection results based on bounding box parameters, category information, and heading angle information.

[0147] Specifically, the fine offset of the bounding box regression branch is obtained and superimposed onto the bounding box parameters corresponding to the candidate regions to obtain the final 3D bounding box position and size. The probability distribution of the classification branch is obtained, and the category with the highest probability value is selected as the category label for the target, while recording the corresponding confidence score. The orientation angle parameters of the heading angle estimation branch are obtained and associated with the 3D bounding boxes. The 3D bounding boxes, category labels, confidence scores, and orientation angle parameters corresponding to each candidate region are combined to construct a structured 3D target detection result.

[0148] Fine-tuned offsets compensate for residual spatial discrepancies between candidate regions and the ground truth, ensuring the final bounding box approximates the true value in both position and size. Probability maximization selects the most likely class label, transforming the continuous probability distribution into a discrete class output. The joint output of the orientation angle parameter and the bounding box spatial parameters provides a complete spatial pose description of the target. Combination operations integrate the multi-branch outputs into a unified detection result format, facilitating parsing and use by downstream modules.

[0149] The scattered information output from each branch of the detection network is integrated into a complete 3D target detection result, including the target's precise location, size, category, and orientation, forming environmental perception data that can be directly used in path planning and decision control processes. Through fine-tuning and combination of multi-branch information, the output 3D detection result has complete spatial and semantic attributes, which can accurately describe the target's position, size, category, and pose in 3D space.

[0150] In this embodiment of the invention, a multi-task detection network is used to decode the second fused feature in parallel. The bounding box regression branch, classification branch and heading angle estimation branch simultaneously output fine offset, category probability distribution and orientation angle parameters. The outputs of each branch are combined into a three-dimensional detection result containing complete spatial semantic attributes, realizing the end-to-end conversion from fused features to structured environmental perception data.

[0151] Figure 6 Flowchart of the multimodal three-dimensional target detection method provided in this application Figure 5 ,like Figure 6 As shown, this embodiment, based on the above embodiments, provides a detailed explanation of the training process of the dual-branch convolutional neural network in the multimodal 3D target detection method. The method includes:

[0152] S601: Obtain a preset training dataset, input it into a dual-branch convolutional neural network, extract features, and fuse them to obtain multi-stage fused features.

[0153] Specifically, a pre-defined training dataset is obtained, which includes LiDAR point cloud data, vehicle-mounted camera image data, and corresponding 3D ground truth annotations. The LiDAR point cloud data is projected to generate a bird's-eye view. This bird's-eye view and the vehicle-mounted camera image are input in parallel into a dual-branch convolutional neural network. After feature extraction, a first feature map and a second feature map are output. In the candidate box generation stage, pre-defined 3D anchor boxes are projected onto the first and second feature maps. After pooling and fusion, the first fused feature of the candidate box generation stage is output. In the detection stage, 3D candidate regions are obtained through regression filtering based on the first fused feature. These 3D candidate regions are projected onto the first and second feature maps. After pooling and fusion, the second fused feature of the detection stage is output. The first and second fused features are used together as the multi-stage fused feature.

[0154] During the training phase, labeled ground truth values ​​are used to jointly optimize the parameters of each module in the network. The dual-branch network generates fused features from two stages during forward propagation: candidate box generation and detection. These two stages correspond to the coarse-screening and fine-tuning processes of target localization, respectively, and their fused features encode modal complementarity information at different scales. Incorporating these two-stage fused features into the subsequent attention modulation allows the attention mechanism to regulate feature responses at multiple scales.

[0155] By obtaining multi-scale fusion features from the candidate box generation and detection stages of the dual-branch network through forward propagation, the reverse attention mechanism is provided with feature modulation objects at different semantic levels. The multi-stage fusion features cover different spatial scales from the anchor box level to the candidate region level, enabling attention modulation to guide the network to focus on key feature regions at multiple levels and improve the comprehensiveness of the modulation effect.

[0156] S602 generates a reverse attention heatmap based on multi-stage fusion features and completes feature modulation.

[0157] Specifically, the multi-stage fusion features are input into the attention modulation module. During the forward propagation of the training phase, the attention modulation module performs gradient calculation on the fusion features, generates an initial attention heatmap based on the gradient response, performs numerical inversion on the initial attention heatmap and then performs thresholding to obtain the inverse attention weight map. The inverse attention weight map is multiplied element-wise with the original fusion features to output the modulated fusion features.

[0158] During gradient backpropagation, regions with strong gradient responses in the fused features correspond to feature locations that contribute significantly to the loss function. By inverting the gradient responses, regions with high responses are assigned low weights, and regions with low responses are assigned high weights, forcing the network to pay more attention to suppressed regions in subsequent iterations. Thresholding binarizes or softly crops the weight map to remove interference from noisy responses. Element-wise multiplication applies the modulated weights to the original features, completing the reweighting of the feature space.

[0159] By dynamically adjusting the spatial and channel response distribution of the fused features during the training phase, the network gradually focuses on feature regions that were originally ignored but contain discriminative information during the iterative optimization process. The inverse attention modulation does not introduce additional learnable parameters, but only achieves feature optimization through gradient-guided weight redistribution. The modulated features guide the network parameters to be updated in a more robust direction in subsequent loss calculations.

[0160] S603 calculates the loss and updates the network weights based on the modulated fusion features.

[0161] Specifically, the modulated fused features are input into each task branch of the detection network to calculate the bounding box regression loss, classification loss, and heading angle loss. Preset weight coefficients are assigned to each of the three losses, and the weighted loss values ​​are summed to obtain the total loss value. The gradient of the total loss value relative to the learnable parameters of each layer of the network is calculated through backpropagation, and the learnable parameters are updated using the gradient descent optimization algorithm.

[0162] Bounding box regression loss measures the spatial deviation between the predicted bounding box and the ground truth bounding box; classification loss measures the difference between the predicted class probability distribution and the ground truth class; and heading angle loss measures the angular deviation between the predicted heading and the ground truth heading. These three losses constrain the network's learning objective from different dimensions. Weighted summation unifies the multi-task losses into a scalar total loss, facilitating gradient calculation. The gradient descent optimizer uses the derivative of the total loss with respect to the parameters to guide parameter updates along the direction of loss reduction.

[0163] By jointly optimizing the network parameters through a multi-task loss function, the dual-branch network, the fusion module, and the branches of the detection network converge collaboratively during training. After multiple rounds of iterative training, the parameters of each module of the network are gradually optimized under the guidance of the reverse attention mechanism, thereby improving detection accuracy while maintaining computational efficiency in the inference stage.

[0164] Figure 7 Flowchart of the multimodal three-dimensional target detection method provided in this application Figure 6 ,like Figure 7 As shown, in some optional embodiments, the above-mentioned S602 includes:

[0165] S6021, gradient calculation is performed on multi-stage fusion features based on gradient-weighted class activation mapping to obtain an initial attention heatmap.

[0166] Specifically, the multi-stage fused features are input into the detection network and forward propagation is performed to calculate the total loss. Backpropagation is then executed to obtain the gradient values ​​of the total loss relative to each spatial location and channel in the multi-stage fused features. Global average pooling is then performed on the gradient values ​​within each channel to obtain the gradient weight coefficient for that channel. Finally, the gradient weight coefficients of each channel are multiplied element-wise with the corresponding channel's fused features, and the results are summed along the channel dimension to generate an initial attention heatmap.

[0167] Gradient-weighted activation mapping quantifies the contribution of each channel to the loss function using gradient information from backpropagation. Channels with larger gradient values ​​indicate that their feature responses are closely related to the loss value and have a significant impact on the detection results. Global average pooling compresses the gradient distribution within a channel into a single weight coefficient, used to measure the overall importance of that channel. The weighted summation operation overlays the channel importance information onto the spatial dimension, generating a heatmap reflecting the importance of spatial locations.

[0168] By extracting the saliency distribution in the spatial dimension from the multi-stage fusion features, an attention heatmap is generated to identify important regions in the feature map. Gradient-weighted class activation mapping can generate attention heatmaps without additional network parameters. The calculation process only involves gradient information in backpropagation and does not increase the computational burden of forward propagation.

[0169] S6022, perform inversion processing on the initial attention heatmap to obtain the inverse attention heatmap.

[0170] Specifically, an initial attention heatmap is obtained. The numerical distribution in the initial attention heatmap reflects the importance of each spatial location to the loss function; the higher the value, the greater the contribution of that location to the detection result. The initial attention heatmap is then inverted element by element. Specifically, the negative value of each position in the heatmap is calculated, and the inverted values ​​are normalized back to a preset numerical range to obtain the reverse attention heatmap.

[0171] The inversion operation transforms high-value regions in the original heatmap into low-value regions and vice versa, achieving a complete reversal of attention focus. Normalization ensures that the numerical distribution after inversion remains within a stable range, preventing numerical overflow or gradient vanishing. The inverted heatmap guides the network to focus on regions with weaker responses in the original network during subsequent training, thereby prompting the network to uncover overlooked complementary features.

[0172] By converting the forward attention heatmap into a reverse attention weight distribution, subsequent thresholding and feature modulation can be applied to the suppressed spatial locations in the original features. By inverting the process, the direction of attention modulation is made opposite to the direction of the original gradient response, forcing the network to explore feature regions that have not been fully learned during training, thereby enhancing the diversity of feature expression.

[0173] Figure 7 Flowchart of the multimodal three-dimensional target detection method provided in this application Figure 6 ,like Figure 7 As shown, in some optional embodiments, the above-mentioned S602 includes:

[0174] S6023, thresholding is performed on the reverse attention heatmap to obtain the first heatmap.

[0175] Specifically, the values ​​at each location in the backattention heatmap are distributed within a certain range. A threshold is set, which can be a predetermined fixed value or an adaptive value dynamically calculated based on the numerical distribution of the backattention heatmap. By comparing the values ​​at each location in the backattention heatmap with the threshold, locations above the threshold are marked as effective modulation locations, and locations below or equal to the threshold are marked as non-modulation locations. A first heatmap is generated based on the comparison results. In the first heatmap, the values ​​at effective modulation locations are assigned effective weight values, and the values ​​at non-modulation locations are assigned unit weight values.

[0176] In the reverse attention heatmap, some locations have small numerical amplitudes, and the corresponding feature responses may originate from background noise rather than target features. Modulating these areas could introduce interference. Thresholding filters out low-confidence attention responses by setting a numerical threshold, retaining only regions with saliency meeting a preset standard for modulation, thus reducing the negative impact of noise on the training process. The unit weight value in the first heatmap keeps the features of non-modulated locations unchanged, avoiding unnecessary feature distortion.

[0177] By filtering out noise and sparsifying the inverse attention heatmap, a weight map containing effective modulation positions is generated; thresholding suppresses the interference of low-confidence attention responses, enabling feature modulation to focus on spatial regions with clear semantic meaning, thereby improving the accuracy and effectiveness of modulation operations.

[0178] S6024 multiplies the first heatmap with the multi-stage fusion features element by element to complete feature modulation.

[0179] Specifically, the first heatmap generated after thresholding is obtained, and its spatial resolution is consistent with that of the multi-stage fusion feature. The weight values ​​of each spatial location in the first heatmap are multiplied by the corresponding vectors in the multi-stage fusion feature, with each channel sharing the weight value of the same spatial location. After element-wise multiplication of all spatial locations, the modulated fusion feature is output.

[0180] Element-wise multiplication scales the response intensity of the fused features using weights corresponding to spatial locations. In the first heatmap, the weights at effective modulation locations deviate from unit values, enhancing or suppressing the feature response in that region. At non-modulation locations, the weights are unit values, and the feature response remains unchanged. In the modulated fused features, feature regions previously ignored by the original network are given higher response intensity, while overly focused feature regions are appropriately suppressed, achieving a rebalancing of the feature space.

[0181] The first heatmap obtained by thresholding is applied to the fused features in the form of spatial weights to achieve selective enhancement and suppression of feature responses; element-wise multiplication is a parameterless operation that does not introduce additional learnable parameters, thus maintaining computational efficiency in the inference stage while modulating the feature distribution.

[0182] Figure 8 Flowchart of the multimodal three-dimensional target detection method provided in this application Figure 7 ,like Figure 8 As shown, in some optional embodiments, the above-mentioned S603 includes:

[0183] S6031, calculate the bounding box regression loss, classification loss and heading angle loss based on the modulated fusion features respectively.

[0184] Specifically, the modulated fused features are input into the bounding box regression branch of the detection network to calculate the difference between the predicted bounding box and the ground truth bounding box, and the bounding box regression loss is calculated based on the difference. The modulated fused features are input into the classification branch of the detection network to calculate the difference between the predicted class probability distribution and the ground truth class label, and the classification loss is calculated based on the difference. The modulated fused features are input into the heading angle estimation branch of the detection network to calculate the angle difference between the predicted heading angle and the ground truth heading angle, and the heading angle loss is calculated based on the angle difference.

[0185] The bounding box regression loss uses a smoothed absolute value loss function, which exhibits squared loss characteristics when the prediction bias is small and absolute value loss characteristics when the bias is large. This ensures gradient smoothness with small biases while limiting the gradient magnitude with large biases. The classification loss uses a negative log-likelihood loss function, measuring the cross-entropy between the predicted and true distributions. The heading angle loss uses a smoothed absolute value loss of the angle difference, taking into account the periodic symmetry of the heading angle. These three losses constrain the optimization direction of network parameters from different supervisory signals.

[0186] By quantifying the prediction errors of the three task dimensions—bounding box localization accuracy, class discrimination accuracy, and orientation angle estimation accuracy—various components are provided for the calculation of the total loss. The three task losses supervise the output quality of different branches of the network, enabling the detection network to simultaneously optimize its localization, classification, and orientation estimation capabilities during training, thus avoiding bias caused by a single task dominating the network.

[0187] S6032 obtains the total loss by weighted summation and updates the network weights based on the total loss.

[0188] Specifically, the values ​​of the bounding box regression loss, classification loss, and heading angle loss are obtained, along with preset weight coefficients for each loss. The bounding box regression loss is multiplied by its weight coefficient to obtain the weighted regression loss; the classification loss is multiplied by its weight coefficient to obtain the weighted classification loss; and the heading angle loss is multiplied by its weight coefficient to obtain the weighted heading angle loss. The three weighted losses are summed to obtain the total loss value. The gradient of the total loss value relative to the parameters of each network layer is calculated through backpropagation, and the optimizer updates the network parameters based on the gradient direction and magnitude.

[0189] Different tasks exhibit varying loss magnitudes and convergence speeds. By recalibrating each loss function using preset weighting coefficients configured according to the optimization priorities of each task, a balance is maintained during joint optimization. Weighted summation merges multiple optimization objectives into a single scalar objective, enabling backpropagation to simultaneously consider the optimization directions of each task. The gradient descent algorithm utilizes the partial derivatives of the parameters with respect to the total loss to guide parameter updates, ensuring coordinated convergence of all tasks during iteration.

[0190] Through multi-task weighted joint optimization, the network modules gradually converge to a balanced state that takes into account localization accuracy, classification accuracy, and orientation estimation accuracy during the training process; the multi-task loss is unified into a total loss and drives the network parameter update, completing the closed loop of parameter optimization in the training iteration.

[0191] This invention employs a pre-set training dataset to drive the forward propagation of a dual-branch network to obtain multi-stage fusion features. It utilizes gradient-weighted class activation mapping to generate an initial attention heatmap and performs numerical inversion and thresholding. The reverse attention weight map is multiplied element-wise with the fusion features to complete feature modulation. Then, based on the modulated features, the multi-task loss is calculated and the network weights are updated by weighted summation. This allows the network to gradually focus on complementary feature regions ignored by the original network during training, thereby improving detection accuracy without increasing the parameters and computational load of the inference stage.

[0192] Figure 9 This is a schematic diagram of the structure of the multimodal three-dimensional target detection device provided in this application, as shown below. Figure 9 As shown, the multimodal three-dimensional target detection device 90 provided in this embodiment includes:

[0193] Acquisition module 901 is used to acquire bird's-eye view images from the LiDAR and images from the vehicle-mounted camera;

[0194] The feature analysis module 902 is used to extract features from the LiDAR bird's-eye view and the vehicle camera image through a dual-branch convolutional neural network to obtain a first feature map and a second feature map.

[0195] Processing module 903 is used to project a preset 3D anchor frame onto a first feature map and a second feature map respectively, and obtain a first fused feature after multimodal feature fusion processing;

[0196] The filtering module 904 is used to perform regression filtering on the first fusion feature to obtain multiple three-dimensional candidate regions;

[0197] The processing module 903 is also used to project the three-dimensional candidate region onto the first feature map and the second feature map respectively, and obtain the second fused feature after multimodal feature fusion processing;

[0198] The generation module 905 is used to generate three-dimensional target detection results based on the second fusion feature.

[0199] In one possible implementation, the processing module 903 is specifically used for:

[0200] The two local features output by pooling after projection are convolved to reduce dimensionality, resulting in two dimensionality-reduced features.

[0201] The fused features are obtained by averaging the two dimensionality-reduced features element by element.

[0202] In one possible implementation, the filtering module 904 is specifically used for:

[0203] Boundary box regression and target calculation are performed on the first fused feature to obtain the boundary box offset and target score;

[0204] The preset 3D anchor frame is corrected based on the bounding box offset, and suppression filtering is performed in combination with the target score to obtain multiple 3D candidate regions.

[0205] In one possible implementation, the generation module 905 is specifically used for:

[0206] The bounding box parameters, category information, and heading angle information are calculated based on the second fusion feature.

[0207] 3D target detection results are generated based on bounding box parameters, category information, and heading angle information.

[0208] In one possible implementation, it also includes:

[0209] A preset training dataset is input into a dual-branch convolutional neural network to obtain multi-stage fused features;

[0210] Generate a reverse attention heatmap based on multi-stage fusion features and complete feature modulation;

[0211] The loss is calculated based on the modulated fusion features, and the network weights are updated.

[0212] In one possible implementation, generating a reverse attention heatmap includes:

[0213] Gradient calculation is performed on multi-stage fused features based on gradient-weighted class activation mapping to obtain an initial attention heatmap;

[0214] The initial attention heatmap is inverted to obtain the reverse attention heatmap.

[0215] In one possible implementation, feature modulation is performed, including:

[0216] Thresholding is applied to the reverse attention heatmap to obtain the first heatmap;

[0217] The first heatmap is multiplied element-wise with the multi-stage fusion features to complete feature modulation.

[0218] In one possible implementation, the loss is calculated based on the modulated fusion features, including:

[0219] Based on the modulated fusion features, the bounding box regression loss, classification loss and heading angle loss are calculated respectively.

[0220] The total loss is obtained by weighted summation, and the network weights are updated based on the total loss.

[0221] The multimodal three-dimensional target detection device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0222] Figure 10 A schematic diagram of the structure of the electronic device provided in this application. Figure 10 As shown, the electronic device 100 provided in this embodiment includes at least one processor 1001 and a memory 1002. Optionally, the electronic device 100 further includes a communication component 1003. The processor 1001, memory 1002, and communication component 1003 are connected via a bus 1004.

[0223] In the specific implementation process, at least one processor 1001 executes computer execution instructions stored in memory 1002, causing at least one processor 1001 to perform the above method.

[0224] The specific implementation process of processor 1001 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0225] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0226] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0227] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0228] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0229] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0230] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0231] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0232] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0233] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0234] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0235] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0236] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0237] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A multimodal three-dimensional target detection method, characterized in that, include: Acquire aerial views from LiDAR and images from vehicle-mounted cameras; The first feature map and the second feature map are obtained by extracting features from the bird's-eye view of the LiDAR and the image of the vehicle camera through a dual-branch convolutional neural network. The preset 3D anchor frame is projected onto the first feature map and the second feature map respectively, and the first fused feature is obtained after multimodal feature fusion processing; The first fused feature is subjected to regression screening to obtain multiple three-dimensional candidate regions; The three-dimensional candidate regions are projected onto the first feature map and the second feature map respectively, and the second fused feature is obtained after multimodal feature fusion processing; Three-dimensional target detection results are generated based on the second fusion feature.

2. The method according to claim 1, characterized in that, Multimodal feature fusion processing includes: The two local features output by pooling after projection are convolved to reduce dimensionality, resulting in two dimensionality-reduced features. The two dimensionality-reduced features are averaged element-wise to obtain the fused features.

3. The method according to claim 1, characterized in that, Regression screening processes include: Boundary box regression and target calculation are performed on the first fused feature to obtain the boundary box offset and target score; The preset 3D anchor frame is corrected based on the bounding box offset, and suppression filtering is performed in combination with the target score to obtain multiple 3D candidate regions.

4. The method according to claim 1, characterized in that, Generate 3D target detection results, including: Based on the second fusion feature, the bounding box parameters, category information, and heading angle information are calculated. The three-dimensional target detection results are generated based on the bounding box parameters, category information, and heading angle information.

5. The method according to claim 1, characterized in that, Also includes: A preset training dataset is input into a dual-branch convolutional neural network to obtain multi-stage fused features; Generate a reverse attention heatmap based on multi-stage fusion features and complete feature modulation; The loss is calculated based on the modulated fusion features, and the network weights are updated.

6. The method according to claim 5, characterized in that, Generate a reverse attention heatmap, including: Gradient calculation is performed on the multi-stage fusion features based on gradient-weighted class activation mapping to obtain an initial attention heatmap; The initial attention heatmap is inverted to obtain the reverse attention heatmap.

7. The method according to claim 6, characterized in that, Complete the feature modulation, including: The reverse attention heatmap is thresholded to obtain a first heatmap; The first heatmap is multiplied element-wise with the multi-stage fusion feature to complete feature modulation.

8. The method according to claim 5, characterized in that, The loss is calculated based on the modulated fusion features, including: Based on the modulated fusion features, the bounding box regression loss, classification loss, and heading angle loss are calculated respectively. The total loss is obtained by weighted summation, and the network weights are updated based on the total loss.

9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-8.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-8.

Images