Multimodal spatio-temporal fusion three-dimensional target detection method and system

Abstract

This invention provides a multimodal spatiotemporal fusion method and system for 3D target detection, relating to the field of target detection technology. In the multimodal and spatiotemporal fusion architecture, it innovatively introduces decoding branches for each modality and a current quality signal autonomously constructed from each initial detection result. This current quality signal serves as a unified adjustment mechanism, dynamically allocating feature fusion weights in both the temporal fusion and cross-modal fusion stages, breaking the limitation of traditional fusion methods that overly rely on implicit learning. Even when facing common challenges in real-world autonomous driving scenarios such as sparse LiDAR point clouds, camera obstruction by rain and fog, and glare from strong light, it can accurately detect targets. Without adding any external supervision labels, it can significantly improve the overall robustness, stability, and reliable localization capability of 3D target detection in complex scenarios.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a multimodal spatiotemporal fusion three-dimensional target detection method and system. Background Technology

[0002] In scenarios such as autonomous driving and mobile robotics, 3D object detection is used to locate and identify surrounding traffic participants based on data collected by sensors, serving as a crucial foundation for environmental perception and decision-making control. Common sensors include LiDAR and cameras, with LiDAR providing accurate spatial geometric information and cameras providing rich semantic and textural information. By fusing information from LiDAR and cameras using multimodal fusion detection methods, detection accuracy and stability can be improved.

[0003] However, in real-world road environments, sensors are often affected by various degradation or failure factors. For example, lidar may experience sparse point clouds, limited field of view, partial occlusion, or loss of entire frames; cameras may suffer from missing viewpoints, image degradation due to adverse weather conditions such as rain, fog, and snow, or glare or obstruction from strong light. Existing multimodal fusion detection methods may experience significant performance degradation in sensor degradation scenarios, and may even lead to errors in target detection results due to negative fusion. Summary of the Invention

[0004] This invention provides a multimodal spatiotemporal fusion three-dimensional target detection method and system to address the deficiencies in related technologies.

[0005] This invention provides a multimodal spatiotemporal fusion three-dimensional target detection method, comprising:

[0006] Acquire multimodal sensor data at the current moment, wherein the multimodal sensor data includes at least lidar point cloud data and camera image data;

[0007] The lidar point cloud data and the camera image data are encoded separately to obtain multimodal feature codes;

[0008] Based on the predetermined target query features and the multimodal feature encoding, at least two current query features corresponding to the modal decoding branches are determined, and at least two initial detection results are determined based on each of the current query features; each initial detection result includes a category detection result;

[0009] Based on the detection results of each category, a current quality signal is constructed; the current quality signal is used to characterize the reliability of the current query feature under the corresponding modality decoding branch.

[0010] Based on each of the current quality signals, each of the current query features and the corresponding historical query features under the modal decoding branch are fused in a quality-aware time series to obtain the time series fused features under each of the modal decoding branches. Then, the time series fused features under each of the modal decoding branches are fused across modalities to obtain the comprehensive query features.

[0011] Based on the comprehensive query features, the 3D target detection result at the current moment is determined.

[0012] According to the multimodal spatiotemporal fusion three-dimensional target detection method provided by the present invention, each current quality signal includes a query-level quality signal and / or a frame-level quality signal, and each category detection result includes a category probability distribution;

[0013] The construction of the current quality signal based on the detection results of each category includes:

[0014] Based on the aforementioned category probability distribution, the detection confidence and detection uncertainty are determined.

[0015] Based on the detection confidence and the detection uncertainty, the query-level quality signal is determined, and based on the query-level quality signal, the frame-level quality signal is determined.

[0016] The multimodal spatiotemporal fusion three-dimensional target detection method provided by the present invention further includes:

[0017] Obtain historical information related to the corresponding historical query features under each modal decoding branch, wherein the historical information includes historical pose information, historical velocity information, and historical quality signal;

[0018] The step of performing quality-aware temporal fusion of each current query feature with the corresponding historical query features under the modal decoding branch based on each current quality signal to obtain temporal fusion features under each modal decoding branch includes:

[0019] For any modal decoding branch, based on the current quality signal corresponding to the modal decoding branch, a prior bias is applied, and the current query feature corresponding to the modal decoding branch and the historical query feature under the modal decoding branch are fused in a quality-aware time series to obtain the time series fused feature under the modal decoding branch.

[0020] The prior bias includes at least one of time decay prior, pose alignment prior, motion consistency prior, and history quality prior.

[0021] According to the multimodal spatiotemporal fusion three-dimensional target detection method provided by the present invention, the step of performing cross-modal fusion of the temporal fusion features under each modality decoding branch to obtain comprehensive query features includes:

[0022] Select a target modal decoding branch from each of the modal decoding branches, determine a query vector based on the temporal fusion features under the target modal decoding branch, and determine a key vector and a value vector based on the splicing or combination results of the temporal fusion features under each modal decoding branch.

[0023] Based on the current quality signals, the query vector, the key vector, and the value vector, a cross-modal attention mechanism is applied to perform cross-modal fusion on the temporal fusion features under each modal decoding branch to obtain the comprehensive query features.

[0024] According to the present invention, a multimodal spatiotemporal fusion three-dimensional target detection method is provided, wherein based on each current quality signal, the query vector, the key vector, and the value vector, a cross-modal attention mechanism is applied to perform cross-modal fusion of the temporal fusion features under each modal decoding branch to obtain the comprehensive query features, including:

[0025] Based on the query vector, the key vector, and the value vector, attention weights are calculated by applying the current quality signals.

[0026] Based on the attention weights, the temporal fusion features under each modal decoding branch are fused across modalities to obtain the comprehensive query features.

[0027] According to the multimodal spatiotemporal fusion three-dimensional target detection method provided by the present invention, each initial detection result further includes a target bounding box attribute detection result;

[0028] The step of performing cross-modal fusion of the temporal fusion features under each modal decoding branch to obtain comprehensive query features also includes:

[0029] Based on the target bounding box attribute detection results, a geometrically consistent prior is constructed;

[0030] Based on the geometric consistency prior, the temporal fusion features under each modal decoding branch are fused across modalities to obtain comprehensive query features.

[0031] According to the multimodal spatiotemporal fusion three-dimensional target detection method provided by the present invention, the step of determining the three-dimensional target detection result at the current moment based on the comprehensive query features includes:

[0032] The comprehensive query features are input into the 3D target detection head to obtain the 3D target detection result output by the 3D target detection head.

[0033] The present invention also provides a multimodal spatiotemporal fusion three-dimensional target detection system, comprising:

[0034] The data acquisition module is used to acquire multimodal sensor data at the current moment, wherein the multimodal sensor data includes at least lidar point cloud data and camera image data;

[0035] The encoding module is used to encode the lidar point cloud data and the camera image data respectively to obtain multimodal feature codes;

[0036] The decoding module is used to determine the current query features corresponding to at least two modality decoding branches based on the pre-determined target query features and the multimodal feature encoding, and to determine at least two initial detection results based on each of the current query features by applying the at least two modality decoding branches; each of the initial detection results includes a category detection result;

[0037] A quality signal construction module is used to construct a current quality signal based on the detection results of each category; the current quality signal is used to characterize the reliability of the current query feature under the corresponding modality decoding branch;

[0038] The feature fusion module is used to perform quality-aware temporal fusion of each current query feature and the corresponding historical query feature under the modal decoding branch based on each current quality signal, to obtain the temporal fusion feature under each modal decoding branch, and to perform cross-modal fusion of the temporal fusion feature under each modal decoding branch to obtain the comprehensive query feature;

[0039] The target detection module is used to determine the three-dimensional target detection result at the current moment based on the comprehensive query features.

[0040] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multimodal spatiotemporal fusion three-dimensional target detection method as described above.

[0041] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal spatiotemporal fusion three-dimensional target detection method as described above.

[0042] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multimodal spatiotemporal fusion three-dimensional target detection method as described above.

[0043] The multimodal spatiotemporal fusion 3D target detection method and system provided by this invention innovatively introduces each modal decoding branch and a current quality signal autonomously constructed from each initial detection result into the multimodal and spatiotemporal fusion architecture. This current quality signal serves as a unified adjustment mechanism, dynamically allocating feature fusion weights in both the temporal fusion and cross-modal fusion stages, breaking the limitation of traditional fusion methods that overly rely on implicit learning. Even when facing common single-modal or local sensor degradation problems in real-world autonomous driving scenarios, such as sparse LiDAR point clouds, camera obstruction by rain and fog, and strong light glare, it can accurately detect targets. Without adding any external supervision labels, it can significantly improve the overall robustness, stability, and reliable localization capability of 3D target detection in complex scenarios. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is one of the flowcharts of the multimodal spatiotemporal fusion three-dimensional target detection method provided by the present invention.

[0046] Figure 2 This is the second flowchart of the multimodal spatiotemporal fusion three-dimensional target detection method provided by the present invention.

[0047] Figure 3 This is a schematic diagram of the structure of the multimodal spatiotemporal fusion three-dimensional target detection system provided by the present invention.

[0048] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0050] To alleviate the performance degradation of existing multimodal fusion detection methods in sensor degradation scenarios, and even the error in target detection results due to negative fusion, existing technologies usually adopt specific structures, such as gating in the feature space, using expert routing, or introducing temporal aggregation. However, their fusion decisions often rely on implicit learning of the structure and lack a reusable and shareable quality signal between different modalities and different time data frames, making it difficult to make unified decisions in cross-modal fusion and temporal fusion.

[0051] Therefore, there is an urgent need for a multimodal spatiotemporal fusion 3D target detection method that can obtain quality signals to guide cross-modal fusion and temporal fusion, so as to improve the reliability of multimodal 3D detection in sensor degradation scenarios without introducing additional supervision or significant computational overhead.

[0052] Figure 1 This is a flowchart illustrating the multimodal spatiotemporal fusion three-dimensional target detection method provided in this embodiment of the invention, as shown below. Figure 1 As shown, the method includes:

[0053] S1, acquire the multimodal sensor data at the current moment, wherein the multimodal sensor data includes at least lidar point cloud data and camera image data;

[0054] S2, the lidar point cloud data and the camera image data are encoded respectively to obtain multimodal feature codes;

[0055] S3, based on the pre-determined target query features and the multimodal feature encoding, determine the current query features corresponding to at least two modal decoding branches, and based on each of the current query features, apply at least two modal decoding branches to determine at least two initial detection results; each of the initial detection results includes a category detection result;

[0056] S4, Based on the detection results of each category, construct a current quality signal; the current quality signal is used to characterize the reliability of the current query feature under the corresponding modality decoding branch;

[0057] S5, based on each of the current quality signals, perform quality-aware temporal fusion of each of the current query features and the corresponding historical query features under the modal decoding branch to obtain temporal fusion features under each of the modal decoding branches, and perform cross-modal fusion of the temporal fusion features under each of the modal decoding branches to obtain comprehensive query features;

[0058] S6. Based on the comprehensive query features, determine the three-dimensional target detection result at the current moment.

[0059] Specifically, the multimodal spatiotemporal fusion three-dimensional target detection method provided in this embodiment of the invention is executed by a multimodal spatiotemporal fusion three-dimensional target detection system. This system can be configured in a computer, which can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0060] First, step S1 is executed to acquire the multimodal sensor data at the current moment. This multimodal sensor data includes at least lidar point cloud data and camera image data. Lidar point cloud data refers to point cloud data acquired and transmitted via lidar; this point cloud data can be a single-frame point cloud or a pre-processed sparse voxel representation. Camera image data refers to multi-view image data acquired and transmitted via a camera.

[0061] In addition, historical multimodal sensor data or historical information obtained from historical multimodal sensor data can be acquired. Let the current time be t, and the lidar point cloud data at the current time be denoted as t. The current camera image data is recorded as Historical multimodal sensor data is recorded as and , where k is the number of data frames in the historical multimodal sensor data, and v is the number of camera viewpoints.

[0062] This historical information can include historical query features, historical pose information, historical velocity information, and historical quality signals. Historical query features are used to characterize the targets present in the multimodal sensor data at a historical moment. Historical pose information refers to the position and orientation of the targets in the multimodal sensor data at a historical moment. Historical velocity information refers to the velocity of the targets in the multimodal sensor data at a historical moment. Historical quality signals refer to the quality signals of the multimodal sensor data at a historical moment corresponding to each candidate target, determined by the quality module in the target detection model.

[0063] Understandably, candidate targets are predetermined targets used to identify targets present in the multimodal sensor data at each time step.

[0064] Then, step S2 is executed, which inputs the multimodal sensor data into the target detection model and outputs the three-dimensional target detection result at the current moment through the target detection model.

[0065] The target detection model may include a multimodal feature encoder, at least two modal decoding branches, a quality module, a temporal fusion module, a cross-modal fusion module, and a 3D target detection head.

[0066] Multimodal feature encoders can include image encoders and radar encoders. Image encoders can use convolutional neural networks and feature pyramid networks, while radar encoders can use voxelization and sparse convolutional networks, point cloud backbone networks, etc.

[0067] Image encoders can encode camera image data to obtain image feature codes, while radar encoders can encode lidar point cloud data to obtain point cloud feature codes. Image feature codes and point cloud feature codes can be combined to form multimodal feature codes.

[0068] This step uses a dedicated encoder to process data from different modalities, which maximizes the extraction of unique deep semantic features and spatial geometric features of each modality, providing high-quality feature representations for subsequent query interactions.

[0069] Next, step S3 is executed, which uses the pre-determined target query features and multimodal feature encoding to determine the current query features corresponding to at least two modality decoding branches. The pre-determined target query features can be learnable query vectors, learnable reference points, or query sets generated from candidate proposals. Multiple target query features can be included, and each target query feature corresponds one-to-one with a candidate target; that is, each target query feature corresponds to a candidate target and is used to characterize the corresponding candidate target.

[0070] The modal decoding branch includes at least an image decoding branch corresponding to image feature encoding and a point cloud decoding branch corresponding to point cloud feature encoding. The image decoding branch is used to decode the image feature encoding, and the point cloud decoding branch is used to decode the point cloud feature encoding. In addition, it may also include a fusion decoding branch corresponding to both image feature encoding and point cloud feature encoding, which is used to fuse and decode the image feature encoding and point cloud feature encoding.

[0071] Each target query feature is input into its respective modality decoding branch, and the multimodal feature encoding is also input into its corresponding modality decoding branch. Each modality decoding branch can output an initial detection result based on the input target query features and the corresponding modality feature encoding. Thus, each modality decoding branch can output an initial detection result, which includes a category detection result and a detection confidence score. The category detection result can be the probability distribution of the target belonging to each candidate target category in the corresponding modality data detected by the modality decoding branch. The detection confidence score refers to the reliability of the initial detection result, which can be determined by the category probability distribution. Furthermore, the initial detection result can also include a target bounding box attribute detection result, which refers to the geometric attribute information of the 3D target bounding box, and can include one or more attributes such as the center position, size, orientation, and velocity of the 3D target bounding box.

[0072] This step involves setting up multiple modal decoding branches for independent decoding and detection, which fully preserves the unique judgment criteria of each modality and provides the necessary basis for subsequent evaluation of the reliability of each modal decoding branch at the current moment.

[0073] Next, step S4 is executed to construct a corresponding current quality signal using the detection results for each category. Each category detection result can correspond to a current quality signal, which characterizes the reliability of the current query feature under the corresponding modal decoding branch and is used to suppress interference from background queries and uncertainty detection. For example, a corresponding current quality signal can be constructed using each category detection result. This current quality signal can simultaneously consider the detection confidence and uncertainty corresponding to the category detection result. The detection confidence reflects whether the target represented by the current query feature is a valid foreground target in the multimodal sensor data at the current moment. Uncertainty reflects the dispersion of the category probability distribution in the category detection result, and can be obtained through the entropy of the category probability distribution, the difference between the maximum probability and the second-largest probability, or other indicators reflecting the dispersion of the classification distribution.

[0074] This step innovatively extracts explicit current quality signals directly from the category detection results, without relying on additional degradation labels or manual quality annotations, and can reflect the perceived reliability of the current modality in real time and accurately at the current moment.

[0075] Next, step S5 is executed, which uses the current quality signals to perform quality-aware temporal fusion of the current query features with the historical query features under the corresponding modality decoding branch, resulting in temporal fusion features under each modality decoding branch. These historical query features refer to the query features at historical moments stored in the cache queue.

[0076] Quality-aware temporal fusion refers to a temporal fusion process that uses the current quality signal as a gating or adjusting weight to adaptively determine the fusion ratio between the current query features and historical query features. Temporal fusion features refer to single-branch query features obtained after fusion along the temporal dimension.

[0077] Cross-modal fusion refers to the operation of interacting and fusing temporal fusion features obtained from different modal decoding branches in spatial or modal dimensions. Comprehensive query features refer to the final feature vector containing the richest and most reliable environmental information after dual fusion of temporal and spatial multimodal features.

[0078] In each modal decoding branch, the fusion weights of the current query feature and the historical query feature are adaptively adjusted using the current quality signal. For example, when the current quality signal indicates that the data at the current moment is degraded or of low quality, the fusion weight of the historical query feature is increased; when the current quality signal indicates that the data at the current moment is clear and reliable, the fusion weight of the current query feature is increased, thereby obtaining the temporal fusion feature under each modal decoding branch.

[0079] Subsequently, an attention mechanism can be used to perform cross-modal fusion of temporal fusion features under each modal decoding branch. In the attention mechanism of cross-modal fusion, the contribution of different modal decoding branches is also adaptively weighted using the current quality signal to suppress the weight of temporal fusion features under the low-quality modal decoding branch and enhance the weight of temporal fusion features under the high-quality modal decoding branch, ultimately obtaining the comprehensive query features.

[0080] This step enables dual adaptive anti-interference fusion across both temporal and cross-modal dimensions. In degraded scenarios such as temporary malfunctions of LiDAR or cameras, severe weather, or occlusion, it can automatically reduce interference from degraded data and fully utilize historical query features, effectively preventing detection performance crashes caused by negative fusion.

[0081] Finally, step S6 is executed to determine the 3D target detection result at the current moment using the comprehensive query features. For example, the comprehensive query features can be input into the 3D target detection head, which then outputs the 3D target detection result. This 3D target detection result refers to the final perception conclusions output by the 3D target detection head, including the target's category, 3D spatial coordinates, size, orientation angle, and detection confidence level in the surrounding environment.

[0082] The comprehensive query features are input into the 3D target detection head, which calculates and outputs the final 3D target detection result through regression and classification fully connected layers or multilayer perceptrons.

[0083] This step, through the final decoding of refined and enhanced comprehensive query features, can significantly improve the recall rate of 3D targets and the prediction accuracy of target box positions in complex and harsh environments.

[0084] The multimodal spatiotemporal fusion 3D target detection method provided in this invention innovatively introduces each modal decoding branch and a current quality signal autonomously constructed from each initial detection result into the multimodal and spatiotemporal fusion architecture. This current quality signal serves as a unified adjustment mechanism, dynamically allocating feature fusion weights in both the temporal fusion and cross-modal fusion stages, thus overcoming the limitation of traditional fusion methods that overly rely on implicit learning. Even when facing common challenges in real-world autonomous driving scenarios such as sparse LiDAR point clouds, camera obstruction by rain and fog, and glare from strong light, it can accurately detect targets. Without requiring any external supervision labels, it can significantly improve the overall robustness, stability, and reliable localization capability of 3D target detection in complex scenarios.

[0085] Based on the above embodiments, each current quality signal includes a query-level quality signal and / or a frame-level quality signal, and each category detection result includes a category probability distribution;

[0086] The construction of the current quality signal based on the detection results of each category includes:

[0087] Based on the aforementioned category probability distribution, the detection confidence and detection uncertainty are determined.

[0088] Based on the detection confidence and the detection uncertainty, the query-level quality signal is determined, and based on the query-level quality signal, the frame-level quality signal is determined.

[0089] Specifically, each current quality signal includes a query-level quality signal and / or a frame-level quality signal. The query-level quality signal is the quality signal calculated for a single target query feature, used to measure the reliability of that target query feature under the current modality branch, and can be used to adjust the fusion weights for each target query feature. The frame-level quality signal is the overall quality evaluation index obtained by aggregating all query-level quality signals within each frame of data, used to provide an overall characterization of the quality signal of the entire frame or the entire modality.

[0090] For each modal decoding branch, the output class probability distribution can be represented as:

[0091] ;

[0092] in, Let represent the unnormalized classification score vector of the i-th target query feature at time t, output by the last decoding layer in this modality decoding branch, where softmax is the normalization function. This represents the probability value corresponding to the i-th target query feature at time t.

[0093] Detection confidence refers to a numerical value that reflects the probability or certainty that the target query feature is a valid foreground target. In the category probability distribution, the highest probability value is selected as the detection confidence.

[0094] The detection uncertainty can be calculated using the following formula:

[0095] ;

[0096] ;

[0097] in, This represents the detection uncertainty corresponding to the i-th target query feature at time t, which is achieved by introducing... It can make H is the entropy function, and C is the number of known target categories in the object detection model. Let c be the probability value of the target category c corresponding to the query feature of the i-th target at time t.

[0098] After obtaining the detection confidence and detection uncertainty, the query-level quality signal can be determined by combining these two indicators. For example, the query-level quality signal can be expressed as:

[0099] ;

[0100] in, Let be the query-level quality signal corresponding to the i-th target query feature at time t. To test the confidence level.

[0101] This query-level quality signal can enhance the quality of target query features with high detection confidence and low uncertainty. Furthermore, when it is necessary to evaluate the overall frame quality, the frame-level quality signal can be calculated by averaging, taking the maximum value, or using other aggregation methods, using each target query feature within the entire frame, i.e., the query-level quality signal corresponding to the current query feature.

[0102] In this embodiment of the invention, by combining detection confidence and detection uncertainty to construct an explicit current quality signal, the reliability of each target query feature can be evaluated more comprehensively and accurately, avoiding misjudgments caused by overconfidence in the target detection model due to a single detection confidence. At the same time, the introduction of frame-level quality signals can provide a basis for the quality assessment of the overall modality or frame.

[0103] Based on the above embodiments, it also includes:

[0104] Obtain historical information related to the corresponding historical query features under each modal decoding branch, wherein the historical information includes historical query features, historical pose information, historical velocity information, and historical quality signals;

[0105] The step of performing quality-aware temporal fusion of each current query feature with the corresponding historical query features under the modal decoding branch based on each current quality signal to obtain temporal fusion features under each modal decoding branch includes:

[0106] For any modal decoding branch, based on the current quality signal corresponding to the modal decoding branch, a prior bias is applied, and the current query feature corresponding to the modal decoding branch and the historical query feature under the modal decoding branch are fused in a quality-aware time series to obtain the time series fused feature under the modal decoding branch.

[0107] The prior bias includes at least one of time decay prior, pose alignment prior, motion consistency prior, and history quality prior.

[0108] Specifically, prior bias refers to the rules or constraints that are pre-set to adjust the weights of historical query features during time-series fusion. Prior bias can be obtained through physical laws or historical experience and is used to improve the rationality and stability of fusion.

[0109] When performing time-series fusion, for any modal decoding branch, first obtain its corresponding current quality signal and historical information related to the corresponding historical query features under that modal decoding, and then construct a prior bias based on the historical information.

[0110] Subsequently, a temporal attention mechanism or a weighted aggregation mechanism is used to weight and aggregate historical query features, resulting in aggregated historical features. The aggregation weights of historical query features can be adjusted using prior biases. For example, a time decay prior can be applied, increasing the aggregation weight of historical query features closer to the current moment; a pose alignment prior can be applied, adjusting the aggregation weights of historical query features based on the alignment between the current target pose and historical pose information; a motion consistency prior can be applied, predicting the target's current position based on its historical velocity and suppressing historical query features that do not conform to reasonable motion trajectories; and a historical quality prior can be applied, reducing the aggregation weights of low-quality historical query features based on historical quality signals.

[0111] After obtaining the historical aggregation features, the current query features and historical aggregation features can be weighted and fused to obtain the time-series fused features. Specifically, the fusion weights of the current query features and historical aggregation features can be gated using the current quality signal; when the current quality signal is low, the fusion weight of the historical aggregation features is increased, and when the current quality signal is high, the fusion weight of the current query features is increased.

[0112] In this embodiment of the invention, by introducing multiple prior biases in temporal fusion and combining them with adaptive gating adjustment based on the current quality signal, unreasonable historical interference can be effectively filtered out. Furthermore, when the current frame degrades, the fusion weights of historical aggregation features can be fully and reasonably utilized to improve fusion stability and enhance the stability and robustness of the target detection model in the temporal dimension.

[0113] Based on the above embodiments, the step of performing cross-modal fusion on the temporal fusion features under each modal decoding branch to obtain comprehensive query features includes:

[0114] Select a target modal decoding branch from each of the modal decoding branches, determine a query vector based on the temporal fusion features under the target modal decoding branch, and determine a key vector and a value vector based on the splicing or combination results of the temporal fusion features under each modal decoding branch.

[0115] Based on the current quality signals, the query vector, the key vector, and the value vector, a cross-modal attention mechanism is applied to perform cross-modal fusion on the temporal fusion features under each modal decoding branch to obtain the comprehensive query features.

[0116] Specifically, the target modal decoding branch refers to a specific modal branch that is selected as the dominant party in information query when conducting cross-modal interactions. For example, an image modal decoding branch containing rich semantic information or a point cloud modal decoding branch with accurate spatial information can be selected as the target modal decoding branch.

[0117] The query vector is the feature vector representing the query request in the cross-modal attention mechanism, used to match information from other modalities. The key vector is the vector representing the feature identifier of the queried information in the cross-modal attention mechanism, used to calculate the matching degree with the query vector. The value vector is the vector representing the actual extracted and fused information content in the cross-modal attention mechanism.

[0118] After obtaining the target modality decoding branch, the temporal fusion features under the target modality decoding branch can be used to generate query vectors through linear transformations and other methods. Simultaneously, the temporal fusion features under each modality decoding branch are concatenated to obtain a concatenated result, or the temporal fusion features under each modality decoding branch are combined in other ways to obtain a combined result. Then, using the concatenated or combined result, key vectors and value vectors are generated respectively through linear transformations and other methods.

[0119] The query vector, key vector, value vector, and current quality signals are input into the cross-modal attention mechanism module. By calculating the matching degree between the query vector and the key vector, and combining the adjustment of the current quality signals, the value vector is weighted and summed to realize the interaction and aggregation of feature information from different modalities, and finally output the comprehensive query features.

[0120] In this embodiment of the invention, a cross-modal attention mechanism is used to achieve deep interaction and aggregation of features from different modalities, and a query vector is generated by specifying a target modality. This enables the extraction of valuable information from multimodal spliced ​​features in an organized manner, providing effective architectural support for the generation of comprehensive query features.

[0121] Based on the above embodiments, the step of applying a cross-modal attention mechanism to perform cross-modal fusion of the temporal fusion features under each modal decoding branch, based on each current quality signal, the query vector, the key vector, and the value vector, to obtain the comprehensive query features, includes:

[0122] Based on the query vector, the key vector, and the value vector, attention weights are calculated by applying the current quality signals.

[0123] Based on the attention weights, the temporal fusion features under each modal decoding branch are fused across modalities to obtain the comprehensive query features.

[0124] Specifically, attention weight refers to the coefficient value calculated in the cross-modal attention mechanism to measure the strength of the correlation between the query vector and each key vector. This coefficient value determines the proportion of the corresponding value vector in the comprehensive query features.

[0125] When applying a cross-modal attention mechanism, the similarity score between the query vector and the key vector is first calculated. Based on this, each current quality signal is introduced as a bias term or scaling factor into the attention score calculation. For example, the current quality signal can be converted into an additive bias term added to the similarity score, or multiplied as a scaling factor, thus giving higher-quality modal features higher similarity scores and lower-quality modal features lower similarity scores. Then, the adjusted similarity scores are normalized using the softmax function to obtain the attention weights. Finally, the calculated attention weights are used to perform a weighted summation of the value vectors to complete cross-modal fusion and output the comprehensive query features.

[0126] In this embodiment of the invention, by directly applying the current quality signal to the calculation process of attention weights, the contribution of reliable modes is adaptively amplified and the interference of degenerate modes is reduced during cross-modal feature fusion, thereby achieving true quality-guided fusion and improving the reliability of fused features.

[0127] Based on the above embodiments, each initial detection result further includes a target bounding box attribute detection result; the step of performing cross-modal fusion on the temporal fusion features under each modal decoding branch to obtain comprehensive query features further includes:

[0128] Based on the target bounding box attribute detection results, a geometrically consistent prior is constructed;

[0129] Based on the geometric consistency prior, the temporal fusion features under each modal decoding branch are fused across modalities to obtain comprehensive query features.

[0130] Specifically, geometric consistency prior refers to a constraint or bias information calculated based on the spatial and geometric similarity between the target box attribute detection results obtained from different modal decoding branches, used to measure whether different target boxes point to the same physical entity.

[0131] By comparing the target bounding box attribute detection results obtained from different modal decoding branches, indices such as center distance, size similarity, and orientation similarity between target bounding boxes can be calculated. For example, target bounding boxes with closer center distances and more similar sizes and orientations have higher geometric consistency scores, thus constructing a geometric consistency prior.

[0132] When performing cross-modal fusion, the constructed geometric consistency prior is introduced as an additional bias term into the weight calculation of the cross-modal attention mechanism. Alternatively, this geometric consistency prior can be used to filter target boxes before cross-modal fusion, such as filtering out target boxes that are too far away or have too large attribute differences. In this way, temporal fusion features that are spatially close and have consistent geometric attributes can have higher fusion weights when performing cross-modal fusion, ultimately resulting in comprehensive query features that incorporate geometric spatial constraints.

[0133] In this embodiment of the invention, a geometric consistency prior is constructed based on the target box attribute detection results to introduce the cross-modal fusion process. This can effectively avoid erroneously fusing spatially unrelated or spatially dissimilar temporal fusion features, thereby improving the accuracy and rationality of cross-modal feature aggregation.

[0134] Based on the above embodiments, when training the target detection model, random modality discarding or degradation enhancement strategies can be used to process the training samples, so that the target detection model is exposed to various sensor degradation scenarios during the training process, thereby improving its robustness in actual degradation scenarios.

[0135] At least one degradation method can be randomly selected in each training iteration to randomly discard points from the LiDAR point cloud data, such as randomly discarding some points or a subset of the point cloud. Random view masking can also be applied to the camera image data input, such as randomly masking some camera views or applying an occlusion mask to a local area. Simulated degradation perturbations, such as noise, blurring, and occlusion, can also be applied to the LiDAR point cloud data and camera image data.

[0136] Random modality drop-off or degradation enhancement strategies do not rely on additional degradation labels or explicit sensor quality annotations, and can enable target detection models to learn to stably construct detection quality signals in degradation scenarios and perform fusion accordingly.

[0137] like Figure 2 The diagram shown is a complete flowchart of the multimodal spatiotemporal fusion three-dimensional target detection method provided in an embodiment of the present invention. The method includes:

[0138] Acquire the current LiDAR point cloud data and camera image data;

[0139] The camera image data is encoded using an image encoder to obtain image feature codes, and the lidar point cloud data is encoded using a radar encoder to obtain point cloud feature codes.

[0140] Using the pre-determined target query features and image feature encoding and point cloud feature encoding, the current query features corresponding to the image decoding branch, point cloud decoding branch and fusion decoding branch are determined. Then, using the current query features, the image decoding branch, point cloud decoding branch and fusion decoding branch are applied to determine each initial detection result.

[0141] Construct the current quality signal using the test results of each category;

[0142] Using the current quality signals, the current query features are fused with the historical query features under the corresponding modal decoding branch to obtain the temporal fusion features under each modal decoding branch. Then, the temporal fusion features under each modal decoding branch are fused across modalities to obtain the comprehensive query features.

[0143] By utilizing comprehensive query features, the 3D target detection result at the current moment is determined.

[0144] like Figure 3 As shown, based on the above embodiments, this embodiment of the invention provides a multimodal spatiotemporal fusion three-dimensional target detection system, including:

[0145] Data acquisition module 31 is used to acquire multimodal sensor data at the current moment, wherein the multimodal sensor data includes at least lidar point cloud data and camera image data;

[0146] Encoding module 32 is used to encode the lidar point cloud data and the camera image data respectively to obtain multimodal feature encoding;

[0147] The decoding module 33 is used to determine the current query features corresponding to at least two modality decoding branches based on the pre-determined target query features and the multimodal feature encoding, and to determine at least two initial detection results based on each of the current query features by applying the at least two modality decoding branches; each of the initial detection results includes a category detection result;

[0148] The quality signal construction module 34 is used to construct a current quality signal based on the detection result of each category; the current quality signal is used to characterize the reliability of the current query feature under the corresponding modality decoding branch;

[0149] The feature fusion module 35 is used to perform quality-aware temporal fusion of each current query feature and the corresponding historical query feature under the modal decoding branch based on each current quality signal, to obtain temporal fusion features under each modal decoding branch, and to perform cross-modal fusion of the temporal fusion features under each modal decoding branch to obtain comprehensive query features;

[0150] The target detection module 36 is used to determine the three-dimensional target detection result at the current moment based on the comprehensive query features.

[0151] Specifically, the functions of each module in the multimodal spatiotemporal fusion three-dimensional target detection system provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0152] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the multimodal spatiotemporal fusion three-dimensional target detection method provided in the above embodiments.

[0153] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0154] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the multimodal spatiotemporal fusion three-dimensional target detection method provided in the above embodiments.

[0155] In another aspect, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the multimodal spatiotemporal fusion three-dimensional target detection method provided in the above embodiments. This computer-readable storage medium can be either a non-transitory computer-readable storage medium or a transient computer-readable storage medium, and is not specifically limited herein.

[0156] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0157] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal spatiotemporal fusion three-dimensional target detection method, characterized in that, include: Acquire multimodal sensor data at the current moment, wherein the multimodal sensor data includes at least lidar point cloud data and camera image data; The lidar point cloud data and the camera image data are encoded separately to obtain multimodal feature codes; Based on the predetermined target query features and the multimodal feature encoding, at least two current query features corresponding to the modal decoding branches are determined, and at least two initial detection results are determined based on each of the current query features; each initial detection result includes a category detection result; Based on the detection results for each of the aforementioned categories, a current quality signal is constructed; The current quality signal is used to characterize the reliability of the current query feature under the corresponding modality decoding branch; Based on each of the current quality signals, each of the current query features and the corresponding historical query features under the modal decoding branch are fused in a quality-aware time series to obtain the time series fused features under each of the modal decoding branches. Then, the time series fused features under each of the modal decoding branches are fused across modalities to obtain the comprehensive query features. Based on the comprehensive query features, the 3D target detection result at the current moment is determined; Also includes: Obtain historical information related to the corresponding historical query features under each modal decoding branch, wherein the historical information includes historical pose information, historical velocity information, and historical quality signal; The step of performing quality-aware temporal fusion of each current query feature with the corresponding historical query features under the modal decoding branch based on each current quality signal to obtain temporal fusion features under each modal decoding branch includes: For any modal decoding branch, based on the current quality signal corresponding to the modal decoding branch, a prior bias is applied, and the current query feature corresponding to the modal decoding branch and the historical query feature under the modal decoding branch are fused in a quality-aware time series to obtain the time series fused feature under the modal decoding branch. The prior bias includes at least one of time decay prior, pose alignment prior, motion consistency prior, and history quality prior.

2. The multimodal spatiotemporal fusion three-dimensional target detection method according to claim 1, characterized in that, Each of the current quality signals includes a query-level quality signal and / or a frame-level quality signal, and each of the category detection results includes a category probability distribution; The construction of the current quality signal based on the detection results of each category includes: Based on the aforementioned category probability distribution, the detection confidence and detection uncertainty are determined. Based on the detection confidence and the detection uncertainty, the query-level quality signal is determined, and based on the query-level quality signal, the frame-level quality signal is determined.

3. The multimodal spatiotemporal fusion three-dimensional target detection method according to claim 1, characterized in that, The cross-modal fusion of the temporal fusion features under each modal decoding branch to obtain comprehensive query features includes: Select a target modal decoding branch from each of the modal decoding branches, determine a query vector based on the temporal fusion features under the target modal decoding branch, and determine a key vector and a value vector based on the splicing or combination results of the temporal fusion features under each modal decoding branch. Based on the current quality signals, the query vector, the key vector, and the value vector, a cross-modal attention mechanism is applied to perform cross-modal fusion on the temporal fusion features under each modal decoding branch to obtain the comprehensive query features.

4. The multimodal spatiotemporal fusion three-dimensional target detection method according to claim 3, characterized in that, Based on the current quality signals, the query vector, the key vector, and the value vector, a cross-modal attention mechanism is applied to perform cross-modal fusion of the temporal fusion features under each modal decoding branch to obtain the comprehensive query features, including: Based on the query vector, the key vector, and the value vector, attention weights are calculated by applying the current quality signals. Based on the attention weights, the temporal fusion features under each modal decoding branch are fused across modalities to obtain the comprehensive query features.

5. The multimodal spatiotemporal fusion three-dimensional target detection method according to any one of claims 1-4, characterized in that, Each of the initial detection results also includes target bounding box attribute detection results; The step of performing cross-modal fusion of the temporal fusion features under each modal decoding branch to obtain comprehensive query features also includes: Based on the target bounding box attribute detection results, a geometrically consistent prior is constructed; Based on the geometric consistency prior, the temporal fusion features under each modal decoding branch are fused across modalities to obtain comprehensive query features.

6. The multimodal spatiotemporal fusion three-dimensional target detection method according to any one of claims 1-4, characterized in that, The determination of the 3D target detection result at the current moment based on the comprehensive query features includes: The comprehensive query features are input into the 3D target detection head to obtain the 3D target detection result output by the 3D target detection head.

7. A multimodal spatiotemporal fusion three-dimensional target detection system, characterized in that, include: The data acquisition module is used to acquire multimodal sensor data at the current moment, wherein the multimodal sensor data includes at least lidar point cloud data and camera image data; The encoding module is used to encode the lidar point cloud data and the camera image data respectively to obtain multimodal feature codes; The decoding module is used to determine the current query features corresponding to at least two modality decoding branches based on the pre-determined target query features and the multimodal feature encoding, and to determine at least two initial detection results based on each of the current query features by applying the at least two modality decoding branches; each of the initial detection results includes a category detection result; A quality signal construction module is used to construct the current quality signal based on the detection results of each of the categories; The current quality signal is used to characterize the reliability of the current query feature under the corresponding modality decoding branch; The feature fusion module is used to perform quality-aware temporal fusion of each current query feature and the corresponding historical query feature under the modal decoding branch based on each current quality signal, to obtain the temporal fusion feature under each modal decoding branch, and to perform cross-modal fusion of the temporal fusion feature under each modal decoding branch to obtain the comprehensive query feature; The target detection module is used to determine the three-dimensional target detection result at the current moment based on the comprehensive query features; The data acquisition module is also used for: Obtain historical information related to the corresponding historical query features under each modal decoding branch, wherein the historical information includes historical pose information, historical velocity information, and historical quality signal; The feature fusion module is specifically used for: For any modal decoding branch, based on the current quality signal corresponding to the modal decoding branch, a prior bias is applied, and the current query feature corresponding to the modal decoding branch and the historical query feature under the modal decoding branch are fused in a quality-aware time series to obtain the time series fused feature under the modal decoding branch. The prior bias includes at least one of time decay prior, pose alignment prior, motion consistency prior, and history quality prior.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multimodal spatiotemporal fusion three-dimensional target detection method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal spatiotemporal fusion three-dimensional target detection method as described in any one of claims 1-6.

Images