Multi-modal space-time fusion three-dimensional target detection method and system

Abstract

The invention provides a multi-modal space-time fusion three-dimensional target detection method and system, relates to the technical field of target detection, and innovatively introduces each modal decoding branch and a current quality signal autonomously constructed by each initial detection result in a multi-modal and space-time fusion architecture. And the current quality signal is used as a unified adjustment mechanism, and feature fusion weights are dynamically distributed in a time sequence fusion link and a cross-modal fusion link, so that the limitation that a traditional fusion method excessively depends on network implicit learning is broken through. In the face of common single-mode or local sensor degradation problems such as sparse point cloud of a laser radar, rain and fog shielding of a camera and strong light glare in a real automatic driving scene, target detection can be accurately carried out, and under the condition that any external supervision label does not need to be added, the target detection efficiency is improved. The overall robustness and stability of three-dimensional target detection and the reliable positioning capability in a complex scene can be greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and in particular to a multi-modal spatio-temporal fusion three-dimensional target detection method and system. BACKGROUND

[0002] In the scenarios of autonomous driving, mobile robots, etc., three-dimensional target detection is used to locate and identify surrounding traffic participants according to data collected by sensors, and is an important basis for realizing environmental perception and decision control. Common sensors include lidar and camera, among which lidar provides accurate spatial geometric information, and camera provides rich semantic and texture information. Through a multi-modal fusion detection method, the information provided by lidar and camera can be fused to improve detection accuracy and stability.

[0003] However, in real road environments, sensors are often affected by various degradation or failure factors, such as: point cloud sparseness, limited field of view, local occlusion or whole frame loss of lidar; missing view angle, imaging degradation caused by adverse weather such as rain, snow, etc., strong light glare or occlusion, etc. of camera. The existing multi-modal fusion detection method may have a significant performance decline in the sensor degradation scene, and even cause target detection results to be wrong due to negative fusion. SUMMARY

[0004] The present application provides a multi-modal spatio-temporal fusion three-dimensional target detection method and system to solve the defects in the related art.

[0005] The present application provides a multi-modal spatio-temporal fusion three-dimensional target detection method, comprising: acquiring multi-modal sensor data at the current time, the multi-modal sensor data at least including lidar point cloud data and camera image data; respectively encoding the lidar point cloud data and the camera image data to obtain multi-modal feature encoding; determining current query features corresponding to at least two modal decoding branches based on pre-determined target query features and the multi-modal feature encoding, and determining at least two initial detection results based on each current query feature; each initial detection result includes a class detection result; based on each class detection result, constructing a current quality signal; the current quality signal is used to represent the reliability of the current query feature under the corresponding modal decoding branch; based on each current quality signal, performing quality-aware temporal fusion on each current query feature and historical query features under the corresponding modal decoding branch to obtain temporal fusion features under each modal decoding branch, and performing cross-modal fusion on the temporal fusion features under each modal decoding branch to obtain comprehensive query features; Determine a three-dimensional target detection result of the current moment based on the comprehensive query feature.

[0006] According to the method, each current quality signal comprises a query-level quality signal and / or a frame-level quality signal, and each category detection result comprises a category probability distribution. The method further comprises the following steps of: Determine a detection confidence and a detection uncertainty based on the category probability distribution. Determine the query-level quality signal based on the detection confidence and the detection uncertainty, and determine the frame-level quality signal based on the query-level quality signal.

[0007] According to the method, the method further comprises the following steps of: Obtain historical information related to the historical query feature under each modality decoding branch, wherein the historical information comprises historical pose information, historical speed information and historical quality signals. The method further comprises the following steps of: For any modality decoding branch, perform quality-aware temporal fusion on the current query feature corresponding to the any modality decoding branch and the historical query feature under the any modality decoding branch based on the current quality signal corresponding to the any modality decoding branch, to obtain the temporal fusion feature under the any modality decoding branch. The prior bias comprises at least one of a time decay prior, a pose alignment prior, a motion consistency prior and a historical quality prior.

[0008] According to the method, the method further comprises the following steps of: Select a target modality decoding branch from the modality decoding branches, determine a query vector based on the temporal fusion feature under the target modality decoding branch, and determine a key vector and a value vector based on a splicing result or a combination result of the temporal fusion features under the modality decoding branches. Perform cross-modality fusion on the temporal fusion features under the modality decoding branches based on the cross-modality attention mechanism, to obtain the comprehensive query feature.

[0009] According to the multi-modal spatio-temporal fusion three-dimensional target detection method provided by the application, the cross-modal attention mechanism is applied based on each current quality signal, the query vector, the key vector and the value vector, the time sequence fusion features under each modal decoding branch are cross-modal fused to obtain the comprehensive query feature, which comprises: Based on the query vector, the key vector and the value vector, the attention weight is calculated by applying each current quality signal; Based on the attention weight, the time sequence fusion features under each modal decoding branch are cross-modal fused to obtain the comprehensive query feature.

[0010] According to the multi-modal spatio-temporal fusion three-dimensional target detection method provided by the application, each initial detection result further comprises a target frame attribute detection result; The cross-modal fusion of the time sequence fusion features under each modal decoding branch to obtain the comprehensive query feature further comprises: Based on the target frame attribute detection result, a geometric consistency prior is constructed; Based on the geometric consistency prior, the time sequence fusion features under each modal decoding branch are cross-modal fused to obtain the comprehensive query feature.

[0011] According to the multi-modal spatio-temporal fusion three-dimensional target detection method provided by the application, the three-dimensional target detection result of the current time is determined based on the comprehensive query feature, which comprises: The comprehensive query feature is input into a three-dimensional target detection head to obtain the three-dimensional target detection result output by the three-dimensional target detection head.

[0012] The application also provides a multi-modal spatio-temporal fusion three-dimensional target detection system, which comprises: A data acquisition module is configured to acquire multi-modal sensor data at a current time, wherein the multi-modal sensor data at least comprises laser radar point cloud data and camera image data; An encoding module is configured to encode the laser radar point cloud data and the camera image data respectively to obtain multi-modal feature encoding; A decoding module is configured to determine current query features corresponding to at least two modal decoding branches based on pre-determined target query features and the multi-modal feature encoding, and to determine at least two initial detection results by applying at least two modal decoding branches based on each current query feature; each initial detection result comprises a category detection result; a quality signal construction module configured to construct a current quality signal based on each of the category detection results, the current quality signal being used to represent a reliability degree of the current query feature under the corresponding modal decoding branch; a feature fusion module configured to perform quality-aware temporal fusion on each of the current query features and historical query features under the corresponding modal decoding branch based on the current quality signals, to obtain temporal fusion features under each of the modal decoding branches, and to perform cross-modal fusion on the temporal fusion features under each of the modal decoding branches to obtain comprehensive query features; a target detection module configured to determine a three-dimensional target detection result of the current time based on the comprehensive query features.

[0013] The application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the multi-modal spatio-temporal fusion three-dimensional target detection method according to any one of the above when executing the computer program.

[0014] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the multi-modal spatio-temporal fusion three-dimensional target detection method according to any one of the above.

[0015] The application further provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the multi-modal spatio-temporal fusion three-dimensional target detection method according to any one of the above.

[0016] The multi-modal spatio-temporal fusion three-dimensional target detection method and system provided by the application introduce each modal decoding branch and the current quality signal constructed by each initial detection result in a multi-modal and spatio-temporal fusion architecture, and use the current quality signal as a unified adjustment mechanism to dynamically allocate feature fusion weights in the temporal fusion link and the cross-modal fusion link, thereby breaking the limitation of traditional fusion methods which excessively rely on network implicit learning. When facing the single-modal or local sensor degradation problems such as laser radar point cloud sparseness, camera rain and fog shielding, and strong light glare which are common in real automatic driving scenes, the application can accurately perform target detection, and can greatly improve the overall robustness, stability and reliable positioning ability in complex scenes of three-dimensional target detection without adding any external supervision label. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present application or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or the related art description. Obviously, the drawings described below are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0018] Figure 1 is one of the flowcharts of the multi-modal spatio-temporal fusion three-dimensional target detection method provided by the present application.

[0019] Figure 2 is another flowchart of the multi-modal spatio-temporal fusion three-dimensional target detection method provided by the present application.

[0020] Figure 3 is a structural schematic diagram of the multi-modal spatio-temporal fusion three-dimensional target detection system provided by the present application.

[0021] Figure 4 is a structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0022] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort belong to the protection scope of the present application.

[0023] In order to alleviate the performance decline of the existing multi-modal fusion detection method in the sensor degradation scene, and even the error of the target detection result caused by the negative fusion, the existing technology usually adopts a specific structure, such as gating in the feature space, using expert routing or introducing time aggregation, but the fusion decision often depends on the implicit learning of the structure, and there is a lack of a quality signal that can be reused and shared between different modalities and different time data frames, so it is difficult to make unified decisions in cross-modal fusion and time series fusion.

[0024] Therefore, there is an urgent need for a multi-modal spatio-temporal fusion three-dimensional target detection method that can obtain a quality signal for guiding cross-modal fusion and time series fusion, so as to improve the reliability of multi-modal three-dimensional detection in the sensor degradation scene without introducing additional supervision or significant computational overhead.

[0025] Figure 1 The flowchart of the multi-modal spatio-temporal fusion three-dimensional target detection method provided in the embodiments of the present application is shown in FIG. Figure 1 The method comprises the following steps. S1, acquire multi-modal sensor data at the current time, the multi-modal sensor data at least comprising laser radar point cloud data and camera image data; S2, encode the laser radar point cloud data and the camera image data respectively to obtain multi-modal feature encoding; S3, determine current query features corresponding to at least two modal decoding branches based on pre-determined target query features and the multi-modal feature encoding, and apply at least two modal decoding branches based on each current query feature to determine at least two initial detection results; each initial detection result comprises a category detection result; S4, construct a current quality signal based on each category detection result; the current quality signal is used to represent the reliability of the current query feature under the corresponding modal decoding branch; S5, based on each current quality signal, perform quality-aware temporal fusion on each current query feature and historical query features under the corresponding modal decoding branch to obtain temporal fusion features under each modal decoding branch, and perform cross-modal fusion on the temporal fusion features under each modal decoding branch to obtain comprehensive query features; S6, determine a three-dimensional target detection result at the current time based on the comprehensive query features.

[0026] Specifically, the multi-modal spatio-temporal fusion three-dimensional target detection method provided in the embodiment of the application has a multi-modal spatio-temporal fusion three-dimensional target detection system as an execution subject, which can be configured in a computer, which can be a local computer or a cloud computer, and the local computer can be a computer, a tablet, etc., which is not specifically limited here.

[0027] Firstly, step S1 is performed to acquire multi-modal sensor data at the current time, which at least comprises laser radar point cloud data and camera image data. The laser radar point cloud data refers to point cloud data collected and transmitted by a laser radar, which can be a single frame of point cloud or a pre-processed sparse voxel representation. The camera image data refers to multi-view image data collected and transmitted by a camera.

[0028] In addition, historical multi-modal sensor data or historical information obtained from the historical multi-modal sensor data can also be acquired. Let the current time be t, the laser radar point cloud data at the current time be denoted as , the camera image data at the current time be denoted as , and the historical multi-modal sensor data be denoted as and , where k is the number of data frames in the historical multi-modal sensor data, and v is the number of views of the camera.

[0029] The historical information can include historical query features, historical pose information, historical speed information, and historical quality signals. The historical query features are used to represent the target existing in the multi-modal sensor data at the historical moment, the historical pose information refers to the position and attitude of the target in the multi-modal sensor data at the historical moment, the historical speed information refers to the speed of the target in the multi-modal sensor data at the historical moment, and the historical quality signal refers to the quality signal of each candidate target corresponding to the multi-modal sensor data at the historical moment determined by the quality module in the target detection model.

[0030] It can be understood that the candidate target is each target determined in advance, from which the target existing in the multi-modal sensor data at each moment is determined.

[0031] Then step S2 is performed, and the multi-modal sensor data can be input to the target detection model, and a three-dimensional target detection result at the current moment is output by the target detection model.

[0032] The target detection model can include a multi-modal feature encoder, at least two modal decoding branches, a quality module, a time sequence fusion module, a cross-modal fusion module, and a three-dimensional target detection head.

[0033] The multi-modal feature encoder can include an image encoder and a radar encoder. The image encoder can adopt a convolutional neural network and a feature pyramid network, etc., and the radar encoder can adopt a voxelization and a sparse convolution network, a point cloud backbone network, etc.

[0034] The image encoder can encode the camera image data to obtain image feature encoding, and the radar encoder can encode the laser radar point cloud data to obtain point cloud feature encoding. The image feature encoding and the point cloud feature encoding can constitute multi-modal feature encoding.

[0035] In this step, a dedicated encoder is used to process data of different modalities respectively, which can maximize the extraction of depth semantic features and spatial geometric features unique to each modality, and provide high-quality feature representation for subsequent query interaction.

[0036] Thereafter, step S3 is performed, and the current query features corresponding to the at least two modal decoding branches are determined by using the pre-determined target query features and the multi-modal feature encoding. The pre-determined target query features are learnable query vectors, learnable reference points, or a query set generated by candidate proposals. The target query features can include multiple target query features, each target query feature corresponds to one candidate target, and each target query feature is used to represent the corresponding candidate target.

[0037] The modal decoding branch at least includes an image feature encoding corresponding image decoding branch and a point cloud feature encoding corresponding point cloud decoding branch, the image decoding branch is used for decoding the image feature encoding, and the point cloud decoding branch is used for decoding the point cloud feature encoding. In addition, a fusion decoding branch corresponding to the image feature encoding and the point cloud feature encoding can also be included, and the fusion decoding branch is used for fusion decoding of the image feature encoding and the point cloud feature encoding.

[0038] Each target query feature is input into each modal decoding branch, and the multi-modal feature encoding is input into the corresponding modal decoding branch. Each modal decoding branch can output a corresponding initial detection result according to the input target query feature and the feature encoding of the corresponding modal. Thus, each modal decoding branch can output an initial detection result, which includes a category detection result and a detection confidence. The category detection result can be a category probability distribution of the target belonging to each candidate target in the corresponding modal data detected by the corresponding modal decoding branch. The detection confidence refers to the reliability of the initial detection result, which can be determined by the category probability distribution. In addition, the initial detection result can also include a target box attribute detection result, which refers to the geometric attribute information of the three-dimensional target box, and can include one or more of the center position, size, orientation, speed and other attributes of the three-dimensional target box.

[0039] In this step, independent decoding and detection are performed by setting multiple modal decoding branches, which can fully retain the judgment basis of each modal and provide necessary basis for subsequent evaluation of the reliability of each modal decoding branch at the current time.

[0040] Thereafter, step S4 is performed to construct a corresponding current quality signal using each category detection result. Each category detection result can correspond to the construction of a current quality signal, which is used to represent the reliability of the current query feature under the corresponding modal decoding branch and to suppress the interference of background queries and uncertain detection. For example, each category detection result can be used to construct a corresponding current quality signal. The current quality signal can simultaneously consider the detection confidence corresponding to the category detection result and the uncertainty. The detection confidence can reflect whether the target represented by the current query feature is an effective foreground target in the multi-modal sensor data at the current time. The uncertainty can reflect the dispersion degree of the category probability distribution in the category detection result, which can be obtained by the entropy, the difference between the maximum probability and the second largest probability, or other indicators reflecting the dispersion degree of the classification distribution.

[0041] In this step, the explicit current quality signal is innovatively extracted directly from the category detection result, without relying on additional degradation labels or artificial quality labels, which can accurately reflect the perception reliability of the current modal at the current time in real time.

[0042] After that, step S5 is performed, and the current query feature is quality-aware time sequence fused with the historical query feature under the corresponding modal decoding branch by using the current quality signal to obtain the time sequence fused feature under each modal decoding branch. The historical query feature refers to the query feature of the historical time stored in the cache queue.

[0043] The quality-aware time sequence fusion refers to a time sequence fusion process of adaptively determining the fusion proportion of the current query feature and the historical query feature by using the current quality signal as a gate or a weight adjustment. The time sequence fused feature refers to a single branch query feature obtained after fusion in the time sequence dimension.

[0044] The cross-modal fusion refers to an operation of interacting and fusing the time sequence fused features obtained by different modal decoding branches in the spatial or modal dimension. The comprehensive query feature refers to a final feature vector containing the most abundant and reliable environmental information after time sequence and spatial multi-modal double fusion.

[0045] In each modal decoding branch, the fusion weight of the current query feature and the historical query feature is adaptively adjusted by using the current quality signal. For example, when the current quality signal indicates that the data at the current time is degraded or of low quality, the fusion weight of the historical query feature is increased; when the current quality signal indicates that the data at the current time is clear and reliable, the fusion weight of the current query feature is increased, thereby obtaining the time sequence fused feature under each modal decoding branch.

[0046] Subsequently, the attention mechanism can be used to perform cross-modal fusion on the time sequence fused features under each modal decoding branch. In the attention mechanism of cross-modal fusion, the current quality signal is also used to adaptively weight the contributions of different modal decoding branches, so as to suppress the weight of the time sequence fused feature under the modal decoding branch of low quality and enhance the weight of the time sequence fused feature under the modal decoding branch of high quality, and finally obtain the comprehensive query feature.

[0047] This step can realize double adaptive anti-interference fusion in the time sequence dimension and the cross-modal dimension. In the degradation scenarios such as temporary failure of laser radar or camera, severe weather or occlusion, the interference of degraded data can be automatically reduced and the historical query feature can be fully called, so that the detection performance collapse caused by negative fusion can be effectively avoided.

[0048] Finally, step S6 is performed, and the three-dimensional target detection result at the current time is determined by using the comprehensive query feature. For example, the comprehensive query feature can be input into the three-dimensional target detection head, and the three-dimensional target detection result is output by the three-dimensional target detection head. The three-dimensional target detection result refers to the final perception conclusion of the class, three-dimensional spatial position coordinates, size, orientation angle and detection confidence of the target in the surrounding environment output by the three-dimensional target detection head.

[0049] The comprehensive query feature is input into a three-dimensional target detection head, and the three-dimensional target detection head calculates and outputs a final three-dimensional target detection result through regression and classification full connection layers or multi-layer perceptrons.

[0050] This step can significantly improve the three-dimensional target recall rate and the prediction accuracy of the target frame position in complex and harsh environments through final decoding of the purified and enhanced comprehensive query feature.

[0051] The multi-modal spatio-temporal fusion three-dimensional target detection method provided by the embodiments of the present application innovatively introduces each modal decoding branch and the current quality signal autonomously constructed by each initial detection result in the multi-modal and spatio-temporal fusion architecture, and uses the current quality signal as a unified adjustment mechanism to dynamically allocate feature fusion weights in the time sequence fusion link and the cross-modal fusion link, thereby breaking the limitation of the traditional fusion method which excessively relies on implicit learning of the network. When facing the single-modal or local sensor degradation problems such as laser radar point cloud sparseness, camera rain and fog shielding, strong light glare and the like which are common in real automatic driving scenes, the target detection can also be accurately performed, and the overall robustness, stability and reliable positioning ability in complex scenes of the three-dimensional target detection can be greatly improved without adding any external supervised labels.

[0052] On the basis of the above-mentioned embodiments, each of the current quality signals comprises a query-level quality signal and / or a frame-level quality signal, and each of the category detection results comprises a category probability distribution. The current quality signal is constructed based on each of the category detection results, comprising: The detection confidence and the detection uncertainty are determined based on the category probability distribution; The query-level quality signal is determined based on the detection confidence and the detection uncertainty, and the frame-level quality signal is determined based on the query-level quality signal.

[0053] Specifically, each current quality signal comprises a query-level quality signal and / or a frame-level quality signal, the query-level quality signal refers to a quality signal calculated for a single target query feature, which is used to measure the reliability of the target query feature in the current modal branch and can be used to adjust the fusion weight for each target query feature. The frame-level quality signal refers to an overall quality evaluation index obtained by aggregating all query-level quality signals in each frame of data, which is used to represent the overall quality signal of the entire frame or the entire modal.

[0054] For each modal decoding branch, the output category probability distribution can be represented as: ; wherein, denotes the unnormalized classification score vector of the i-th target query feature at time t, which is output by the last decoding layer in the modal decoding branch, and softmax is a normalization function, denotes the probability value corresponding to the i-th target query feature at time t.

[0055] The detection confidence refers to a value reflecting the probability or certainty degree that the target query feature is an effective foreground target.

[0056] The detection uncertainty can be calculated by the following formula: ; ; wherein, denotes the detection uncertainty corresponding to the i-th target query feature at time t, which is introduced by , so that H is an entropy function, C is the number of target categories known by the target detection model, is the probability value of the target category c corresponding to the i-th target query feature at time t.

[0057] After obtaining the detection confidence and the detection uncertainty, the query-level quality signal can be determined by comprehensively considering the two indicators, for example, the query-level quality signal can be represented as: ; wherein, is the query-level quality signal corresponding to the i-th target query feature at time t, is the detection confidence.

[0058] The query-level quality signal can make the target query feature with high detection confidence and low uncertainty have higher quality. Further, when it is necessary to evaluate the overall frame quality, the query-level quality signals corresponding to each target query feature in the current frame, i.e. the current query feature, can be utilized to calculate the frame-level quality signal by averaging, taking the maximum value or other aggregation methods.

[0059] In the embodiments of the present application, the current quality signal is constructed by combining the detection confidence and the detection uncertainty, which can more comprehensively and accurately evaluate the reliability of each target query feature, avoid the misjudgment caused by the single detection confidence when the target detection model is overconfident, and the introduction of the frame-level quality signal can provide a basis for the quality evaluation of the overall modal or frame.

[0060] On the basis of the above-mentioned embodiments, further comprising: acquire historical information related to the corresponding historical query feature under each of the modal decoding branches, the historical information including historical query features, historical pose information, historical speed information, and historical quality signals; The quality-aware temporal fusion of the current query feature and the historical query feature under the corresponding modal decoding branch is performed based on the current quality signal of the corresponding modal decoding branch to obtain the temporal fusion feature under the corresponding modal decoding branch. For any modal decoding branch, the quality-aware temporal fusion of the current query feature and the historical query feature under the corresponding modal decoding branch is performed based on the current quality signal of the corresponding modal decoding branch, and a prior bias is applied to obtain the temporal fusion feature under the corresponding modal decoding branch. The prior bias includes at least one of a time decay prior, a pose alignment prior, a motion consistency prior, and a historical quality prior.

[0061] Specifically, the prior bias refers to a rule or constraint condition for adjusting the weight of the historical query feature when performing temporal fusion. The prior bias can be obtained through physical laws or historical experience to improve the rationality and stability of the fusion.

[0062] When performing temporal fusion, for any modal decoding branch, the corresponding current quality signal and the historical information related to the corresponding historical query feature under the modal decoding are first acquired, and the prior bias is constructed according to the historical information.

[0063] Thereafter, a temporal attention mechanism or a weighted aggregation mechanism is used to perform weighted aggregation on the historical query feature to obtain a historical aggregation feature. The aggregation weight of the historical query feature can be adjusted through the prior bias. For example, a time decay prior can be applied to make the aggregation weight of the historical query feature closer to the current time greater; a pose alignment prior can be applied to adjust the aggregation weight of the historical query feature according to the alignment of the target pose at the current time and the historical pose information; a motion consistency prior can be applied to predict the current position of the target according to the historical speed, and to suppress the historical query feature that does not conform to the reasonable motion trajectory; and a historical quality prior can be applied to reduce the aggregation weight of the low-quality historical query feature according to the historical quality signal.

[0064] After obtaining the historical aggregation feature, the current query feature and the historical aggregation feature can be weighted and fused to obtain the temporal fusion feature. The fusion weight of the current query feature and the historical aggregation feature can be adjusted by the current quality signal. When the current quality signal is low, the fusion weight of the historical aggregation feature is increased, and when the current quality signal is high, the fusion weight of the current query feature is increased.

[0065] In the embodiment of the present application, by introducing various prior biases in the temporal fusion and combining the current quality signal for adaptive gating adjustment, unreasonable historical interference can be effectively filtered out, and when degradation occurs in the current frame, the fusion weight of the historical aggregation feature can be fully and reasonably utilized to improve the fusion stability, thereby enhancing the stability and robustness of the target detection model in the temporal dimension.

[0066] On the basis of the above-mentioned embodiment, the cross-modal fusion of the temporal fusion features under each of the modal decoding branches is performed to obtain comprehensive query features, comprising: A target modal decoding branch is selected from each of the modal decoding branches, and a query vector is determined based on the temporal fusion features under the target modal decoding branch, and a key vector and a value vector are determined based on a splicing result or a combination result of the temporal fusion features under each of the modal decoding branches; Based on each of the current quality signals, the query vector, the key vector and the value vector, a cross-modal attention mechanism is applied to perform cross-modal fusion on the temporal fusion features under each of the modal decoding branches to obtain the comprehensive query features.

[0067] Specifically, the target modal decoding branch refers to a specific modal branch selected as the information query leading party when performing cross-modal interaction, for example, an image modal decoding branch containing rich semantic information or a point cloud modal decoding branch with accurate spatial information can be selected as the target modal decoding branch.

[0068] The query vector refers to a feature vector representing a query request in the cross-modal attention mechanism, which is used to match the information of other modalities. The key vector refers to a vector representing the feature identifier of the queried information in the cross-modal attention mechanism, which is used to calculate the matching degree with the query vector. The value vector refers to a vector representing the actual extracted and fused information content in the cross-modal attention mechanism.

[0069] After obtaining the target modal decoding branch, the temporal fusion features under the target modal decoding branch can be used to generate the query vector through linear transformation or other methods. Meanwhile, the temporal fusion features under each of the modal decoding branches are spliced to obtain a splicing result, or the temporal fusion features under each of the modal decoding branches are combined in other ways to obtain a combination result, and the splicing result or the combination result is used to generate the key vector and the value vector through linear transformation or other methods.

[0070] The query vector, the key vector, the value vector and each of the current quality signals are input into the cross-modal attention mechanism module, the matching degree between the query vector and the key vector is calculated, and the value vector is weighted and summed by combining the adjustment of the current quality signal, so as to realize the interaction and aggregation of different modal feature information, and finally output the comprehensive query feature.

[0071] In the embodiment of the present application, the cross-modal attention mechanism is used to realize the deep interaction and aggregation of different modal features, and the query vector is generated by specifying the target modal, which can extract valuable information from the multi-modal splicing features in an organized way, and provide effective architecture support for the generation of comprehensive query features.

[0072] On the basis of the above-mentioned embodiment, the cross-modal fusion of the time sequence fusion features under each modal decoding branch is performed based on the cross-modal attention mechanism to obtain the comprehensive query feature, including: The attention weight is calculated based on the query vector, the key vector and the value vector, and each current quality signal; The cross-modal fusion of the time sequence fusion features under each modal decoding branch is performed based on the attention weight to obtain the comprehensive query feature.

[0073] Specifically, the attention weight refers to a coefficient value for measuring the correlation between the query vector and each key vector in the cross-modal attention mechanism, which determines the proportion of the corresponding value vector in the comprehensive query feature.

[0074] When the cross-modal attention mechanism is applied, the similarity score between the query vector and the key vector is first calculated. On this basis, each current quality signal is introduced as a bias term or a scaling factor into the calculation of the attention score. For example, the current quality signal is converted into an additive bias term added to the similarity score, or multiplied by the similarity score as a scaling coefficient, so that the modal feature with higher quality obtains a higher similarity score, and the modal feature with lower quality obtains a lower similarity score. Then, the adjusted similarity score is normalized by using the softmax function to obtain the attention weight. Finally, the value vector is weighted and summed using the calculated attention weight to complete the cross-modal fusion and output the comprehensive query feature.

[0075] In the embodiment of the present application, by directly applying the current quality signal to the calculation process of the attention weight, the contribution of the reliable modal is adaptively amplified and the interference of the degenerative modal is reduced during the cross-modal feature fusion, realizing the quality-guided fusion in a true sense and improving the reliability of the fusion feature.

[0076] On the basis of the above-mentioned embodiment, each of the initial detection results further includes a target frame attribute detection result; and the cross-modal fusion of the time sequence fusion features under each modal decoding branch to obtain the comprehensive query feature further includes: Based on the target frame attribute detection result, a geometric consistency prior is constructed; Based on the geometric consistency prior, the time sequence fusion features under each modality decoding branch are cross-modality fused to obtain comprehensive query features.

[0077] Specifically, the geometric consistency prior refers to constraint or bias information calculated based on the spatial and geometric similarity between the target box attribute detection results obtained by different modality decoding branches, which is used to measure whether different target boxes point to the same physical entity.

[0078] By comparing the target box attribute detection results obtained by different modality decoding branches, the center distance, size similarity, and orientation angle similarity between target boxes can be calculated. For example, the closer the center distance, the more similar the size and orientation of the target boxes, and the higher the geometric consistency score corresponding to the target boxes. In this way, the geometric consistency prior is constructed.

[0079] When performing cross-modality fusion, the constructed geometric consistency prior is introduced as an additional bias term into the weight calculation of the cross-modality attention mechanism, or the geometric consistency prior is used to filter target boxes before cross-modality fusion, such as filtering target boxes that are too far apart or have too large attribute differences. In this way, time sequence fusion features that are close in space and consistent in geometric attributes have higher fusion weights when performing cross-modality fusion, and finally comprehensive query features combined with geometric spatial constraints are obtained.

[0080] In the embodiments of the present application, the geometric consistency prior is constructed by target box attribute detection results and introduced into the cross-modality fusion process, which can effectively avoid the incorrect fusion of time sequence fusion features that are not related in space or have large attribute differences, and improve the accuracy and rationality of cross-modality feature aggregation.

[0081] On the basis of the above-mentioned embodiments, when training the target detection model, a random modality discard or degradation enhancement strategy can be used to process the training samples, so that the target detection model is exposed to multiple sensor degradation scenarios during the training process, to improve the robustness in actual degradation scenarios.

[0082] At least one degradation mode can be randomly selected in each training iteration, random discard is performed on the lidar point cloud data, for example, part of the points or point cloud subsets are randomly discarded, random view shielding is performed on the camera image data input, for example, part of the camera views or local regions are randomly shielded by applying a shielding mask. Noise, blur, occlusion, etc. can also be applied to the lidar point cloud data and the camera image data to simulate degradation disturbances.

[0083] The random modality discard or degradation enhancement strategy does not rely on additional degradation labels or explicit sensor quality annotations, and can promote the target detection model to learn to stably construct detection quality signals under degradation conditions and complete fusion accordingly.

[0084] As Figure 2 shown, a complete flowchart of a multi-modal spatio-temporal fusion three-dimensional target detection method provided by an embodiment of the application is shown, and the method comprises: acquiring laser radar point cloud data and camera image data at a current time; encoding the camera image data using an image encoder to obtain image feature encoding, and encoding the laser radar point cloud data using a radar encoder to obtain point cloud feature encoding; determining current query features corresponding to image decoding branches, point cloud decoding branches and fusion decoding branches using pre-determined target query features and the image feature encoding and the point cloud feature encoding, and using the current query features, applying the image decoding branches, the point cloud decoding branches and the fusion decoding branches to determine initial detection results; constructing current quality signals using the category detection results; using the current quality signals to perform quality-aware temporal fusion of the current query features and historical query features under corresponding modal decoding branches to obtain temporal fusion features under the modal decoding branches, and performing cross-modal fusion on the temporal fusion features under the modal decoding branches to obtain comprehensive query features; using the comprehensive query features to determine a three-dimensional target detection result at the current time.

[0085] As Figure 3 shown, on the basis of the above-mentioned embodiments, a multi-modal spatio-temporal fusion three-dimensional target detection system is provided in an embodiment of the application, and the system comprises: a data acquisition module 31 configured to acquire multi-modal sensor data at a current time, wherein the multi-modal sensor data at least comprises laser radar point cloud data and camera image data; an encoding module 32 configured to encode the laser radar point cloud data and the camera image data respectively to obtain multi-modal feature encoding; a decoding module 33 configured to determine current query features corresponding to at least two modal decoding branches based on pre-determined target query features and the multi-modal feature encoding, and based on the current query features, apply the at least two modal decoding branches to determine at least two initial detection results; each of the initial detection results comprises a category detection result; a quality signal construction module 34 configured to construct current quality signals based on each of the category detection results; the current quality signals are used to represent the reliability of the current query features under corresponding modal decoding branches; The feature fusion module 35 is configured to perform quality-aware temporal fusion of each current query feature and historical query features under a corresponding modal decoding branch based on the current quality signals, to obtain temporal fusion features under each modal decoding branch, and perform cross-modal fusion on the temporal fusion features under each modal decoding branch to obtain comprehensive query features. The target detection module 36 is configured to determine a three-dimensional target detection result of the current moment based on the comprehensive query features.

[0086] Specifically, the functions of each module in the multi-modal spatio-temporal fusion three-dimensional target detection system provided in the embodiments of the present application are one-to-one corresponding to the operation processes of each step in the method embodiment described above, and the effects achieved are consistent. For details, refer to the above embodiments, and the present application will not be repeated here.

[0087] Figure 4 An example of an entity structure diagram of an electronic device is shown in FIG. 8. Figure 4 As shown in FIG. 8, the electronic device can include a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 can communicate with each other through the communications bus 840. The processor 810 can invoke the logical instructions in the memory 830 to execute the multi-modal spatio-temporal fusion three-dimensional target detection method provided in each embodiment described above.

[0088] In addition, the logical instructions in the memory 830 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application or the part that contributes to the related art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0089] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the multi-modal spatio-temporal fusion three-dimensional target detection method provided in each of the above embodiments.

[0090] In yet another aspect, the present application also provides a computer readable storage medium, which stores a computer program, the computer program is executed by a processor to implement the multi-modal spatio-temporal fusion three-dimensional target detection method provided in each of the above embodiments. The computer readable storage medium can be a non-transitory computer readable storage medium or a transitory computer readable storage medium, which is not specifically limited here.

[0091] The system embodiments described above are only illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0092] From the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and necessary general hardware platform, and of course, it can also be realized by hardware. Based on such understanding, the above technical solutions or the part that contributes to the related art can be embodied in the form of software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the method described in each embodiment or some part of the embodiment.

[0093] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-modal spatio-temporal fusion three-dimensional object detection method, characterized in that, The method comprises: acquiring multi-modal sensor data at a current time, the multi-modal sensor data comprising at least lidar point cloud data and camera image data; encoding the lidar point cloud data and the camera image data respectively to obtain multi-modal feature encoding; based on a predetermined target query feature and the multi-modal feature encoding, determining current query features corresponding to at least two modal decoding branches, and based on each of the current query features, determining at least two initial detection results; each of the initial detection results comprises a category detection result; based on each of the category detection results, constructing a current quality signal; the current quality signal is used to represent the reliability of the current query feature under the corresponding modal decoding branch; based on each of the current quality signals, performing quality-aware temporal fusion on each of the current query features and historical query features under the corresponding modal decoding branch to obtain temporal fusion features under each of the modal decoding branches, and performing cross-modal fusion on the temporal fusion features under each of the modal decoding branches to obtain comprehensive query features; based on the comprehensive query features, determining a three-dimensional target detection result at the current time.

2. The multi-modal spatio-temporal fusion three-dimensional object detection method according to claim 1, wherein, each of the current quality signals comprises a query-level quality signal and / or a frame-level quality signal, and each of the category detection results comprises a category probability distribution; the current quality signal is constructed based on each of the category detection results, comprising: based on the category probability distribution, determining a detection confidence and a detection uncertainty; based on the detection confidence and the detection uncertainty, determining the query-level quality signal, and based on the query-level quality signal, determining the frame-level quality signal. 3.The multi-modal spatio-temporal fusion three-dimensional object detection method of claim 1, wherein, It also comprises: acquiring historical information related to the corresponding historical query features under each of the modal decoding branches, the historical information comprising historical pose information, historical speed information and historical quality signals; based on each of the current quality signals, performing quality-aware temporal fusion on each of the current query features and historical query features under the corresponding modal decoding branch to obtain temporal fusion features under each of the modal decoding branches, comprising: for any modal decoding branch, based on the current quality signal corresponding to the any modal decoding branch, applying a prior bias to perform quality-aware temporal fusion on the current query feature corresponding to the any modal decoding branch and the historical query feature under the any modal decoding branch to obtain the temporal fusion feature under the any modal decoding branch; the prior bias comprises at least one of a time decay prior, a pose alignment prior, a motion consistency prior and a historical quality prior.

4. The multi-modal spatio-temporal fusion three-dimensional object detection method of claim 1, wherein, the cross-modal fusion of the temporal fusion features under each of the modal decoding branches to obtain comprehensive query features comprises: selecting a target modal decoding branch from each of the modal decoding branches, determining a query vector based on the temporal fusion feature under the target modal decoding branch, and determining a key vector and a value vector based on a splicing result or a combination result of the temporal fusion features under each of the modal decoding branches; Based on each of the current quality signals, the query vector, the key vector, and the value vector, a cross-modal attention mechanism is applied to perform cross-modal fusion on the time sequence fusion features under each of the modal decoding branches to obtain the comprehensive query feature.

5. The multi-modal spatio-temporal fusion three-dimensional object detection method according to claim 4, characterized in that, The cross-modal attention mechanism is applied to perform cross-modal fusion on the time sequence fusion features under each of the modal decoding branches to obtain the comprehensive query feature based on each of the current quality signals, the query vector, the key vector, and the value vector. Attention weights are calculated based on the query vector, the key vector, and the value vector and each of the current quality signals. The time sequence fusion features under each of the modal decoding branches are cross-modal fused based on the attention weights to obtain the comprehensive query feature.

6. The multi-modal spatio-temporal fusion three-dimensional object detection method according to any one of claims 1-5, characterized in that, Each of the initial detection results further includes a target box attribute detection result. The cross-modal fusion of the time sequence fusion features under each of the modal decoding branches to obtain the comprehensive query feature further includes: A geometric consistency prior is constructed based on the target box attribute detection result. The time sequence fusion features under each of the modal decoding branches are cross-modal fused based on the geometric consistency prior to obtain the comprehensive query feature.

7. The multi-modal spatio-temporal fusion three-dimensional object detection method according to any one of claims 1-5, characterized in that, The comprehensive query feature is input into a three-dimensional target detection head to obtain the three-dimensional target detection result output by the three-dimensional target detection head. The comprehensive query feature is input into a three-dimensional target detection head to obtain the three-dimensional target detection result output by the three-dimensional target detection head. 8.A multi-modal spatio-temporal fusion three-dimensional object detection system, characterized in that, The data acquisition module is configured to acquire multi-modal sensor data at a current time, the multi-modal sensor data including at least laser radar point cloud data and camera image data. The encoding module is configured to encode the laser radar point cloud data and the camera image data respectively to obtain multi-modal feature encodings. The decoding module is configured to determine current query features corresponding to at least two modal decoding branches based on a pre-determined target query feature and the multi-modal feature encodings, and determine at least two initial detection results based on each of the current query features and at least two modal decoding branches. Each of the initial detection results includes a class detection result. The quality signal construction module is configured to construct a current quality signal based on each of the class detection results. The current quality signal is used to represent the reliability of the current query feature under the corresponding modal decoding branch. The feature fusion module is configured to perform quality-aware time sequence fusion on each of the current query features and historical query features under the corresponding modal decoding branch based on each of the current quality signals to obtain time sequence fusion features under each of the modal decoding branches, and perform cross-modal fusion on the time sequence fusion features under each of the modal decoding branches to obtain a comprehensive query feature. The target detection module is configured to determine a three-dimensional target detection result at the current time based on the comprehensive query feature. The processor executes the computer program to implement the multi-modal spatio-temporal fusion three-dimensional target detection method of any one of claims 1-7.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, ​ 10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to implement the multi-modal spatio-temporal fusion three-dimensional target detection method in any one of claims 1-7.

Images