A 3D target detection method based on multi-frame temporal information

By using a multi-frame temporal information fusion method based on the TSIF3D model, the detection difficulties caused by occlusion and motion in BEV3D target detection are solved, improving detection accuracy and efficiency while reducing computational costs.

CN118736558BActive Publication Date: 2026-11-17XI AN JIAOTONG UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410664005.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-27
Publication Date
2026-11-17
Estimated Expiration
2044-05-27

AI Technical Summary

Technical Problem

Existing 3D target detection methods in the BEV3D field are difficult to perform accurately and efficiently due to factors such as target object occlusion and the movement of the vehicle itself and the target object. Furthermore, traditional temporal information fusion methods suffer from long-term memory failure and high computational costs.

Method used

A 3D target detection method based on multi-frame temporal information is adopted. The method uses the TSIF3D model and combines a feature pyramid module, a multi-scale spatial cross-attention encoder module, a long-short-term feature encoder module, and a historical query fusion decoder module. The long-short-term feature encoder is used to fuse spatial and temporal information of historical frame features, and the historical query fusion decoder module fuses temporal information in the decoding stage to improve detection accuracy.

Benefits of technology

This improved the detection accuracy of occluded and moving objects, reduced computational costs, and enhanced the model's detection performance and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118736558B_ABST
    Figure CN118736558B_ABST
Patent Text Reader

Abstract

This invention discloses a 3D object detection method based on multi-frame temporal information, comprising: acquiring a 3D image to be detected; inputting the 3D image to be detected into a TSIF3D model to obtain 3D object detection results, wherein the TSIF3D model is trained using a training set, which consists of collected historical 3D images, and the TSIF3D model comprises a feature pyramid module, a multi-scale spatial cross-attention encoder module, a long short-term feature encoder module, and a historical query fusion decoder module. This invention addresses the problems of insufficient detection accuracy caused by long-term memory failure and failure to utilize query-level temporal information in traditional temporal fusion methods, while reducing computational costs and improving object detection performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of 3D target detection technology, and in particular relates to a 3D target detection method based on multi-frame temporal information. Background Technology

[0002] Currently, most 3D object detection research focuses on single-frame images. In single-frame image-based 3D object detection methods, the emphasis is on how to better model the spatial information within the image. However, besides spatial information, temporal information is also crucial for 3D object detection tasks. For example, without temporal clues, it is difficult to accurately infer the speed of moving objects or the size of occluded objects from static image features. Modeling the spatiotemporal information of multiple frames can compensate for these deficiencies.

[0003] Furthermore, for the detection of occluded objects in images, the information in a single frame is insufficient to capture the occluded portion, making it difficult for the detection head to accurately predict its size and position. However, in multi-frame images, feature information of the occluded object in historical frames can be used to complete the feature information of the current frame, thus improving detection performance. Existing models for temporal information fusion can be divided into feature-level fusion methods and query-level fusion methods.

[0004] In summary, in the field of BEV3D object detection, the accuracy and efficiency of object detection are difficult to achieve due to factors such as object occlusion, egomotion, and the movement of both the vehicle and the object. Existing research utilizes temporal information to optimize detection accuracy and performance; however, the complex dynamic characteristics of multi-frame images make temporal information modeling extremely difficult, traditional methods offer limited improvement in detection accuracy, and the performance loss caused by complex fusion calculation modules is detrimental to practical applications. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a 3D target detection method based on multi-frame temporal information. This method solves the problems of insufficient detection accuracy caused by long-term memory failure and failure to utilize query-level temporal information in traditional temporal fusion methods, while also reducing computational costs and improving target detection performance.

[0006] To achieve the above objectives, the present invention provides a 3D target detection method based on multi-frame temporal information, comprising:

[0007] Acquire the 3D image to be detected;

[0008] The 3D image to be detected is input into the TSIF3D model to obtain the 3D target detection result. The TSIF3D model is trained using a training set, which consists of historical 3D images. The TSIF3D model is composed of a feature pyramid module, a multi-scale spatial cross-attention encoder module, a long short-term feature encoder module, and a historical query fusion decoder module.

[0009] Optionally, the 3D image to be detected is input into the TSIF3D model to obtain the 3D object detection result, including:

[0010] Based on the feature pyramid module, multi-view feature maps are obtained;

[0011] The multi-view feature map is input into the multi-scale spatial cross-attention encoder module to obtain the current frame feature map;

[0012] The historical frame features and the current frame feature map are input into the long-short-term feature encoder module to obtain the fused feature map;

[0013] Based on the fused feature map and the historical query fusion decoder module, the 3D target detection results are obtained.

[0014] Optionally, the historical features and the current frame feature map are input into the long short-term feature encoder module to obtain the fused feature map, including:

[0015] A short-term feature encoder is used to fuse features from adjacent historical frames to obtain fused spatial information features.

[0016] The historical frame features are fused using a long-term feature encoder to obtain fused temporal information features;

[0017] The spatial and temporal information features are stacked and fused using convolutional layers to obtain a fused feature map.

[0018] Optionally, the method for obtaining the fused spatial information features is as follows:

[0019]

[0020] in, The fused spatial information features are represented by pos, which is the position index on the BEV plane. For short-term queries, Features of the current frame Features of historical frames.

[0021] Optionally, the method for obtaining the fused temporal information features is as follows:

[0022]

[0023] in, For the fused time information features, For long-term BEV feature queries, W r To reduce the number of channels in the network, r represents the reduction rate and the default value is set to 4. Let N be the set of all historical frame features, where t is the current frame time and N is the number of historical frame features.

[0024] Optionally, based on the fused feature map and the historical query fusion decoder module, the 3D target detection results are obtained including:

[0025] Obtain historical query results and predefined queries, and merge the historical query results and predefined queries to obtain merged time-series information queries;

[0026] The fused temporal information query and fused feature map are input into the historical query fusion decoder module to obtain the 3D target detection results.

[0027] Optionally, merging the historical query results and predefined queries includes:

[0028] The historical query results are input into the motion sensing layer normalization network to obtain the aligned historical query results;

[0029] The aligned historical query results are added bitwise to the predefined query to obtain the fused time-series information query.

[0030] Optionally, the historical query results are input into the motion sensing layer normalization network to obtain aligned historical query results, including:

[0031] Input the pose transformation matrix, time information and velocity information, perform a flattening operation on the pose transformation matrix, time information and velocity information, input the flattened pose transformation matrix, time information and velocity information into the linear layer, and obtain the first affine transformation vector and the second affine transformation vector;

[0032] After performing layer normalization on the historical query results, the aligned historical query results are calculated by combining them with the first affine transformation vector and the second affine transformation vector.

[0033] Optionally, the first affine transformation vector is:

[0034]

[0035] Where γ is the first affine transformation vector. Let v be the pose transformation matrix from time t-1 to time t, v be the velocity information, Δt be the time interval between two frames, and Linear1 be the linear layer network.

[0036] The second affine transformation vector is:

[0037]

[0038] Where β is the second affine transformation vector, and Linear2 is a linear layer network;

[0039] The aligned historical query results are as follows:

[0040]

[0041] in, For the aligned historical query results, LN indicates layer normalization. These are historical query results.

[0042] Compared with the prior art, the present invention has the following advantages and technical effects:

[0043] 1. In the field of BEV3D object detection, the accuracy and efficiency of object detection are difficult to achieve due to factors such as occlusion of target objects and the motion of both the vehicle and the target object. Existing research utilizes temporal information to optimize and supplement detection accuracy and performance. However, due to the complex dynamic characteristics of multi-frame images, modeling temporal information is extremely difficult, and traditional methods offer limited compensation for detection accuracy. Furthermore, the performance loss caused by complex fusion calculation modules is detrimental to practical applications. This invention proposes a method that addresses this problem by designing two modules to model the detection information, thereby improving detection performance and accuracy.

[0044] 2. Traditional methods, in order to utilize more information from historical frames to address issues such as target occlusion and motion, often iteratively fuse features from historical frames pairwise or stack all features along the channels. This can lead to long-term memory failure or computational cost issues, resulting in low computational efficiency and failing to significantly improve the model's detection accuracy. The method proposed in this invention uses a long short-term memory feature encoder module, which can utilize features from historical frames over a longer time span without reducing model performance, thereby improving the detection accuracy of occluded and moving targets.

[0045] 3. Traditional methods rarely focus on temporal information during the decoding stage. The decoder only uses predefined queries to perform target queries on BEV features, which makes it difficult to solve the problem of detecting moving target objects. This invention proposes a Historical Query Fusion Decoder Module (HQFD), which explores a query-level temporal fusion method by fusing historical object queries into the current query.

[0046] 4. This invention enables the model to effectively model temporal features and decouple the motion information of self-vehicles and moving target objects, thereby improving the detection accuracy of occluded target objects and moving target objects. Attached Figure Description

[0047] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0048] Figure 1 This is a flowchart of a 3D target detection method based on multi-frame temporal information according to an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the overall structure of TSFF3D according to an embodiment of the present invention;

[0050] Figure 3 This is a structural diagram of the long-short-term feature encoder according to an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of the short-term feature encoder structure according to an embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram of the long-term feature encoder structure according to an embodiment of the present invention;

[0053] Figure 6 This is a schematic diagram of the historical query fusion decoder module structure according to an embodiment of the present invention;

[0054] Figure 7 This is a schematic diagram of the motion sensing layer normalization module structure according to an embodiment of the present invention;

[0055] Figure 8 This is a schematic diagram illustrating the detection results of subsets of visibility at different levels according to an embodiment of the present invention. Figure 8 (a) is a schematic diagram of the recall results. Figure 8 (b) is a schematic diagram of the model's displacement detection results. Figure 8 (c) is a schematic diagram of the model's detection results for the direction. Figure 8 (d) is a schematic diagram of the model's detection results for velocity;

[0056] Figure 9 This is a schematic diagram of the visualization results in the BEV space according to an embodiment of the present invention. Figure 9 (a) is a schematic diagram of the BEVFormer-S visualization results. Figure 9 (b) is a schematic diagram of the BEVFormer visualization results. Figure 9 (c) is a schematic diagram of the TSIF3D visualization results;

[0057] Figure 10 This is a schematic diagram of the image visualization of the detection results in an embodiment of the present invention. Detailed Implementation

[0058] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0059] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0060] This invention proposes a 3D target detection method based on multi-frame temporal information, such as... Figure 1 As shown, the specific steps include:

[0061] Acquire the 3D image to be detected;

[0062] The 3D image to be detected is input into the TSIF3D model. Figure 2 The system obtains 3D object detection results. The TSIF3D model is trained using a training set, which consists of historical 3D images. The TSIF3D model is composed of a feature pyramid module, a multi-scale spatial cross-attention encoder module, a long and short term feature encoder module, and a historical query fusion decoder module.

[0063] Specifically, to ensure the model possesses the ability to detect target objects in real-world autonomous driving scenarios, it needs to be trained using collected real-world scene data to enable accurate and efficient target detection. The data can be organized into the nuScenes dataset format for model preprocessing and feature extraction, or it can be trained directly using the nuScenes dataset. Training can utilize computing hardware such as GPUs.

[0064] Furthermore, the 3D image to be detected is input into the TSIF3D model to obtain 3D object detection results, including:

[0065] Based on the feature pyramid module, obtain multi-view feature maps;

[0066] The multi-view feature maps are input into the multi-scale spatial cross-attention encoder module to obtain the feature map of the current frame.

[0067] Input the historical frame features and the current frame feature map into the long-short-term feature encoder module to obtain the fused feature map;

[0068] The 3D object detection results are obtained based on the fusion feature map and historical query fusion decoder module.

[0069] Specifically, this invention obtains the features of the current frame through a multi-scale spatial cross-attention feature encoder (MSCAE). For historical frame features, this invention uses features stored in memory. To eliminate positional offsets caused by the vehicle's own motion, this invention aligns historical frame features with current frame features using a pose transformation matrix, resulting in aligned historical frame features. Will and These are input together into the Long Short-Term Feature Encoder (LSTFE), where the short-term feature encoder is used to process features from adjacent historical frames. The fusion process yields features that incorporate spatial information. Use a long-term feature encoder to process features from all historical frames. The fusion process yields features that incorporate time information. Next and Connect along the channel dimension and obtain the result through BEV convolution. Next The input is fed into the Historical Query Fusion Decoder (HQFD) module proposed in this invention. Within the HQFD module, this invention designs a motion-aware layer normalization module to add additional motion information to the historical query results, fusing it with predefined queries to obtain features incorporating spatiotemporal information. Decoding is then performed. By introducing the feature-level temporal information fusion method LSTFE and the query-level temporal information fusion method HQFD, the TSIF3D method proposed in this invention further improves the detection accuracy of the model by utilizing historical temporal information.

[0070] Furthermore, the historical features and the current frame feature map are input into the long short-term feature encoder module to obtain the fused feature map, including:

[0071] A short-term feature encoder is used to fuse features from adjacent historical frames to obtain fused spatial information features.

[0072] The features of historical frames are fused using a long-term feature encoder to obtain the fused temporal information features;

[0073] Spatial and temporal features are stacked and then fused using convolutional layers to obtain a fused feature map.

[0074] Specifically, for single-frame images, modeling BEV features mainly focuses on the contextual information between the BEV space and multiple viewpoints; while for multi-frame images, the focus is more on the contextual association information between the current frame and historical frames. To obtain the relationship between current frame features and historical frame features, some methods fuse features from adjacent frames. While this method can solve the problem of occluded targets, it cannot obtain longer-term temporal information. Therefore, some methods consider introducing historical frame features with longer time series, iteratively fusing historical frame features pairwise or stacking all features along the channels. Although this fuses features with longer time series, it can lead to long-term memory failure or computational cost issues, resulting in limited improvement in model detection accuracy. To address these problems, this invention proposes a Long Short-Term Feature Encoder (LSTFE), whose overall structure is as follows: Figure 3 As shown.

[0075] This invention proposes a temporal feature encoder consisting of two independent, specialized branches: a Short-term Feature Encoder (STFE) and a Long-term Feature Encoder (LTFE). These branches are used to acquire spatial and temporal information from historical frame features, respectively, and to achieve better temporal fusion of BEV features without compromising model performance. The input to the STFE is short-term BEV features. and long-term BEV characteristics Composition. LSTFE first analyzes the BEV features of all historical frames. The time information location embedding is added, and then the current frame features are completed by using the temporal information of historical frame features through STFE and LTFE.

[0076] Furthermore, due to the strong correlation of spatial information between adjacent frames, the short-term feature encoder primarily performs feature fusion on spatial information. The short-term feature encoder uses BEV features from adjacent frames. and current frame BEV features As input, a richer spatial representation is constructed in the BEV space. Its network structure is as follows: Figure 4 As shown, from the input set It consists of convolutional layers and deformable attention networks.

[0077] The input to STFE is the BEV feature at time t-1 after pose alignment. BEV characteristics at time t First of all and Connecting along the feature channel dimension yields intermediate features of size H×W×2C. These intermediate features are then subjected to dimensionality reduction via convolutional layers to obtain pre-fused features, which are then combined with… The summed inputs are fed into a deformable self-attention module for further encoding and fusion of features to obtain the final BEV features. As shown in formula (1):

[0078]

[0079] in, The fused spatial information features are represented by pos, which is the position index on the BEV plane. For short-term queries, Features of the current frame These are features from historical frames. However, the features obtained after short-term feature fusion encoders... Although it incorporates spatial information, it cannot analyze the same target object in... The invention models the temporal correlation information between the features on the surface and other BEV features, and therefore designs a long-term time encoder (LTFE) for the temporal information in long-term historical features.

[0080] Furthermore, the Long-Term Feature Encoder (LTFE) is used to process the entire historical BEV feature set. To perceive long-term historical motion information, increase localization potential, and help to more accurately locate and estimate the velocity of moving targets, the network structure of LTFE is as follows: Figure 5 As shown, the feature input set It consists of a channel reduction operation layer, a deformable attention layer, and a feature bitwise addition operation.

[0081] For long-term motion information, LTFE only focuses on the spatial motion of the same objects from the BEV perspective, without concerning itself with the semantic information of the same objects. Therefore, the long-term temporal encoder first processes the input set... Channel reduction is used to prune semantic information, which significantly reduces the number of parameters in LTFE, accelerating model training and inference. Given a learnable long-term BEV query... and parameters are The channel reduction layer is then processed using a variable attention mechanism to analyze the long-term historical BEV features after channel reduction. and Long-term temporal information fusion is performed, followed by bitwise addition and merging of all pairwise fused features, as shown in Formula 2:

[0082]

[0083] In the formula: For the fused time information features, For long-term BEV feature queries, W r To reduce the number of channels in the network, r represents the reduction rate and the default value is set to 4. Let N be the set of all historical frame features, where t is the current frame time and N is the number of historical frame features.

[0084] Thus, features incorporating spatial information were obtained. and features that incorporate time information After that and The features are stacked along the channel dimension and fused using 3×3 convolution to achieve complementarity of spatial and temporal information, ultimately yielding the current frame features that fuse the spatial and temporal information of historical frames.

[0085] Furthermore, based on the fusion feature map and historical query fusion decoder module, the 3D object detection results are obtained as follows:

[0086] Retrieve historical query results and predefined queries, and merge the historical query results and predefined queries to obtain merged time-series information queries;

[0087] The fused temporal information query and fused feature map are input into the historical query fusion decoder module to obtain the 3D object detection results.

[0088] Furthermore, the fusion of historical query results and predefined queries includes:

[0089] The historical query results are input into the motion perception layer normalization network to obtain the aligned historical query results;

[0090] The aligned historical query results are added bitwise to the predefined query to obtain the fused time-series information query.

[0091] Furthermore, the historical query results are input into the motion perception layer normalization network to obtain aligned historical query results, including:

[0092] Input the pose transformation matrix, time information and velocity information, perform a flattening operation on the pose transformation matrix, time information and velocity information, input the flattened pose transformation matrix, time information and velocity information into the linear layer, and obtain the first affine transformation vector and the second affine transformation vector;

[0093] After performing layer normalization on the historical query results, the aligned historical query results are calculated by combining them with the first affine transformation vector and the second affine transformation vector.

[0094] Specifically, existing research on the utilization of temporal information mostly focuses on feature-level fusion of the correlation information between historical frame features and current frame features, with few methods addressing the utilization of temporal information during the decoding stage. A major reason for this is that traditional anchor-box-based detection methods make it difficult to apply temporal information during decoding. Therefore, a new detection method is urgently needed. In existing methods, the decoder only uses a predefined query Q to perform target queries on BEV features, which is insufficient for detecting moving objects. To address these issues, this invention proposes a Historical Query Fusion Decoder Module (HQPD), which explores query-level temporal fusion methods by fusing historical object queries into the current query.

[0095] HQFD processes historical query results and merges them with predefined queries to form a refined query. This compensates for biases caused by the movement of target objects, thereby improving the model's detection accuracy for moving targets. Its overall structure is as follows: Figure 6 As shown, the historical query results of the previous frame image are first obtained. In order to align them with the current frame, the historical query results are input into the motion perception layer normalization network for motion bias elimination. The aligned historical query results are then fused with the predefined query by bitwise addition to obtain a fused query that incorporates historical temporal information. This fused query is then input into the Detr decoder along with the current frame features obtained through MSCAE to obtain the query results and provide historical temporal information for the feature decoding of the next frame.

[0096] In the decoder, each object query learns to specifically handle target regions in the BEV plane, thus each output query contains rich semantic information from that target region. When detecting features in the current frame, refining these output historical queries and fusing them with predefined object queries on the BEV feature map, rather than starting with predefined queries, makes it easier for the model to detect targets. For example, if an object query detects a stationary object in a previous historical frame, its historical object query results can help it quickly locate the object without having to search other regions in the BEV plane again. On the other hand, historical information from moving objects also helps in locating and detecting their velocity. For the BEV features of the previous frame... The detection is performed using a predefined object query, as shown in formula (3).

[0097]

[0098] In the formula: Q is a predefined query, The query result is obtained after decoding; the decoder is the decoder. The query result is from the previous frame. First, based on the self-pose matrix E of the previous frame... t-1And the self-pose matrix E of the current frame t Calculate the pose transformation matrix Its calculation expression is shown in (4).

[0099]

[0100] Assuming the target objects to be detected are all static, then the query results in the previous frame... The alignment to the current frame can be displayed, and its calculation method is shown in formula (5):

[0101]

[0102] In the formula: This is the historical query result after alignment by the pose transformation matrix. Some methods align the historical query result to the current frame using formula (5), and then fuse it with a predefined query to compensate for the deviation caused by the vehicle's own motion. This apparent motion compensation method solves the problem of detecting stationary target objects to a certain extent. However, in actual autonomous driving scenarios, the target objects to be detected are not all stationary, but more often dynamic target objects, such as moving vehicles and pedestrians.

[0103] Therefore, this invention considers incorporating more information into historical queries to eliminate biases caused by the motion of dynamic target objects. Inspired by Conditional Layer Normalization, this invention designs a Motion Awareness Layer Normalization (MLN) module to model the motion information of the target object. A schematic diagram of the MLN network structure is shown below. Figure 7 As shown, the inputs are the pose transformation matrix, time information, and velocity information. After flattening, the feature vectors are input into two linear layers to obtain affine transformation vectors γ and β. Then, the historical query results are normalized by layers and calculated with the affine transformation vectors γ and β to obtain the aligned historical query results.

[0104] Similar to conditional layer normalization, which allows scaling and offset factors to depend on additional conditional information (such as category labels or other attributes), the MLN module dynamically generates these parameters based on the target object's motion properties. Motion properties include the target object's pose transformation matrix, which reflects the changes in the target object's position and orientation in the world coordinate system; time information, which can be used to calculate the target object's trajectory; and velocity information, which represents the magnitude and direction of the target object's velocity.

[0105] Similar to conditional layer normalization, which allows scaling and offset factors to depend on additional conditional information (such as category labels or other attributes), the MLN module dynamically generates these parameters based on the target object's motion properties. Motion properties include the target object's pose transformation matrix, which reflects the changes in the target object's position and orientation in the world coordinate system; time information, which can be used to calculate the target object's trajectory; and velocity information, which represents the magnitude and direction of the target object's velocity.

[0106] In MLN, not only is the pose transformation of the target object to the current frame considered, but also... Time and velocity information are also considered. MLN first flattens these motion attribute information so that they can be processed through linear layers. Then, the flattened attribute information is processed through two linear layers and converted into affine transformation vectors γ and β, as shown in Equations (6) and (7).

[0107]

[0108]

[0109] In the formula: For additional motion attribute information, where This represents the pose transformation matrix from time t-1 to time t, where v represents velocity information and Δt represents the time interval between two frames. Then, an affine transformation is performed to obtain... By adjusting the historical query results through affine transformation, the model can not only understand the current state of the object, but also perceive the object's motion information. As shown in formula (8),

[0110]

[0111] In the formula: LN represents layer normalization. For example... Figure 6 As shown, the processed BEV query results from the previous frame are... With predefined query Q t Integration, on BEV characteristics Perform the query as shown in formula (9):

[0112]

[0113] This invention quantitatively compares the experimental results of the proposed TSIF3D method with the aforementioned comparative methods on the nuScenes dataset. To ensure fairness in the comparison, this invention controls that the proposed TSIF3D method and all comparative methods use the same backbone network for feature extraction.

[0114] Table 1 presents the experimental results of the proposed TSIF3D method and the comparison method on the nuScenes dataset test set. All methods in the table use a ResNet101 network pre-trained on the COCO dataset as the backbone network. View represents the viewpoint, Perspective represents 3D object detection from the perspective feature map, BEV represents 3D object detection from the BEV feature map, Images represents the number of input images, and Temporal indicates whether temporal information is introduced.

[0115] Table 1. Experimental results of different methods on the nuScenes dataset.

[0116]

[0117] Table 1 shows that the TSIF3D method proposed in this invention achieved the highest detection accuracy in all comparative experiments. Compared with FCOS3D, DETR3D, BEVFormer-S (which do not incorporate temporal information), and the TSBEV3D method proposed in the previous chapter, TSIF3D significantly improves detection accuracy. Compared with BEVFormer, which also incorporates temporal information, TSIF3D improves NDS and mAP by 2.2% and 1.8%, respectively, and decreases the mAVE index by 0.042 m / s. Compared with the TSBEV3D method, TSIF3D utilizes temporal information by introducing the LSTFE module and HQFD module, achieving significant progress in all three evaluation metrics. It improves NDS and mAP by 6.5% and 4.1%, respectively, and decreases the mAVE index by 0.545 m / s. The experimental results in Table 1 demonstrate the effectiveness of the TSIF3D method proposed in this invention in incorporating temporal information.

[0118] Furthermore, to evaluate the detection performance of TSIF3D for targets with different occlusion levels, this invention divides the nuScenes validation set into four subsets {0-40%, 40-60%, 60-80%, 80-100%} based on the official visibility tags provided by nuScenes. The experimental results are as follows. Figure 8 As shown.

[0119] from Figure 8(a) It can be seen that compared to DETR3D and BEVFormer-S without temporal information, TSIF3D with temporal information has a higher recall rate across all subsets, especially in the subset with the lowest visibility (0-40%), where it improves by more than 7%. Compared to BEVFormer, which also enhances temporal information, TSIF3D further improves recall across different subsets by fusing longer temporal information. (b), (d), and (e) It can be seen that temporal information is beneficial for the model to detect displacement, orientation, and velocity. (c) and (f) It can be seen that after introducing temporal information, the model's performance in target scale prediction and attribute prediction is even lower than DETR3D and BEVFormer-S without temporal information in subsets with high visibility. Therefore, it can be concluded that introducing temporal information is detrimental to the prediction of target object scale and attributes. In general, the impact of time information is mainly reflected in the following aspects: (1) The introduction of time information greatly improves the accuracy of velocity estimation; (2) Time information makes the prediction of the position and direction of objects more accurate; (3) Since time information contains past object clues, TSIF3D achieves higher recall on severely occluded objects.

[0120] In this invention, to further verify the effectiveness of the proposed method, this section presents a comparative visualization analysis of the experimental results of BEVFormer-S, BEVFormer, and the TSIF3D method proposed in this invention. The visualization results in BEV space are shown below. Figure 9 As shown in (a)-(c), the visualization results of projecting the detection results back into the image are as follows: Figure 10 As shown.

[0121] from Figure 9 As can be seen from the visualization results, the BEVFormer-S without temporal fusion exhibits significant deviations in the localization of moving objects, as shown by the detection result of the target indicated by the arrow in the figure. While BEVFormer, through cyclical temporal feature fusion, improves localization accuracy to some extent, certain deviations still exist. In contrast, the TSIF3D method proposed in this invention almost perfectly matches the ground truth in its detection result for the moving target. The visualization results also demonstrate that the TSIF3D method, which fuses multi-frame temporal information using a long-short-term feature fusion module and a historical query fusion module, outperforms the cyclical fusion method used in BEVFormer.

[0122] from Figure 10As shown in the first row, BEVFormer-S, without temporal information fusion, exhibits a significant deviation between its predicted bounding box and the ground truth bounding box for the car in motion. In contrast, BEVFormer with temporal information fusion and the proposed TSIF3D provide more accurate detection results for this target, with TSIF3D's prediction of the target angle being closer to the ground truth than BEVFormer. Furthermore, the second row shows that for the detection of moving pedestrians, TSIF3D provides a more accurate prediction of both the predicted position and angle compared to BEVFormer-S and BEVFormer. Figure 9 and Figure 10 The visualization results qualitatively verified the comparison of the quantitative experimental results in Table 1, further demonstrating the effectiveness of the TSIF3D method in time series information fusion.

[0123] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A 3D target detection method based on multi-frame temporal information, characterized in that, include: Acquire the 3D image to be detected; The 3D image to be detected is input into the TSIF3D model to obtain the 3D target detection result. The TSIF3D model is trained by a training set, which consists of historical 3D images. The TSIF3D model is composed of a feature pyramid module, a multi-scale spatial cross-attention encoder module, a long short-term feature encoder module, and a historical query fusion decoder module. The 3D image to be detected is input into the TSIF3D model to obtain the 3D object detection results, including: Based on the feature pyramid module, multi-view feature maps are obtained; The multi-view feature map is input into the multi-scale spatial cross-attention encoder module to obtain the current frame feature map; The historical frame features and the current frame feature map are input into the long-short-term feature encoder module to obtain the fused feature map; Based on the fused feature map and historical query fusion decoder module, 3D target detection results are obtained; The historical features and the current frame feature map are input into the long short-term feature encoder module to obtain the fused feature map, including: A short-term feature encoder is used to fuse features from adjacent historical frames to obtain fused spatial information features. The historical frame features are fused using a long-term feature encoder to obtain fused temporal information features; The spatial and temporal information features are stacked and fused using convolutional layers to obtain a fused feature map. The short-term feature encoder consists of an input set It consists of convolutional layers and deformable attention networks; The input to the short-term feature encoder is the pose-aligned... BEV characteristics at any time and current BEV characteristics at any time First of all and Connect along the feature channel dimension to obtain The intermediate features are then used to reduce the dimensionality of the intermediate features through convolutional layers to obtain the preliminarily fused features, which are then combined with... The summed inputs are fed into a deformable self-attention module for further encoding and fusion of features to obtain the final BEV features. ; The long-term feature encoder feature input set It consists of a channel reduction operation layer, a deformable attention layer, and a feature bitwise addition operation; Long-term time encoders first process the input set Channel reduction is used to prune semantic information, given a learnable long-term BEV query. and parameters are The channel reduction layer is then processed using a variable attention mechanism to analyze the long-term historical BEV features after channel reduction. and Long-term temporal information fusion is performed, followed by bitwise addition and merging of all pairwise fused features to obtain features that incorporate temporal information. ; The historical query fusion decoder module includes a motion perception layer normalization network and a Detr decoder; The historical query fusion decoder module first obtains the historical query result of the previous frame image, inputs the historical query result into the motion perception layer normalization network to obtain the aligned historical query result; performs a bitwise addition operation with the aligned historical query result and a predefined query to obtain the fused temporal information query, and then inputs it into the Detr decoder along with the current frame features obtained through MSCAE to obtain the query result and provide historical temporal information for the feature decoding of the next frame.

2. The 3D target detection method based on multi-frame temporal information according to claim 1, characterized in that, The method for obtaining the fused spatial information features is as follows: in, For the fused spatial information features, For the position index on the BEV plane, For short-term queries, Features of the current frame Features from the previous frame.

3. The 3D target detection method based on multi-frame temporal information according to claim 2, characterized in that, The method for obtaining the fused temporal information features is as follows: in, For the fused time information features, For long-term BEV feature queries, To reduce network bandwidth, Indicates the reduction rate. Let N be the set of all historical frame features, where t is the current frame time and N is the number of historical frame features.

4. The 3D target detection method based on multi-frame temporal information according to claim 1, characterized in that, Based on the fused feature map and historical query fusion decoder module, the 3D object detection results are obtained as follows: Obtain historical query results and predefined queries, and merge the historical query results and predefined queries to obtain merged time-series information queries; The fused temporal information query and fused feature map are input into the historical query fusion decoder module to obtain the 3D target detection results.

5. The 3D target detection method based on multi-frame temporal information according to claim 1, characterized in that, The historical query results are input into the motion sensing layer normalization network to obtain aligned historical query results, including: Input the pose transformation matrix, time information and velocity information, perform a flattening operation on the pose transformation matrix, time information and velocity information, input the flattened pose transformation matrix, time information and velocity information into the linear layer, and obtain the first affine transformation vector and the second affine transformation vector; After performing layer normalization on the historical query results, the aligned historical query results are calculated by combining them with the first affine transformation vector and the second affine transformation vector.

6. The 3D target detection method based on multi-frame temporal information according to claim 5, characterized in that, The first affine transformation vector is: in, Let be the first affine transformation vector. From Time's up The pose transformation matrix at time t. For speed information, The time interval between two frames. It is a linear layer network; The second affine transformation vector is: in, The second affine transformation vector, It is a linear layer network; The aligned historical query results are as follows: in, For the aligned historical query results, Representation layer normalization, These are historical query results.

Citation Information

Patent Citations

  • Multi-target tracking method and device based on aerial view angle, storage medium and equipment

    CN115457288A

  • Traffic flow prediction method and system based on trend space-time diagram convolution, and medium

    CN116895157A