A 3D target tracking method for multimodal autonomous driving based on spatiotemporal fusion

Abstract

The present invention discloses a multimodal autonomous driving three-dimensional target tracking method based on time-space fusion, which belongs to the field of autonomous driving technology. The present invention mainly includes the following steps: generation of time-space fusion features; generation of multimodal fused BEV features; generation of single-frame three-dimensional target detection results; matching of target detection and target tracking; generation of single-frame three-dimensional target tracking results. Based on the three-dimensional target tracking method proposed in the present invention, the time-space features can be fused for three-dimensional target tracking tasks, and different sensor data can be fused into a unified bird's-eye view feature. Compared with the model that processes time dimension features and space dimension features separately, this model can more effectively fuse time and space features, and the fusion of multimodal can further improve the accuracy and robustness of target tracking, and enhance the safety of the autonomous driving system.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and in particular relates to a three-dimensional target tracking method for multimodal autonomous driving based on time-space fusion. Background Art

[0002] The rapid development of autonomous driving technology is driving the automotive industry into a new era. In this process, 3D object tracking technology plays a vital role. 3D object tracking refers to the continuous and precise position tracking of various objects (such as vehicles, pedestrians, bicycles, etc.) around a moving vehicle to ensure that the vehicle can accurately perceive and understand its surroundings.

[0003] In the autonomous driving system, 3D target tracking plays a key role. First, it provides the vehicle with real-time environmental perception capabilities, allowing the vehicle to promptly detect and track various obstacles and pedestrians around it, thereby avoiding potential collision risks. Second, 3D target tracking technology can help the vehicle more accurately predict the behavior of other targets, such as predicting the movement direction and speed of pedestrians, so as to better plan the vehicle's driving path and actions. In addition, 3D target tracking can also improve the vehicle's autonomous decision-making capabilities, allowing the vehicle to more flexibly respond to complex and changing traffic scenarios.

[0004] With the continuous advancement of autonomous driving technology, 3D target tracking technology is also constantly being improved and optimized. Traditional sensor data-based target tracking methods have gradually been replaced by deep learning-based methods. Deep learning-based 3D target tracking technology can use a large amount of data for training, thereby obtaining more accurate and stable target tracking results.

[0005] In autonomous driving systems, common sensors include lidar, cameras, radars, etc., each of which has different advantages and limitations. By fusing their data together, the defects of various sensors can be compensated, thereby improving the accuracy and stability of target tracking. The advantages of multimodality are mainly reflected in its enhanced environmental perception ability, improved target recognition and classification accuracy, enhanced system robustness, and improved system safety. Therefore, multimodal processing is very helpful in improving the performance of three-dimensional target tracking.

[0006] At the same time, for three-dimensional target tracking, single-frame features are usually unable to learn the continuity of target motion. Therefore, the fusion of spatiotemporal features can effectively improve the accuracy of target tracking and the continuity of target motion, enhance the system's adaptability to dynamic environments, and reduce the possibility of false detection or missed detection. Summary of the invention

[0007] In response to the problems in the prior art, the present invention proposes a multimodal autonomous driving three-dimensional target tracking method based on spatiotemporal fusion, which can improve the three-dimensional target tracking capability in autonomous driving scenarios.

[0008] The technical solution adopted by the present invention is as follows:

[0009] The present invention proposes a multi-modal autonomous driving three-dimensional target tracking method based on time-space fusion, comprising the following steps:

[0010] S1: The laser radar and camera arranged on the autonomous driving vehicle acquire the point cloud modal data and image modal data of the current frame and the previous two frames in real time, and extract the multi-frame BEV features of the point cloud modality and image modality respectively;

[0011] S2: Fusing the multi-frame BEV features of the point cloud modality and the image modality to obtain the multi-modal fused BEV features of the current frame;

[0012] S3: Based on the BEV features of the multimodal fusion of the current frame, the candidate region generation network is used to detect the 3D target of the current frame. The obtained 3D target detection result includes a series of 3D candidate frames of the current frame.

[0013] S4: performing a region of interest pooling operation on the 3D target detection result of the current frame and the 3D target tracking result of the previous frame to obtain a corresponding feature vector; concatenating the feature vector of the 3D target detection result of the current frame and the feature vector of the 3D target tracking result of the previous frame to obtain a matching matrix, and based on the matching matrix, using a self-attention mechanism to obtain a matching result;

[0014] S5: Update the 3D target tracking result of the current frame according to the matching result obtained in S4;

[0015] S6: Repeat S1-S5 to continuously iterate the three-dimensional target tracking results to achieve tracking of three-dimensional targets around the vehicle during autonomous driving.

[0016] The present invention also proposes a multimodal autonomous driving three-dimensional target tracking system based on time-space fusion, which is used to implement the above-mentioned three-dimensional target tracking method.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] The present invention designs a complete set of multimodal autonomous driving three-dimensional target tracking methods based on spatiotemporal fusion, including the fusion of spatiotemporal features, the fusion of multimodal features, target detection and target tracking. In the spatiotemporal feature fusion stage, the method can process data in the time dimension and the space dimension, and fuse them into the BEV feature space, which can effectively improve the accuracy of target tracking. In the multimodal feature fusion stage, the data of multiple sensors can be effectively fused, combining the advantages of different sensors. In the target detection and target tracking stages, the method can complete the target detection task of the current frame, and match it with the target tracking result of the previous frame to complete the target tracking task of the current frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a structural block diagram of a multi-modal autonomous driving three-dimensional target tracking method based on spatiotemporal fusion proposed by the present invention;

[0020] Figure 2 This is a flow chart of a multimodal autonomous driving three-dimensional target tracking method based on spatiotemporal fusion proposed by the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] In the autonomous driving system, three-dimensional target tracking technology plays a key role. At present, the three-dimensional target tracking technology based on deep learning has gradually become the mainstream. The present invention proposes a multimodal autonomous driving three-dimensional target tracking method based on time-space fusion, which enhances the environmental perception ability, improves the accuracy of target recognition and classification, enhances the robustness of the system and improves the safety of the system. At the same time, the time-space features are integrated, which can effectively improve the accuracy of target tracking and the continuity of target movement, enhance the system's adaptability to dynamic environments, and reduce the possibility of false detection or missed detection.

[0023] The flowchart of the multi-modal autonomous driving three-dimensional target tracking method based on time-space fusion of the present invention is as follows: Figure 1 As shown, the following steps are included:

[0024] S1: collect point cloud modal data and image modal data in real time, select the data of the current frame and its previous two frames to extract multi-frame BEV features of point cloud modality and image modality respectively;

[0025] S2: Multimodal feature fusion obtains the multimodal fused BEV feature of the current frame;

[0026] S3: Use the candidate region generation network to detect the three-dimensional target in the current frame;

[0027] S4: Combine the region of interest pooling operation with the self-attention mechanism to match the 3D target detection result of the current frame with the 3D target tracking result of the previous frame;

[0028] S5: Update the three-dimensional target tracking result according to the matching result;

[0029] Repeat the above steps to iteratively update the three-dimensional target tracking results to achieve tracking of three-dimensional targets around the vehicle during autonomous driving.

[0030] The structural block diagram of the present invention is as follows Figure 2 As shown, the specific implementation steps of the multimodal autonomous driving three-dimensional target tracking method based on time-space fusion include:

[0031] S1. Place lidar and cameras on the autonomous vehicle to obtain the multimodal data of the latest three frames of lidar and camera, including point cloud modal data collected by the lidar and image modality data collected by the camera The corresponding three frames of lidar data are: , , , where the most recent frame is called the current frame. Similarly, the corresponding three frames of camera data are , , . Use different feature generation networks to generate multi-frame mixed modal data and Converted to the BEV perspective, each frame of the BEV features of each modality has the same spatial resolution , the same feature dimension , the specific calculation formula is as follows:

[0032]

[0033]

[0034] in, Represents the BEV features of the point cloud modality, Represents the BEV features of the image modality. Similar to the input data, , , Represent the BEV features of the latest three frames respectively.

[0035] S2. The multi-frame BEV features of different modalities obtained in S1 are used to learn the weights for fusing BEV features of different modalities, and the BEV features of different modalities are adaptively fused together in an appropriate manner, including:

[0036] (2.1) Generate an adaptive fusion weight of a corresponding point cloud modality BEV feature from the multi-frame BEV feature of the LiDAR point cloud modality , to achieve the time dimension fusion under the point cloud mode, its expression is as follows:

[0037]

[0038] in It is a multi-level four-dimensional convolutional neural network. After downsampling and upsampling, an adaptive fusion weight with the same receptive field as the original BEV feature is obtained.

[0039] (2.2) Generate an adaptive fusion weight of the corresponding image modality BEV feature from the multi-frame BEV feature of the camera image modality , to achieve the time dimension fusion under the image modality, its expression is as follows:

[0040]

[0041] in It is a multi-level four-dimensional convolutional neural network. After downsampling and upsampling, an adaptive fusion weight with the same receptive field as the original BEV feature is obtained.

[0042] (2.3). Normalize the weights of (2.1) and (2.2), and the expression is as follows:

[0043] [ , ,

[0044] in, Represents the normalization function, which is implemented by SoftMax function in this model.

[0045] (2.4). The BEV features of the last three frames of point cloud modality and image modality and Perform frame-by-frame superposition to generate preliminary fused BEV features , further generating a corresponding adaptive fusion weight , which is expressed as follows:

[0046]

[0047]

[0048] in It is a multi-layer convolutional neural network. After downsampling and upsampling, an adaptive fusion weight with the same receptive field as the original BEV feature is obtained.

[0049] (2.5). Adaptive fusion weight Normalized and fused with multi-frame modality BEV features Multiply them together and get the multimodal fused BEV features in the form of residual, which is expressed as follows:

[0050]

[0051] in represents the normalized adaptive fusion weight, Represents the multi-frame BEV features of multi-modal fusion, They represent the BEV features of multimodal fusion of the latest three frames respectively.

[0052] S3. Through the multi-frame BEV features of multi-modal fusion obtained in S2, the adaptive fusion BEV features of the most recent frame are selected to complete the 3D target detection task of the current frame through the Region Proposal Network (RPN), and a series of 3D candidate boxes are generated. The target detection results will be used to match the target tracking results of the previous frame, including:

[0053] (3.1) Input multi-frame BEV features of multi-modal fusion and select the adaptive fusion BEV features of the nearest frame As input, after multi-layer convolution feature extraction, the high-dimensional features of the nearest frame are obtained. , which is expressed as follows:

[0054]

[0055] in Represents multiple layers of convolution for feature extraction.

[0056] (3.2). The high-dimensional features obtained in step (3.1) After being processed by a detection head and a classification head, a series of 3D candidate frames of the current frame are obtained. And the corresponding detection category , complete the 3D target detection task of the current frame, and its expression is as follows:

[0057]

[0058]

[0059] in Indicates the detection head. Represents the classification head. Each 3D candidate box represents a 3D detected object in the current scene, and the corresponding detection category can be used to display the category information of the tracked target when outputting the tracking image.

[0060] S4 obtains the features of the 3D candidate frame in the multi-frame BEV feature map obtained in S2 through the 3D target detection result of the current frame obtained in S3 and the 3D target tracking result of the previous frame through the pooling operation of the region of interest, and uses the BEV features to match the target detection and target tracking results, specifically including:

[0061] (4.1) The 3D target detection results of the current frame obtained by S3, including the 3D candidate frames of the current frame and the corresponding detection categories, are pooled in the region of interest to obtain all the 3D candidate frames of the current frame. BEV features on feature map:

[0062] First, for each 3D candidate box , extend the candidate box outward a certain distance in the three directions of length, width and height to obtain the expanded three-dimensional candidate box , to obtain more spatial information around the 3D object. , according to its position in the BEV feature map of multimodal fusion, select a set of feature vectors corresponding to it in the BEV feature map, and use the maximum pooling method to obtain a one-dimensional feature vector representing this three-dimensional candidate box. For a scene with M detected objects in a certain frame, all three-dimensional candidate boxes in the scene are The BEV features on the feature map are , where C represents the dimension of BEV features.

[0063] (4.2). The 3D target tracking result of the previous frame , after the pooling operation of the region of interest, we get All 3D candidate boxes in BEV features on feature map:

[0064] The 3D target tracking result of the previous frame includes N tracked objects and their 3D pre-selected boxes in the previous frame. First, for each target tracking result, the 3D candidate box , extend the candidate box outward a certain distance in the three directions of length, width and height to obtain the expanded three-dimensional candidate box , to obtain more spatial information around the 3D object. , according to its position in the BEV feature map, select a set of feature vectors corresponding to it in the BEV feature map, and use the maximum pooling method to obtain a one-dimensional feature vector representing this 3D candidate frame. For a scene with N tracked objects in a frame, all 3D candidate frames in the scene are BEV features on feature map , where C represents the dimension of BEV features.

[0065] (4.3) The two sets of feature vectors of the target detection task and the target tracking task obtained by (4.1) and (4.2) are concatenated in feature dimension to obtain the matching matrix between the detection result and the tracking result, which is expressed as follows:

[0066]

[0067] For the matching matrix L, it is mapped to the query vector of the self-attention mechanism through linear transformation , key vector , value vector , the matching relationship between the detection result and the tracking result is analyzed through the self-attention mechanism. This process is based on the Transformer structure calculation, and its expression is as follows:

[0068]

[0069]

[0070] in, Used to calculate the correlation matrix between the two; the σ function uses the Softmax function to normalize the correlation matrix; It is used to compress high-dimensional features to low dimensions. The final output features It is used to indicate the matching relationship between the detection result and the tracking result, including the matching results between each pre-selected box in the detection result and each pre-selected box in the tracking result.

[0071] S5. The target detection result obtained in S4 is matched with the target tracking result, and the target tracking result is continuously iterated, and finally the three-dimensional target tracking result of the multi-frame input is output, which specifically includes:

[0072] (5.1). Output matrix obtained by S4 The target detection result of the current frame is matched with the target tracking result of the previous frame. The matching results can be divided into three categories, and the 3D target tracking results are updated according to different matching results:

[0073] Category 1: For a pre-selected box in the detection result, if it matches the pre-selected box of a tracked object in the tracking result, the pre-selected box of the tracked object in the tracking result is updated to the pre-selected box in the detection result;

[0074] Category 2: If a pre-selected box in the detection result does not match the pre-selected box of any tracked object in the tracking result, it is judged that a new detection object has entered the scene, and the object and its pre-selected box are added to the tracking result;

[0075] Category 3: If the pre-selected box of the tracked object in the tracking result does not match any pre-selected box in the detection result, it is judged that the tracked object has left the scene, and whether to delete its tracking track is determined based on its existence time; if its departure time exceeds the threshold, the object and its pre-selected box are deleted from the tracking result, and if the departure time does not exceed the threshold, the tracked object and its pre-selected box are still retained.

[0076] (5.2). The 3D target tracking result of the current frame obtained by (5.1) will be used in the target tracking method of the next frame, as the target tracking result of the previous frame to match the target detection result of the next frame. The target tracking result is continuously updated iteratively to achieve the tracking of the 3D target around the vehicle during the autonomous driving process.

[0077] In this embodiment, a multimodal autonomous driving three-dimensional target tracking system based on spatiotemporal fusion is also provided, which is used to implement the above embodiment. The terms "module", "unit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the system described in the following embodiments is preferably implemented in software, it is also possible to implement hardware, or a combination of software and hardware.

[0078] The multimodal autonomous driving three-dimensional target tracking system based on time-space fusion includes:

[0079] A feature extraction module is used to collect point cloud modal data and image modal data, and extract multi-frame BEV features of point cloud modalities and image modalities;

[0080] Feature fusion module, used to fuse multi-frame BEV features of point cloud modality and image modality;

[0081] A three-dimensional detection module is used to detect three-dimensional targets based on the BEV features of the current frame multimodal fusion;

[0082] The matching update module is used to perform a region of interest pooling operation on the 3D target detection result of the current frame and the 3D target tracking result of the previous frame to obtain the corresponding feature vector, and obtain the matching result based on the obtained feature vector using a self-attention mechanism;

[0083] The result updating module is used to update the three-dimensional target tracking result of the current frame according to the matching result.

[0084] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0085] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory by the processor of any device with data processing capabilities and run.

[0086] The above description is only a specific embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the concept of the present application. For example, the above features are replaced with the technical features with similar functions disclosed in the present application (but not limited to) to form a technical solution.

Claims

1. A multimodal autonomous driving three-dimensional target tracking method based on spatiotemporal fusion, characterized in that: The following steps are involved: S1: The laser radar and camera arranged on the autonomous driving vehicle are used to obtain the point cloud modal data and image modal data of the current frame and the previous two frames in real time, and the multi-frame BEV features of the point cloud modality and image modality are extracted respectively; S2: Fusing the multi-frame BEV features of the point cloud modality and the image modality to obtain the multi-modal fused BEV features of the current frame; the S2 includes: 2.1) Using a multi-layer convolutional neural network, the adaptive fusion weights of the point cloud modality BEV features with the same receptive field are generated by downsampling and upsampling the multi-frame BEV features of the point cloud modality ; 2.2) Using a multi-layer convolutional neural network, by downsampling and upsampling the multi-frame BEV features of the image modality, an adaptive fusion weight of the BEV features of the image modality with the same receptive field is generated. ; 2.3) Normalize the adaptive fusion weights generated in 2.1) and 2.2) to obtain the normalized adaptive fusion weights ; 2.4) Using normalized adaptive fusion weights , the BEV features of the point cloud modality And the corresponding frame image mode Perform weighted summation to generate the BEV features of the three frames ,in and Respectively represent the BEV features of the 1st, 2nd and 3rd frames of the point cloud modality. Using a multi-layer convolutional neural network, the BEV features of the three frames are initially fused. Downsampling and upsampling are performed to generate an adaptive fusion weight with the same receptive field ; 2.5) Adaptive fusion weight Normalization, according to the normalized adaptive fusion weights BEV features initially fused with the current frame Get the BEV features of multimodal fusion of the current frame: ； in Represents the BEV features of multimodal fusion of the current frame; S3: Based on the BEV features of the multimodal fusion of the current frame, the candidate region generation network is used to detect the 3D target of the current frame. The obtained 3D target detection result includes a series of 3D candidate frames of the current frame. S4: performing a region of interest pooling operation on the 3D target detection result of the current frame and the 3D target tracking result of the previous frame to obtain a corresponding feature vector; concatenating the feature vector of the 3D target detection result of the current frame and the feature vector of the 3D target tracking result of the previous frame to obtain a matching matrix, and based on the matching matrix, using a self-attention mechanism to obtain a matching result; S5: Update the 3D target tracking result of the current frame according to the matching result obtained in S4; S6: Repeat S1-S5 to continuously iterate the three-dimensional target tracking results to achieve tracking of three-dimensional targets around the vehicle during autonomous driving.

2. The multimodal autonomous driving three-dimensional target tracking method based on spatiotemporal fusion according to claim 1, characterized in that: In S1, the step of extracting the multi-frame BEV features of the point cloud modality and the multi-frame BEV features of the image modality respectively includes: for the point cloud modality data of the current frame and the two frames before it and image modality data , respectively using feature generation networks of point cloud modality and image-based feature generation networks Convert to BEV perspective and obtain multi-frame BEV features of point cloud modality Multi-frame BEV features of image modalities ,in Respectively represent the BEV features of the 1st, 2nd and 3rd frames of the point cloud modality, Represent the BEV features of the 1st, 2nd and 3rd frames of the image modality respectively.

3. The multimodal autonomous driving three-dimensional target tracking method based on spatiotemporal fusion according to claim 1, characterized in that: In the multi-frame BEV features of the point cloud modality and the multi-frame BEV features of the image modality, each frame of BEV features has the same spatial resolution. and the same feature dimensions .

4. The multimodal autonomous driving three-dimensional target tracking method based on spatiotemporal fusion according to claim 1, characterized in that: The S3 includes: 3.1) The BEV features of the current frame multimodal fusion are extracted through multi-layer convolution to obtain the high-dimensional features of the current frame. ; 3.2) High-dimensional features of the current frame After being processed by a detection head and a classification head respectively, a series of 3D candidate frames and corresponding detection categories of the current frame are obtained, and the 3D candidate frames and their detection categories are used as the 3D object detection results of the current frame.

5. The multimodal autonomous driving three-dimensional target tracking method based on spatiotemporal fusion according to claim 1, characterized in that: The S4 includes: 4.1) Perform a region of interest pooling operation on the 3D target detection result of the current frame to obtain the feature vector of all 3D candidate frames in the multimodal fused BEV feature of the current frame. , as the feature vector of the three-dimensional target detection result of the current frame, where M represents the number of three-dimensional targets in the current frame, and C represents the dimension of the BEV feature of the multimodal fusion; 4.2) Perform the same region of interest pooling operation as 4.1) on the 3D target tracking result of the previous frame to obtain the feature vector of all 3D candidate frames in the multimodal fused BEV feature of the previous frame. , as the feature vector of the 3D target tracking result of the previous frame, where N represents the number of 3D targets in the previous frame; 4.3) Concatenate the feature vectors of the three-dimensional target detection results and the three-dimensional target tracking results obtained in 4.1) and 4.2) in terms of feature dimensions to obtain a matching matrix L between the detection results and the tracking results; 4.4). The matching matrix L is mapped to the query vector of the self-attention mechanism through linear transformation , key vector Sum value vector , the matching results of 3D target detection results and 3D target tracking results are obtained by using Transformer-based structural decoding: ； ； Among them, σ represents the activation function, represents an MLP network used to compress high-dimensional features into low-dimensional ones. Represents the output matrix, which contains the matching results between each pre-selected box in the detection result and the pre-selected box of each object in the tracking result.

6. The multimodal autonomous driving three-dimensional target tracking method based on spatiotemporal fusion according to claim 5 is characterized in that: In the above 4.1) and 4.2), the region of interest pooling operation includes: For each 3D candidate box in the 3D target detection result or 3D target tracking result to be processed, the candidate box is extended outward along the three directions of length, width and height to obtain an extended 3D candidate box; for the extended 3D candidate box, a group of BEV feature vectors corresponding to it in the multimodal fused BEV features are selected, and the maximum pooling method is adopted to obtain a one-dimensional BEV feature vector representing the 3D candidate box.

7. The multimodal autonomous driving three-dimensional target tracking method based on spatiotemporal fusion according to claim 5, characterized in that: The S5 includes: For the pre-selected box in the detection result, if it matches the pre-selected box of an object in the tracking result, the pre-selected box of the object in the tracking result is updated to the corresponding pre-selected box in the detection result; If there is a pre-selected box in the detection result that does not match the pre-selected box of any object in the tracking result, it is judged that there is a new object entering the scene, and the object and its pre-selected box are added to the tracking result; If the pre-selected box of an object in the tracking result does not match any pre-selected box in the detection result, it is determined that the tracked object has left the scene, and whether to retain the object in the tracking result is determined based on its leaving time.

8. The multimodal autonomous driving three-dimensional target tracking method based on spatiotemporal fusion according to claim 7 is characterized in that: The decision of whether to retain the object in the tracking result based on its departure time is as follows: if the departure time of the object exceeds a preset threshold, the object and its pre-selected box are deleted from the tracking result; if the departure time does not exceed the threshold, the object and its pre-selected box are still retained.

9. A multi-modal autonomous driving three-dimensional target tracking system based on time-space fusion, characterized in that: For implementing the three-dimensional target tracking method according to claim 1, the system comprises: A feature extraction module is used to collect point cloud modal data and image modal data, and extract multi-frame BEV features of point cloud modalities and image modalities; Feature fusion module, used to fuse multi-frame BEV features of point cloud modality and image modality; A three-dimensional detection module is used to detect three-dimensional targets based on the BEV features of the current frame multimodal fusion; The matching update module is used to perform a region of interest pooling operation on the 3D target detection result of the current frame and the 3D target tracking result of the previous frame to obtain the corresponding feature vector, and obtain the matching result based on the obtained feature vector using a self-attention mechanism; The result updating module is used to update the three-dimensional target tracking result of the current frame according to the matching result.

Images