A Deep Learning-Based Multimodal Feature-Interactive 3D Multi-Target Tracking Method

By using point cloud adaptive diffusion fusion and multi-level data association modules, the problems of information loss and poor robustness in 3D multi-target tracking are solved, achieving more efficient and reliable multi-target tracking and improving the perception and safety of autonomous driving systems.

CN118447354BActive Publication Date: 2026-11-17QINGDAO UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410616024.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-17
Publication Date
2026-11-17
Estimated Expiration
2044-05-17

AI Technical Summary

Technical Problem

Existing 3D multi-target tracking methods suffer from information loss and poor robustness during the multimodal feature fusion stage, and the accuracy and robustness of data association methods are insufficient, leading to target identity switching and trajectory fragmentation.

Method used

The system employs a point cloud adaptive diffusion fusion module, a dynamic sampling homogeneous region fusion module guided by RoI features, and a confidence-guided multi-level data association module. It densifies point cloud features through an inverse distance weight aggregation diffusion method, combines a diffusion query fusion module and an attention mechanism for feature interaction, and constructs a multi-level association matrix for data association through confidence guidance.

Benefits of technology

It significantly improves the discriminative power of features and the accuracy of data association, enhances the performance and robustness of multimodal 3D multi-target tracking, and strengthens the perception capabilities and safety of autonomous vehicles.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118447354B_ABST
    Figure CN118447354B_ABST
Patent Text Reader

Abstract

This invention relates to a deep learning-based multimodal feature interaction 3D multi-target tracking method, belonging to the fields of autonomous driving and artificial intelligence. The method comprises three innovative modules: a point cloud adaptive diffusion fusion module, a RoI feature-guided dynamic sampling homogeneous region fusion module, and a confidence-guided multi-level data association module. It utilizes depth completion to obtain pseudo-point clouds, combined with the RoI feature-guided dynamic sampling homogeneous region fusion module, to achieve fine-grained feature fusion and reduce computational load. This method enables full interaction of multimodal features, significantly improves feature discriminativeness, and effectively solves the problems of missing and incorrect matching in data association. It significantly enhances the efficiency and robustness of multimodal 3D multi-target tracking, improves the perception capabilities and safety of autonomous vehicles, and promotes the rapid development and widespread application of autonomous driving technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of autonomous driving and artificial intelligence, specifically to a deep learning-based multimodal feature interaction 3D multi-target tracking method. Background Technology

[0002] With the continuous advancement of deep learning, neural networks, and image processing technologies, computer vision has also experienced rapid development, and its application in the field of autonomous driving is becoming increasingly widespread. Among these technologies, 3D multi-object tracking is crucial for the perception of autonomous vehicles and is one of the key technologies for achieving safe, efficient, and intelligent autonomous driving systems. 3D multi-object tracking systems can detect the surrounding environment in real time, including other vehicles, pedestrians, bicycles, etc. By monitoring the position and dynamic information of these targets in real time, autonomous driving systems can better avoid collisions and predict the behavior of other road users, thereby improving road safety. Furthermore, by tracking the position and speed of other vehicles, autonomous driving systems can more intelligently plan routes, change lanes, and select speeds to optimize traffic flow, reduce traffic congestion, and improve road efficiency. Therefore, the importance of 3D multi-object tracking in autonomous driving technology is evident.

[0003] Existing 3D multi-object tracking methods, in the multimodal feature fusion stage, commonly employ two main approaches: decision-level fusion and voxel-level fusion. Both strategies project point features from the 3D point cloud space onto the 2D image space, fusing them with 2D pixel features. This direct projection of 3D points into 2D space inevitably leads to loss of semantic information in the image. Furthermore, decision-level fusion performance is often limited, and feature-level fusion has a coarse granularity, resulting in suboptimal performance. Secondly, multi-object tracking scenarios often involve complex target occlusion and small target sizes, which introduce a large number of noise points when extracting features from such targets, reducing the discriminative power of the target features.

[0004] Furthermore, existing data association methods mostly construct association matrices based on location information, ignoring target appearance information and other influencing factors. This results in poor robustness and discriminative power of the association matrices, leading to target identity switching and trajectory fragmentation. Moreover, when performing multi-level data association, existing methods all use a single similarity metric, which can cause target mismatches and missed matches, thereby reducing the accuracy of data association and the robustness of the tracking task. Summary of the Invention

[0005] This invention addresses the shortcomings of existing 3D multi-target tracking methods by providing a deep learning-based multimodal feature interaction 3D multi-target tracking method. This method comprises three innovative modules: a point cloud adaptive diffusion fusion module, a dynamic sampling homogeneous region fusion module guided by RoI features, and a confidence-guided multi-level data association module. This enables full interaction of multimodal features, significantly improves feature discriminativeness, and effectively solves the problems of missing and incorrect matching in data association. It significantly enhances the efficiency and robustness of multimodal 3D multi-target tracking, improves the perception capabilities and safety of autonomous vehicles, and promotes the rapid development and widespread application of autonomous driving technology.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The deep learning-based multimodal feature interaction 3D multi-target tracking method provided by the present invention includes the following steps:

[0007] A comprehensive network model for a multimodal 3D multi-target tracking method for autonomous driving is constructed, including a point cloud adaptive diffusion fusion module, a dynamic sampling homogeneous region fusion module guided by RoI features, and a confidence-guided multi-level data association module.

[0008] Discriminative target features are obtained based on a point cloud adaptive diffusion fusion module and a dynamic sampling homogeneous region fusion module guided by RoI feature fusion, specifically including the following:

[0009] S1 uses a point cloud adaptive diffusion fusion module to perform preliminary fusion of 2D image features and 3D point cloud features: after upscaling the image features to 3D space, the sparse point cloud is first densified by the inverse distance weight aggregation diffusion method, and then the diffusion query fusion module is used to fuse multimodal feature information to obtain fused features containing more information and reduce information loss during fusion.

[0010] S2 fusion RoI feature-guided dynamic sampling homogeneous region fusion module: First, based on the fused RoI features obtained from the point cloud adaptive diffusion fusion module, pseudo-point clouds are cropped to obtain pseudo-point cloud RoI features. Then, dynamic sampling is performed only on each pseudo-point cloud RoI feature to reduce the amount of computation.

[0011] Dynamic sampling is implemented based on the number of valid points in each RoI feature and the distance of the target in the scene. In order to improve the discriminativeness of the features, the pseudo point cloud RoI features and the point cloud image RoI features are divided into features. An attention mechanism is used to achieve dual fusion of local and global features to obtain more discriminative target features.

[0012] The S3 confidence-guided multi-level data association module improves the accuracy of data association and the robustness of tracking.

[0013] Preferably, after establishing the point cloud adaptive diffusion fusion module, the input information data is processed to generate 3D image RoI voxel features and 3D point cloud voxel features respectively; the input information data in the overall network model of the multimodal 3D multi-target tracking method includes two parts: 3D LiDAR point cloud information data and 2D camera image information data.

[0014] Preferred methods for obtaining discriminative target features from input information data include the following:

[0015] For 3D LiDAR point cloud information data, it is first voxelized, and then 3D sparse convolution with different lengths of 1, 2, 4 and 8 is used to encode the feature of the point cloud data to obtain the corresponding 3D point cloud voxel features.

[0016] For 2D camera image information data, there are two sub-branches. One is to upscale the 2D image information to 3D space and then use the point cloud adaptive diffusion fusion module to fuse it with the 3D point cloud voxel features to obtain the point cloud image RoI features. The other is to use the depth completion method to map the pixels in the 2D camera image to 3D space to obtain pseudo point clouds. Then, the features extracted by feature aggregation and 3D sparse convolution are voxelized to obtain pseudo point cloud RoI features.

[0017] Finally, the RoI features of the point cloud image and the RoI features of the pseudo point cloud are divided according to the region of interest. An attention mechanism is used on the divided features to achieve finer-grained fusion, resulting in more discriminative target features.

[0018] Preferably, the specific steps of the multi-level data association module in step S3 include:

[0019] The discriminative target features obtained in step 2 are fed into the embedding branch to obtain the target location information and appearance information required for data association. Then, based on the location information and appearance information, location affinity matrix and appearance affinity matrix are constructed respectively for association matching.

[0020] Based on the confidence level of the detection results, they are divided into high-confidence and low-confidence parts. Then, for the high-confidence detection results, a discriminative association matrix is ​​constructed using the location affinity matrix, appearance affinity matrix, and detection box aspect ratio affinity matrix for association matching. For the low-confidence detection results, three-level data association and multiple similarity measures are used for data association to improve the data association and robustness of the tracking task.

[0021] Preferably, in the adaptive diffusion of point clouds, the inverse distance weighted aggregation diffusion method is used to densify the point cloud features. The specific steps are as follows:

[0022] First, based on the number of valid point features in each voxel block, the voxel blocks are divided into empty voxel blocks, sparse voxel blocks, and dense voxel blocks. Empty voxel blocks contain no valid point features, dense voxel blocks contain five valid point features, and the rest are sparse voxel blocks. After division, empty voxel blocks are processed by aggregating the features of adjacent voxel points and calculating the average to complete their features. The specific formula is shown below:

[0023]

[0024] Where P i This represents the effective point features selected from adjacent voxels along the x-axis, y-axis, and z-axis, respectively.

[0025] For a sparse voxel block, arbitrarily select an effective point P0. Then, select effective feature points P1 and P2 from each of the two adjacent voxel blocks along the x-axis. Similarly, select effective feature points P3, P4, P5, and P6 along the y-axis and z-axis. Calculate the Euclidean distances L1, L2, L3, L4, L5, and L6 between P1, P2, P3, P4, P5, P6 and P0, respectively. Calculate the weights α1, α2, α3, α4, α5, and α6 for each feature point during aggregation based on the Euclidean distances. However, since features with greater distances have less influence on the aggregated features, it is necessary to recalculate the final weight α1 used for feature aggregation based on the weights obtained from the Euclidean distances. * α2 * α3 * α4 * α5 * α6 * The calculation formula is shown below:

[0026]

[0027] The final diffusion result is obtained by weighted summation of the aggregation weights and effective point features, as shown in the following formula:

[0028]

[0029] To ensure the randomness of diffusion, random diffusion can be achieved by randomly selecting effective points in sparse voxel blocks and adjacent voxel blocks according to the above formula.

[0030] In addition, to reduce the computational cost of the model, a dynamic diffusion method is used, which determines the amount of diffusion for each sparse voxel block based on the difference in effective points between the sparse voxel block and the dense voxel block, thus avoiding ineffective diffusion and over-diffusion and effectively reducing the computational cost of the model.

[0031] Preferably, a point cloud feature densification method is used to obtain the point cloud feature. A diffusion query fusion module is designed based on an attention mechanism to enable more full interaction between the point cloud feature and the image feature. The overall structure of the diffusion query fusion module is as follows:

[0032] The densed point cloud features are diffused into point cloud voxel features F. D As the query in the diffusion query fusion, i.e. the Query in the attention mechanism, the image voxel features F I As keys and values, i.e., the Key and Value in the attention mechanism; and using the attention mechanism to fuse them to obtain fused voxel features containing more information;

[0033] Specifically, a diffusion query fusion module is designed using a multi-head self-attention layer, and a corresponding weight matrix is ​​preset for each self-attention head i. The weight matrix here is pre-trained; then the preset weight matrix in each head and the query F are... D 、key F I Sum F I Multiplying them yields three learnable linear transformations for each feature: Q j K j V j The specific formula is shown below:

[0034]

[0035] Obtain linear change Q j K j V j Then, the similarity score S between point cloud features and image features can be calculated. j The higher the similarity between targets, the higher the score should be; the lower the similarity between targets, the lower the score should be. The score is highest when the point cloud features and image features represent the same target. The specific formula is shown below:

[0036]

[0037] Where d k This represents the scaling factor, whose value is the length of the input feature vector. After obtaining the similarity score, the point cloud features and image features are weighted and summed to obtain the final fused feature F. f The specific formula is shown below:

[0038]

[0039] Where n represents the number of features involved in feature interaction;

[0040] The above steps achieve the fusion of point cloud features and image features, enhance the interaction between features, and obtain fused features that contain more information.

[0041] Preferably, step S2, which integrates the RoI feature-guided dynamic sampling homogeneous region fusion module, determines the number of samples based on the number of valid points in the 3D RoI and the relative distance of the target in the scene, thereby achieving dynamic sampling. Specifically, this includes the following:

[0042] First, the 3D RoI in the pseudo-point cloud is projected onto the 2D image using a camera transformation matrix to obtain the image search region. Then, each virtual point in each search region is evaluated. If the 3D coordinates of a virtual point within the search region are inside any 3D RoI, the virtual point is considered valid. If the 3D coordinates of a virtual point are not inside any 3D RoI, the virtual point is considered invalid and is not counted. After determining the number of valid points in each 3D RoI using the above method, the magnitude factor δ of each 3D RoI is dynamically calculated using the relative number, as shown in the following formula:

[0043]

[0044] Where i is the number of each 3D RoI, m is the number of valid points in the 3D RoI with the most valid points, N represents the number of valid points in the current 3D RoI, n represents the number of 3D RoIs in the current scene, and ω is a constant between 0 and 1.

[0045] The number of samples is determined based on their distance from the location. The formula for calculating the distance factor ε is shown below:

[0046]

[0047] Where i is the number of each 3D RoI, ρ and σ are weight constants, l and L are the lengths of the search region and the entire pseudo point cloud, respectively, and w and W are the widths of the search region and the entire pseudo point cloud, respectively. After obtaining the magnitude factor δ and the distance factor ε, the two are weighted and summed to obtain the final sampling rate S, as shown in the following formula.

[0048] S i =αδ i +βε i α+β=1, α,β<1;

[0049] Here, α and β represent the importance of the magnitude factor δ and the distance factor ε. If a better sampling effect can be achieved by increasing the number of valid points, then the proportion of α is increased and the proportion of β is decreased. Conversely, if a better sampling effect can be achieved by considering the distance of the scene location, then the proportion of α is decreased and the proportion of β is increased.

[0050] Preferably, the confidence-guided multi-level data association module in step S3 includes: step S3.1, a first-stage data association guided by multiple information. The first-stage data association involves associating and matching high-confidence detection boxes. The association matrix of the high-confidence detection boxes consists of three parts: appearance affinity matrix, location affinity matrix, and detection box aspect ratio affinity matrix. Among them, appearance affinity M... qpp The position affinity matrix M is obtained by arranging the outputs of the embedded branches using the softmax function. pos Using detection box D B And trajectory prediction box P B The IoU and Euclidean distance between them are calculated, and the specific calculation formula is shown in the following formula:

[0051]

[0052] Where D B and P B D represents the trajectory prediction box and the detection box, respectively. c (x D y D ) and P c (x P y P ( ) represent the center points of the detection box and the trajectory prediction box, respectively, and E represents the Euclidean distance, which is calculated as follows:

[0053]

[0054] Detection frame aspect ratio affinity matrix M lwr The aspect ratio between the detection box and the trajectory prediction box is calculated, as shown in the following formula:

[0055]

[0056] Where R i r represents the difference between the aspect ratio of the current trajectory prediction box and the aspect ratio of the i-th detection box. p The aspect ratio of the trajectory prediction bounding box, r D The aspect ratio of the detection box is represented by the following formula:

[0057]

[0058] L P and L D W represents the lengths of the trajectory prediction box and the detection box, respectively. P and W D Let M and α represent the widths of the trajectory prediction box and the detection box, respectively. In summary, the final correlation matrix M = αM qpp +βM pos +γMlwr , where α+β+γ=1.

[0059] Preferably, the confidence-guided multi-level data association module in step S3 includes: step S3.2 multi-similarity metric-guided multi-level data association, where the second-stage data association involves associating and matching low-confidence detection boxes, and a three-level data association is designed; wherein, the first-stage data association directly uses the position affinity matrix M from the first-stage data association. pos Data association is performed; the goal of secondary data association is to identify targets that can only be detected by the lidar detector and targets that do not match in the primary data association. These targets often have high similarity to the background or fail to match due to large inter-frame displacement, resulting in a zero IoU between the trajectory prediction box and the detection box. Therefore, secondary data association uses GIOU to calculate the position affinity matrix, and the calculation formula is shown in the following formula:

[0060]

[0061] A c =l*w

[0062] Where D B and P B A represents the detection box and the trajectory prediction box, respectively. c Let l represent the smallest enclosing rectangle of the bounding detection box and the trajectory prediction box, where l and w represent the length and width of the smallest enclosing rectangle, respectively;

[0063] The goal of Level 3 data association is to identify targets that can only be detected by the camera detector. These targets are often located in the background of the scene. Therefore, Level 3 data association calculates the location affinity matrix based on the ratio of Euclidean distance to the diagonal length of the smallest enclosing rectangle. The calculation formula is shown below:

[0064]

[0065] The beneficial effects of this invention are:

[0066] The method proposed in this invention has shown significant beneficial effects in the field of multimodal 3D multi-target tracking, specifically in the following aspects:

[0067] (1) The multimodal 3D multi-target tracking method for unmanned driving of the present invention effectively solves the problem of information loss caused by the projection of 3D information into 2D space in traditional methods by using a point cloud adaptive diffusion fusion module. The sparse point cloud is densed by using the inverse distance weight aggregation diffusion method and combined with the diffusion query fusion module to realize more fine-grained multimodal feature fusion, thereby improving the integrity and accuracy of feature information.

[0068] (2) The multimodal 3D multi-target tracking method for unmanned driving of the present invention integrates a dynamic sampling homogeneous region fusion module guided by RoI features, which not only reduces the computational load but also improves the robustness of feature fusion. By guiding the cropping of pseudo-point clouds according to the fused RoI features and combining it with a dynamic sampling strategy, the poor robustness problem that may be caused by sampling based solely on distance is effectively avoided. At the same time, the dual fusion of local and global features is achieved by utilizing an attention mechanism, further enhancing the discriminative power of target features.

[0069] (3) The multimodal 3D multi-target tracking method for autonomous driving of the present invention significantly improves the accuracy of data association through a confidence-guided multi-similarity measurement data association module. Different processing strategies are divided according to the confidence of the detection box, and various information is introduced to construct the association matrix, which effectively solves the problems of missing matching and incorrect matching. At the same time, multi-stage data association and different similarity measurement methods are designed to address the complexity of the scene, further improving the robustness of tracking. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of the overall network framework of one embodiment of the present invention;

[0071] Figure 2 This is a schematic diagram of a lidar branch structure according to an embodiment of the present invention;

[0072] Figure 3 This is a schematic diagram of a point cloud voxelized structure according to an embodiment of the present invention;

[0073] Figure 4 This is a schematic diagram of point cloud data visualization according to an embodiment of the present invention;

[0074] Figure 5 This is a schematic diagram of an image branch structure according to an embodiment of the present invention;

[0075] Figure 6 This is a schematic diagram of the point cloud adaptive diffusion fusion module structure according to an embodiment of the present invention;

[0076] Figure 7 This is a schematic diagram of a three-dimensional image voxel feature generation process according to an embodiment of the present invention;

[0077] Figure 8 This is a flowchart of a three-dimensional point cloud voxel feature generation process according to an embodiment of the present invention;

[0078] Figure 9 This is a schematic diagram of the inverse distance weighted aggregation diffusion method in one embodiment of the present invention;

[0079] Figure 10 This is a schematic diagram of the diffusion query fusion module structure according to an embodiment of the present invention;

[0080] Figure 11 This is a schematic diagram of the structure of a dynamic sampling homogeneous region fusion module according to an embodiment of the present invention;

[0081] Figure 12 This is a schematic diagram of a homogeneous region fusion module structure according to an embodiment of the present invention;

[0082] Figure 13 This is a schematic diagram of a confidence-guided multi-level data association structure according to an embodiment of the present invention;

[0083] Figure 14 This is a schematic diagram of the first stage of data association according to an embodiment of the present invention;

[0084] Figure 15 This is a schematic diagram of the second stage of data association in one embodiment of the present invention. Detailed Implementation

[0085] To facilitate understanding of the technical means, inventive features, and achieved effects of this invention, the technical solutions in the embodiments of this invention will be further described clearly and completely below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0086] Example 1

[0087] The present invention provides a deep learning-based multimodal feature interaction 3D multi-target tracking method, comprising the following steps: constructing an overall network model for an autonomous driving multimodal 3D multi-target tracking method, such as... Figure 1 The schematic diagram of the specific network structure shown includes a point cloud adaptive diffusion fusion module, a dynamic sampling homogeneous region fusion module guided by RoI features, and a confidence-guided multi-level data association module.

[0088] Discriminative target features are obtained based on a point cloud adaptive diffusion fusion module and a dynamic sampling homogeneous region fusion module guided by RoI feature fusion, specifically including the following:

[0089] The S1 point cloud adaptive diffusion fusion module performs preliminary fusion of 2D image features and 3D point cloud features to reduce information loss during the fusion of different modalities. The specific steps are as follows: data collected by the vehicle-mounted LiDAR and camera are input into their respective branch modules, as shown in the schematic diagram below. Figure 2As shown, after upscaling the image features to 3D space, the sparse point cloud is first densified using an inverse distance weighted aggregation diffusion method. Then, the diffusion query fusion module is used to fuse multimodal feature information to obtain fused features containing more information, reducing information loss during fusion.

[0090] After establishing the point cloud adaptive diffusion fusion module, the input information data is processed to generate 3D image RoI voxel features and 3D point cloud voxel features, respectively. The input information data in the overall network model of the multimodal 3D multi-target tracking method includes two parts: 3D LiDAR point cloud information data and 2D camera image information data.

[0091] Specific methods for obtaining discriminative target features from input information data include the following:

[0092] For 3D LiDAR point cloud data, it is first voxelized, and then 3D sparse convolutions with different lengths of 1, 2, 4, and 8 are used to encode the feature of the point cloud data to obtain the corresponding 3D point cloud voxel features, as follows:

[0093] Voxelization—To improve data processing efficiency and extract multi-scale, multi-level local feature information, point clouds are divided into regular voxels. Specifically, taking autonomous vehicles as the center, for input point clouds... P∈Υ N×4 N represents the number of point clouds, and 4 represents the four attributes of each point: its 3D coordinates [x, y, z] and reflectance [r]. The point cloud ranges from [0, 70.4m] on the x-axis, from [-40m, 40m] on the y-axis, and from [-3m, 1m] on the z-axis. Each voxel has a size of [0.05m, 0.05m, 0.1m]. The entire point cloud space is divided into 1408*1600*40 voxel blocks. Furthermore, due to the irregularity of the point cloud distribution, each voxel block may contain multiple point cloud data points or none at all. The voxelized structure diagram of the point cloud is shown below. Figure 3 As shown.

[0094] like Figure 4 As shown, the existing 3D sparse convolution method is used to extract features from all voxel blocks using four sparse convolutions with strides of 1, 2, 4, and 8. In sparse convolution, the convolution operation is only performed at the positions of non-zero values ​​in the input data, which reduces computation and improves efficiency.

[0095] For 2D camera image information data, the image data acquired by the vehicle-mounted camera is used as the input source and input into two branch modules of the camera image, as shown in the specific structure. Figure 5As shown, it includes two sub-branches. One is to upscale 2D image information to 3D space and then use a point cloud adaptive diffusion fusion module to fuse it with 3D point cloud voxel features to obtain point cloud image RoI features. The other is to map pixels in the 2D camera image to 3D space using a depth completion method to obtain pseudo-point clouds. Then, the features extracted through feature aggregation and 3D sparse convolution are voxelized to obtain pseudo-point cloud RoI features.

[0096] Finally, the RoI features of the point cloud image and the RoI features of the pseudo point cloud are divided according to the region of interest. An attention mechanism is used on the divided features to achieve finer-grained fusion, resulting in more discriminative target features and avoiding information loss to a certain extent.

[0097] S1.1 Establishing a point cloud adaptive diffusion fusion module

[0098] After upscaling image features to 3D space, the sparse point cloud features are first densified using an inverse distance weighted aggregation diffusion method. Then, a diffusion query fusion attention mechanism is used to interactively fuse the densed point cloud features and 3D image features, thereby obtaining more information-rich target features. A schematic diagram of its point cloud adaptive diffusion fusion module structure is shown below. Figure 6 As shown, RoI features containing richer semantic information help guide the cropping of pseudo-point cloud RoI regions and subsequent finer-grained RoI-by-RoI fusion, thereby obtaining more discriminative target features, improving the accuracy of data association, and ultimately achieving more robust tracking.

[0099] S1.1.1 Generate 3D image RoI voxel features: Use a 2D backbone network to extract features from the input camera image to obtain image RoI features F rich in semantic information, and convert the image RoI planar features F into truncated cone features G.

[0100] Specifically, the vector F corresponding to each pixel (m, n) in the RoI feature map F of the image is... m,n via Depth BinD m,n The perspective ray along the truncated cone of the image is projected into 3D space. The Depth Bin D is obtained by discretizing the depth map using the Linear Increasing Depth Discretization (LID) method. The image RoI feature F and Depth Bin D are processed using an outer product to generate the truncated cone feature G. Each pixel (m, n) can have its corresponding truncated cone feature G obtained using the following formula. m,n Specifically in Let (m, n) represent the outer product, and (m, n) represent the index of each image pixel feature. Next, trilinear interpolation is used to transform the truncated cone space features G to the three-dimensional space I.

[0101] Specifically, in order to obtain the image RoI voxel feature I corresponding to the i-th grid i First, the centroid of the image truncated cone feature G is sampled and converted based on the calibration matrix CM. in and Indicate the 3D position of the i-th grid in the truncated cone feature G and the image RoI voxel feature I, then... Trilinear interpolation is performed on the neighborhood to obtain Finally, this process is used to construct the 3D image RoI voxel feature I at each spatial index i. The specific process is as follows: Figure 7 As shown.

[0102] S1.1.2 Generating 3D Point Cloud Voxel Features: First, the acquired 3D laser point cloud data is voxelized. The size of each voxel is set to [0.05, 0.05, 0.1], for a total of 16,000 voxels. Each voxel contains information about five points, and the information of each point consists of four parts: coordinates x, y, z, and reflection intensity r. If a voxel contains fewer than five points, it is first padded with 0s and then deleted. If a voxel contains more than five points, five points are randomly retained, and the rest are discarded. The segmented point cloud voxels are then downsampled using four sparse convolutions at multiple scales to obtain the final 3D point cloud voxel features. The downsampling factors are 1, 2, 4, and 8, and the number of feature channels after each downsampling is 16, 32, 64, and 128, respectively. The features of all points in each voxel are used to represent the features of each voxel. The final 3D point cloud voxel features are composed of the features of all points. A detailed flowchart is shown below. Figure 8 As shown.

[0103] S1.1.3 Perform adaptive diffusion of point clouds: such as Figure 9 As shown, to alleviate the problem of sparse point cloud features, the point cloud adaptive diffusion uses an inverse distance weighted aggregation diffusion method to densify the point cloud features. The specific steps are as follows:

[0104] First, based on the number of valid point features in each voxel block, the voxel blocks are divided into empty voxel blocks, sparse voxel blocks, and dense voxel blocks. Empty voxel blocks contain no valid point features, dense voxel blocks contain five valid point features, and the rest are sparse voxel blocks. After division, empty voxel blocks are processed by aggregating the features of adjacent voxel points and calculating the average to complete their features. The specific formula is shown below:

[0105]

[0106] Where P i This represents the effective point features selected from adjacent voxels along the x-axis, y-axis, and z-axis, respectively.

[0107] For a sparse voxel block, arbitrarily select an effective point P0. Then, select effective feature points P1 and P2 from each of the two adjacent voxel blocks along the x-axis. Similarly, select effective feature points P3, P4, P5, and P6 along the y-axis and z-axis. Calculate the Euclidean distances L1, L2, L3, L4, L5, and L6 between P1, P2, P3, P4, P5, P6 and P0, respectively. Calculate the weights α1, α2, α3, α4, α5, and α6 for each feature point during aggregation based on the Euclidean distances. However, since features with greater distances have less influence on the aggregated features, it is necessary to recalculate the final weight α1 used for feature aggregation based on the weights obtained from the Euclidean distances. * α2 * α3 * α4 * α5 * α6 * The calculation formula is shown below:

[0108]

[0109] The final diffusion result is obtained by weighted summation of the aggregation weights and effective point features, as shown in the following formula:

[0110]

[0111] To ensure the randomness of diffusion, random diffusion can be achieved by randomly selecting effective points in sparse voxel blocks and adjacent voxel blocks according to the above formula.

[0112] In addition, to reduce the computational cost of the model, a dynamic diffusion method is used, which determines the amount of diffusion for each sparse voxel block based on the difference in effective points between the sparse voxel block and the dense voxel block, thus avoiding ineffective diffusion and over-diffusion and effectively reducing the computational cost of the model.

[0113] S1.1.4 Diffusion Query Fusion Module Structure: Diffusion of the densed point cloud features into point cloud voxel features F D As the query in the diffusion query fusion, i.e. the Query in the attention mechanism, the image voxel features F I As keys and values, i.e., the Key and Value in the attention mechanism; and using the attention mechanism to fuse them to obtain fused voxel features containing more information.

[0114] Specifically, a diffusion query fusion module is designed using a multi-head self-attention layer, and a corresponding weight matrix is ​​preset for each self-attention head i. The weight matrix here is pre-trained; then the preset weight matrix in each head and the query F are... D 、key F I Sum FI Multiplying them yields three learnable linear transformations for each feature: Q j K j V j The specific formula is shown below:

[0115]

[0116] Obtain linear change Q j K j V j Then, the similarity score S between point cloud features and image features can be calculated. j The higher the similarity between targets, the higher the score should be; the lower the similarity between targets, the lower the score should be. The score is highest when the point cloud features and image features represent the same target. The specific formula is shown below:

[0117]

[0118] Where d k This represents the scaling factor, whose value is the length of the input feature vector. After obtaining the similarity score, the point cloud features and image features are weighted and summed to obtain the final fused feature F. f The specific formula is shown below:

[0119]

[0120] Where n represents the number of features involved in feature interaction;

[0121] The above steps achieve the fusion of point cloud features and image features, enhancing the interaction between features and resulting in fused features that contain more comprehensive information. A schematic diagram of the overall diffusion query fusion module structure is shown below. Figure 10 As shown.

[0122] Example 2

[0123] S2, a dynamic sampling homogeneous region fusion module guided by RoI features, specifically includes the following:

[0124] S2.1 RoI dynamic sampling determines the number of samples based on the number of valid points in the 3D RoI and the relative distance of the target in the scene, thus achieving dynamic sampling. The pseudo-point cloud is cropped based on the fused RoI features obtained from the point cloud adaptive diffusion fusion module, resulting in pseudo-point cloud RoI features. Then, dynamic sampling is performed only on each pseudo-point cloud RoI feature, reducing computational load. A schematic diagram of its dynamic sampling homogeneous region fusion module is shown below. Figure 11As shown. Utilizing the unique characteristics of pseudo-point clouds—that each virtual point in a pseudo-point cloud consists of 3D coordinates, RGB information, and 2D coordinates ([x, y, z, r, g, b, u, v])—the 3D coordinates of a point can be obtained from its 2D coordinates without reprojecting the 2D point into 3D space. To determine whether a virtual point in a pseudo-point cloud is within a 3D RoI, the same method as determining whether a virtual point in an image is within a 3D RoI can be used.

[0125] In this process, dynamic sampling is performed based on the number of valid points in each RoI feature and the distance of the target in the scene. To improve the discriminativeness of the features, the pseudo-point cloud RoI features and the point cloud image RoI features are divided into features. An attention mechanism is used to achieve dual fusion of local and global features, resulting in more discriminative target features.

[0126] First, the 3D RoI in the pseudo-point cloud is projected onto the 2D image using a camera transformation matrix to obtain the image search region. Then, each virtual point in each search region is evaluated. If the 3D coordinates of a virtual point within the search region are inside any 3D RoI, the virtual point is considered valid. If the 3D coordinates of a virtual point are not inside any 3D RoI, the virtual point is considered invalid and is not counted. After determining the number of valid points in each 3D RoI using the above method, the magnitude factor δ of each 3D RoI is dynamically calculated using the relative number, as shown in the following formula:

[0127]

[0128] Where i is the number of each 3D RoI, m is the number of valid points in the 3D RoI with the most valid points, N represents the number of valid points in the current 3D RoI, n represents the number of 3D RoIs in the current scene, and ω is a constant between 0 and 1.

[0129] The number of samples is determined based on their distance from the location. The formula for calculating the distance factor ε is shown below:

[0130]

[0131] Where i is the number of each 3D RoI, ρ and σ are weight constants, l and L are the lengths of the search region and the entire pseudo-point cloud, respectively, and w and W are the widths of the search region and the entire pseudo-point cloud, respectively. After obtaining the magnitude factor δ and the distance factor ε, the two are weighted and summed to obtain the final sampling rate S, as shown in the following formula:

[0132] S i =αδ i +βεi α+β=1, α,β<1;

[0133] Here, α and β represent the importance of the magnitude factor δ and the distance factor ε. If a better sampling effect can be achieved by increasing the number of valid points, the proportion of α should be increased and the proportion of β decreased. Conversely, if a better sampling effect can be achieved by considering the distance of the scene, the proportion of α should be decreased and the proportion of β increased. Experimental results show that optimal performance is achieved when ω in the magnitude factor δ is 0.75, ρ and σ in the distance factor ε are 1.5 and 3 respectively, and α and β in the sampling rate S are 0.65 and 0.35 respectively.

[0134] S2.2 Establishing a homogeneous region fusion module using a homogeneous region fusion method: A sparse pseudo-point cloud RoI is obtained through RoI dynamic sampling. Then, the sparse pseudo-point cloud RoI is voxelized and 3D sparse convolution is used for feature extraction to obtain the pseudo-point cloud RoI features F. v Since pseudo-point clouds and point clouds exist in the same space, the RoI features F of point cloud images... P And pseudo-point cloud RoI features F v The representation is the same. Therefore, each pseudo-point cloud RoI feature F v And each point cloud image RoI feature F P There is a one-to-one correspondence between them, and for pseudo-point cloud RoI features F v RoI features of point cloud images F P Synchronization will not change the correspondence between the two.

[0135] To achieve finer-grained feature fusion and improve the discriminative power of the fused features, a homogeneous region fusion method is proposed. For example... Figure 12 As shown, the specific steps are as follows:

[0136] F for each pseudo-point cloud RoI feature v RoI features of point cloud images F P First, a feature region division is performed, dividing each RoI feature into 4*4*4 local features. Then, an attention module is used to focus each local feature F in the point cloud image RoI. P ′ and all local features F in the corresponding pseudo-point cloud RoI v Interactive fusion is performed to obtain the local RoI fusion feature F. f ′, thus the local RoI fusion feature F f Each local feature in the point cloud image contains richer feature information; at the same time, the attention mechanism pays more attention to the local features F of the RoI in the point cloud image. P F' High similarity pseudo-point cloud RoI local features vThis effectively enhances the saliency of each local feature in the fused RoI features. To further enhance the fused RoI features F... f The discriminative power of the global features F of the point cloud image RoI P * And pseudo-point cloud RoI global features F v * Attention is also used for fusion to obtain the global RoI fusion feature F. f * ; and fuse local RoI features F f ′ and global RoI fusion feature F f * The final fused RoI feature F is obtained by performing a weighted summation. f The weights are predicted using an attention mechanism.

[0137] Example 3

[0138] The S3 confidence-guided multi-level data association module improves the accuracy of data association and the robustness of tracking. Specific steps include: feeding the discriminative target features obtained in step 2 into the embedding branch to obtain the target location and appearance information required for data association; then constructing location affinity matrices and appearance affinity matrices based on the location and appearance information for association matching. Its structural diagram is shown below. Figure 13 As shown. The specific steps are as follows:

[0139] S3.1 The first stage of data association guided by multiple information involves associating and matching high-confidence detection boxes, such as... Figure 14 As shown, the correlation matrix of a high-confidence detection box consists of three parts: appearance affinity matrix, location affinity matrix, and aspect ratio affinity matrix. Among these, appearance affinity M... qpp The position affinity matrix M is obtained by arranging the outputs of the embedded branches using the softmax function. pos Using detection box D B And trajectory prediction box P B The IoU and Euclidean distance between them are calculated, and the specific calculation formula is shown in the following formula:

[0140]

[0141] Where D B and P B D represents the trajectory prediction box and the detection box, respectively. c (x D y D ) and P c (x P y P( ) represent the center points of the detection box and the trajectory prediction box, respectively, and E represents the Euclidean distance, which is calculated as follows:

[0142]

[0143] Detection frame aspect ratio affinity matrix M lwr The aspect ratio between the detection box and the trajectory prediction box is calculated, as shown in the following formula:

[0144]

[0145] Where R i r represents the difference between the aspect ratio of the current trajectory prediction box and the aspect ratio of the i-th detection box. p The aspect ratio of the trajectory prediction bounding box, r D The aspect ratio of the detection box is represented by the following formula:

[0146]

[0147] L P and L D W represents the lengths of the trajectory prediction box and the detection box, respectively. P and W D Let M and α represent the widths of the trajectory prediction box and the detection box, respectively. In summary, the final correlation matrix M = αM qpp +βM pos +γM lwr Where α+β+γ=1. Experimental results show that the optimal performance is achieved when the confidence threshold τ=0.75, the correlation matrix weights α=0.5, β=0.35, and γ=0.15.

[0148] S3.2 Multi-level data association guided by multiple similarity measures. The second stage of data association involves matching low-confidence detection boxes. A three-level data association system is designed, such as... Figure 15 As shown; where, the first-level data association directly uses the location affinity matrix M from the first-stage data association. pos Data association is performed; the goal of secondary data association is to identify targets that can only be detected by the lidar detector and targets that do not match in the primary data association. These targets often have high similarity to the background or fail to match due to large inter-frame displacement, resulting in a zero IoU between the trajectory prediction box and the detection box. Therefore, secondary data association uses GIOU to calculate the position affinity matrix, and the calculation formula is shown in the following formula:

[0149]

[0150] A c =l*w

[0151] Where DB and P B A represents the detection box and the trajectory prediction box, respectively. c Let l represent the smallest enclosing rectangle of the bounding detection box and the trajectory prediction box, where l and w represent the length and width of the smallest enclosing rectangle, respectively;

[0152] The goal of Level 3 data association is to identify targets that can only be detected by the camera detector. These targets are often located in the background of the scene. Therefore, Level 3 data association calculates the location affinity matrix based on the ratio of Euclidean distance to the diagonal length of the smallest enclosing rectangle. The calculation formula is shown below:

[0153]

[0154] For low-confidence detection results, three-level data association and multiple similarity measures are used for data association. This approach takes a more comprehensive view of complex scenarios, effectively solves the problem of matching failure caused by low or zero IoU, reduces the problem of target identity switching and trajectory fragmentation caused by occlusion and other issues, and improves the data association and robustness of tracking tasks.

[0155] In summary, the deep learning-based multimodal feature interaction 3D multi-target tracking method of this invention, through the synergistic effect of innovative modules such as point cloud adaptive diffusion fusion, dynamic sampling homogeneous region fusion guided by RoI features, and confidence-guided multi-similarity metric data association, minimizes information loss, achieves finer-grained feature fusion, significantly reduces overall computational load, and effectively solves the problems of missed matching and incorrect matching caused by the use of a single similarity metric or insufficient discriminativeness of the association matrix in current data association. This improves the accuracy of data association and the robustness of the tracking task, thereby enhancing the performance of multimodal 3D multi-target tracking. This invention has broad application prospects in fields such as autonomous driving and intelligent monitoring.

[0156] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions claimed by the present invention.

Claims

1. A deep learning-based multimodal feature interaction 3D multi-target tracking method, characterized in that, Includes the following steps: A comprehensive network model for a multimodal 3D multi-target tracking method for autonomous driving is constructed, including a point cloud adaptive diffusion fusion module, a dynamic sampling homogeneous region fusion module guided by RoI features, and a confidence-guided multi-level data association module. Discriminative target features are obtained based on a point cloud adaptive diffusion fusion module and a dynamic sampling homogeneous region fusion module guided by RoI feature fusion, specifically including the following: S1 uses a point cloud adaptive diffusion fusion module to perform preliminary fusion of 2D image features and 3D point cloud features: after upscaling the image features to 3D space, the sparse point cloud is first densified by the inverse distance weight aggregation diffusion method, and then the diffusion query fusion module is used to fuse multimodal feature information to obtain fused features containing more information and reduce information loss during fusion. S2 fusion RoI feature-guided dynamic sampling homogeneous region fusion module: First, based on the fused RoI features obtained from the point cloud adaptive diffusion fusion module, pseudo-point clouds are cropped to obtain pseudo-point cloud RoI features. Then, dynamic sampling is performed only on each pseudo-point cloud RoI feature to reduce the amount of computation. Dynamic sampling is implemented based on the number of valid points in each RoI feature and the distance of the target in the scene. In order to improve the discriminativeness of the features, the pseudo point cloud RoI features and the point cloud image RoI features are divided into features. An attention mechanism is used to achieve dual fusion of local and global features to obtain more discriminative target features. The number of samples is determined based on the number of valid points in the 3D RoI and the relative distance of the target in the scene to achieve dynamic sampling, specifically including the following: First, the 3D RoI in the pseudo-point cloud is projected onto a 2D image using a camera transformation matrix to obtain the image search region. Then, each virtual point in each search region is evaluated individually. If the 3D coordinates of a virtual point within the search region are inside any 3D RoI, the virtual point is considered valid. If the 3D coordinates of a virtual point are not inside any 3D RoI, the virtual point is considered invalid and is not counted. After determining the number of valid points in each 3D RoI using the above method, the magnitude factor of each 3D RoI is dynamically calculated using relative quantities. The specific formula is as follows: ; in Number each 3D RoI, This represents the number of valid points in the 3D RoI that contains the most valid points. This indicates the number of valid points in the current 3D RoI. This indicates the number of 3D RoIs in the current scene. It is a constant between 0 and 1; The number of samples is determined based on their distance from the location, using a distance factor. The specific calculation formula is shown below: ; in Number each 3D RoI, and Let l and L be the lengths of the search region and the entire pseudo-point cloud, respectively, and w and W be the widths of the search region and the entire pseudo-point cloud, respectively; then, the magnitude factor is obtained. and distance factor Then, the two are weighted and summed to obtain the final sampling rate S, as shown in the following formula: ; in and Indicates magnitude factor and distance factor The importance of this factor lies in increasing the number of valid points. If a better sampling effect can be achieved by increasing the number of valid points, then... The proportion of, reduced The proportion of [samples] should be reduced; conversely, if better sampling can be achieved based on the distance of the scene location, then [the proportion of samples] should be reduced. The proportion increased The proportion; The S3 confidence-guided multi-level data association module improves the accuracy of data association and the robustness of tracking.

2. The deep learning-based multimodal feature interaction 3D multi-target tracking method according to claim 1, characterized in that, After establishing the point cloud adaptive diffusion fusion module, the input information data is processed to generate 3D image RoI voxel features and 3D point cloud voxel features respectively; the input information data in the overall network model of the multimodal 3D multi-target tracking method includes two parts: 3D LiDAR point cloud information data and 2D camera image information data.

3. The deep learning-based multimodal feature interaction 3D multi-target tracking method according to claim 2, characterized in that, Specific methods for obtaining discriminative target features from input information data include the following: For 3D LiDAR point cloud information data, it is first voxelized, and then 3D sparse convolution with different lengths of 1, 2, 4 and 8 is used to encode the feature of the point cloud data to obtain the corresponding 3D point cloud voxel features. For 2D camera image information data, there are two sub-branches. One is to upscale the 2D image information to 3D space and then use the point cloud adaptive diffusion fusion module to fuse the features with the 3D point cloud voxel features to obtain the point cloud image RoI features. One approach is to map pixels in a 2D camera image to a 3D space using depth completion to obtain a pseudo-point cloud. Then, the features extracted through feature aggregation and 3D sparse convolution are voxelized to obtain the pseudo-point cloud RoI features. Finally, the RoI features of the point cloud image and the RoI features of the pseudo point cloud are divided according to the region of interest. An attention mechanism is used on the divided features to achieve finer-grained fusion, resulting in more discriminative target features.

4. The deep learning-based multimodal feature interaction 3D multi-target tracking method according to claim 3, characterized in that, The specific steps of the multi-level data association module in step S3 include: The discriminative target features obtained in step 2 are fed into the embedding branch to obtain the target location information and appearance information required for data association. Then, based on the location information and appearance information, location affinity matrix and appearance affinity matrix are constructed respectively for association matching. Based on the confidence level of the detection results, they are divided into high-confidence and low-confidence parts. Then, for the high-confidence detection results, a discriminative association matrix is ​​constructed using the location affinity matrix, appearance affinity matrix, and detection box aspect ratio affinity matrix for association matching. For the low-confidence detection results, three-level data association and multiple similarity measures are used for data association to improve the data association and robustness of the tracking task.

5. The deep learning-based multimodal feature interaction 3D multi-target tracking method according to claim 4, characterized in that, In point cloud adaptive diffusion, the inverse distance weighted aggregation diffusion method is used to densify the point cloud features. The specific steps are as follows: First, based on the number of valid point features in each voxel block, the voxel blocks are divided into empty voxel blocks, sparse voxel blocks, and dense voxel blocks. Empty voxel blocks contain no valid point features, dense voxel blocks contain five valid point features, and the rest are sparse voxel blocks. After division, empty voxel blocks are processed by aggregating the features of adjacent voxel points and calculating the average to complete their features. The specific formula is shown below: ; in This represents the effective point features selected from adjacent voxels along the x-axis, y-axis, and z-axis, respectively. For a sparse voxel block, arbitrarily select an effective point. Select one valid feature point from each of two adjacent voxel blocks along the x-axis. and Select effective point features along the y-axis and z-axis in the same manner. , , and ; and calculate respectively , , , , , and Euclidean distance between , , , , , The weights of each point's features are calculated based on Euclidean distance during aggregation. , , , , , However, since features that are farther apart have less impact on the aggregated features, it is necessary to recalculate the weights used for feature aggregation based on the weights derived from the Euclidean distance. , , , , , ; The calculation formula is shown below: ; The final diffusion result is obtained by weighted summation of the aggregation weights and effective point features, as shown in the following formula: ; To ensure the randomness of diffusion, random diffusion can be achieved by randomly selecting effective points in sparse voxel blocks and adjacent voxel blocks according to the above formula. In addition, to reduce the computational cost of the model, a dynamic diffusion method is used, which determines the amount of diffusion for each sparse voxel block based on the difference in effective points between the sparse voxel block and the dense voxel block, thus avoiding ineffective diffusion and over-diffusion and effectively reducing the computational cost of the model.

6. The deep learning-based multimodal feature interaction 3D multi-target tracking method according to claim 5, characterized in that, A point cloud feature density enhancement method is obtained by using point cloud adaptive diffusion. A diffusion query fusion module is designed based on an attention mechanism to enable more thorough interaction between point cloud features and image features. The overall structure of the diffusion query fusion module is as follows: Diffusion of point cloud features into point cloud voxel features after densification As the query in diffusion query fusion, i.e. the Query in the attention mechanism, it incorporates image voxel features. As keys and values, i.e., the Key and Value in the attention mechanism; and using the attention mechanism to fuse them to obtain fused voxel features containing more information; Specifically, a diffusion query fusion module is designed using a multi-head self-attention layer to provide each self-attention head with a specific function. Preset the corresponding weight matrix respectively , , The weight matrix here is pre-trained; Then, the preset weight matrix in each header and the query... ,key Sum Multiplying them yields three learnable linear transformations for each feature: , , The specific formula is shown below: , , ; A linear change was obtained , , Then, the similarity score between point cloud features and image features can be calculated. The higher the similarity between targets, the higher the score should be; the lower the similarity between targets, the lower the score should be. The score is highest when the point cloud features and image features represent the same target. The specific formula is shown below: ; in This represents the scaling factor, whose value is the length of the input feature vector. After obtaining the similarity score, the point cloud features and image features are weighted and summed to obtain the final fused features. The specific formula is shown below: ; Where n represents the number of features involved in feature interaction; The above steps achieve the fusion of point cloud features and image features, enhance the interaction between features, and obtain fused features that contain more information.

7. The deep learning-based multimodal feature interaction 3D multi-target tracking method according to claim 4, characterized in that, Step S3, the confidence-guided multi-level data association module, includes: Step S3.1, the first-stage data association guided by multiple information. The first-stage data association involves associating and matching high-confidence detection boxes. The association matrix of the high-confidence detection boxes consists of three parts: appearance affinity matrix, location affinity matrix, and detection box aspect ratio affinity matrix. Among these, appearance affinity... The position affinity matrix is ​​obtained by arranging the outputs of the embedding branch using the softmax function. Using detection box and trajectory prediction box The IoU and Euclidean distance between them are calculated, and the specific calculation formula is shown in the following formula: ; in These represent the trajectory prediction box and the detection box, respectively. and Let represent the center points of the detection box and the trajectory prediction box, respectively, and E represent the Euclidean distance, calculated as follows: ; Aspect Ratio Affinity Matrix of Detection Frame The aspect ratio between the detection box and the trajectory prediction box is calculated, as shown in the following formula: ; in This represents the difference between the aspect ratio of the current trajectory prediction box and the aspect ratio of the i-th detection box. This indicates the aspect ratio of the trajectory prediction box. The aspect ratio of the detection box is represented by the following formula: ; and These represent the lengths of the trajectory prediction box and the detection box, respectively. and These represent the widths of the trajectory prediction box and the detection box, respectively; in summary, the final correlation matrix... ,in .

8. The deep learning-based multimodal feature interaction 3D multi-target tracking method according to claim 7, characterized in that, Step S3, the confidence-guided multi-level data association module, includes: Step S3.2, multi-similarity metric-guided multi-level data association. The second-stage data association involves associating and matching low-confidence detection boxes, and a three-level data association is designed. The first-stage data association directly uses the location affinity matrix from the first-stage data association. Data association is performed; the goal of secondary data association is to identify targets that can only be detected by the lidar detector and targets that do not match in the primary data association. These targets often have high similarity to the background or fail to match due to large inter-frame displacement, resulting in a zero IoU between the trajectory prediction box and the detection box. Therefore, secondary data association uses GIOU to calculate the position affinity matrix, and the calculation formula is shown in the following formula: ; ; in and These represent the detection box and the trajectory prediction box, respectively. Let l represent the smallest enclosing rectangle of the bounding detection box and the trajectory prediction box, where l and w represent the length and width of the smallest enclosing rectangle, respectively; The goal of Level 3 data association is to identify targets that can only be detected by the camera detector. These targets are often located in the background of the scene. Therefore, Level 3 data association calculates the location affinity matrix based on the ratio of Euclidean distance to the diagonal length of the smallest enclosing rectangle. The calculation formula is shown below: 。