A target tracking method based on fusion of vision and radar point clouds

Abstract

The application belongs to the technical field of target tracking, and particularly relates to a target tracking method based on fusion of vision and radar point clouds, wherein through the use of perspective projection, three-dimensional LiDAR points are projected to the coordinate plane of a two-dimensional image to obtain the two-dimensional features P of the point cloud; a double-flow residual network is designed, the aligned point cloud and image are respectively subjected to instance segmentation by using a point cloud center segmentation method and a 2D image segmentation method, the segmented instances are fused, and two independent MLPs are introduced to perform instance recognition and calibration reference frame. Finally, a double-mode instance correction module is input to correct the instance label. In the target tracking module, the weighted center of the instance is used for inter-frame tracking target image matching and trajectory management, so that the unmanned aerial vehicle can detect, locate and track the target of interest and the environmental target, the problems in single-mode recognition and positioning and other problems are alleviated, and the performance of multi-target visual positioning and tracking is improved.

Description

Technical Field

[0001] This invention relates to the field of target tracking technology, specifically to a target tracking method based on the fusion of vision and radar point clouds. Background Technology

[0002] In today's field of drone inspection, target detection and tracking are among the key technologies. With the development and widespread adoption of detection technologies, more and more sensors are being applied to drones, providing rich environmental information for target identification. For example, lidar can provide precise depth information, aiding in accurate target localization; while drones equipped with cameras can capture high-resolution images, providing rich semantic features such as shape, texture, and color for the detection system. However, traditional single-sensor methods, such as target detection and tracking technologies relying solely on lidar or cameras, have significant limitations. For instance, existing traditional foreign object identification technologies for power transmission lines are generally based on a single sensor, often failing to meet requirements in terms of accuracy, speed, robustness, and adaptability to complex environmental changes. Multimodal data fusion of radar point clouds and two-dimensional images is crucial for drones' environmental perception in complex environments. Lidar point clouds can provide depth information and accurately locate the absolute position of targets, but they suffer from drawbacks such as sparsity, disorder, and limited range. They have lower accuracy for distant and small-scale objects, and their sparsity easily leads to missed detections and misidentifications.

[0003] Images and video streams captured by visual cameras can provide rich semantic features and temporal information such as shape, texture, and color, but lack depth information. These shortcomings can affect the environmental perception and real-time localization performance of drones, thus interfering with subsequent planning and decision-making tasks. In the face of complex scenes and multimodal perception data, the sufficiency of data fusion and the robustness of correlation remain pressing issues to be addressed.

[0004] This invention designs a target tracking network based on the fusion of vision and radar point cloud data, enabling UAVs to accurately identify and track inspection targets in complex power transmission environments. By fusing image and point cloud data, and comprehensively utilizing the rich color and texture information of the images as well as the precise distance and spatial information of the point clouds, the limitations of single sensors are overcome, improving the accuracy, stability, and robustness of inspections. This technology can be widely applied in various industries, including production and inspection. Summary of the Invention

[0005] The purpose of this invention is to provide a target tracking method based on the fusion of vision and radar point clouds to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a target tracking method based on the fusion of vision and radar point clouds, the specific steps of which are as follows:

[0007] Step 1: By using perspective projection, the 3D LiDAR points are projected onto the coordinate plane of the 2D image to obtain the 2D features P of the point cloud. Then, a multimodal alignment and cropping module (MAC) is introduced to combine the RGB image and the projected point cloud features. Fine-grained alignment is performed to find the overlapping parts of the multimodal inputs, and the RGB image and projected point cloud features are cropped to obtain a compact aligned input.

[0008] Step 2: Design a dual-stream network for image data stream and two-dimensional point set feature stream. For the incomplete bimodal information after alignment, use point cloud center segmentation method and two-dimensional segmentation method to perform instance segmentation and fusion respectively.

[0009] Step 3: In the image instance segmentation backbone, feature extraction is performed on the features of the image stream fused with the radar stream. The CenterMask network is used to segment the image instances corresponding to the two-dimensional point sets using 2D instance segmentation. Specifically, n instance masks are generated from the RGB single-view image using the 2D instance segmentation model, corresponding to M respectively. i , where i = 1, 2, ..., m, and the point sets corresponding to each instance in the two-dimensional point set feature are segmented using image instance boxes, and are regarded as image segmentation instances;

[0010] Step 4: In the sparse instance segmentation backbone, the cluster center method is used to segment point cloud instances. The point cloud is voxelized and converted into a set of voxels. Then, a sparse voxel feature extractor is used as the backbone to extract voxel features. Given voxel features, they are converted into point features. Finally, a lightweight point-level MLP is used to perform point-level classification and center scoring. Points with classification scores higher than the threshold τ are retained as foreground points, and the point set is grouped into 3D instances.

[0011] Step 5: The two segmentation backbones fuse the image instance and the LiDAR instance;

[0012] Step Six: Finally, in the visual information processing module, a multimodal instance correction module is designed to match the updated instances to the real 3D bounding boxes, minimizing the noise impact of camera segmentation instances.

[0013] Step 7: After instance correction, the correction results are used as input to the tracking target map to complete the matching association between the trajectory and the detection. When there is occlusion, tracking using only the previous stage will lead to frequent label ID switching. To overcome this problem and ensure that the target maintains the same label ID throughout the tracking process, a tracking target map matching based on the feature set of points after instance segmentation is designed to construct the tracking target map, store the lost target trajectory, and determine the status of newly generated, tracked, and disappeared trajectories; the instance center point Center(N) is used. i The obtained 3D features do not need to be transferred to the current frame coordinates, and relatively accurate cross-frame matching can be directly achieved, simplifying the matching method and reducing memory usage; for unmatched targets d∈D t D t-1 A fully connected 3D volume feature matching is performed between target k∈K in the tracking target map. If the matching is successful, p inherits the label ID of target h. If the target matching fails in frame t, it is determined to be a new target and a new label ID is assigned. If the target matching fails in frame t-1, it is determined to be a pre-disappearing target and added to the disappearing target map K. If target k fails to match d in consecutive frames, it is determined to be a disappearing target.

[0014] In the target detection and trajectory state matching, for matching detections and trajectories, the trajectory is updated; for unmatched detections, a new trajectory is initialized; for unmatched trajectories, they are retained for τ frames. If they reappear within τ frames, they are updated; if they do not reappear, the trajectory is terminated. This completes the tracking of multiple targets.

[0015] Preferably, in step one, the formula for calculating the two-dimensional feature P of the point cloud is: Where C represents the number of channels in the projected point cloud, H is the height, and W is the width; projecting the point cloud P onto a two-dimensional image plane yields two-dimensional point set features. The calculation formula is: K is the projection intrinsic parameter matrix of the camera, and [R|t] is the extrinsic parameter matrix.

[0016] Preferably, in step two, the dual-stream network consists of an image instance segmentation backbone and a sparse instance segmentation backbone. X is the Lidar stream, and X is the camera stream.

[0017] Preferably, in step three, the features of the k points in the i-th instance extracted by the camera are:

[0018] Preferably, in step four, each foreground point predicts a surrogate point representing the center of its associated object. The set of cluster centers is denoted as C. Then, the cluster centers are connected using the CCL (Connected Components Labeling) algorithm to partition the instances. In this algorithm, if the distance between two cluster centers is less than a threshold, they are considered connected. CCL groups adjacent cluster centers into n groups. Each instance center is mapped one-to-one with a foreground point, and partitioned using the corresponding foreground point set to construct C. i Corresponding instance N i ; where the features of the k points in the i-th instance extracted by each radar are Then, two sparse instance recognition modules, SIR (Sparse Instance Recognition), were used to... and Feature extraction was performed separately for each. and

[0019] Preferably, in step five, let These are image instance features segmented from the camera stream; C represents the number of channels, and H and W represent the height and width of the feature map, respectively; let... The instance features in the LiDAR stream are fused into F. fuse The calculation formula is:

[0020] F fuse =F C ∪F L =[F C ;F L ]

[0021] Fusion of dual-modal features F fuse After expanding the number of channels, the fused feature information is processed by introducing a self-attention mechanism with residuals to enhance the interaction of dual-modal instance features and retain the instance feature flow information;

[0022] F * =F fuse +selfAttn(F fuse )

[0023] in For the updated instance features, where the total number of instances is f = m + n; subsequently, two separate MLPs are designed to classify and determine the bounding boxes of the camera stream instance features and radar instance features extracted by the SIR module, respectively.

[0024] Preferably, in step six, the instances are allocated in two stages. The first stage is based on LiDAR allocation, for instance N of radar flow prediction. iCalculate its weighted center point as the instance center point Center(N) i The weight is determined by the classification score;

[0025]

[0026] in, It is instance N i The points in the image correspond to the foreground point classification scores, while It is instance N i For each point in the instance, let the number of points in the instance be q; use the weighted center point to perform the point-in-box assignment strategy, that is, if the weighted center point of an instance falls within a certain 3D bounding box, then assign the instance to that bounding box;

[0027] Because the depth and noise of camera instances are much greater than those of point cloud instances, to address the issue of unallocated camera instances from the previous stage, a camera-based allocation process is continued, calculating M for each remaining unallocated camera instance. i The IoU score S(M) between the 2D mask and the 2D projections of all bounding boxes i The instance is assigned to the corresponding 3D bounding box if the maximum IoU value exceeds the set threshold.

[0028] S(M i ,GT)=IoU 2d (V i GT 2d )

[0029] Among them, M i For a camera segmentation instance, GT is the bounding box, and V... i For camera instance M i Two-dimensional projection, GT 2d This is the two-dimensional projection of the bounding box.

[0030] Preferably, in step seven, the calculation formula is as follows:

[0031] Y = y aff +y dis

[0032] y aff =S aff |D t -D t-1 |

[0033] y dis =S dis (D t ,K)+S dis (D t-1 ,K)

[0034] K = K + S dis (D t-1 ,K)

[0035] Where Y represents the target being tracked, y aff Indicates the target being matched between two adjacent frames, y dis S represents the target matched with the tracking map. aff S is a score for the similarity of matched targets in two adjacent frames. dis The similarity score is given to the targets that fail to match between two adjacent frames and the targets in the tracking map, where K is the total number of targets that have disappeared from the map.

[0036] Preferably, in step seven, the total loss in the target detection and recognition loss calculation includes the classification and regression losses of RPN and RCNN, expressed as:

[0037] L total =L rpn +L rcnn

[0038] For the classification loss of RPN, this method uses the following formula to calculate:

[0039] FL(p t )=-α t (1-p t ) γ log(p t )

[0040] Where pt is the probability of the predicted class, i.e., the output probability of the sigmoid function, with α and γ used to balance positive and negative samples; L1 loss (mean absolute error MAE) is used to optimize the ground truth bounding boxes; after optimizing the regressed 3D detection boxes (x, y, z, l, h, w, θ) with L1 loss, the regression loss is calculated; that is, the RPN loss L... rpn Class loss L cls Add bounding box loss L reg :

[0041] L rpn =L cls +L reg

[0042]

[0043]

[0044] E is the cross-entropy loss, and S is the L1 loss; and Represents the 3D ground truth bounding box and residual offset; the RCNN loss is the sum of the classification loss, regression loss, and association module loss. The classification loss and regression loss are calculated in the same way as the RPN loss, and the association module loss uses the L1 loss.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] This application performs multimodal feature fusion on radar point clouds and images, and builds an intelligent perception system based on a visual information processing module and a target tracking module;

[0047] In the visual information processing module, the input two-modal features are first cropped and aligned to preprocess the data.

[0048] This method designs a dual-stream residual network to segment instances in the aligned point cloud and image using point cloud center segmentation and 2D image segmentation methods, respectively. The segmented instances are then fused, and two independent MLPs are introduced for instance recognition and bounding box calibration, respectively. Finally, the data is input into a dual-modal instance correction module to correct the instance labels. In the target tracking module, the weighted center of instances is used for inter-frame target map matching and trajectory management, enabling the UAV to detect, locate, and track targets of interest and environmental targets. This alleviates the problems of single-modal recognition and localization, as well as other existing issues, and improves the performance of multi-target visual localization and tracking.

[0049] This method provides a high-precision visual perception approach that is adaptable to complex background environments such as substations and field transmission lines. It improves detection performance while reducing false detection and false negative rates, and has good engineering application prospects and practical promotion value. Attached Figure Description

[0050] Figure 1 This is a roadmap for target localization and tracking technology based on the fusion of vision and point cloud. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Example:

[0053] Please see Figure 1 The present invention provides a technical solution:

[0054] A target tracking method based on the fusion of visual and radar point cloud data is proposed. The specific steps of this target tracking method are as follows:

[0055] Step 1: By using perspective projection, the 3D LiDAR points are projected onto the coordinate plane of the 2D image to obtain the 2D features P of the point cloud. Then, a multimodal alignment and clipping module (MAC) is introduced to reduce computational costs and avoid the computational resource consumption caused by direct projection of the point cloud. This combines the RGB image and the projected point cloud features. Fine-grained alignment is performed to find overlapping parts of the multimodal inputs. The RGB image and projected point cloud features are cropped to obtain a compact aligned input. Because the RGB image contains more semantic and texture information, it has a higher weight for recognizing small and distant targets, so it is retained entirely. For the projected point cloud, since the area outside the RGB camera's field of view is not suitable for multimodal perception processing, only the points within the camera's field of view are retained as input for subsequent algorithms.

[0056] Step 2: Design a dual-stream network for image data stream and two-dimensional point set feature stream. For the incomplete bimodal information after alignment, use point cloud center segmentation method and two-dimensional segmentation method to perform instance segmentation and fusion respectively.

[0057] Step 3: In the image instance segmentation backbone, since image features contain a lot of detailed information about objects and radar features contain spatial information about objects, feature extraction is performed on the features of the image stream fused with the radar stream. The Center Mask network is used to segment 2D instances to extract image instances corresponding to two-dimensional point sets. Geometrically, this process is equivalent to upscaling a 2D segmentation mask to the space of a 3D view frustum and collecting all 3D points falling into this view frustum. Specifically, n instance masks are generated from the RGB single-view image through the 2D instance segmentation model, corresponding to M respectively. i , where i = 1, 2, ..., m, and the point sets corresponding to each instance in the two-dimensional point set feature are segmented using image instance boxes, and are regarded as image segmentation instances;

[0058] Step 4: In the sparse instance segmentation backbone, the cluster center method is used to segment point cloud instances. The point cloud is voxelized and converted into a set of voxels. Then, a sparse voxel feature extractor is used as the backbone to extract voxel features. Given voxel features, they are converted into point features. Finally, a lightweight point-level MLP is used to perform point-level classification and center scoring. Points with classification scores higher than the threshold τ are retained as foreground points, and the point set is grouped into 3D instances.

[0059] Step 5: The two segmentation backbones fuse the image instance and the LiDAR instance;

[0060] Step Six: Finally, in the visual information processing module, a multimodal instance correction module is designed to match the updated instances to the real 3D bounding boxes, minimizing the noise impact of camera segmentation instances.

[0061] Step 7: After instance correction, the correction results are used as input to the tracking target map to complete the matching association between the trajectory and the detection. When there is occlusion, tracking using only the previous stage will lead to frequent label ID switching. To overcome this problem and ensure that the target maintains the same label ID throughout the tracking process, a tracking target map matching based on the feature set of points after instance segmentation is designed to construct the tracking target map, store the lost target trajectory, and determine the status of newly generated, tracked, and disappeared trajectories; the instance center point Center(N) is used. i The obtained 3D features do not need to be transferred to the current frame coordinates, and relatively accurate cross-frame matching can be directly achieved, simplifying the matching method and reducing memory usage; for unmatched targets d∈D t D t-1 A fully connected 3D volume feature matching is performed between target k∈K in the tracking target map. If the matching is successful, p inherits the label ID of target h. If the target matching fails in frame t, it is determined to be a new target and a new label ID is assigned. If the target matching fails in frame t-1, it is determined to be a pre-disappearing target and added to the disappearing target map K. If target k fails to match d in consecutive frames, it is determined to be a disappearing target.

[0062] In the target detection and trajectory state matching, for matching detections and trajectories, the trajectory is updated; for unmatched detections, a new trajectory is initialized; for unmatched trajectories, they are retained for τ frames. If they reappear within τ frames, they are updated; if they do not reappear, the trajectory is terminated. This completes the tracking of multiple targets.

[0063] In step one, the formula for calculating the two-dimensional feature P of the point cloud is: Where C represents the number of channels in the projected point cloud, H is the height, and W is the width; projecting the point cloud P onto a two-dimensional image plane yields two-dimensional point set features. The calculation formula is: K is the projection intrinsic parameter matrix of the camera, and [R|t] is the extrinsic parameter matrix.

[0064] In step two, the two-stream network consists of an image instance segmentation backbone and a sparse instance segmentation backbone. X is the Lidar stream, and X is the camera stream.

[0065] In step three, the features of the k points in the i-th instance extracted by the camera are:

[0066] In step four, each foreground point predicts a surrogate point representing the center of its associated object. The set of cluster centers is denoted as C. Then, the CCL (Connected Components Labeling) algorithm is used to connect the cluster centers and partition the instances. In this algorithm, if the distance between two cluster centers is less than a threshold, they are considered connected. CCL groups adjacent cluster centers into n groups. Each instance center is mapped one-to-one with a foreground point, and the corresponding foreground point set is used for partitioning, constructing C0. i Corresponding instance N i ; where the features of the k points in the i-th instance extracted by each radar are Then, two sparse instance recognition modules, SIR (Sparse Instance Recognition), were used to... and Feature extraction was performed separately for each. and

[0067] In step five, let These are image instance features segmented from the camera stream; C represents the number of channels, and H and W represent the height and width of the feature map, respectively; let... The instance features in the LiDAR stream are fused into F. fuse The calculation formula is:

[0068] F fuse =F C ∪F L =[F C ;F L ]

[0069] Fusion of dual-modal features F fuse After expanding the number of channels, the fused feature information is processed by introducing a self-attention mechanism with residuals to enhance the interaction of dual-modal instance features and retain the instance feature flow information;

[0070] F * =F fuse +selfAttn(F fuse )

[0071] in For the updated instance features, where the total number of instances is f = m + n; subsequently, two separate MLPs are designed to classify and determine the bounding boxes of the camera stream instance features and radar instance features extracted by the SIR module, respectively.

[0072] In step six, the instance allocation is divided into two stages. The first stage is LiDAR-based allocation, for instance N of radar flow prediction. iCalculate its weighted center point as the instance center point Center(N) i The weight is determined by the classification score;

[0073]

[0074] in, It is instance N i The points in the image correspond to the foreground point classification scores, while It is instance N i For each point in the instance, let the number of points in the instance be q; use the weighted center point to perform the point-in-box assignment strategy, that is, if the weighted center point of an instance falls within a certain 3D bounding box, then assign the instance to that bounding box;

[0075] Because the depth and noise of camera instances are much greater than those of point cloud instances, to address the issue of unallocated camera instances from the previous stage, a camera-based allocation process is continued, calculating M for each remaining unallocated camera instance. i The IoU score S(M) between the 2D mask and the 2D projections of all bounding boxes i The instance is assigned to the corresponding 3D bounding box if the maximum IoU value exceeds the set threshold.

[0076] S(M i ,GT)=IoU 2d (V i GT 2d )

[0077] Among them, M i For a camera segmentation instance, GT is the bounding box, and V... i For camera instance M i Two-dimensional projection, GT 2d This is the two-dimensional projection of the bounding box.

[0078] In step seven, the calculation formula is as follows:

[0079] Y = y aff +y dis

[0080] y aff =S aff |D t -D t-1 |

[0081] y dis =S dis (D t ,K)+S dis (D t-1 ,K)

[0082] K = K + S dis (D t-1 ,K)

[0083] Where Y represents the target being tracked, y aff Indicates the target being matched between two adjacent frames, y dis S represents the target matched with the tracking map. aff S is a score for the similarity of matched targets in two adjacent frames. dis The similarity score is given to the targets that fail to match between two adjacent frames and the targets in the tracking map, where K is the total number of targets that have disappeared from the map.

[0084] In step eight, the total loss in the target detection and recognition loss calculation includes the classification and regression losses of RPN and RCNN, expressed as:

[0085] L total =L rpn +L rcnn

[0086] For the classification loss of RPN, this method uses the following formula to calculate:

[0087] FL(p t )=-α t (1-p t ) γ log(p t )

[0088] Where pt is the probability of the predicted class, i.e., the output probability of the sigmoid function, with α and γ used to balance positive and negative samples; L1 loss (mean absolute error MAE) is used to optimize the ground truth bounding boxes; after optimizing the regressed 3D detection boxes (x, y, z, l, h, w, θ) with L1 loss, the regression loss is calculated; that is, the RPN loss L... rpn Class loss L cls Add bounding box loss L reg :

[0089] L rpn =L cls +L reg

[0090]

[0091]

[0092] E is the cross-entropy loss, and S is the L1 loss; and Represents the 3D ground truth bounding box and residual offset; the RCNN loss is the sum of the classification loss, regression loss, and association module loss. The classification loss and regression loss are calculated in the same way as the RPN loss, and the association module loss uses the L1 loss.

[0093] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or basic characteristics. Therefore, the embodiments should be considered exemplary and non-limiting in all respects. The scope of the invention is defined by the appended claims rather than the foregoing description. Therefore, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0094] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A target tracking method based on the fusion of vision and radar point clouds, characterized in that, The specific steps of this target tracking method based on the fusion of vision and radar point clouds are as follows: Step 1: By using perspective projection, the 3D LiDAR points are projected onto the coordinate plane of the 2D image to obtain the 2D features P of the point cloud. Then, a multimodal alignment and cropping module is introduced to combine the RGB image and the projected point cloud features. Fine-grained alignment is performed to find the overlapping parts of the multimodal inputs, and the RGB image and projected point cloud features are cropped to obtain a compact aligned input. Step 2: Design a dual-stream network for image data stream and two-dimensional point set feature stream. For the incomplete bimodal information after alignment, use point cloud center segmentation method and two-dimensional segmentation method to perform instance segmentation and fusion respectively. Step three: image instance segmentation backbone, for image stream fusion radar stream feature extraction, using CenterMask network 2D instance segmentation out of two-dimensional point set corresponding image instance, specifically: through 2D instance segmentation model from the single view of RGB image generates n instance mask, respectively corresponding to M i Where i = 1,2, …, m, and use image instance frame to project the point set of the region corresponding to each instance in the two-dimensional point set feature, and regard it as image segmentation instance; Step 4: In the sparse instance segmentation backbone, the cluster center method is used to segment point cloud instances. The point cloud is voxelized and converted into a set of voxels. Then, a sparse voxel feature extractor is used as the backbone to extract voxel features. Given voxel features, they are converted into point features. Finally, a lightweight point-level MLP is used to perform point-level classification and center scoring. Points with classification scores higher than the threshold τ are retained as foreground points, and the point set is grouped into 3D instances. Step 5: The two segmentation backbones fuse the image instance and the LiDAR instance; Step Six: Finally, in the visual information processing module, a multimodal instance correction module is designed to match the updated instances to the real 3D bounding boxes, minimizing the noise impact of camera segmentation instances. Step 7: After instance correction, the correction result is used as the input of the tracking target map and the matching association between the trajectory and the detection is completed. When there is occlusion, tracking using only the previous stage will cause the label ID to switch frequently. In order to overcome this problem and ensure that the target maintains the same label ID throughout the tracking process, a tracking target map matching based on the feature set of points after instance segmentation is designed to construct the tracking target map, store the lost target trajectory, and determine the status of newly generated, tracked and updated, and disappeared trajectories. Using the instance center point Center(N) i The obtained 3D features do not need to be transferred to the current frame coordinates, and relatively accurate cross-frame matching can be directly achieved, simplifying the matching method and reducing memory usage; for unmatched targets d∈D t D t-1 A fully connected 3D volume feature matching is performed between target k∈K in the tracking target map. If the matching is successful, p inherits the label ID of target h. If the target matching fails in frame t, it is determined to be a new target and a new label ID is assigned. If the target matching fails in frame t-1, it is determined to be a pre-disappearing target and added to the disappearing target map K. If target k fails to match d in consecutive frames, it is determined to be a disappearing target. In the target detection and trajectory state matching, for matching detections and trajectories, the trajectory is updated; for unmatched detections, a new trajectory is initialized; for unmatched trajectories, they are retained for τ frames. If they reappear within τ frames, they are updated; if they do not reappear, the trajectory is terminated. This completes the tracking of multiple targets.

2. The target tracking method based on the fusion of vision and radar point clouds according to claim 1, characterized in that: In step one, the formula for calculating the two-dimensional feature P of the point cloud is: Where C represents the number of channels in the projected point cloud, H is the height, and W is the width; projecting the point cloud P onto a two-dimensional image plane yields two-dimensional point set features. The calculation formula is K is the projection intrinsic parameter matrix of the camera, and [R|t] is the extrinsic parameter matrix.

3. The target tracking method based on the fusion of vision and radar point clouds according to claim 1, characterized in that: In step two, the two-stream network consists of an image instance segmentation backbone and a sparse instance segmentation backbone. X is the Lidar stream, and X is the camera stream.

4. The target tracking method based on the fusion of vision and radar point clouds according to claim 1, characterized in that: In step three, the features of the k points in the i-th instance extracted by the camera are:

5. The target tracking method based on the fusion of vision and radar point clouds according to claim 1, characterized in that: In step four, each foreground point predicts a proxy point representing the center of its belonging object. The set of cluster centers is denoted as C. Then, the cluster centers are connected by using the CCL algorithm to divide the instances. In this algorithm, if the distance between two cluster centers is less than a threshold, they are considered connected. CCL groups adjacent cluster centers into n groups. Assign a one-to-one correspondence between instance centers and foreground points, and partition the data using the corresponding foreground point set to construct C. i Corresponding instance N i ; where the features of the k points in the i-th instance extracted by each radar are Then, two sparse instance identification modules, the SIR module, were used to... and Feature extraction was performed separately for each. and 6. The target tracking method based on the fusion of vision and radar point clouds according to claim 1, characterized in that: In step five, let... These are image instance features segmented from the camera stream; C represents the number of channels, and H and W represent the height and width of the feature map, respectively; let... The instance features in the LiDAR stream are fused into F. fuse The calculation formula is: F fuse =F C ∪F L =[F C ;F L ] Fusion of dual-modal features F fuse After expanding the number of channels, the fused feature information is processed by introducing a self-attention mechanism with residuals to enhance the interaction of dual-modal instance features and retain the instance feature flow information; F * =F fuse +selfAttn(F fuse ) in For the updated instance features, where the total number of instances is f = m + n; subsequently, two separate MLPs are designed to classify and determine the bounding boxes of the camera stream instance features and radar instance features extracted by the SIR module, respectively.

7. The target tracking method based on the fusion of vision and radar point clouds according to claim 1, characterized in that: In step six, the instance allocation is divided into two stages. The first stage is LiDAR-based allocation, for instance N of radar flow prediction. i Calculate its weighted center point as the instance center point Center(N) i The weight is determined by the classification score; in, It is instance N i The points in the image correspond to the foreground point classification scores, while It is instance N i For each point in the instance, let the number of points in the instance be q; use the weighted center point to perform the point-in-boundary assignment strategy, that is, if the weighted center point of an instance falls within a certain 3D bounding box, then assign the instance to that bounding box; Because the depth and noise of camera instances are much greater than those of point cloud instances, to address the issue of unallocated camera instances from the previous stage, a camera-based allocation process is continued, calculating M for each remaining unallocated camera instance. i The IoU score S(M) between the 2D mask and the 2D projections of all bounding boxes i The instance is assigned to the corresponding 3D bounding box if the maximum IoU value exceeds the set threshold. S(M i ,GT)=IoU 2d (V i ,GT 2d ) Among them, M i For a camera segmentation instance, GT is the bounding box, and V... i For camera instance M i Two-dimensional projection, GT 2d This is the two-dimensional projection of the bounding box.

8. The target tracking method based on the fusion of vision and radar point clouds according to claim 1, characterized in that: In step seven, the calculation formula is as follows: Y=y aff +and dis y aff =S aff |D t -D t-1 | y dis =S dis (D t ,K)+S dis (D t-1 ,K) K=K+S dis (D t-1 ,K) Where Y represents the target being tracked, y aff Indicates the target matched between two adjacent frames, y dis S represents the target matched with the tracking map. aff S is a score for the similarity of matched targets in two adjacent frames. dis The similarity score is given to the targets that fail to match between two adjacent frames and the targets in the tracking map, where K is the total number of targets that have disappeared from the map.

9. The target tracking method based on the fusion of vision and radar point clouds according to claim 1, characterized in that: In step seven, the total loss in the target detection and recognition loss calculation includes the classification and regression losses of RPN and RCNN, expressed as: L total =L rpn +L rcnn For the classification loss of RPN, this method uses the following formula to calculate: FL(p t )=-a t (1-p t ) γ log(p t ) Where pt is the probability of the predicted class, i.e., the output probability of the sigmoid function, with α and γ used to balance positive and negative samples; L1 loss (mean absolute error MAE) is used to optimize the ground truth bounding boxes; after optimizing the regressed 3D detection boxes with L1 loss, the regression loss is calculated; i.e., RPN loss L rpn Class loss L cls Add bounding box loss L reg : L rpn =L cls +L reg L cls =-a t (1-C p ) γ log(C p ) E is the cross-entropy loss, and S is the L1 loss; and Represents the 3D ground truth bounding box and residual offset; the RCNN loss is the sum of the classification loss, regression loss, and association module loss. The classification loss and regression loss are calculated in the same way as the RPN loss, and the association module loss uses the L1 loss.

Images