Target tracking method based on visible light, infrared and lidar data fusion

Abstract

This invention discloses a target tracking method based on the fusion of visible light, infrared, and lidar data. It acquires visible light camera data, infrared camera data, and lidar data from a vehicle; it fuses the two-dimensional image data obtained from the visible light and infrared cameras to enhance image target information and improve the detection rate and accuracy of target detection; it uses detection algorithms to detect and extract two-dimensional and three-dimensional target information from the fused image and lidar point cloud data respectively; it projects the point cloud targets onto the image, and then performs target state filtering by associating data with targets at different times based on the point cloud target detection boxes and motion state information, while simultaneously supplementing with image-detected targets, thus completing the fusion tracking of image targets and point cloud targets.

Description

Technical Field

[0001] This invention relates to the field of target tracking technology, specifically to a target tracking method based on the fusion of visible light, infrared and lidar data, applicable to tracking multiple targets using multiple sensors, mainly visible light, infrared and lidar, in complex environments. Background Technology

[0002] Modern battlefield environments are complex and ever-changing. In actual combat, the monitoring range of systems is expanding, the types of targets to be detected are increasing, and the movement characteristics of targets are becoming more complex. Single sensors are easily interfered with, resulting in information loss and uncertainty, making it impossible to guarantee stable reconnaissance missions. Therefore, collaborative detection by sensor systems is necessary. A multi-sensor fusion tracking method should be established, utilizing the complementarity and redundancy of sensor performance to expand the temporal, spatial, and frequency coverage of sensor detection, improve the confidence level of target detection decisions, reduce ambiguity, enhance reliability and fault tolerance, and strengthen the overall system's resistance to interference. Summary of the Invention

[0003] The present invention aims to solve the problems of the prior art mentioned above and proposes a target tracking method based on the fusion of visible light, infrared and lidar data.

[0004] The technical solution of this invention is to fuse visible light images and infrared images to enhance image target information and improve the detection rate and accuracy of target detection. Then, target information of the image and point cloud is detected and extracted from the fused image and point cloud data respectively. The point cloud target is projected onto the image. Then, based on the point cloud target detection box and motion state information, the target at different time is correlated and the target state is filtered. At the same time, the target is supplemented by the image detection target, thus completing the fusion tracking of image target and point cloud target.

[0005] A target tracking method based on the fusion of visible light, infrared, and lidar data includes the following steps:

[0006] Step 1: Feature extraction, fusion, and feature reconstruction are performed on the acquired visible light and infrared images using residual networks and autoencoders to achieve image fusion, enhance image target information, and improve the detection rate and accuracy of target detection;

[0007] Step 2: Use a detection algorithm to perform target detection on the fused image, extract the two-dimensional target information on the image, and use a point cloud detection algorithm to perform target detection on the acquired LiDAR point cloud data to obtain three-dimensional target information.

[0008] Step 3: Predict the state of the target and perform target state filtering by associating the three-dimensional target data at different times.

[0009] Step 4: Perform secondary data association using 2D targets to supplement any missing parts of the 3D targets. Then, complete the fusion tracking of 2D and 3D targets through state updates and trajectory lifecycle management.

[0010] Preferably, step 1 includes the following steps:

[0011] Step 1.1 involves spatially registering the visible light image and the infrared image. Using the camera's translation and rotation matrices, and with the visible light image coordinates as the reference coordinates, the infrared image data is transformed to visible light coordinates for unification. If coordinate system information is unavailable, mathematical morphology methods can be used to extract the edges of the infrared and visible light images separately, resulting in coarse edge images. Then, the SURF algorithm is used to extract feature points from the two edge images. Based on prior knowledge of slope consistency between correct matching point pairs, feature point matching is performed. Finally, the least squares method is used to obtain the affine transformation model parameters and achieve the registration of the two images.

[0012] Step 1.2, Training the autoencoder. Visible light and infrared images are input into the autoencoder for training. The encoder has three convolutional layers, each with an output that is fused and simultaneously serves as the input to the decoder. The autoencoder is trained independently, with the following loss function:

[0013] L auto =L pixel +λL ssim (1)

[0014]

[0015] L ssim =1-SSIM(Output,Input) (3)

[0016]

[0017] in:

[0018] L pixel Pixel loss; The Frobenius norm of the output and input images, i.e., the square root of the sum of squares of corresponding elements; L ssim SSIM: Structural similarity loss; μ: Mean; σ: Variance

[0019] Step 1.3: The fusion network uses a residual network, taking the encoder output as input, and performs convolution and concatenation on the input visible light features and infrared features respectively, and then performs three convolution layers to obtain Conv5; at the same time, it concatenates and convolves the input visible light features and infrared features, and concatenates this convolution layer with Conv5 through a fully connected layer, and finally outputs the fused features as the input of the decoder.

[0020] Step 1.4, Infrared and Visible Light Overall Fusion Model: Place the fusion network between the encoder and decoder, train the model, using visible light and infrared images as input and the fused image as output. The model loss function is as follows:

[0021] L fvusion =αL vi +(1-α)L ir (5)

[0022] L vi =1-SSIM(O,I) vi (6)

[0023]

[0024] in:

[0025] vi and ir represent the visible light image and infrared image, respectively; L is the loss function; O is the output fused image; M is the number of depth features; B(m) is the balanced loss function vector, initially set to (1, 10, 100, 1000); w is the relative weight matrix; and Φ is the feature matrix.

[0026] Visible light images contain rich background details, so the main calculation is of structural loss, while infrared images have rich feature information, so the main calculation is of feature loss.

[0027] Preferably, step 2 includes the following steps:

[0028] Step 2.1: Use image target detection algorithm and point cloud target detection algorithm to obtain target type, target bounding box, and target azimuth information from the image, and target type, target 3D bounding box (target position and target length, width and height), and target azimuth information from the point cloud, respectively.

[0029] Step 2.2: Set the coordinate system of the visible light camera to the world coordinate system. Project the three-dimensional target onto this coordinate system using the translation and rotation matrices of the sensor. Then, based on the three-dimensional target detection box and motion state information, a preliminary trajectory model is established.

[0030] Preferably, step 3 includes the following steps:

[0031] Step 3.1: Predict the state of the target and represent the state of the object's trajectory as an 11-dimensional vector T = (x, y, z, θ, l, w, h, s, v). x v y v z ), where v x v y v z This represents the velocity of an object in three-dimensional space. The predicted trajectory is based on a constant velocity model x. t =x t-1 +v x y t =y t-1 +v y , z t =z t-1 +v z ,therefore,

[0032] Step 3.2: Associate the detected 3D targets with the predicted targets using the following matching rules:

[0033]

[0034]

[0035] in: This indicates the position and size information of a three-dimensional target; It is the Euclidean distance between each target in the i-th frame and each target in the j-th frame; This indicates the angle of rotation around the camera's y-axis; yes and The cosine distance between direction vectors.

[0036] Step 3.3: Based on the above association rules, construct the 3D detection target D. t With predicted trajectory The affinity matrix is ​​used to obtain T. match , T unmatch , These represent the matched trajectory targets and detected targets, as well as the unmatched trajectory targets and detected targets.

[0037] Step 3.4, for the matched 3D detection target According to Bayes' theorem, we treat it as a new trajectory point and update the state of trajectory T.

[0038] Preferably, step 4 includes the following steps:

[0039] Step 4.1: Match the two-dimensional target with the unmatched trajectory target T. unmatchMatching is performed on the two-dimensional projection in its coordinate system, and the correlation metric uses IoU to calculate the correlation degree, resulting in the matched two-dimensional target.

[0040] Step 4.2, for the matched two-dimensional target Compare it with the unmatched 3D targets An IoU correlation analysis is performed on the two-dimensional projection of its coordinate system. If a certain threshold is reached, it is considered as the target and included in the trajectory. If the threshold is not reached, and the three-dimensional target of the target with the highest correlation is matched again within three frames, it is also considered as the target and included in the trajectory.

[0041] Step 4.3, for the remaining unmatched 3D targets that have not been supplemented by 2D targets. If no match is found within three consecutive frames, it is considered a new target, i.e., the start of a new trajectory. The above two steps are called three-dimensional regression.

[0042] Step 4.4, for unmatched trajectory targets T umatch_2 If no match is found within three consecutive frames, the trajectory is considered to have ended.

[0043] This invention provides a target tracking method based on the fusion of visible light, infrared, and lidar data. It acquires visible light camera data, infrared camera data, and lidar data from a vehicle; it fuses the two-dimensional image data obtained from the visible light and infrared cameras to enhance image target information and improve the detection rate and accuracy of target detection; it uses detection algorithms to detect and extract two-dimensional and three-dimensional target information from the fused image and lidar point cloud data respectively; it projects the point cloud targets onto the image, and then performs target state filtering by associating data with targets at different times based on the point cloud target detection boxes and motion state information, while simultaneously supplementing with image-detected targets, thus completing the fusion tracking of image targets and point cloud targets.

[0044] The beneficial effects of this invention are that it combines the advantages of visible light, infrared, and lidar, integrates information from several heterogeneous sensors, utilizes the complementarity and redundancy of sensor performance, expands the information acquisition range of the sensor group, improves the confidence of target detection decisions, reduces ambiguity, enhances reliability and fault tolerance, strengthens the system's anti-interference capability, and improves the accuracy of target tracking. Attached Figure Description

[0045] Figure 1 This is the overall framework diagram of the present invention;

[0046] Figure 2 This is a flowchart illustrating the present invention;

[0047] Figure 3This is a model diagram of the infrared and visible light fusion proposed in this invention;

[0048] Figure 4 This is a flowchart of the infrared and visible light fusion proposed in this invention;

[0049] Figure 5 This is a structural diagram of the image-point cloud fusion tracking proposed in this invention;

[0050] Figure 6 This is a flowchart of the image-point cloud fusion tracking proposed in this invention. Detailed Implementation

[0051] The present invention will be further described in detail below with reference to embodiments, but the scope of protection of the present invention is not limited thereto.

[0052] This invention provides a target tracking method based on the fusion of visible light, infrared, and lidar data. It acquires visible light camera data, infrared camera data, and lidar data from a vehicle; it fuses the two-dimensional image data obtained from the visible light and infrared cameras to enhance image target information and improve the detection rate and accuracy of target detection; it uses detection algorithms to detect and extract two-dimensional and three-dimensional target information from the fused image and lidar point cloud data respectively; it projects the point cloud targets onto the image, and then performs target state filtering by associating data with targets at different times based on the point cloud target detection boxes and motion state information, while simultaneously supplementing with image-detected targets, thus completing the fusion tracking of image targets and point cloud targets.

[0053] Specifically, the method includes the following steps:

[0054] Step 100: The acquired visible light and infrared images are fused by performing feature extraction, fusion, and feature reconstruction through residual networks and autoencoders to enhance the target information in the images and improve the detection rate and accuracy of target detection.

[0055] Step 100 includes the following steps:

[0056] Step 110: Spatial registration is performed on the visible light image and the infrared image. By using the translation and rotation matrices of the camera and taking the coordinates of the visible light image as the reference coordinates, the infrared image data is converted to visible light coordinates for unification.

[0057] Step 120: Train the autoencoder. Input the visible light and infrared images separately into the autoencoder for training. The encoder has three convolutional layers, each with an output that is fused and simultaneously serves as the input to the decoder. The autoencoder is trained independently.

[0058] Step 130: The fusion network uses a residual network, taking the encoder output as input, and performs convolution and concatenation on the input visible light features and infrared features respectively, and then performs three convolution layers to obtain Conv5; at the same time, it concatenates and convolves the input visible light features and infrared features, and concatenates this convolution layer with Conv5 through a fully connected layer, and finally outputs the fused features as the input of the decoder.

[0059] Step 140, Infrared and Visible Light Overall Fusion Model: Place the fusion network between the encoder and decoder, train the model, using visible light images and infrared images as inputs and the fused image as output.

[0060] Step 200: Use a detection algorithm to perform target detection on the fused image, extract two-dimensional target information from the image, and use a point cloud detection algorithm to perform target detection on the acquired LiDAR point cloud data to obtain three-dimensional target information.

[0061] Step 200 includes the following steps:

[0062] Step 210: Use image target detection algorithm and point cloud target detection algorithm to obtain target type, target bounding box, and target azimuth information from the image and target type, target 3D bounding box (target position and target length, width and height), and target azimuth information from the point cloud, respectively.

[0063] Step 220: Set the coordinate system of the visible light camera to the world coordinate system, project the three-dimensional target onto the coordinate system using the translation and rotation matrices of the sensor, and then establish a preliminary trajectory model based on the three-dimensional target detection box and motion state information.

[0064] Step 300: Perform fusion tracking on the fused image and point cloud data. First, predict the state of the target, and then perform target state filtering by associating the 3D target data at different times.

[0065] Step 300 includes the following steps:

[0066] Step 310: Predict the state of the target.

[0067] Step 320: Associate the detected 3D target with the predicted target.

[0068] Step 330, construct the three-dimensional detection target D t With predicted trajectory The affinity matrix is ​​used to obtain T. match , T unmatch , These represent the matched trajectory targets and detected targets, as well as the unmatched trajectory targets and detected targets.

[0069] Step 340, for the matched 3D detection target According to Bayes' theorem, we treat it as a new trajectory point and update the state of trajectory T.

[0070] Step 400: Secondary data association is performed using the two-dimensional target to supplement any missing parts of the three-dimensional target. Then, the fusion tracking of the two-dimensional and three-dimensional targets is completed through state updates and trajectory lifecycle management.

[0071] Step 400 includes the following steps:

[0072] Step 410: Match the two-dimensional target with the unmatched trajectory target T. unmatch Matching is performed on the two-dimensional projection in its coordinate system, and the correlation metric uses IoU to calculate the correlation degree, resulting in the matched two-dimensional target.

[0073] Step 420, for the matched two-dimensional target Compare it with the unmatched 3D targets An IoU correlation analysis is performed on the two-dimensional projection of its coordinate system. If a certain threshold is reached, it is considered as the target and included in the trajectory. If the threshold is not reached, and the three-dimensional target of the target with the highest correlation is matched again within three frames, it is also considered as the target and included in the trajectory.

[0074] Step 430, for the remaining unmatched 3D targets that have not been supplemented by 2D targets. If no match is found within three consecutive frames, it is considered a new target, marking the start of a new trajectory.

[0075] Step 440, for unmatched trajectory target T unmatch_2 If no match is found within three consecutive frames, the trajectory is considered to have ended.

[0076] This invention provides an embodiment, taking vehicle tracking as an example:

[0077] like Figure 1 As shown, the infrared and visible light images are first spatially registered. By using the camera's translation and rotation matrices and taking the coordinates of the visible light image as reference coordinates, the infrared data is converted to visible light coordinates for unification.

[0078] Subsequently, a visible light-infrared fusion model is used to fuse the visible light image and the infrared image into a single image, such as... Figure 3 , Figure 4 As shown.

[0079] The model consists of a pre-trained autoencoder: visible light and infrared images are input into the autoencoder for training. The encoder has three convolutional layers, each with an output that is fused and simultaneously serves as the input to the decoder. The loss function of the autoencoder is as follows: The autoencoder L is pre-trained using the following loss function. auto =L pixel +λL ssim

[0080]

[0081] L ssim = 1 - SSIM(Output, Input)

[0082]

[0083] Where: L pixel Indicates pixel loss; L represents the Frobenius norm of the output and input images, which is the square root of the sum of the squares of the corresponding elements; ssim σ represents the structural similarity loss; SSIM represents the structural similarity measure; μ represents the mean; σ represents the variance.

[0084] After the autoencoder is pre-trained, it is fixed, and a fusion model is inserted for training. The fusion network uses a residual network, taking the encoder output as input, and performs convolution and concatenation on the input visible light and infrared features respectively, followed by three convolutional layers to obtain the Conv5 convolutional layer; simultaneously, it concatenates and convolves the input visible light and infrared features, and concatenates this convolutional layer with the Conv5 layer through a fully connected layer, finally outputting the fused features as the input to the decoder. The loss function of the fusion network is as follows:

[0085] L fusion =αL vi +(1-α)L ir

[0086] L vi =1-SSIM(O,I) vi )

[0087]

[0088] Where: vi and ir represent visible light image and infrared image respectively; L represents loss function; O represents output fused image; M is the number of depth features; B(m) is the balanced loss function vector, initially set to (1, 10, 100, 1000); w is relative weight matrix, and Φ is feature matrix.

[0089] Visible light images contain richer background details, so structural loss is primarily calculated, while infrared images have richer feature information, so feature loss is primarily calculated. Each fusion network has the same structure, but different inputs, outputs, and parameters. The fusion network is placed between the encoder and decoder to train the model, using visible light and infrared images as inputs and the fused image as the output.

[0090] After outputting the fused image, a detection algorithm is used to detect the 2D targets in the fused image. The detection algorithm obtains target type, target bounding box, and target azimuth information from the image, and target type, 3D bounding box (target position and dimensions), and target azimuth information from the point cloud. The coordinate system of the visible light camera is set as the world coordinate system. The 3D targets are projected onto this coordinate system using the sensor's translation and rotation matrices. After projecting the 3D targets onto the image coordinate system, a trajectory model is established. Fusion tracking is then performed, such as... Figure 5 , Figure 6 As shown.

[0091] To predict the target state, the object trajectory is represented as an 11-dimensional vector T = (x, y, z, θ, l, w, h, s, v). x v y v z ), where v x v y v z This represents the velocity of an object in three-dimensional space. The predicted trajectory is based on a constant velocity model x. t =x t-1 +v x y t =y t-1 +v y , z t =z t-1 +v z ,therefore,

[0092] The first stage involves matching 3D targets using the following rules:

[0093]

[0094]

[0095] in: This indicates the position and size information of a three-dimensional target; It is the Euclidean distance between each target in the i-th frame and each target in the j-th frame; This indicates the angle of rotation around the camera's y-axis; yes and The cosine distance between direction vectors.

[0096] Based on the above association rules, construct the 3D detection target D. t With predicted trajectory The affinity matrix is ​​used to obtain T. match , T unmatch , These represent the matched trajectory targets and detected targets, as well as the unmatched trajectory targets and detected targets. For the matched 3D detected targets... According to Bayes' theorem, we treat it as a new trajectory point and update the state of trajectory T.

[0097] In the second stage, the two-dimensional target is compared with the unmatched trajectory target T. unmatch Matching is performed on the two-dimensional projection in its coordinate system, and the correlation metric uses IoU to calculate the correlation degree, resulting in the matched 2D detected target. and unmatched T unmatch_2 The two-dimensional target is compared with the unmatched trajectory target T. umatch Matching is performed on the two-dimensional projection in its coordinate system, and the correlation metric uses IoU to calculate the correlation degree, resulting in the matched two-dimensional detected target. and unmatched T umatch_2 .

[0098] For the matched 2D detection target Compare it with unmatched 3D detection targets Perform an IoU (Inter-Origin Union) analysis on its two-dimensional projection in its coordinate system, and obtain the results based on an 80% matching threshold. and Will It is considered that the target is included in the trajectory.

[0099] Two-dimensional targets that have not reached the threshold when associated with three-dimensional targets. These are considered new trajectory points corresponding to the current trajectory and included in the trajectory. If the threshold is not reached, and the three-dimensional target with the highest correlation is... If a target is matched again within three frames, it is also considered as the target and included in the trajectory, and the trajectory points within those three frames are updated. For the remaining unmatched 3D detected targets that have not been supplemented by 2D targets... If no match is found within three consecutive frames, it is considered a new target, marking the start of a new trajectory. For an unmatched trajectory target T... umatch_2 If no match is found within three consecutive frames, the trajectory is considered to have ended.

[0100] The final T tFor all targets in frame t, each labeled with its ID and other parameters, the trajectory model is used to predict the trajectory. Proceed to the next frame for matching and tracking.

[0101] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0102] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A target tracking method based on the fusion of visible light, infrared, and lidar data, characterized in that, The method includes the following steps: Step 1: Image fusion is achieved by performing feature extraction, fusion, and feature reconstruction on the acquired visible light and infrared images using a residual network and an autoencoder; Step 1 includes the following steps: Step 1.1: Spatial registration of the visible light image and the infrared image is performed. Using the camera's translation and rotation matrices, and with the visible light image coordinates as the reference coordinates, the infrared image data is converted to visible light coordinates for unification. If coordinate system information is unavailable, edges are extracted from both the infrared and visible light images using mathematical morphology methods to obtain coarse edge images. Then, feature points are extracted from the two edge images using the SURF algorithm. Based on prior knowledge of slope consistency between correct matching point pairs, feature point matching is performed. Finally, the affine transformation model parameters are obtained using the least squares method to achieve registration of the two images. Step 1.2, Training the autoencoder: Visible light and infrared images are input into the autoencoder for training. The encoder has three convolutional layers, each with an output that is fused and simultaneously serves as the input to the decoder. The autoencoder is trained independently, with the following loss function: THE auto =L pixel +λL ssim (1) L ssim =1-SSIM(Output,Input) (3) in: L pixel Pixel loss; The Frobenius norm of the output and input images, i.e., the square root of the sum of squares of corresponding elements; L ssim SSIM: Structural similarity loss; μ: mean; σ: variance; Step 1.3: The fusion network uses a residual network, taking the encoder output as input, and performs convolution and concatenation on the input visible light features and infrared features respectively, and then performs three convolution layers to obtain Conv5; at the same time, the input visible light features and infrared features are concatenated and convolved, and this convolution layer is concatenated with Conv5 through a fully connected layer, and finally the fused features are output as the input of the decoder. Step 1.4, Infrared and Visible Light Overall Fusion Model: Place the fusion network between the encoder and decoder, train the model, using visible light and infrared images as input and the fused image as output. The model loss function is as follows: L fusion =αL vi +(1-α)L ir (5) L vi =1-SSIM(O,I vi ) (6) in: vi and ir represent the visible light image and infrared image, respectively; L is the loss function; O is the output fused image; M is the number of depth features; B(m) is the balanced loss function vector, initially set to (1, 10, 100, 1000); w is the relative weight matrix; and Φ is the feature matrix. Step 2: Use a detection algorithm to perform target detection on the fused image, extract the two-dimensional target information on the image, and use a point cloud detection algorithm to perform target detection on the acquired LiDAR point cloud data to obtain three-dimensional target information; Step 3: Perform fusion tracking on the fused image and point cloud data. First, predict the state of the target, and then perform data association and state filtering on the 3D target at different times. Step 3 includes the following steps: Step 3.1: Predict the state of the target and represent the state of the object's trajectory as an 11-dimensional vector T = (x, y, z, θ, l, w, h, s, v). x ,v y ,v z ), where v x ,v y ,v z This represents the velocity of an object in three-dimensional space; the predicted trajectory is based on a constant velocity model x. t =x t-1 +v x y t =y t-1 +v y , z t =z t-1 +v z ,therefore, Step 3.2: Associate the detected 3D targets with the predicted targets using the following matching rules: in: This indicates the position and size information of a three-dimensional target; It is the Euclidean distance between each target in the i-th frame and each target in the j-th frame; This indicates the angle of rotation around the camera's y-axis; yes and Cosine distance between direction vectors; Step 3.3: Construct the 3D detection target D based on the association rules. t With predicted trajectory The affinity matrix is ​​used to obtain T. match , T unmatch , These represent the matched trajectory targets and detected targets, as well as the unmatched trajectory targets and detected targets. Step 3.4, for the matched 3D detection target According to Bayes' theorem, treat it as a new trajectory point and update the state of trajectory T; Step 4: Perform secondary data association through two-dimensional targets to supplement the missing parts of three-dimensional targets, and then complete the fusion tracking of two-dimensional and three-dimensional targets through state updates and trajectory lifecycle management.

2. The target tracking method based on visible light, infrared, and lidar data fusion according to claim 1, characterized in that, Step 2 includes the following steps: Step 2.1: Use image target detection algorithm and point cloud target detection algorithm to obtain target type, target bounding box, and target azimuth information from the image and target type, target 3D bounding box, and target azimuth information from the point cloud, respectively; Step 2.2: Set the coordinate system of the visible light camera to the world coordinate system. Project the three-dimensional target onto this coordinate system using the translation and rotation matrices of the sensor. Then, based on the three-dimensional target detection box and motion state information, establish a preliminary trajectory model.

3. The target tracking method based on visible light, infrared, and lidar data fusion according to claim 1, characterized in that, Step 4 includes the following steps: Step 4.1: Match the two-dimensional target with the unmatched trajectory target T. unmatch Matching is performed on the two-dimensional projection in its coordinate system, and the correlation metric uses IoU to calculate the correlation degree, resulting in the matched two-dimensional target. Step 4.2, for the matched two-dimensional target Compare it with the unmatched 3D targets An IoU correlation analysis is performed on the two-dimensional projection of its coordinate system. If the target reaches the set threshold, it is considered as the target and included in the trajectory. If the target with the highest correlation is matched again within three frames, it is also considered as the target and included in the trajectory. Step 4.3, for the remaining unmatched 3D targets that have not been supplemented by 2D targets. If no match is found within three consecutive frames, it is considered a new target, i.e., the start of a new trajectory; Step 4.4, for unmatched trajectory targets T unmatch_2 If no match is found within three consecutive frames, the trajectory is considered to have ended.

Images