Multi-target recognition and tracking methods, systems, devices and media

By combining two-dimensional features and three-dimensional point cloud motion vectors in a multimodal fusion method, the accuracy and robustness issues of target recognition and tracking in autonomous driving are solved, achieving more efficient target recognition and tracking results.

CN118397600BActive Publication Date: 2026-11-17CHANGAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410577112.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-10
Publication Date
2026-11-17
Estimated Expiration
2044-05-10

AI Technical Summary

Technical Problem

In the field of autonomous driving, the information acquired by a single sensor is limited and it is difficult to fully perceive the state of the target. Multimodal fusion has problems such as time synchronization, spatial registration and heterogeneous data fusion, which affect the accuracy and robustness of target recognition and tracking.

Method used

The system combines the two-dimensional features of the target with the motion vectors in the three-dimensional point cloud space for tracking and verification. The two-dimensional processing module obtains the two-dimensional features, the three-dimensional prediction module performs clustering and motion vector prediction, and the target prediction module performs spatial relationship matching. Multimodal data is fused to improve the accuracy and robustness of recognition and tracking.

Benefits of technology

It reduces algorithm complexity, improves the robustness and accuracy of target recognition and tracking, and solves the challenges of traditional methods through multimodal data fusion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118397600B_ABST
    Figure CN118397600B_ABST
Patent Text Reader

Abstract

This invention belongs to the field of multi-sensor multimodal computer vision technology, and relates to a method, system, device, and medium for multi-target recognition and tracking. It acquires the two-dimensional features of the target, tracks and associates the target based on these features, and simultaneously draws bounding boxes for the target, obtaining a two-dimensional image with classification label boxes. It then acquires the target's three-dimensional point cloud information and initial motion vectors, clusters the three-dimensional point cloud information, and segments it into different three-dimensional point sets based on the clustering results. The motion vectors of the three-dimensional point sets are calculated to predict the target's motion trend, obtaining a three-dimensional point set at the predicted position. Based on the acquired and predicted three-dimensional point sets, a front view of the target's three-dimensional point cloud is obtained. Finally, spatial relationship matching between the front view of the three-dimensional point cloud and the two-dimensional image yields the predicted target image. This invention combines the target's two-dimensional features and the motion vectors in the three-dimensional point cloud space for target tracking and verification, improving the robustness and accuracy of target recognition and tracking.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of multi-sensor multimodal computer vision technology, and relates to a method, system, device and medium for multi-target recognition and tracking. Background Technology

[0002] In the field of autonomous driving, accurate and efficient target recognition and tracking technologies are crucial. Autonomous vehicles need to detect and track targets such as vehicles, pedestrians, and obstacles on the road in real time to ensure driving safety and intelligent decision-making. Although deep learning has driven the rapid development of target recognition and tracking algorithms, it still faces some challenges, such as occlusion, changes in lighting, and target deformation, which may lead to errors or loss in recognition and tracking. Furthermore, the information acquired by a single sensor is limited, making it difficult to comprehensively perceive the target's state.

[0003] By fusing multiple sensors (such as vision and 3D LiDAR) to obtain complementary target information, the accuracy and robustness of target recognition and tracking can be significantly improved. For example, vision sensors provide appearance and semantic information of the target, while LiDAR can accurately measure the target's position and motion information. Multimodal fusion can maximize the advantages of each sensor, thereby greatly improving the accuracy and robustness of target recognition and tracking.

[0004] However, multimodal fusion also faces some challenges, such as time synchronization, spatial registration, and heterogeneous data fusion, which in turn affect the perception performance, decision-making accuracy, and safety of multimodal fusion autonomous driving systems. Summary of the Invention

[0005] In order to overcome the shortcomings of the prior art, the present invention aims to provide a multi-target recognition and tracking method, system, device and medium. The present invention combines the two-dimensional features of the target and the motion vector of the three-dimensional point cloud space to track and verify the target, thereby improving the robustness and accuracy of target recognition and tracking.

[0006] To achieve the above objectives, the present invention employs the following technical solution:

[0007] In a first aspect, the present invention provides a multi-target identification and tracking method, comprising the following steps:

[0008] The two-dimensional features of the target are obtained, and the target is tracked and associated based on the two-dimensional features. At the same time, bounding boxes are drawn for the target to obtain a two-dimensional image with classification label boxes.

[0009] The target's 3D point cloud information and initial motion vector are acquired. The 3D point cloud information is clustered and segmented into different 3D point sets according to the clustering results. The motion vectors of the 3D point sets are calculated. The target's motion trend is predicted using the target's initial motion vector, the motion vectors of the 3D point sets, and the 3D point sets. The 3D point set at the predicted position is obtained. The 3D point cloud image of the target is obtained based on the acquired 3D point set and the predicted 3D point set.

[0010] The three-dimensional point cloud image is converted into a three-dimensional point cloud front view, and the spatial relationship between the three-dimensional point cloud front view and the two-dimensional image is matched to obtain the predicted target image.

[0011] Secondly, the present invention provides a multi-target recognition and tracking system, comprising a two-dimensional processing module, a three-dimensional prediction module, and a target prediction module connected in sequence, wherein:

[0012] Two-dimensional processing module: used to acquire the two-dimensional features of the target, track and associate the target based on the two-dimensional features, and draw bounding boxes for the target to obtain a two-dimensional image with classification label boxes;

[0013] 3D prediction module: used to acquire the 3D point cloud information of the target, cluster the 3D point cloud information and segment it into different 3D point sets according to the clustering results, calculate the motion vector of the 3D point set, use the motion vector and the 3D point set to predict the motion trend of the target, obtain the 3D point set of the predicted position, and obtain the 3D point cloud image of the target based on the acquired 3D point set and the predicted 3D point set.

[0014] Target prediction module: Used to convert a 3D point cloud image into a 3D point cloud front view, and to perform spatial relationship matching between the 3D point cloud front view and a 2D image to obtain the predicted target image.

[0015] Thirdly, the present invention provides an electronic device, comprising: a processor; a memory for storing computer program instructions; and steps for implementing a multi-target recognition and tracking method when executing the computer program.

[0016] Fourthly, the present invention provides a storage medium storing computer program instructions, which are loaded and executed by a processor, wherein the processor performs a multi-target recognition and tracking method.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] 1. The method of this invention acquires the two-dimensional features of the target, tracks and associates the target based on these features, and simultaneously draws a bounding box for the target, obtaining a two-dimensional image with a classification label box; acquires the three-dimensional point cloud information and initial motion vector of the target, clusters the three-dimensional point cloud information and segments it into different three-dimensional point sets based on the clustering results, calculates the motion vector of the three-dimensional point sets, and uses the target's initial motion vector, the motion vector of the three-dimensional point sets, and the three-dimensional point sets to predict the target's motion trend, obtaining a three-dimensional point set at the predicted position; and obtains a three-dimensional point cloud image of the target based on the acquired and predicted three-dimensional point sets; converts the three-dimensional point cloud image into a three-dimensional point cloud front view, and performs spatial relationship matching between the three-dimensional point cloud front view and the two-dimensional image to obtain the predicted target image. By combining the two-dimensional and three-dimensional images through spatial relationships, multi-modal data is fused, reducing the algorithm complexity compared to traditional methods. This invention combines the target's two-dimensional features and the motion vector of the three-dimensional point cloud space to track and verify the target, improving the robustness and accuracy of target recognition and tracking.

[0019] 2. The system of this invention includes a two-dimensional processing module, a three-dimensional prediction module, and a target prediction module connected in sequence. The two-dimensional processing module acquires the two-dimensional features of the target, tracks and associates the target based on these features, and simultaneously draws a bounding box for the target, obtaining a two-dimensional image with a classification label box. The three-dimensional prediction module acquires the three-dimensional point cloud information of the target, clusters the three-dimensional point cloud information, segments it into different three-dimensional point sets based on the clustering results, calculates the motion vectors of the three-dimensional point sets, predicts the motion trend of the target using the motion vectors and the three-dimensional point sets, obtains the three-dimensional point set of the predicted position, and obtains a three-dimensional point cloud image of the target based on the acquired and predicted three-dimensional point sets. The target prediction module converts the three-dimensional point cloud image into a three-dimensional point cloud front view, and performs spatial relationship matching between the three-dimensional point cloud front view and the two-dimensional image to obtain a predicted target image. These modules cooperate with each other, combining the two-dimensional features of the target and the motion vectors in the three-dimensional point cloud space to track and verify the target, thereby improving the robustness and accuracy of target recognition and tracking.

[0020] 3. The device and medium of the present invention can also track and verify the target by combining the two-dimensional features of the target and the motion vector of the three-dimensional point cloud space, which can improve the robustness and accuracy of target recognition and tracking. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method of the present invention;

[0022] Figure 2 This is a diagram illustrating an embodiment of the method of the present invention;

[0023] Figure 3 This is a system module diagram of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] The present invention will now be described in further detail with reference to the accompanying drawings:

[0027] See Figure 1 and Figure 2 This invention discloses a multi-target recognition and tracking method, comprising the following steps:

[0028] S1. Obtain the two-dimensional features of the target, track and associate the target based on the two-dimensional features, and draw bounding boxes for the target to obtain a two-dimensional image with classification label boxes;

[0029] Preferably, the two-dimensional features of the target are obtained, and the target is tracked and associated based on the two-dimensional features. At the same time, bounding boxes are drawn for the target to obtain a two-dimensional image with classification label boxes, as follows:

[0030] Acquire the target's input image or video frame;

[0031] Use the YOLO object detection algorithm to detect objects in input images or video frames;

[0032] Convolutional neural networks are typically used to extract feature vectors of targets.

[0033] Input the target's feature vector into DeepOCSORT, and use DeepOCSORT to track and associate the target;

[0034] Draw bounding boxes for the tracked and associated targets to obtain a 2D image with category label boxes.

[0035] Preferably, the YOLO object detection algorithm is used to detect objects in the input image or video frame, as follows:

[0036] YOLO predicts targets in input images by dividing them into S×S grid cells. Each cell predicts B bounding boxes, along with their corresponding confidence scores and class probabilities. Based on these confidence scores and class probabilities, targets are detected in the input images or video frames.

[0037] Preferably, the target's feature vector is input into DeepOCSORT, and DeepOCSORT is used to track and associate the target, as follows:

[0038] Input the target's feature vector into DeepOCSORT;

[0039] The correlation between the target and targets in previous frames is calculated based on the target's characteristics. The correlation is used to determine whether a new target has appeared. If a new target appears, it is identified using an online classifier and added to the tracking management.

[0040] Maintain the target's trajectory and continuously update the target's position coordinates;

[0041] Output the target's location coordinates and unique ID.

[0042] S2. Obtain the target's 3D point cloud information and initial motion vector, cluster the 3D point cloud information and segment it into different 3D point sets according to the clustering results, calculate the motion vector of the 3D point set, use the target's initial motion vector, the motion vector of the 3D point set and the 3D point set to predict the target's motion trend, obtain the 3D point set of the predicted position, and obtain the target's 3D point cloud image based on the obtained 3D point set and the predicted 3D point set.

[0043] Preferably, the 3D point cloud information of the target is acquired, the 3D point cloud information is clustered, and the data is segmented into different 3D point sets based on the clustering results, as follows:

[0044] The target's 3D point cloud information is acquired, the ground point cloud information is separated from the 3D point cloud information, the remaining 3D point cloud information is clustered, and the clustering results are segmented into different 3D point sets using a point cloud segmentation algorithm based on PointNet++.

[0045] Preferably, the motion vector of the 3D point set is calculated, and the motion trend of the target is predicted using the initial motion vector of the target, the motion vector of the 3D point set, and the 3D point set. The predicted 3D point set is then obtained, and a 3D point cloud image of the target is obtained based on the acquired and predicted 3D point sets, as follows:

[0046] By comparing the initial motion vector of the target, the motion vector of the 3D point set, and the predicted 3D point set, the relative position of each 3D point set in the preceding and following frames is determined using an initial distance threshold and detection and tracking information based on the 2D image target. The average motion vector of the 3D point set is calculated by combining the time difference between frames. The point in the point set that is closest to the sensor is used as the representative of the 3D point set and assigned the average motion vector of the 3D point set. The 3D point set of the predicted position of the target is obtained based on the average motion vector.

[0047] The target's 3D point cloud image is obtained based on the acquired 3D point set and the predicted 3D point set;

[0048] The 3D point cloud image is converted into a front view for display, which facilitates the next step of fusion processing with the 2D image.

[0049] S3. Convert the 3D point cloud image into a 3D point cloud front view, and perform spatial relationship matching between the 3D point cloud front view and the 2D image to obtain the predicted target image.

[0050] Preferably, the predicted target image is obtained by spatially matching the front view of the 3D point cloud with the 2D image, as follows:

[0051] By matching the objects in the 2D image with the 3D point cloud front view based on their spatial positional relationship, the objects in the 2D image are matched with the 3D point cloud front view. The classification label boxes of the objects in the 2D image are assigned to the corresponding 3D point cloud front view, thereby achieving the purpose of content recognition of the 3D point cloud front view. The distance information and motion vectors of the 3D point cloud front view are then assigned to the 2D image processing network to obtain the predicted target image.

[0052] See Figure 1 In another feasible embodiment of the present invention, the following modifications are made as appropriate. The multi-target recognition and tracking method acquires the two-dimensional features of the target, tracks and associates the target based on these features, and simultaneously draws bounding boxes for the target to obtain a two-dimensional image with classification label boxes. It then acquires the three-dimensional point cloud information of the target, clusters the three-dimensional point cloud information, and segments it into different three-dimensional point sets based on the clustering results. It calculates the motion vectors of the three-dimensional point sets, uses the motion vectors and three-dimensional point sets to predict the target's motion trend, and obtains the three-dimensional point set at the predicted position. Based on the acquired and predicted three-dimensional point sets, it obtains a three-dimensional point cloud image of the target. The three-dimensional point cloud image is converted into a three-dimensional point cloud front view, and spatial relationship matching is performed between the three-dimensional point cloud front view and the two-dimensional image to obtain a predicted target image. By combining two-dimensional and three-dimensional images through spatial relationships, multi-modal data is fused, reducing algorithm complexity compared to traditional methods. This invention combines the two-dimensional features of the target and the motion vectors of the three-dimensional point cloud space to track and verify the target, improving the robustness and accuracy of target recognition and tracking.

[0053] Example 1:

[0054] See Figure 1 and Figure 2 This invention discloses a multi-target recognition and tracking method, which is specifically carried out according to the following steps:

[0055] S1. Obtain the input image or video frame;

[0056] S2. Object Detection. The YOLO object detection algorithm is used to detect objects in images or frames. YOLO is a single-stage object detection algorithm whose core idea is to treat object detection as a regression problem.

[0057] YOLO predicts the input image by dividing it into S×S grid cells. Each cell predicts B bounding boxes, along with their corresponding confidence scores and class probabilities.

[0058] S3. Feature Extraction. This involves extracting features from the detected target, typically using convolutional neural networks to extract feature vectors, with ResNet and similar network structures being commonly used.

[0059] S4. Target Association and Tracking. The extracted feature vectors are input into the DeepOCSORT algorithm for tracking;

[0060] S4-1, Data Association. Detected targets are associated with tracked targets by calculating Mahalanobis distance.

[0061] Mahalanobis distance formula:

[0062]

[0063] Where y is the measured value, μ is the mean, and S is the covariance matrix.

[0064] S4-2, State Prediction. A Kalman filter is used to predict the target's position and state in the next frame.

[0065] Kalman filter time update equation:

[0066] x(k+1)=Fx(k)+Bv(k)

[0067]

[0068]

[0069] Kalman filter state update equation:

[0070]

[0071]

[0072]

[0073] Where x(k+1) represents the system state vector at the next time step (k+1), x(k) represents the system state vector at the current time step k, v(k) represents the input signal, F is the transformation matrix, and B is the control matrix. u represents the predicted state value at time t. t-1 W is the control input vector at time t-1. t It's process noise. It is a priori estimate of covariance, Q t It is the variance of the process noise, P t-1 F represents the error covariance matrix of the system state estimate at the previous time t-1. T K represents the transpose of the state transition matrix F. t The Kalman gain describes the correction weight of the measurement for the state variable, H. T This represents the transpose of the measurement matrix H, where H is the measurement matrix that transforms state variables into predicted observations, and R is the measurement noise covariance. z represents the prior state estimate. t P represents the actual observed value; t This shows the error covariance matrix of the system state estimate at the current time t;

[0074] S4-3, New Target Initialization. Initialize newly appearing targets and assign them a unique ID;

[0075] S4-4, State Update. Update the state of the tracked target based on the motion model.

[0076] S5. Visualization. Use the tracking results output by DeepOCSORT to draw bounding boxes and label the corresponding IDs;

[0077] S6. 3D Point Cloud Clustering. Separate the ground point set, cluster the remaining point set, segment the point cloud into different point sets, and assign unique IDs to them;

[0078] 3D point cloud segmentation uses a PointNet++-based point cloud segmentation algorithm. The input is n 3D points (x, y, z), and multi-scale features are extracted through the PointNet++ network.

[0079] f(x1,x2,…,x n )=γ(maxpool(h(x1),…,h(x n )))

[0080] Where h(x) n) represents the learned local features, and γ represents the PointNet++ network weights. Finally, point-level segmentation prediction is performed through a fully connected layer.

[0081] S7. 3D Point Set Motion Vector. By comparing the point cloud data of the current frame with those of the previous and next frames, the relative position of each point set in the previous and next frames is determined using an initial distance threshold and information based on 2D image target detection and tracking. The average motion vector of the point set is calculated by combining the time difference between frames. The point in the point set that is closest to the sensor is used as the representative of the point set and assigned the average motion vector of the point set.

[0082] S8. Convert the processed 3D point cloud image into a front view for display, which will facilitate the next step of fusion processing with the 2D image.

[0083] S9. Image Fusion. By leveraging the spatial relationship between the front view of a 2D image and a 3D point cloud image, and utilizing the object classification bounding boxes generated by the 2D image processing network, objects in the 2D image are matched with point sets in the 3D point cloud image. The classification bounding boxes of the objects in the 2D image are then assigned to the corresponding 3D point sets, achieving the purpose of content recognition in the 3D point cloud image. Furthermore, 3D distance information and motion vectors are incorporated into the 2D image processing network, enriching the usable information of the 2D target detection and tracking network.

[0084] S10. Establish a cross-validation network. A 3D point set with motion vectors and classification bounding boxes is used to supplement and correct information for tracking objects in 2D images, improving the accuracy of target tracking, by leveraging the spatial predictability of the point set. The high accuracy of 2D image recognition methods is used to verify and correct the label information of the 3D point set. Its main function is to generate initial motion vectors and assign labels to newly detected 3D point sets.

[0085] S11, Optimize velocity initialization and update. During tracker initialization, the measured initial motion vector of the target is directly passed to the tracker as the target's initial state, instead of defaulting to zero velocity. During tracking in each frame, for the detection box matched by the tracker, the initial motion vector measured by the LiDAR for that detection box is obtained again. The Kalman filter combines its predicted motion vector with the initial motion vector measured by the LiDAR, updating the velocity state through Kalman gain to predict the target's motion trend.

[0086] S12 optimizes the velocity difference penalty term. During the linear assignment process of the associate function, additional motion vector information is added as a penalty term to the cost matrix calculation. The motion vector difference between each pair of detection boxes and tracking boxes on the x and y axes is calculated, and the motion vector difference is standardized and mapped to the range of 0-1 to convert it into cost (1-Δv). The associate function is one of the components of the Kalman filter and is mainly used to handle the data association problem in multi-target tracking.

[0087] S13, optimize process noise control. Set the process noise covariance matrix Q of the tracker and increase the noise value of the motion vector-related DIM to indicate the uncertainty of the motion vector. In the tracker's predict function, combine prior motion vector knowledge to predict the state, instead of relying entirely on the motion model. In tracking management, for trackers that have not been matched for a long time, if their prior motion vector is non-zero, they can be retained instead of being directly deleted.

[0088] S14, the above processing steps are continued for subsequent frames to achieve real-time detection, identification and tracking of the target.

[0089] Example 2:

[0090] See Figure 1 and Figure 2 This embodiment discloses a multi-target recognition and tracking method, which is carried out according to the following steps:

[0091] Step 1: Use the YOLO algorithm to perform object detection on the input image and obtain the bounding boxes and category information of the objects in the image, such as vehicles, pedestrians, bicycles, traffic lights, obstacles, infrastructure, etc.

[0092] Step 2: Use a feature extraction network to extract target features, such as ResNet.

[0093] The feature representation of the detected target is extracted using the deep feature extraction module of the DeepOCSORT algorithm;

[0094] Step 3: Using the YOLO detection results and the target features obtained through the feature extraction network as input, DeepOCSORT is used to associate and track the target.

[0095] Step 4: Data association is performed by calculating feature similarity, linking newly detected targets with already tracked targets;

[0096] Step 5: For newly appearing targets, initialize a Kalman filter and record its state;

[0097] Step 6: Use a Kalman filter to predict the position and motion state of the tracked target in the current frame;

[0098] Step 7: Match the predicted state with the current detection result and associate it with the previous and next frames of the same target;

[0099] Step 8: Based on the matching results, update the state estimate of each target using a Kalman filter;

[0100] Step 9: Perform ground segmentation on the input 3D point cloud image and remove point sets belonging to the ground.

[0101] Step 10: Cluster and segment the remaining non-ground point cloud data, dividing the point cloud into different point sets and numbering them;

[0102] Step 11: Calculate the motion vector for each point set, which is used for distance determination and motion trend prediction;

[0103] Step 12: Compare the point cloud data of the current frame with those of the previous and next frames, and combine the two-dimensional target detection and tracking results to determine the relative position of each point set in the previous and next frames;

[0104] Step 13: Calculate the average relative velocity of each point set based on the inter-frame time difference;

[0105] Step 14: In each point set, select the point closest to the sensor as the representative point and assign the average relative velocity to the point set;

[0106] Step 15: Using motion vector information, predict the motion trend of each point set and give the possible location in subsequent frames;

[0107] Step 16: Convert the processed 3D point cloud image into a front view display to facilitate fusion with the 2D image;

[0108] Step 17: Using the predicted bounding boxes of objects in the 2D image, associate them with the corresponding 3D point sets;

[0109] Step 18: Assign the classification label boxes of the two-dimensional image objects to the corresponding three-dimensional point sets to achieve three-dimensional point cloud content recognition;

[0110] Step 19: Provide the distance and motion vector information of the 3D point cloud to the 2D target detection and tracking network to enrich the input information available to it.

[0111] Step 20, optimize velocity initialization and update. When initializing the tracker, the measured initial motion vector of the target is directly passed to the initial state of the tracker, instead of the default zero velocity; during tracking in each frame, for the detection box matched by the tracker, the LiDAR measured motion vector of the detection box is obtained again, and the Kalman filter will combine its predicted motion vector and the LiDAR measurement value to update the velocity state in the form of Kalman gain.

[0112] Step 21: Optimize the velocity difference penalty term. During the linear allocation process of the associate function, additional motion vector information is added as a penalty term to the cost matrix calculation. The motion vector difference between each pair of detection boxes and tracking boxes on the x and y axes is calculated, and the motion vector difference is standardized and mapped to the range of 0-1 to convert it into cost (1-Δv).

[0113] Step 22, optimize process noise control. Set the process noise covariance matrix Q of the tracker and increase the noise value of the velocity-related DIM to indicate the uncertainty of velocity. In the tracker's predict function, combine prior velocity knowledge to predict the state, instead of relying entirely on the motion model. In target tracking management, for trackers that have not matched for a long time, if their prior motion vector is non-zero, they can be retained instead of being directly deleted.

[0114] Step 23: Continue using the above processing flow for subsequent frames to achieve real-time detection, identification and tracking of the target.

[0115] In step 20, during tracker initialization, the measured initial motion vector of the target is directly passed to the tracker's initial state instead of the default zero velocity. When the relative velocity state of the internal dynamic model is initialized to zero by default, the tracker will remain stationary and predict motion even after the target disappears and the velocity state is not updated, failing to simulate continuous target motion. To improve this, we can extract the target's motion vector information during tracker initialization and directly assign the motion vector to the tracker's initial state. This allows the tracker to continuously use the measured initial velocity for more accurate motion prediction even after the target disappears, which is more reasonable than the default zero-velocity initial state, even with noise in the velocity measurement. Furthermore, during tracking in each frame, the LiDAR-measured velocity of the matched detection box is retrieved again. The Kalman filter combines its predicted velocity with the LiDAR measurement value and updates the velocity state using Kalman gain. This approach provides the tracker with prior information about continuous motion, helping it to make more accurate and stable state predictions during target disappearance.

[0116]

[0117]

[0118] In step 21, during the linear allocation process of the associate function, additional motion vector information is added as a penalty term to the cost matrix calculation. The original cost matrix consists of two parts:

[0119] iou_cost: Cost of IOU overlap of anchor frames

[0120] reid_cost: The cost of appearance similarity calculated using the appearance model.

[0121] Now, a third term, speed_cost, is introduced. This term calculates the velocity difference between each pair of detection and tracking boxes along the x and y axes. The velocity difference is standardized and mapped to a range of 0-1, which is then converted into a cost. The larger the velocity difference, the higher the cost. During linear assignment to find pairs, box pairs with large velocity differences receive a higher cost penalty, making them less likely to be matched. This method introduces a constraint based on velocity information, identifying box pairs with matching velocities and improving matching accuracy.

[0122]

[0123]

[0124] In step 32, the process noise covariance matrix Q of the tracker is set, and the noise value of the motion vector correlation matrix DIM is increased to indicate the uncertainty of the motion vector. This makes the tracker rely more on measurement than prior knowledge. In the tracker's predict function, the state is predicted by incorporating prior motion vector knowledge, rather than relying entirely on the motion model. In tracking management, for trackers that have not been matched for a long time, if their prior motion vector is non-zero, they can be retained instead of being directly deleted.

[0125]

[0126]

[0127] Example 3:

[0128] See Figure 1 and Figure 2 This invention discloses a multi-target recognition and tracking method, which is specifically carried out according to the following steps:

[0129] Step 1: Use the YOLO-DeepOCSORT algorithm to perform target detection and tracking on the 2D image to obtain the target's bounding box, category, and motion trajectory information.

[0130] Step 2: Acquire 3D point cloud image data based on LiDAR.

[0131] Step 3: Perform point clustering and segmentation on the 3D point cloud data to obtain different point sets.

[0132] Step 4: Calculate the motion vector for each point set and predict the motion trend.

[0133] Step 5: Based on spatial location relationships, match the two-dimensional image detection results with the three-dimensional point cloud point set.

[0134] Step 6: Assign the category labels of the two-dimensional image targets to the corresponding three-dimensional point sets to complete the content recognition of the three-dimensional point cloud.

[0135] Step 7: Use the motion vector information from the 3D point cloud image to supplement the performance deficiencies of 2D target tracking.

[0136] Step 8: Utilize the advantages of 2D image content recognition to supplement the shortcomings of 3D point cloud content recognition.

[0137] Step 9: Use motion prediction from 3D point clouds to authenticate and correct targets in the 2D image.

[0138] See Figure 3 Based on the above method, the present invention also discloses a multi-target recognition and tracking system, comprising a two-dimensional processing module, a three-dimensional prediction module, and a target prediction module connected in sequence, wherein:

[0139] Two-dimensional processing module: used to acquire the two-dimensional features of the target, track and associate the target based on the two-dimensional features, and draw bounding boxes for the target to obtain a two-dimensional image with classification label boxes;

[0140] 3D prediction module: used to acquire the 3D point cloud information of the target, cluster the 3D point cloud information and segment it into different 3D point sets according to the clustering results, calculate the motion vector of the 3D point set, use the motion vector and the 3D point set to predict the motion trend of the target, obtain the 3D point set of the predicted position, and obtain the 3D point cloud image of the target based on the acquired 3D point set and the predicted 3D point set.

[0141] Target prediction module: Used to convert a 3D point cloud image into a 3D point cloud front view, and to perform spatial relationship matching between the 3D point cloud front view and a 2D image to obtain the predicted target image.

[0142] See Figure 3In another feasible embodiment of the present invention, the following modifications are made as needed. It includes a two-dimensional processing module, a three-dimensional prediction module, and a target prediction module connected in sequence. The two-dimensional processing module acquires the two-dimensional features of the target, tracks and associates the target based on these features, and simultaneously draws a bounding box for the target, obtaining a two-dimensional image with a classification label box. The three-dimensional prediction module acquires the three-dimensional point cloud information of the target, clusters the three-dimensional point cloud information, segments it into different three-dimensional point sets based on the clustering results, calculates the motion vectors of the three-dimensional point sets, predicts the motion trend of the target using the motion vectors and the three-dimensional point sets, obtains the three-dimensional point set of the predicted position, and obtains the three-dimensional point cloud image of the target based on the acquired and predicted three-dimensional point sets. The target prediction module converts the three-dimensional point cloud image into a three-dimensional point cloud front view, and performs spatial relationship matching between the three-dimensional point cloud front view and the two-dimensional image to obtain the predicted target image. These modules cooperate with each other, and by combining the two-dimensional features of the target and the motion vectors of the three-dimensional point cloud space to track and verify the target, the robustness and accuracy of target recognition and tracking can be improved.

[0143] An electronic device includes: a processor; a memory for storing computer program instructions; and steps for implementing a multi-target recognition and tracking method when executing the computer program.

[0144] A storage medium storing computer program instructions, which are loaded and executed by a processor, wherein the processor performs a multi-target recognition and tracking method.

[0145] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0147] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0149] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A method of multi-target recognition and tracking, characterized by, Includes the following steps: The two-dimensional features of the target are obtained, and the target is tracked and associated based on the two-dimensional features. At the same time, bounding boxes are drawn for the target to obtain a two-dimensional image with classification label boxes. The process involves acquiring the target's 3D point cloud information and initial motion vector, clustering the 3D point cloud information, and segmenting it into different 3D point sets based on the clustering results. The acquired target's initial motion vector, the motion vectors of the 3D point sets, and the predicted 3D point sets are compared. Using an initial distance threshold and detection and tracking information based on the 2D image target, the relative position of each 3D point set across consecutive frames is determined. The average motion vector of the 3D point set is calculated by combining the time difference between frames. The point closest to the sensor in the point set is used as the representative of the 3D point set, and its average motion vector is assigned. The predicted 3D point set of the target's position is obtained based on the average motion vector. A 3D point cloud image of the target is obtained from the acquired and predicted 3D point sets. This 3D point cloud image is then converted into a front view for display, facilitating subsequent fusion processing with the 2D image. The three-dimensional point cloud image is converted into a three-dimensional point cloud front view. By matching the objects in the two-dimensional image with the three-dimensional point cloud front view based on the spatial positional relationship between the two-dimensional image and the three-dimensional point cloud front view, the classification label box of the object in the two-dimensional image is assigned to the corresponding three-dimensional point cloud front view, so as to achieve the purpose of content recognition of the three-dimensional point cloud front view. The distance information and motion vector of the three-dimensional point cloud front view are given to the two-dimensional image processing network to obtain the predicted target image.

2. The multi-target recognition and tracking method according to claim 1, characterized in that, The process involves acquiring the two-dimensional features of the target, tracking and associating the target based on these features, and simultaneously drawing bounding boxes for the target to obtain a two-dimensional image with classification label boxes, as detailed below: Acquire the target's input image or video frame; Use the YOLO object detection algorithm to detect objects in input images or video frames; Use a convolutional neural network to extract the feature vector of the target; Input the target's feature vector into DeepOCSORT, and use DeepOCSORT to track and associate the target; Draw bounding boxes for the tracked and associated targets to obtain a 2D image with category label boxes.

3. The multi-target recognition and tracking method according to claim 2, characterized in that, The YOLO object detection algorithm is used to detect objects in the input image or video frame, specifically as follows: YOLO predicts targets in input images by dividing them into S×S grid cells. Each cell predicts B bounding boxes, along with their corresponding confidence scores and class probabilities. Based on these confidence scores and class probabilities, targets are detected in the input images or video frames.

4. The multi-target recognition and tracking method according to claim 2, characterized in that, The process involves inputting the target's feature vector into DeepOCSORT, and then using DeepOCSORT to track and associate the target, as detailed below: Input the target's feature vector into DeepOCSORT; The correlation between the target and targets in previous frames is calculated based on the target's characteristics. The correlation is used to determine whether a new target has appeared. If a new target appears, it is identified using an online classifier and added to the tracking management. Maintain the target's trajectory and continuously update the target's position coordinates; Output the target's location coordinates and unique ID.

5. The multi-target identification and tracking method according to claim 1, characterized in that, The process involves acquiring the target's 3D point cloud information and initial motion vector, clustering the 3D point cloud information, and segmenting it into different 3D point sets based on the clustering results, as detailed below: The target's 3D point cloud information and initial motion vector are obtained. The ground point cloud information in the 3D point cloud information is separated. The remaining 3D point cloud information is clustered. The clustering results are segmented into different 3D point sets using a point cloud segmentation algorithm based on PointNet++.

6. A multi-target recognition and tracking system for implementing the method of any one of claims 1 to 5, characterized in that, It includes a two-dimensional processing module, a three-dimensional prediction module, and a target prediction module connected in sequence, wherein: Two-dimensional processing module: used to acquire the two-dimensional features of the target, track and associate the target based on the two-dimensional features, and draw bounding boxes for the target to obtain a two-dimensional image with classification label boxes; 3D prediction module: used to acquire the 3D point cloud information of the target, cluster the 3D point cloud information and segment it into different 3D point sets according to the clustering results, calculate the motion vector of the 3D point set, use the motion vector and the 3D point set to predict the motion trend of the target, obtain the 3D point set of the predicted position, and obtain the 3D point cloud image of the target based on the acquired 3D point set and the predicted 3D point set. Target prediction module: Used to convert a 3D point cloud image into a 3D point cloud front view, and to perform spatial relationship matching between the 3D point cloud front view and a 2D image to obtain the predicted target image.

7. An electronic device, comprising: processor; A memory for storing computer program instructions; characterized in that, when executing the computer program, it implements the steps of the multi-target recognition and tracking method as described in any one of claims 1-5.

8. A storage medium storing computer program instructions, characterized in that, When the computer program instructions are loaded and run by the processor, the processor executes the multi-target recognition and tracking method according to any one of claims 1-5.