Online tracking methods for multiple categories and targets in monitoring scenarios

By employing a multi-level cascaded matching strategy and a fast color feature extraction algorithm, the stability and real-time performance issues of multi-category, multi-target tracking in monitoring scenarios are resolved, achieving efficient tracking in complex environments and reducing computational resource consumption and target mismatch probability.

CN118096828BActive Publication Date: 2026-11-17HANGZHOU ARCVIDEO TECHNOLOGY CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410259797.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-07
Publication Date
2026-11-17
Estimated Expiration
2044-03-07

AI Technical Summary

Technical Problem

In monitoring scenarios, multi-category, multi-target tracking tasks face challenges such as different motion characteristics of different target categories, large size differences, complex occlusion, unstable target detection, and insufficient computing resources. These issues make it difficult for tracking algorithms to achieve stable and real-time tracking in complex environments.

Method used

A multi-level cascaded matching strategy and a disappearing target re-matching strategy are adopted. Combined with fast color histogram feature vector extraction and Hungarian algorithm, a matching method based on intersection-union ratio (IoU) and greedy algorithm is designed. The IoU calculation method is optimized. By using a virtual IoU algorithm for face and license plate, the consumption of computing resources is reduced and the tracking stability and accuracy are improved.

Benefits of technology

It achieves efficient and stable tracking of multiple categories and multiple targets in complex monitoring scenarios, reduces computational resource consumption, improves the robustness and real-time performance of the tracker, and reduces the probability of ID mismatch when similar targets overlap.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118096828B_ABST
    Figure CN118096828B_ABST
Patent Text Reader

Abstract

This invention discloses a multi-category, multi-target online tracking method in a surveillance scenario, comprising the following steps: S10, using a target detector to detect targets in the current frame of a video image, obtaining the detection box coordinates, category, and confidence score of all target categories; S20, using a fast color histogram feature vector extraction algorithm to obtain the color feature vector of the target; S30, dividing the target detection boxes into high-scoring boxes and low-scoring boxes according to the target confidence score, setting different matching priorities and matching strategies for high-scoring boxes and low-scoring boxes; S40, designing a matching method based on intersection-union ratio (IoU) and a greedy algorithm; S50, matching all detected target boxes with all trajectories in the tracker, using a multi-level cascading strategy for multiple rounds of matching; S60, processing the remaining unmatched detection boxes and trajectories for adding and deleting trajectories; S70, updating the status information of all trajectories; S80, performing ID fusion binding of face and human targets to obtain a fused ID.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to an online tracking method for multiple categories and multiple targets in a monitoring scenario. Background Technology

[0002] With the increasing demand for intelligent analysis of video surveillance in fields such as public safety, autonomous driving and smart cities, target detection and target tracking algorithms are playing an increasingly important role in these fields as an important means of obtaining complete time series information and motion trajectory information of targets. At present, detection-based tracking methods are one of the more mainstream target tracking methods in engineering applications. Its characteristic is that it tracks based on the detection box results obtained by the target detection algorithm. The tracking algorithm associates information with the detection box results of the current frame based on historical trajectory information, target pedestrian re-identification (reid) feature information, etc. The tracking itself requires less computational resources and time, and can better meet the needs of online real-time tracking. Due to the high complexity of the environment in the monitoring scene, the following problems and challenges are faced in carrying out multi-category and multi-target tracking tasks: (1) Different categories of targets have different motion characteristics, and their motion speed and nonlinearity of motion trajectory are different. How to take into account the motion prediction of each category in the tracking algorithm is particularly important. (2) Different categories of targets have different sizes. Some categories of targets are small in size, making target detection more difficult and the detection box results less stable, which brings greater challenges to tracking. (3) There is a lot of mutual occlusion between targets. In complex monitoring scenarios, there may be partial or complete occlusion between targets of the same category, targets of different categories, and targets and background. Maintaining the stability of the tracker under occlusion is very challenging. (4) The stability of the target detector. In real-world scenarios, there will inevitably be false detections, false negatives, and inaccurate detection boxes of the target detector. Therefore, the tracker needs to be robust enough. (5) The computational resources required to utilize REID feature information are large. This problem is particularly prominent when the computing power of edge computing devices is insufficient and there are a large number of targets in the scene, making it difficult to achieve multi-channel real-time online tracking. Summary of the Invention

[0003] In view of the above-mentioned problems, the present invention provides a method for online tracking of multiple categories and multiple targets in a monitoring scenario.

[0004] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0005] A method for online tracking of multiple categories and multiple targets in a monitoring scenario includes the following steps:

[0006] S10, Use the object detector to perform object detection on the current frame of the video image, and obtain the detection box coordinates, category and confidence score of all categories of objects;

[0007] S20, use the fast color histogram feature vector extraction algorithm to obtain the color feature vector of the target;

[0008] S30: Based on the target confidence score, the target detection box is divided into high-scoring boxes and low-scoring boxes, and different matching priorities and matching strategies are set for high-scoring boxes and low-scoring boxes;

[0009] S40, a matching method based on intersection-union ratio (IoU) and greedy algorithm is designed, and a virtual IoU algorithm is designed for two categories: face and license plate.

[0010] S50 matches all detected target boxes with all trajectories in the tracker, employs a multi-level cascading strategy for multiple rounds of matching, and includes a re-matching strategy for disappearing targets;

[0011] S60, Processing the final remaining unmatched detection boxes, processing the final remaining unmatched trajectories, used for adding and deleting trajectories;

[0012] S70, update the status information of all trajectories, calculate the stable width and height values, stable corner points, and moving speed vector in the most recent first time, and finally calculate the coordinates of the trajectory prediction box;

[0013] S80 uses the Hungarian algorithm to match face and human targets in all trajectories. After matching, a mapping table is established between face ID, human ID, and fusion ID. The IDs of face and human targets are fused and bound to obtain the fusion ID.

[0014] In one possible implementation, the categories include pedestrians, motor vehicles, non-motorized vehicles, faces, and license plates.

[0015] In one possible implementation, S20, obtaining the color feature vector of the target using a fast color histogram feature vector extraction algorithm includes: expanding the target detection box inward based on its coordinates, expanding the four borders of the detection box inward by a certain percentage, and selecting the expanded box as the main part of the target for color histogram calculation.

[0016] In one possible implementation, S20, obtaining the target's color feature vector using a fast color histogram feature vector extraction algorithm includes color space compression. For a 3-channel 24-bit BGR original image, 16,777,216 colors are compressed into a 1331-dimensional feature vector. Specific steps include:

[0017] S201, map the 8-bit pixel value range of 0-255 of each channel to the simplified range of 0-10, where each value in the range of 0-9 corresponds to the original pixel value range of 24, and the value in the range of 10 corresponds to the original pixel value range of 16;

[0018] S202, each channel corresponds to a different weight coefficient, where channel B corresponds to 121, channel G corresponds to 11, and channel R corresponds to 1;

[0019] S203, To accelerate calculation, a mapping table between pixel values ​​and simplified region values ​​is established. The value after dividing the pixel value by 8 corresponds to the index position subscript value of the mapping table. Each value in the mapping table is the corresponding simplified interval value multiplied by the channel weight coefficient.

[0020] S204: Pixels within the target area are counted at each stride using a lookup table. The sum of the values ​​obtained from the lookup table for the three color channels of each pixel is the feature vector dimension corresponding to that pixel value. After the feature vector of the color histogram distribution is counted, the coordinate value of each dimension of the vector represents the number of pixel values ​​in that interval within the target area.

[0021] In one possible implementation, the stride is calculated as follows: sampling is performed using alternating rows and columns, and the sampling stride is set according to the width and height of the box, where w and h represent the width and height of the box. The stride is then calculated as follows:

[0022]

[0023] In one possible implementation, S50 matches all detected target boxes with all trajectories in the tracker, employing a multi-level cascading strategy for multiple rounds of matching, and including a re-matching strategy for disappearing targets, specifically including:

[0024] S501, perform the first round of the first batch of matching, select the high-scoring bounding box result set and match it with the predicted bounding boxes of the tracker's existing trajectories of this category that have not been lost;

[0025] S502, perform the first round of second batch matching, select the high-scoring box results and match them with the predicted boxes of the missing trajectories of this category in the tracker, and calculate the similarity score between the trajectory historical color histogram feature vector and the current detection box color histogram feature vector;

[0026] S503, perform the second round of the first batch of matching, select the high-scoring bounding box result set and match it with the predicted bounding boxes of the tracker for the trajectories of this category that have not been lost. The IoU matching threshold is significantly smaller than that of the first round of the first batch of matching.

[0027] S504, perform the second round of the second batch of matching, select the low-scoring bounding box result set and match it with the predicted bounding boxes of the tracker for the trajectories of this category that have not been lost;

[0028] S505, perform the second round of the third batch of matching, select the high-scoring box results and match them with the predicted boxes of the missing trajectories of this category in the tracker. At this time, a dynamic matching threshold is used, and the threshold decreases linearly with the length of the missing time.

[0029] S506, perform the third round of matching, which is to match fast-moving targets. Select the high-scoring box results and match them with the remaining unmatched trajectories of the tracker that have not been lost. The matching strategy is based on the distance between the center points of the detection box and the prediction box, and at the same time calculates the color feature vector similarity.

[0030] S507, perform the fourth round of matching, and continue the processing of the trajectory to be activated and the missing trajectory. Select the trajectory that has been matched but is to be activated in the tracker, and perform IoU matching with the prediction boxes of all missing trajectories in the tracker. At the same time, calculate the distance between the center points of the two.

[0031] S508, the fifth round of the first batch of matching is carried out. In the case of a significant reduction in the detection box due to occlusion, the high-scoring box results are selected and matched with the remaining unmatched trajectories of the tracker that have not been lost. The matching strategy is based on the ratio of the intersection to the minimum (IoMin).

[0032] S509, perform the fifth round of the second batch of matching, select the high-resolution bounding box results and perform IoMin matching with the missing trajector in the remaining unmatched trajector ...

[0033] S510, perform the sixth round of matching, select the high-scoring bounding box results and match the center point distance with the missing trajectories in the remaining unmatched trajectories of the tracker, and calculate the color feature vector similarity.

[0034] In one possible implementation, S501 includes:

[0035] For a target class_n, the set of high-scoring bounding boxes for that class is selected and matched with the predicted bounding boxes of the tracker's existing trajectories that are not missing. The matching strategy is a greedy algorithm based on IoU, where IoU is the intersection of the predicted bounding box and the detection bounding box divided by their union. The formula is as follows:

[0036]

[0037] Where A is the predicted bounding box and B is the detected bounding box;

[0038] If a matching trajectory can be found and the IoU value is greater than the matching threshold IoU_thr_high1, then the detection box is considered to have successfully matched the tracking trajectory, and the matching status of the trajectory is updated to "matched".

[0039] In one possible implementation, S502 includes:

[0040] First, select the high-scoring bounding box result set of class_n and match it with the predicted bounding boxes of the missing trajectories of that class in the tracker's existing trajectories. A greedy algorithm is used to find the maximum matching IoU. If the IoU value of the matched trajectory is greater than the matching threshold IoU_thr_high1, check if the matched trajectory is at the image edge. If it is at the edge, select the frame with the highest score in its historical confidence score cache list before it disappeared. Calculate the similarity score between the historical color histogram feature vector of this frame and the color histogram feature vector of the current detection box. The similarity calculation method is as follows: Cosine similarity; if the similarity score is greater than the threshold simi_thr1 or the trajectory is not on the edge, then the detection box is considered to have successfully matched the missing trajectory, and the matching status of the trajectory is updated to matched; if the matching IoU value is less than the threshold IoU_thr_match1, then all cached values ​​in the target width and height history cache list of the trajectory are reset. If the missing time of the trajectory exceeds the threshold miss_time_thr1 at the same time, then all cached values ​​in its target box coordinate history cache list and color histogram feature vector history cache list are reset.

[0041] In one possible implementation, S503 includes: selecting a set of high-scoring bounding boxes of class_n, matching them with the predicted bounding boxes of the tracker's existing trajectories of that class that have not been lost; if the IoU value of the matched trajectory is greater than the matching threshold IoU_thr_high2, then the detected bounding box is considered to have successfully matched the tracked trajectory, and the matching status of the trajectory is updated to "matched".

[0042] In one possible implementation, S504 includes: selecting a set of low-scoring bounding boxes of class_n, matching them with the predicted bounding boxes of the tracker's existing trajectories of that class that have not been lost; if the IoU value of the matched trajectory is greater than the matching threshold IoU_thr_low1, then the detected bounding box is considered to have successfully matched the tracked trajectory, and the matching status of the trajectory is updated to "matched".

[0043] In one possible implementation, S505 includes: selecting a set of high-scoring bounding boxes of class_n and matching them with the predicted bounding boxes of the missing trajectories of that class in the tracker's existing trajectories. At this point, a dynamic matching threshold is used, which decreases linearly with the length of the missing time. If the IoU value of the matched trajectory is greater than the matching threshold IoU_thr_high2, it is checked whether the matched trajectory is at the edge of the image. If it is at the edge, the frame with the highest score in the historical cache list of its pre-missing confidence score is selected, and the similarity score between the historical color histogram feature vector of the trajectory corresponding to this frame and the current detection box color histogram feature vector is calculated. The similarity calculation method is cosine similarity. If the similarity score is greater than the threshold simi_thr2 or the trajectory is not at the edge, it is considered that this detection box and this missing trajectory have been successfully matched, and the matching status of the trajectory is updated to matched. If the matching IoU value is less than the threshold IoU_thr_match2, all cached values ​​in the historical cache list of the target width and height of the trajectory are reset. If the missing time of the trajectory exceeds the threshold miss_time_thr2 at the same time, all cached values ​​in the historical cache list of its target box coordinates and the historical cache list of its color histogram feature vector are reset.

[0044] In one possible implementation, S506 includes: selecting a set of high-scoring bounding boxes of category class_n and matching them with the remaining unmatched trajectories of the tracker that are not missing. The matching strategy is not based on IoU but on the distance dist between the center points of the detection box and the predicted box. The distance threshold dist_thr1 is set with different coefficients s_n depending on the category, and the coefficient value needs to be adjusted according to the size of the detection box within the same category. When dist is less than dist_thr1*s_n, the similarity score between the most recent color histogram feature vector cache in the trajectory history and the color histogram feature vector of the current detection box is calculated. If the similarity score is greater than the threshold simi_thr2, the detection box and the trajectory are considered to have successfully matched.

[0045] In one possible implementation, S507 includes: selecting a matched but yet-to-be-activated trajectory from the class_n tracker, performing IoU matching with the predicted bounding boxes of all missing trajectories in the tracker, and simultaneously calculating the distance between their center points. The IoU threshold decreases linearly based on the disappearance time of the missing trajectory. If the IoU value is greater than the threshold IoU_thr_temp and the center point distance is less than the threshold dist_thr_temp, then the frame with the highest score in the historical confidence score cache list before disappearance is selected. The similarity score between the historical color histogram feature vector of this frame and the most recent color histogram feature vector cache of the trajectory to be activated is calculated. If the similarity score is greater than the threshold simi_thr1, then the trajectory to be activated is considered to have successfully matched the missing trajectory. It should be noted that continuation processing is performed because detectors often generate false alarms near real targets. If a new trajectory is generated for this false alarm, it will cause the tracking ID to be interrupted and switched, which is detrimental to the stability of the tracker. The principle of continuation processing is that the generation of new trajectories should be treated as cautiously as possible, especially new trajectories that appear immediately near the missing trajectory. If the trajectory to be activated fails to continue from the missing trajectory, and the activation time exceeds the activation time threshold activate_thr, then a new activated trajectory is generated.

[0046] In one possible implementation, S508 includes: selecting the set of high-scoring bounding boxes of class_n, and matching them with the remaining unmatched trajectories of the tracker that are not missing. The matching strategy is based on IoMin, which is the intersection of the predicted bounding box and the detected bounding box divided by the smaller of the two. The formula is as follows:

[0047]

[0048] If the IoMin value of the matched trajectory is greater than the matching threshold IoU_thr_min1, then the detection box is considered to have successfully matched the tracking trajectory.

[0049] In one possible implementation, S509 includes: selecting a set of high-scoring bounding boxes of class_n and matching them with the missing trajectories in the remaining unmatched trajectories of the tracker. The matching strategy is based on IoMin. If the IoMin value of the matched trajectory is greater than the matching threshold IoU_thr_min2, then the detection box is considered to have successfully matched the tracking trajectory.

[0050] In one possible implementation, S510 includes: selecting the high-scoring bounding box result set of class_n, matching it with the missing trajectory in the remaining unmatched trajectories of the tracker, and calculating the distance between their center points. The distance threshold calculation requires two coefficients: first, a different coefficient s_n is set according to different classes, and within the same class, the coefficient value needs to be adjusted according to the detection box size; second, a coefficient v_n is set according to the trajectory movement speed, with a larger v_n value for faster movement. When the distance is less than the missing time undetected_time * v_n * s_n, the frame with the highest score in the historical cache list of its pre-missing confidence score is selected, and the similarity score between the historical color histogram feature vector of this frame and the current detection box color histogram feature vector is calculated. If the similarity score is greater than the threshold simi_thr1, then the trajectory to be activated is considered to have successfully matched this missing trajectory, and the cache values ​​in all its historical cache lists are reset.

[0051] The present invention offers the following advantages: It optimizes the IoU calculation method based on the characteristics of different target categories, improving the tracking stability for targets with different characteristics; it corrects the predicted bounding box size and predicted motion vector through a predicted bounding box stabilization algorithm, increasing tracking accuracy when the target is partially occluded; it reduces the probability of tracking errors between two target categories by using a face and human target ID binding algorithm; it combines a multi-level cascaded matching strategy with a disappearing target re-matching strategy, and designs a fast target color feature information extraction and similarity calculation method to reduce ID mismatches when similar targets overlap, and improves the matching success rate when a target reappears after disappearing. This allows for reduced computational resource consumption while ensuring continuous and reliable tracking, enabling online real-time tracking. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating the steps of the online tracking method for multiple categories and multiple targets in a monitoring scenario according to an embodiment of the present invention.

[0053] Figure 2 This is a flowchart of a single search in the greedy algorithm method for finding the optimal match in an embodiment of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] See Figure 1The diagram shows a flowchart of the steps of a multi-category, multi-target online tracking method in a monitoring scenario according to an embodiment of the present invention.

[0056] S10, Use an object detector to perform object detection on the current frame of the video image, and obtain the detection box coordinates, category and confidence score of all categories of objects; In a preferred example, the categories of objects include pedestrians, motor vehicles, non-motor vehicles, faces and license plates, etc.

[0057] S20, use the fast color histogram feature vector extraction algorithm to obtain the color feature vector of the target;

[0058] S30: Based on the target confidence score, the target detection box is divided into high-scoring boxes and low-scoring boxes, and different matching priorities and matching strategies are set for high-scoring boxes and low-scoring boxes;

[0059] S40, a matching method based on intersection-union ratio (IoU) and greedy algorithm is designed, and a virtual IoU algorithm is designed for two categories: face and license plate.

[0060] S50 matches all detected target boxes with all trajectories in the tracker, employing a multi-level cascading strategy for multiple rounds of matching, including a re-matching strategy for disappearing targets; the tracker stores all state information for each trajectory.

[0061] S60, Processing the final remaining unmatched detection boxes, processing the final remaining unmatched trajectories, used for adding and deleting trajectories;

[0062] S70, update the status information of all trajectories, calculate the stable width and height values, stable corner points, and moving speed vector in the most recent first time, and finally calculate the coordinates of the trajectory prediction box;

[0063] S80 uses the Hungarian algorithm to match face and human targets in all trajectories. After matching, a mapping table is established between face ID, human ID, and fusion ID. The IDs of face and human targets are fused and bound to obtain the fusion ID.

[0064] The above-described online tracking method for multiple categories and multiple targets in the monitoring scenario combines a multi-level cascaded matching strategy and a disappearing target re-matching strategy, and utilizes a fast target color feature information extraction method and a prediction box stabilization algorithm to achieve efficient and stable online tracking.

[0065] In a multi-category, multi-target online tracking method for monitoring scenarios according to an embodiment of the present invention, a fast color histogram feature vector extraction algorithm is used to obtain the color features of the target, including: expanding the target detection box coordinates inward as the target body; calculating the sampling stride across rows and columns; establishing a mapping table of pixel values ​​and simplified interval values; and quickly generating histogram color feature vectors by looking up the table. To improve the feature representation capability of the color histogram, this embodiment of the present invention uses a method of fusing histograms of all color channel components for calculation. Furthermore, to improve the calculation speed of the subsequent color histogram feature vector similarity distance, this embodiment of the present invention performs color space compression. In a multi-category, multi-target online tracking method for monitoring scenarios according to an embodiment of the present invention, S20, obtaining the color feature vector of the target using the fast color histogram feature vector extraction algorithm includes color space compression. For a 3-channel 24-bit BGR original image, 16,777,216 colors are compressed into a 1331-dimensional feature vector. Specific steps include:

[0066] S201, map the 8-bit pixel value range of 0-255 of each channel to the simplified range of 0-10, where each value in the range of 0-9 corresponds to the original pixel value range of 24, and the value in the range of 10 corresponds to the original pixel value range of 16;

[0067] S202, each channel corresponds to a different weight coefficient, where channel B corresponds to 121, channel G corresponds to 11, and channel R corresponds to 1;

[0068] S203, To accelerate calculation, a mapping table between pixel values ​​and simplified region values ​​is established. The value after dividing the pixel value by 8 corresponds to the index position subscript value of the mapping table. Each value in the mapping table is the corresponding simplified interval value multiplied by the channel weight coefficient.

[0069] S204: Pixels within the target area are counted at each stride using a lookup table. The sum of the values ​​obtained from the lookup table for the three color channels of each pixel is the feature vector dimension corresponding to that pixel value. After the feature vector of the color histogram distribution is counted, the coordinate value of each dimension of the vector represents the number of pixel values ​​in that interval within the target area.

[0070] To reduce the impact of the background area within the detection box on color feature extraction, it is necessary to select the main part of the target as much as possible. In a preferred embodiment, namely S20, obtaining the color feature vector of the target using a fast color histogram feature vector extraction algorithm includes: expanding the detection box inward based on its coordinates, with all four borders of the detection box expanding inward by a certain percentage, and selecting the expanded box as the main part of the target for color histogram calculation. The certain percentage can be 5%-20%, specifically 15%. To accelerate the color histogram calculation, this embodiment of the invention adopts an alternating row and column sampling method, and sets the sampling stride according to the width and height of the box. In a preferred embodiment, w and h represent the width and height of the box, and the stride is calculated as follows:

[0071]

[0072] In a multi-category, multi-target online tracking method for monitoring scenarios according to an embodiment of the present invention, specifically in step S30, a high-score confidence threshold `score_thr1` and a low-score confidence threshold `score_thr2` can be set. Target boxes with a confidence score greater than `score_thr1` are marked as high-score boxes, target boxes with a confidence score less than `score_thr1` but greater than `score_thr2` are marked as low-score boxes, and target boxes with a confidence score lower than `score_thr2` are considered unreliable and will not enter the subsequent tracking and matching process. The high-score confidence threshold `score_thr1` and the low-score confidence threshold `score_thr2` are related to the characteristics of the target detector and can be set based on the test set metrics of target detection and expert experience. Correspondingly, different high-score or low-score confidence thresholds can be set according to the distribution characteristics of confidence scores for different categories of targets.

[0073] In a multi-category, multi-target online tracking method for a monitoring scenario according to an embodiment of the present invention, S50, all detected target boxes and all trajectories in the tracker are matched, and a multi-level cascading strategy is used for multiple rounds of matching, including a re-matching strategy for disappearing targets, specifically including:

[0074] S501, perform the first round of the first batch of matching, select the high-scoring bounding box result set and match it with the predicted bounding boxes of the tracker's existing trajectories of this category that have not been lost;

[0075] S502, perform the first round of second batch matching, select the high-scoring box results and match them with the predicted boxes of the missing trajectories of this category in the tracker, and calculate the similarity score between the trajectory historical color histogram feature vector and the current detection box color histogram feature vector;

[0076] S503, perform the second round of the first batch of matching, select the high-scoring bounding box result set and match it with the predicted bounding boxes of the tracker for the trajectories of this category that have not been lost. The IoU matching threshold is significantly smaller than that of the first round of the first batch of matching.

[0077] S504, perform the second round of the second batch of matching, select the low-scoring bounding box result set and match it with the predicted bounding boxes of the tracker for the trajectories of this category that have not been lost;

[0078] S505, perform the second round of the third batch of matching, select the high-scoring box results and match them with the predicted boxes of the missing trajectories of this category in the tracker. At this time, a dynamic matching threshold is used, and the threshold decreases linearly with the length of the missing time.

[0079] S506, perform the third round of matching, which is to match fast-moving targets. Select the high-scoring box results and match them with the remaining unmatched trajectories of the tracker that have not been lost. The matching strategy is based on the distance between the center points of the detection box and the prediction box, and at the same time calculates the color feature vector similarity.

[0080] S507, perform the fourth round of matching, and continue the processing of the trajectory to be activated and the missing trajectory. Select the trajectory that has been matched but is to be activated in the tracker, and perform IoU matching with the prediction boxes of all missing trajectories in the tracker. At the same time, calculate the distance between the center points of the two.

[0081] S508, the fifth round of the first batch of matching is carried out. In the case of a significant reduction in the detection box due to occlusion, the high-scoring box results are selected and matched with the remaining unmatched trajectories of the tracker that have not been lost. The matching strategy is based on the ratio of the intersection to the minimum (IoMin).

[0082] S509, perform the fifth round of the second batch of matching, select the high-resolution bounding box results and perform IoMin matching with the missing trajector in the remaining unmatched trajector ...

[0083] S510, perform the sixth round of matching, select the high-scoring bounding box results and match the center point distance with the missing trajectories in the remaining unmatched trajectories of the tracker, and calculate the color feature vector similarity.

[0084] A method for online tracking of multiple categories and multiple targets in a monitoring scenario according to an embodiment of the present invention further includes, in step S501:

[0085] For a target class_n, the set of high-scoring bounding boxes for that class is selected and matched with the predicted bounding boxes of the tracker's existing trajectories that are not missing. The matching strategy is a greedy algorithm based on IoU, where IoU is the intersection of the predicted bounding box and the detection bounding box divided by their union. The formula is as follows:

[0086]

[0087] Where A is the predicted bounding box and B is the detected bounding box.

[0088] A virtual IoU algorithm was designed for both face and license plate categories. This is because license plates and faces that are close to the camera may move significantly between frames. Ordinary IoU algorithms would calculate IoU values ​​that are too small or even zero, making them unusable as a matching reference. Furthermore, since both faces and license plates are part of the human body and vehicle, the bounding boxes for faces and license plates are expanded proportionally according to the standard proportions of human faces and vehicle license plates to obtain virtual bounding box coordinates. Virtual IoU is then calculated based on these virtual bounding boxes, thereby improving the matching efficiency for both face and license plate categories.

[0089] The method for finding the optimal match using a greedy algorithm, with a flowchart of a single search as follows: Figure 2 As shown, the single search process begins; input box set M and box set N; calculate the maximum IoU (IoU) of the first box BoxM1 in box set M and the sum of all boxes in box set N, IoU_Max1; determine if IoU_Max1 is greater than a threshold. If not, end the single search process; if greater than the threshold, calculate the maximum IoU (IoU) of the box corresponding to IoU_Max1 in set N, BoxN_n, and the sum of all boxes in set M, IoU_Max2; determine if the box corresponding to IoU_Max is BoxM1. If so, output that BoxM1 in set M and BoxN_n in set N have matched successfully, and end the single search process; if the box corresponding to IoU_Max is not BoxM1, move the storage position of BoxM1 to the right, and BoxM2 becomes BoxM1, and continue repeating from calculating the maximum IoU (IoU) of the first box BoxM1 in box set M and the sum of all boxes in box set N. Specifically, the system finds the detection box with the largest IoU (Intersection over Union) between its predicted bounding box and the set of all unmatched detection boxes. This IoU is the maximum value of the IoU between the predicted bounding box and the set of all unmatched trajectory prediction boxes. If no matching detection box is found, the matching priority of the predicted bounding box is shifted to the next trajectory's predicted bounding box. If a matching trajectory is found and the IoU value is greater than the matching threshold IoU_thr_high1, the detection box is considered to have successfully matched the tracking trajectory, and the trajectory's matching status is updated to "matched".

[0090] A method for online tracking of multiple categories and multiple targets in a monitoring scenario according to an embodiment of the present invention further includes, in step S502:

[0091] First, select the high-scoring bounding box result set of class_n and match it with the predicted bounding boxes of the missing trajectories of that class in the tracker's existing trajectories. A greedy algorithm is used to find the maximum matching IoU. If the IoU value of the matched trajectory is greater than the matching threshold IoU_thr_high1, check if the matched trajectory is at the image edge. If it is at the edge, select the frame with the highest score in its historical confidence score cache list before it disappeared. Calculate the similarity score between the historical color histogram feature vector of this frame and the color histogram feature vector of the current detection box. The similarity calculation method is as follows: Cosine similarity; if the similarity score is greater than the threshold simi_thr1 or the trajectory is not on the edge, then the detection box is considered to have successfully matched the missing trajectory, and the matching status of the trajectory is updated to matched; if the matching IoU value is less than the threshold IoU_thr_match1, then all cached values ​​in the target width and height history cache list of the trajectory are reset. If the missing time of the trajectory exceeds the threshold miss_time_thr1 at the same time, then all cached values ​​in its target box coordinate history cache list and color histogram feature vector history cache list are reset.

[0092] In a monitoring scenario, the online tracking method for multiple categories and multiple targets according to an embodiment of the present invention further includes, in step S503: selecting a set of high-scoring bounding boxes of category class_n, matching them with the predicted bounding boxes of the tracker's existing trajectories of that category that have not been lost; if the IoU value of the matched trajectory is greater than the matching threshold IoU_thr_high2, then the detected bounding box is considered to have successfully matched the tracking trajectory, and the matching status of the trajectory is updated to "matched".

[0093] In a monitoring scenario, the online tracking method for multiple categories and multiple targets according to an embodiment of the present invention further includes, in step S504: selecting a set of low-scoring bounding boxes of category class_n, matching them with the predicted bounding boxes of the tracker's existing trajectories of that category that have not been lost; if the IoU value of the matched trajectory is greater than the matching threshold IoU_thr_low1, then the detected bounding box is considered to have successfully matched the tracking trajectory, and the matching status of the trajectory is updated to "matched".

[0094] In a monitoring scenario, the online tracking method for multiple categories and multiple targets according to an embodiment of the present invention further includes, in step S505: selecting a set of high-scoring bounding boxes of category class_n and matching them with the predicted bounding boxes of the missing trajectories of that category in the existing trajectory of the tracker. At this point, a dynamic matching threshold is used, which decreases linearly with the length of the missing time. If the IoU value of the matched trajectory is greater than the matching threshold IoU_thr_high2, it is checked whether the matched trajectory is at the edge of the image. If it is at the edge, the frame with the highest score in the historical cache list of its pre-missing confidence score is selected, and the similarity score between the historical color histogram feature vector of the trajectory corresponding to this frame and the current detection box color histogram feature vector is calculated. The similarity calculation method is cosine similarity. If the similarity score is greater than the threshold simi_thr2 or the trajectory is not at the edge, it is considered that this detection box and this missing trajectory have been successfully matched, and the matching status of the trajectory is updated to matched. If the matching IoU value is less than the threshold IoU_thr_match2, all cached values ​​in the historical cache list of the target width and height of the trajectory are reset. If the missing time of the trajectory exceeds the threshold miss_time_thr2 at the same time, all cached values ​​in the historical cache list of its target box coordinates and the historical cache list of its color histogram feature vector are reset.

[0095] In a monitoring scenario, the online tracking method for multiple categories and targets according to an embodiment of the present invention further includes, in step S506: selecting a set of high-scoring bounding boxes of category class_n and matching them with the remaining unmatched trajectories of the tracker that are not missing. The matching strategy is not based on IoU but on the distance dist between the center points of the detection box and the predicted box. The distance threshold dist_thr1 is set with different coefficients s_n according to different categories, and the coefficient value needs to be adjusted according to the size of the detection box within the same category. When dist is less than dist_thr1*s_n, the similarity score between the most recent color histogram feature vector cache in the trajectory history and the color histogram feature vector of the current detection box is calculated. If the similarity score is greater than the threshold simi_thr2, the detection box and the trajectory are considered to have been successfully matched.

[0096] In a monitoring scenario, a multi-category, multi-target online tracking method according to an embodiment of the present invention further includes, in step S507: selecting a matched but yet-to-be-activated trajectory from the class_n tracker, performing IoU matching with the predicted bounding boxes of all missing trajectories in the tracker, and simultaneously calculating the distance between their center points. The IoU threshold decreases linearly based on the disappearance time of the missing trajectory. If the IoU value is greater than the threshold IoU_thr_temp and the center point distance is less than the threshold dist_thr_temp, then the frame with the highest score in the historical confidence score cache list before disappearance is selected. The similarity score between the historical color histogram feature vector of this frame and the most recent color histogram feature vector cache of the trajectory to be activated is calculated. If the similarity score is greater than the threshold simi_thr1, then the trajectory to be activated is considered to have successfully matched the missing trajectory. It should be noted that this continuation processing is performed because detectors often generate false alarms near the real target. If a new trajectory is generated for this false alarm, it will cause the tracking ID to be interrupted and switched, which is detrimental to the stability of the tracker. The principle of continuation processing is that the generation of new trajectories should be treated with the utmost caution, especially new trajectories that appear immediately near the missing trajectory. If the trajectory to be activated fails to continue from the missing trajectory, and the activation time exceeds the activation time threshold `activate_thr`, then an activated new trajectory is generated.

[0097] In a monitoring scenario, the online tracking method for multiple categories and multiple targets according to an embodiment of the present invention further includes, in step S508: selecting a set of high-scoring bounding boxes of category class_n and matching them with the remaining unmatched trajectories of the tracker that are not missing. The matching strategy is based on IoMin, that is, the intersection of the predicted bounding box and the detected bounding box divided by the smaller of the two, and the formula is:

[0098]

[0099] Where A is the predicted bounding box and B is the detection bounding box. If the IoMin value of the matched trajectory is greater than the matching threshold IoU_thr_min1, then the detection bounding box is considered to have successfully matched the tracking trajectory.

[0100] In a monitoring scenario, the online tracking method for multiple categories and multiple targets according to an embodiment of the present invention further includes, in step S509: selecting a set of high-scoring bounding boxes of category class_n and matching them with the missing trajectories in the remaining unmatched trajectories of the tracker. The matching strategy is based on IoMin. If the IoMin value of the matched trajectory is greater than the matching threshold IoU_thr_min2, then the detection box and the tracking trajectory are considered to have been successfully matched.

[0101] In a monitoring scenario, the online tracking method for multiple categories and targets according to an embodiment of the present invention further includes, in step S510: selecting a set of high-scoring bounding boxes of category class_n, matching them with the missing trajectories among the remaining unmatched trajectories of the tracker, and calculating the distance between their center points. The calculation of the distance threshold requires two coefficients: first, different coefficients s_n are set according to different categories, and within the same category, the coefficient value needs to be adjusted according to the size of the detection box; second, a coefficient v_n is set according to the trajectory movement speed, with a larger v_n value for faster movement speed. When the distance is less than the missing time undetected_time*v_n*s_n, the frame with the highest score in the historical cache list of its pre-missing confidence score is selected, and the similarity score between the historical color histogram feature vector of this frame and the current detection box color histogram feature vector is calculated. If the similarity score is greater than the threshold simi_thr1, then the trajectory to be activated is considered to have successfully matched the missing trajectory, and the cache values ​​in all its historical cache lists are reset.

[0102] In a monitoring scenario, the online tracking method for multiple categories and multiple targets according to an embodiment of the present invention further includes, in step S60, processing the remaining unmatched detection boxes and the remaining unmatched trajectories for adding and deleting trajectories, which includes: processing the remaining unmatched detection boxes, and when their confidence score is higher than the threshold score_thr_new, and their width and height are greater than the minimum target width threshold w_thr_new and the high threshold h_thr_new of the category, then a new trajectory is established, and its status is marked as inactive.

[0103] Process the remaining unmatched trajectories, incrementing their unmatch count by one. If the unmatch count exceeds the maximum undetected time threshold max_undetected_time, delete the trajectory.

[0104] In a multi-category, multi-target online tracking method for monitoring scenarios according to an embodiment of the present invention, further, in S70, updating the state information of all trajectories includes: for a matched trajectory, updating its current coordinate value and confidence score to the coordinate value and confidence score of the matched detection box, and setting the trajectory mismatch count undetected_time to 0. For all trajectories, checking whether the current coordinate value is on the edge of the input image, updating the target's edge status is_edge; recording the confidence score to the historical confidence score cache list, recording the coordinates of the upper left, center point, and upper right of the bounding box to the historical coordinate cache list, and recording the color histogram feature vector to the historical color histogram feature vector list.

[0105] In a multi-category, multi-target online tracking method for monitoring scenarios according to an embodiment of the present invention, further, in S70, the calculation of the most recent stable width and height values, stable corner points, and moving speed vectors within the first time period, and the final calculation of the trajectory prediction box coordinates, includes: calculating virtual width and height values ​​based on the most recent frame width and height values ​​in the trajectory historical width and height cache list, combined with the current matching detection box width and height, and storing them in the historical cache list. The change ratio of the virtual width and height relative to the most recent frame historical width and height must not exceed the threshold wh_range_thr. Then, the average value of the historical width and height is calculated based on the historical width and height cache list as the recent stable width and height value of the target. Three different target moving speed vectors are calculated based on the historical coordinate cache lists for the upper left, center point, and upper right coordinates. The calculation method is a linear fit, i.e., based on the prior assumption that the target's moving trajectory within a short time window can be simplified to a linear path. Simultaneously, the historical mean square errors of the three linear fits are calculated. The one with the smallest historical mean square error is selected as the recent stable coordinate point of the target, and the target moving speed vector corresponding to this point is used as the trajectory's moving vector. The predicted box width and height values ​​are calculated based on the historical width and height cache list, also using a linear fit.

[0106] Update the predicted bounding box coordinates and the recently stable predicted bounding box coordinates of the trajectory. The predicted bounding box coordinates of the trajectory are obtained by adding the trajectory's movement vector to the coordinates of the matched detection box. For a matched trajectory, calculate the horizontal and vertical offsets of the top, bottom, left, and right borders of the previous trajectory predicted bounding box and the current matched detection box, and select the corner point with the smallest horizontal and vertical offset values ​​as the stable point of the trajectory bounding box. Update the recently stable predicted bounding box coordinates of the trajectory based on the coordinates of this point plus the target's recent stable width and height values.

[0107] For unmatched missing trajectories, when the missing duration undetected_time is less than the threshold stable_undetected_time, the current bounding box coordinates of the trajectory are updated to the predicted bounding box coordinates of the previous frame. In addition, the current bounding box coordinates of the trajectory are updated to the recently stable predicted bounding box coordinates of the previous frame.

[0108] In a monitoring scenario, the online tracking method for multiple categories and multiple targets according to an embodiment of the present invention further includes, in step S80, using the Hungarian algorithm to match face targets and human targets in all trajectories. After matching, a mapping table of face ID, human ID, and fused ID is established. The face and human target IDs are fused and bound to obtain the fused ID. The matching of face targets and human targets in all trajectories using the Hungarian algorithm includes: the cost matrix of the Hungarian matching algorithm utilizes the IoU between face and human targets and their relative positions. After calculating the IoU matrix, the distance dist1 between the midpoint of the upper border of the face target and the midpoint of the upper border of the human target is calculated and divided by the height p_h of the human target as a penalty coefficient. The final cost matrix Cost is calculated as follows:

[0109]

[0110] Set the Hungarian matching threshold Hung_thr1, and output the human body and face matching list and unmatched list with the minimum overall cost through the algorithm.

[0111] The process of merging and binding IDs for face and human targets involves: first, establishing a mapping table between human IDs, face IDs, and fusion IDs. For a new human ID appearing for the first time, i.e., checking whether the human body is in the mapping table, the matching face ID from the previous Hungarian matching step is checked. If a matching face exists and that face has an existing corresponding human body, the fusion ID corresponding to that face is used for binding, and a new fusion ID is created to bind the existing human body. Otherwise, a new fusion ID is created and bound to the current human ID.

[0112] If the matched face does not exist or the matched face ID is a new ID, a new fusion ID is created and bound to the human face ID. If the matched face exists and the face is in the relationship table, the fusion ID corresponding to the face is directly bound to the human ID. If the face is not in the relationship table, a new fusion ID is created and bound.

[0113] For a human ID that is not appearing for the first time, check if there is a matching face for that human. If no matching face is found, if a new human is currently matched to the human's old corresponding face, create a new fusion ID and bind it; otherwise, keep the original fusion ID unchanged. If a matching face is found, check if that face is in the relationship table. If the face is not in the relationship table (i.e., it is a newly appearing face or its trajectory status is missing), if the human has no old corresponding face, the matching face is directly bound to the human's fusion ID; otherwise, create a new fusion ID and bind it. If the face is in the relationship table and its trajectory status is not missing, the human and face fusion IDs are bound together. Furthermore, if the face has an old corresponding human, and if the human also has an old corresponding face and that face already has a new matching human, create a new fusion ID and bind it to the old corresponding human; otherwise, bind the old corresponding face fusion ID to the old corresponding human.

[0114] It's important to note that fusing and binding face and human target IDs is primarily due to two reasons: First, individual face tracking can lead to lost or interrupted trajectories due to face turning or turning away, resulting in duplicate trajectories. In this case, human tracking generally performs normally. Second, in densely populated scenes, human detectors can cause numerous false alarms or missed detections due to excessive density. Individual human tracking trajectories often struggle to match and continue correctly, frequently resulting in trajectory breaks. In such situations, faces are less obstructed, leading to more stable tracking trajectories. Therefore, fusing and overlaying human and face tracking information better maintains the tracking stability of both targets.

[0115] The online tracking method for multiple categories and multiple targets in the monitoring scenario set above has at least the following beneficial effects: (1) The IoU calculation method is optimized according to the characteristics of different categories of targets, and the accuracy of matching categories such as face and license plate is improved by using virtual IoU calculation. (2) A fast target histogram color feature information extraction method is designed, which can integrate three-channel color features instead of the traditional single-channel histogram, and significantly improve the feature vector calculation speed through color compression and table lookup method, realizing ultra-lightweight color feature extraction. (3) A cascaded matching strategy with up to 10 levels is designed. The disappearing target re-matching strategy is designed based on color feature similarity comparison. Each level uses different matching priorities and dynamically designed thresholds, and weight factors are designed according to the motion characteristics of different categories of targets, minimizing the situation of target ID mismatch and trajectory breakage. (4) The predicted box stabilization algorithm is designed. The predicted motion vector and predicted box position are calculated through historical data of target trajectory, which increases the success rate of trajectory matching and reduces the impact of drastic changes in target position and size on tracking. (5) The designed face and human target ID binding algorithm can better utilize the tracking characteristics of the two targets, complement each other's tracking trajectory information, and improve the tracking stability of the two targets.

[0116] It should be understood that the exemplary embodiments described herein are illustrative and not restrictive. Although one or more embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A method for online tracking of multiple categories and multiple targets in a monitoring scenario, characterized in that, Includes the following steps: S10, Use the object detector to perform object detection on the current frame of the video image, and obtain the detection box coordinates, category and confidence score of all categories of objects; S20, use the fast color histogram feature vector extraction algorithm to obtain the color feature vector of the target; S30: Based on the target confidence score, the target detection box is divided into high-scoring boxes and low-scoring boxes, and different matching priorities and matching strategies are set for high-scoring boxes and low-scoring boxes; S40, a matching method based on intersection-union ratio (IoU) and greedy algorithm is designed, and a virtual IoU algorithm is designed for two categories: face and license plate. S50 matches all detected target boxes with all trajectories in the tracker, employs a multi-level cascading strategy for multiple rounds of matching, and includes a re-matching strategy for disappearing targets; S60, Processing the final remaining unmatched detection boxes, processing the final remaining unmatched trajectories, used for adding and deleting trajectories; S70, update the status information of all trajectories, calculate the stable width and height values, stable corner points, and moving speed vector in the most recent first time, and finally calculate the coordinates of the trajectory prediction box; S80 uses the Hungarian algorithm to match face targets and human targets in all trajectories. After the matching is completed, a mapping table of face ID, human ID and fusion ID is established. The IDs of face and human targets are fused and bound to obtain the fusion ID. S50 matches all detected target boxes with all trajectories in the tracker, employing a multi-level cascading strategy for multiple rounds of matching, including a re-matching strategy for disappearing targets. Specifically, this includes: S501, perform the first round of the first batch of matching, select the high-scoring bounding box result set and match it with the predicted bounding boxes of the tracker's existing trajectories of this category that have not been lost; S502, perform the first round of second batch matching, select the high-scoring box results and match them with the predicted boxes of the missing trajectories of this category in the tracker, and calculate the similarity score between the trajectory historical color histogram feature vector and the current detection box color histogram feature vector; S503, perform the second round of the first batch of matching, select the high-scoring bounding box result set and match it with the predicted bounding boxes of the tracker for the trajectories of this category that have not been lost. The IoU matching threshold is significantly smaller than that of the first round of the first batch of matching. S504, perform the second round of the second batch of matching, select the low-scoring bounding box result set and match it with the predicted bounding boxes of the tracker for the trajectories of this category that have not been lost; S505, perform the second round of the third batch of matching, select the high-scoring box results and match them with the predicted boxes of the missing trajectories of this category in the tracker. At this time, a dynamic matching threshold is used, and the threshold decreases linearly with the length of the missing time. S506, the third round of matching is performed, which is for fast-moving targets. The high-scoring bounding box results are selected and matched with the remaining unmatched trajectories of the tracker that have not been lost. The matching strategy is based on the distance between the center points of the detection box and the prediction box, and the color feature vector similarity is calculated at the same time. S507, perform the fourth round of matching, and continue the processing of the trajectory to be activated and the missing trajectory. Select the trajectory that has been matched but is to be activated in the tracker, and perform IoU matching with the prediction boxes of all missing trajectories in the tracker. At the same time, calculate the distance between the center points of the two. S508 performs the fifth round of the first batch of matching. In cases where the detection box is significantly reduced due to occlusion, the high-scoring box results are selected and matched with the remaining unmatched trajectories of the tracker that have not been lost. The matching strategy is based on the ratio of the intersection to the minimum (IoMin). S509, perform the fifth round of the second batch of matching, select the high-scoring bounding box results and perform IoMin matching with the missing trajector in the remaining unmatched trajector ... S510, perform the sixth round of matching, select the high-scoring box results and match the missing trajectories in the remaining unmatched trajectories of the tracker with the center point distance, and calculate the color feature vector similarity; In S80, the Hungarian algorithm is used to match face and human targets in all trajectories. The Hungarian matching algorithm cost matrix uses the IoU of face and human targets and their relative positions. After calculating the IoU matrix, the distance dist1 between the midpoint of the upper bounding box of the face target and the midpoint of the upper bounding box of the human target is calculated and divided by the height p_h of the human target as a penalty coefficient. The final cost matrix Cost is then calculated.

2. The online tracking method for multiple categories and multiple targets in a monitoring scenario as described in claim 1, characterized in that, The target categories include pedestrians, motor vehicles, non-motorized vehicles, faces, and license plates.

3. The online tracking method for multiple categories and multiple targets in a monitoring scenario as described in claim 1, characterized in that, S20, the color feature vector of the target is obtained by using the fast color histogram feature vector extraction algorithm, including: expanding the target detection box inward according to the target detection box coordinates, expanding the four borders of the detection box inward by a certain percentage, and selecting the expanded box as the main part of the target for color histogram calculation.

4. The online tracking method for multiple categories and multiple targets in a monitoring scenario as described in claim 1, characterized in that, S20, using a fast color histogram feature vector extraction algorithm to obtain the target's color feature vector includes color space compression. For a 3-channel 24-bit BGR original image, 16,777,216 colors are compressed into a 1331-dimensional feature vector. Specific steps include: S201, map the 8-bit pixel value range of 0-255 of each channel to the simplified range of 0-10, where each value in the range of 0-9 corresponds to the original pixel value range of 24, and the value in the range of 10 corresponds to the original pixel value range of 16; S202, each channel corresponds to a different weight coefficient, where channel B corresponds to 121, channel G corresponds to 11, and channel R corresponds to 1; S203, To accelerate calculation, a mapping table between pixel values ​​and simplified region values ​​is established. The value after dividing the pixel value by 8 corresponds to the index position subscript value of the mapping table. Each value in the mapping table is the corresponding simplified interval value multiplied by the channel weight coefficient. S204: Pixels within the target area are counted at each stride using a lookup table. The sum of the values ​​obtained from the lookup table for the three color channels of each pixel is the feature vector dimension corresponding to that pixel value. After the feature vector of the color histogram distribution is counted, the coordinate value of each dimension of the vector represents the number of pixel values ​​in that interval within the target area.

5. The online tracking method for multiple categories and multiple targets in a monitoring scenario as described in claim 4, characterized in that, The stride is calculated as follows: Sampling is performed using alternating rows and columns, and the stride is set according to the width and height of the frame. w and h represent the width and height of the frame, respectively. The stride is calculated as follows: 。 6. The online tracking method for multiple categories and multiple targets in a monitoring scenario as described in claim 1, characterized in that, S501 includes: For a target class_n, the set of high-scoring bounding boxes for that class is selected and matched with the predicted bounding boxes of the tracker's existing trajectories that are not missing. The matching strategy is a greedy algorithm based on IoU, where IoU is the intersection of the predicted bounding box and the detection bounding box divided by their union. The formula is as follows: Where A is the predicted bounding box and B is the detected bounding box; If a matching trajectory can be found and the IoU value is greater than the matching threshold IoU_thr_high1, then the detection box is considered to have successfully matched the tracking trajectory, and the matching status of the trajectory is updated to "matched".

7. The online tracking method for multiple categories and multiple targets in a monitoring scenario as described in claim 1, characterized in that, S502 includes: First, select the high-scoring bounding box result set of class_n and match it with the predicted bounding boxes of the missing trajectories of that class in the tracker's existing trajectories. A greedy algorithm is used to find the maximum matching IoU. If the IoU value of the matched trajectory is greater than the matching threshold IoU_thr_high1, check if the matched trajectory is at the image edge. If it is at the edge, select the frame with the highest score in its historical confidence score cache list before it disappeared. Calculate the similarity score between the historical color histogram feature vector of this frame and the color histogram feature vector of the current detection box. The similarity calculation method is as follows: Cosine similarity; if the similarity score is greater than the threshold simi_thr1 or the trajectory is not on the edge, then the detection box is considered to have successfully matched the missing trajectory, and the matching status of the trajectory is updated to matched; if the matching IoU value is less than the threshold IoU_thr_match1, then all cached values ​​in the target width and height history cache list of the trajectory are reset. If the missing time of the trajectory exceeds the threshold miss_time_thr1 at the same time, then all cached values ​​in its target box coordinate history cache list and color histogram feature vector history cache list are reset.

8. The online tracking method for multiple categories and multiple targets in a monitoring scenario as described in claim 1, characterized in that, S503 includes: selecting the high-scoring bounding box result set of class_n, matching it with the predicted bounding boxes of the tracker's existing trajectories that are not missing in that class, and if the IoU value of the matched trajectory is greater than the matching threshold IoU_thr_high2, then the detected bounding box is considered to have successfully matched the tracked trajectory, and the matching status of the trajectory is updated to matched.

9. The online tracking method for multiple categories and multiple targets in a monitoring scenario as described in claim 1, characterized in that, S504 includes: selecting the low-scoring bounding box result set of class_n, matching it with the predicted bounding boxes of the tracker's existing trajectories that are not missing in that class, and if the IoU value of the matched trajectory is greater than the matching threshold IoU_thr_low1, then the detected bounding box is considered to have successfully matched the tracked trajectory, and the matching status of the trajectory is updated to matched.

10. The online tracking method for multiple categories and multiple targets in a monitoring scenario as described in claim 1, characterized in that, S505 includes: selecting a set of high-scoring bounding boxes for class_n and matching them with the predicted bounding boxes of missing trajectories of that class in the tracker's existing trajectories; at this time, a dynamic matching threshold is used, which decreases linearly with the length of the missing time. If the IoU value of the matched trajectory is greater than the matching threshold IoU_thr_high2, it is checked whether the matched trajectory is at the edge of the image. If it is at the edge, the frame with the highest score in the historical cache list of its pre-missing confidence score is selected, and the similarity score between the historical color histogram feature vector of the trajectory corresponding to this frame and the color histogram feature vector of the current detection box is calculated. The similarity calculation method is cosine similarity. If the similarity score is greater than the threshold simi_thr2 or the trajectory is not on the edge, the detection box is considered to have successfully matched the missing trajectory, and the matching status of the trajectory is updated to matched. If the matching IoU value is less than the threshold IoU_thr_match2, all cached values ​​in the target width and height history cache list of the trajectory are reset. If the missing time of the trajectory exceeds the threshold miss_time_thr2, all cached values ​​in the target box coordinate history cache list and the color histogram feature vector history cache list are reset.

11. The online tracking method for multiple categories and multiple targets in a monitoring scenario as described in claim 1, characterized in that, S506 includes: selecting the high-scoring bounding box result set of class_n, matching it with the remaining unmatched trajectories of the tracker that are not missing, and the matching strategy is not based on IoU but on the distance dist between the center points of the detection box and the predicted box; the distance threshold dist_thr1 is set with different coefficients s_n according to different classes, and the coefficient value needs to be adjusted according to the size of the detection box within the same class. When dist is less than dist_thr1*s_n, the similarity score between the most recent color histogram feature vector cache in the trajectory history and the color histogram feature vector of the current detection box is calculated. If the similarity score is greater than the threshold simi_thr2, the detection box and the trajectory are considered to have been successfully matched.

12. The online tracking method for multiple categories and multiple targets in a monitoring scenario as described in claim 1, characterized in that, S507 includes: selecting a matched but unactivated trajectory from the class_n tracker, performing IoU matching with the predicted bounding boxes of all missing trajectories in the tracker, and calculating the distance between their center points; wherein the IoU threshold decreases linearly according to the disappearance time of the missing trajectory; if the IoU value is greater than the threshold IoU_thr_temp and the center point distance is less than the threshold dist_thr_temp, then the frame with the highest score in the historical cache list of its pre-disappearance confidence score is selected, and the similarity score between the historical color histogram feature vector of the trajectory corresponding to this frame and the most recent color histogram feature vector cache of the trajectory to be activated is calculated; if the similarity score is greater than the threshold simi_thr1, then the trajectory to be activated is considered to have successfully matched this missing trajectory; if the trajectory to be activated fails to continue with the missing trajectory, and the activation time exceeds the activation time threshold activate_thr, then a new activated trajectory is generated.

13. The online tracking method for multiple categories and multiple targets in a monitoring scenario as described in claim 1, characterized in that, S508 includes: selecting the set of high-scoring bounding boxes of class_n, and matching them with the remaining unmatched tracks from the tracker's remaining unmatched tracks. The matching strategy is based on IoMin, which is the intersection of the predicted bounding box and the detected bounding box divided by the smaller of the two. The formula is as follows: If the IoMin value of the matched trajectory is greater than the matching threshold IoU_thr_min1, then the detection box is considered to have successfully matched the tracking trajectory.

14. The online tracking method for multiple categories and multiple targets in a monitoring scenario as described in claim 1, characterized in that, S509 includes: selecting the set of high-scoring bounding boxes of class_n and matching them with the missing tracks in the remaining unmatched tracks of the tracker. The matching strategy is based on IoMin. If the IoMin value of the matched track is greater than the matching threshold IoU_thr_min2, then the detection box and the tracked track are considered to have been successfully matched.

15. The online tracking method for multiple categories and multiple targets in a monitoring scenario as described in claim 1, characterized in that, S510 includes: selecting the high-scoring bounding box result set of class_n, matching it with the missing trajectory in the remaining unmatched trajectories of the tracker, and calculating the distance between the center points of the two; the calculation of the distance threshold requires two coefficients, one is to set different coefficients s_n according to different categories, and the coefficient value needs to be adjusted according to the detection box size in the same category; the other is to set the coefficient v_n according to the trajectory movement speed, the faster the movement speed, the larger the v_n value. When the distance is less than the missing time undetected_time * v_n * s_n, the frame with the highest score in the historical cache list of its pre-missing confidence score is selected, and the similarity score between the historical color histogram feature vector of the trajectory corresponding to this frame and the current detection box color histogram feature vector is calculated. If the similarity score is greater than the threshold simi_thr1, it is considered that this trajectory to be activated has successfully matched this missing trajectory, and the cache values ​​in all its historical cache lists are reset.

Citation Information

Patent Citations

  • Multi-target tracking method, system and device for joint measurement of center point offset and GIoU distance, and medium

    CN115937260A

  • Multi-target tracking method in dense scene

    CN116883452A