Target detection label smoothing method based on historical information

Abstract

The invention discloses a target detection label smoothing method based on historical information. The method comprises the following steps: inputting a video frame sequence into a trained detector to obtain a detection result; inputting a detection result into a tracker, and performing Kalman filtering prediction on each result trajectory to obtain a unique ID and a tracking result of each target; by utilizing the uniqueness of the track ID and the label information of the detection result, different weights are distributed to the historical frame label and the current frame label, and the label information of the current frame is obtained; and correcting the label of the detection result according to the obtained label information, and outputting a final stable detection label.

Description

Technical Field

[0001] The invention belongs to the technical field of target detection and tracking, and in particular relates to a method for target detection label smoothing based on historical information. Background Art

[0002] With the rapid development of artificial intelligence technology, intelligent monitoring systems and autonomous driving based on computer vision technology have made significant breakthroughs compared with the past, further reducing the waste of human resources and improving the safety of security and transportation. As a key basic technology in these fields, the performance of object detection directly affects the reliability and effectiveness of the system.

[0003] In practical object detection applications, especially when processing dynamic video streams, the classification results of some objects may suddenly change due to motion distortion, occlusion of moving objects, or fluctuations in the detector itself. For example, an object is detected as the same category in consecutive frames, but due to factors such as motion distortion, it is incorrectly classified as another category in a certain frame. Such label mutations may seriously affect the overall stability of the detection system, and thus have a negative impact on subsequent decisions. Therefore, it is necessary to study the smoothing of object detection labels.

[0004] The main disadvantage of traditional label smoothing technology is that it cannot effectively deal with label changes of dynamic targets, especially when the target is distorted, occluded or changes rapidly. Traditional methods usually assume that the target category label is relatively stable in the time series. Therefore, for fast-moving or occluded targets, the label may change suddenly or be inconsistent, resulting in tracking failure or misidentification. In addition, traditional label smoothing methods lack dynamic analysis of the target's motion trajectory and cannot make adaptive adjustments based on the target's historical state. Therefore, it is easy to cause miscorrection or loss of labels in complex scenarios, affecting the overall tracking performance and accuracy. Summary of the invention

[0005] Purpose of the invention: To solve the above problems, the present invention proposes a method for target detection label smoothing based on historical information. By utilizing a tracking algorithm, the detection result can be effectively associated with the trajectory information to generate a unique ID for the detected target. The unique ID of the target generated by the tracking algorithm can be effectively recorded with the historical label information of the current detected target. By weightedly fusing the historical label information of the target with the current detection label information, stable label information can be effectively output, thereby reducing the problem of frequent label jumps in target detection caused by factors such as mutual occlusion of targets and motion blur.

[0006] Technical solution, in order to achieve the above purpose, the present invention proposes a method for target detection label smoothing based on historical information, the method comprising the following steps:

[0007] S1. Take an image or video frame as input, and take the category, confidence, and target box coordinates of the selected target in the image as output to train the model and obtain a detector;

[0008] S2. Input the video frame sequence into the detector to obtain the target detection result, which includes the coordinates, categories and confidence scores of all detected target boxes in the image;

[0009] S3, input the detection result into the target tracker, determine whether the current input is the first frame of the video sequence, if so, create a track list, and generate the initial track of the target based on the target frame coordinates, category, confidence and other information, and store all tracks in the track list. If not, use Kalman filtering to predict all tracks in the track list, obtain the predicted position of the track, and update the track list after the correlation match with the detection result;

[0010] S4, performing data association matching on the detection result and the predicted position of all trajectories in the trajectory list, and obtaining the tracking status and updated trajectory information of each target by using the uniqueness of the trajectory ID;

[0011] S5. For each target in the tracking state, determine whether it is the first time to appear according to the uniqueness of the trajectory ID. If the ID appears for the first time, create a historical category label probability matrix for it based on the currently detected category label and record its category label probability. If the ID does not appear for the first time, use the weighted fusion method to obtain the smooth label of the target for the category label detected in the current frame and the historical category label probability matrix of the trajectory;

[0012] S6. For targets that appear for the first time, directly output their current detection category labels. For targets that do not appear for the first time, after obtaining the smoothed labels, assign them to the target box in the current frame detection result, and output stable detection labels.

[0013] In particular, the detector used in S1 can be a current mainstream target detection algorithm, such as YOLOv10, YOLC, etc., which is a model obtained after training with a general data set or a specific application data set. As long as the detection output of the model can contain the content required by step S2 (such as the category and confidence score of the target), even if these outputs are not originally provided, they can be achieved by modifying the relevant output parts in the detector source code.

[0014] Further, the S3 includes:

[0015] If the input is the first frame, the initial target trajectory is generated according to the target box coordinates, category, confidence and other information, including initializing the coordinates of all trajectories to the detection result coordinates, and initializing the mean and variance of the Kalman filter according to the coordinates. At the same time, it is marked as a tracking state according to the detection box result, and a unique ID is assigned for subsequent matching.

[0016] Further, the S4 includes:

[0017] Step 1: Set a confidence threshold track_thresh. After obtaining the detection frame and confidence information of the target through the detector, judge each detected target. If its confidence is greater than track_thresh, its detection frame is judged as a high-scoring frame. If the confidence is less than track_thresh but higher than 0.1, it is judged as a low-scoring frame. If the confidence is lower than 0.1, it is considered to be a misjudgment by the detector and is directly discarded. The track_thresh is greater than 0.1.

[0018] Step 2: For the high-score frame, calculate the IOU of the high-score frame and the predicted frame, and use the Hungarian algorithm to match the IOU. If the high-score frame matches the predicted frame, update the frame in the tracking trajectory to the high-score frame and assign an ID. If the high-score frame does not match the predicted frame, execute steps 3 and 4.

[0019] Step 3: For the prediction box that fails to match the high-resolution box, if the prediction box is not in the tracking state, it is marked as lost. When the maximum cache frame number exceeds 25 frames, the prediction box information is deleted; if it is in the tracking state, the IOU with the low-resolution box obtained in step 1 is calculated, and the Hungarian algorithm is used to match the IOU. If the match is successful, the box in the tracking trajectory is updated to the low-resolution box and an ID is assigned; if the match fails, the prediction box that fails to match is directly discarded; for the low-resolution box that fails to match, if it fails to match all the prediction boxes that failed to match in step 2, the low-resolution box is discarded;

[0020] Step 4: For the high-score frame that fails to match the predicted frame, match it with the inactive predicted frame. If the match is successful, activate the predicted frame state, mark it as the tracking state, and update the frame to the high-score frame at this time; if the match fails, mark the unmatched predicted frame for deletion. For the high-score frame that has not been matched yet, if the confidence is greater than track_thresh+0.1, create a new tracking track, otherwise discard it.

[0021] Further, the S5 includes:

[0022] According to the number of detection categories k, initialize the historical category label probability matrix His:

[0023]

[0024] Among them, x i Represents the probability of detecting category i. In each frame, the ID of the target successfully tracked by the tracker is recorded, and for the target with the same ID, a current category label probability matrix Cur is generated by its current detection category j, j∈[0,k-1]:

[0025]

[0026] Different weights are assigned to the current category label probability matrix and the historical category label probability matrix. The formula is as follows:

[0027] s=αCur+(1-α)His

[0028] Among them, s is the smoothed category label probability matrix, α represents the weight, and the smaller the value, the more importance is attached to the historical label information.

[0029] The smoothed label L is the category label with the highest probability in the smoothed category label probability matrix. The specific formula is as follows:

[0030] L = max(P Class ), P Class ∈s.

[0031] In particular, in S6, the difference between the probability matrix of the current detection label and the probability matrix of the historical tracking label is that the probability matrix of the current target detection label must be a sparse table, that is, only the probability of the current detection category is α, and the rest are 0, while the probability matrix of the historical tracking label will cause the probabilities of multiple category labels to be non-zero due to the influence of label jumps, and it is a sparse table only under ideal conditions.

[0032] In addition, the present invention proposes a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the methods described above when executing the computer program.

[0033] In addition, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any one of the methods described above is implemented.

[0034] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0035] (1) The present invention utilizes a tracking algorithm to break through the limitation that traditional label smoothing cannot be adaptively adjusted according to the historical state of the target, and effectively integrates detection information and trajectory information, so that the target can still output relatively stable labels in challenging scenarios such as mutual occlusion and motion blur, and has good robustness.

[0036] (2) The present invention obtains smoother label information by weighting and fusing the historical label information of the same target with the current detection information using different weights, thereby effectively suppressing the problem of label mutation in a single frame or a few frames. As time goes by, the probability of mutation will become lower and lower.

[0037] (3) The detector used in the present invention can be a training model of any mainstream algorithm on a general data set or a specific data set, and the tracking algorithm can also be any mainstream tracking algorithm. As long as the two are compatible and matchable, it has a wide range of application value.

[0038] (4) The present invention can effectively improve the stability and accuracy of target detection labels. When the targets are close or occluded, the detector output fluctuations may cause the label to change frequently. The present invention combines historical information with current detection results to smooth and correct the label, effectively avoiding the phenomenon of frequent label changes. Through a simple and efficient method, target tracking and label output are more stable and robust. The present invention can be combined with any mainstream target tracker that generates a unique trajectory ID, and has strong adaptability and wide practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments are briefly introduced below.

[0040] Figure 1 A flowchart of a label smoothing method based on historical information implemented in the present invention;

[0041] Figure 2 Flow chart of the tracking algorithm adopted by the present invention. DETAILED DESCRIPTION

[0042] In order to more clearly illustrate the technical solution of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0043] In actual object detection applications, the classification results of some objects may suddenly change due to occlusion of moving objects or fluctuations of the detector itself. Frequent label mutations may seriously affect the overall stability of the detection system, and thus have a negative impact on subsequent decisions.

[0044] Based on the above ideas, the present invention provides a method for smoothing target detection labels based on historical information. By using a tracking algorithm, the current detection label information of the target is merged with the historical label information to generate smoother labels, effectively reducing the frequent label jumps caused by challenges such as motion occlusion and blur, and outputting more stable labels.

[0045] like Figure 1 As shown, the present invention provides a target detection label smoothing method based on historical information, which specifically includes the following steps:

[0046] Step 1: Use a model that takes an image or video frame as input and outputs the category, confidence, and target box coordinates of the selected target in the image for training as a detector. In a preferred embodiment, the detector is YoloV8, and is trained on drone and bird datasets using official pre-trained weights to obtain the detector of the present invention.

[0047] Step 2: Input the video sequence to be detected into the detector in the form of frame images to obtain the output results of the detector, including target location, confidence and category label information;

[0048] In particular, if the output of a detector does not contain the confidence score and category label information of the target, it can be obtained by modifying its source code.

[0049] In a preferred embodiment, the detector outputs each detection target as a 6-dimensional vector (x1, y1, x2, y2, conf, cls), where the first four values ​​of the vector are the coordinates of the upper left and lower right corners of the detection box, conf is the confidence score, ranging from 0 to 1, and cls is the classification result, which is an integer variable.

[0050] Step 3: If the output image is the first frame, the initial trajectory of the target is generated according to the target frame coordinates, category, confidence and other information, that is, the coordinates of all trajectories are initialized to the detection result coordinates, and the mean and variance of the Kalman filter are initialized according to the coordinates. At the same time, it is marked as a tracking state according to the detection frame result, and a unique ID is assigned for subsequent matching.

[0051] It should be noted that the detector designed by the present invention is not limited to YoloV8. It only needs to output information that can satisfy the tracking algorithm and contain label category information. The tracking algorithm only needs to be compatible with the detector and meet the setting of unique ID generation.

[0052] Step 4: Tracker Follow Figure 2 The process shown associates the detection results with the data, obtains the unique ID of the target, and then initializes the historical category label probability matrix according to the number of categories.

[0053] In a preferred embodiment, the target categories are birds and drones, 0 represents birds, and 1 represents drones. For the first appearance of the target ID, the initialized historical category label probability matrix is ​​[0,0], and the corresponding category probability is set to 1 according to the current detection frame detection label.

[0054] In a preferred embodiment, the ID of the tracked target is 1, the first appearance, and the current detection category is 0, indicating a bird, so the final historical category label probability matrix is ​​initialized to [1,0].

[0055] Step 5: Perform weighted fusion of the detection category label probability matrix of the current frame and the historical category label probability matrix. The specific formula is as follows:

[0056] s=αCur+(1-α)His

[0057] Among them, s is the smoothed category label probability matrix, Cur is the current category label probability matrix generated by the current detection category, His is the historical category label probability matrix, and α represents the weight. The smaller the value, the more importance is attached to the historical label information.

[0058] Finally, the smoothed label L is the category label with the highest probability in the smoothed category label probability matrix. The specific formula is as follows:

[0059] L = max(P Class ), P Class ∈s.

[0060] In a preferred embodiment, 0 represents birds and 1 represents drones. The ID of the tracked target is 1. When it appears for the first time, its current detection category is 0, and its generated Cur is [1,0]. Since it appears for the first time, its His is also [1,0]. Therefore, its smoothed label information probability matrix s is [1,0]. The final smoothed label L is the category index with the highest probability, that is, the index 0 where the probability is 1, indicating a bird.

[0061] In a preferred embodiment, when the tracking ID is 1 and appears for the second time, the detection label at this time is 1, then the generated Cur is [0,1], and its His is [1,0], and the value of α is 0.25, so the smoothed label information probability matrix s is [0.75,0.25]. The final smoothed label L is the category index with the highest probability, that is, the index 0 at the position with a probability of 0.75, which still represents a bird.

[0062] Step 6, according to the final smoothed label L, read the prior category information and output the final detection label. In a preferred embodiment, 0 represents a bird and 1 represents a drone. For the target with ID 1, the first output label is a bird, and the second first detection label is a drone. However, after label smoothing, it is still a bird.

[0063] It should be noted that the main problem to be solved here is the frequent jumps in label categories caused by factors such as occlusion and blur of moving targets, rather than the false alarms that the detector has always had.

[0064] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for target detection label smoothing based on historical information, characterized in that: The following steps are involved: S1. Take an image or video frame as input, and use the category, confidence, and target box coordinates of the selected target in the image as output to train the model to obtain a detector; S2. Input the video frame sequence into the detector to obtain the target detection result, which includes the coordinates, categories and confidence scores of all detected target boxes in the image; S3, input the detection result into the target tracker, determine whether the current input is the first frame of the video sequence, if so, create a track list, and generate the initial track of the target according to the target frame coordinates, category, and confidence information, and store all tracks in the track list; if not, use Kalman filtering to predict all tracks in the track list, obtain the predicted position of the track, and update the track list after the association match with the detection result; S4, performing data association matching on the detection result and the predicted position of all trajectories in the trajectory list, and obtaining the tracking status and updated trajectory information of each target by using the uniqueness of the trajectory ID; S5. For each target in the tracking state, determine whether it is the first appearance of the target based on the uniqueness of the trajectory ID. If the ID is the first appearance, create a historical category label probability matrix for it based on the currently detected category label and record its category label probability; if the ID is not the first appearance, use a weighted fusion method for the category label detected in the current frame and the historical category label probability matrix of the trajectory to obtain a smooth label for the target; S6. For targets that appear for the first time, directly output their current detection category labels. For targets that do not appear for the first time, obtain the smoothed labels and assign them to the target boxes in the current frame detection results, and output stable detection labels.

2. The method for target detection label smoothing based on historical information according to claim 1, characterized in that: The S3 initial trajectory includes initializing the coordinates of all trajectories to the detection result coordinates, initializing the mean and variance of the Kalman filter based on the coordinates, marking it as a tracking state based on the detection frame result, and assigning a unique ID for matching.

3. The method for target detection label smoothing based on historical information according to claim 1, characterized in that: The S4 step includes: Step 1: Set a confidence threshold track_thresh. After obtaining the detection frame and confidence information of the target through the detector, judge each detected target. If its confidence is greater than track_thresh, its detection frame is judged as a high-scoring frame. If the confidence is less than track_thresh but higher than 0.1, it is judged as a low-scoring frame. If the confidence is lower than 0.1, it is considered to be a misjudgment by the detector and is directly discarded. The track_thresh is greater than 0.

1. Step 2: For the high-score frame, calculate the IOU of the high-score frame and the predicted frame, and use the Hungarian algorithm to match the IOU. If the high-score frame matches the predicted frame, update the frame in the tracking trajectory to the high-score frame and assign an ID. If the high-score frame does not match the predicted frame, execute steps 3 and 4. Step 3: For the prediction box that fails to match the high-resolution box, if the prediction box is not in the tracking state, it is marked as lost. When the maximum cache frame number exceeds 25 frames, the prediction box information is deleted; if it is in the tracking state, the IOU with the low-resolution box obtained in step 1 is calculated, and the Hungarian algorithm is used to match the IOU. If the match is successful, the box in the tracking trajectory is updated to the low-resolution box and an ID is assigned; if the match fails, the prediction box that fails to match is directly discarded; for the low-resolution box that fails to match, if it fails to match all the prediction boxes that failed to match in step 2, the low-resolution box is discarded; Step 4: For the high-score frame that fails to match the predicted frame, match it with the inactive predicted frame. If the match is successful, activate the predicted frame state, mark it as the tracking state, and update the frame to the high-score frame at this time; if the match fails, mark the unmatched predicted frame for deletion. For the high-score frame that has not been matched yet, if the confidence is greater than track_thresh+0.1, create a new tracking track, otherwise discard it.

4. The method for target detection label smoothing based on historical information according to claim 1, characterized in that: The S5 includes: According to the number of detection categories k, initialize the historical category label probability matrix His: Among them, x i Indicates the probability of detecting category i. In each frame, the ID of the successfully tracked target output by the tracker is recorded, and for the target with the same ID, a current category label probability matrix Cur is generated by its current detection category j, j∈[0,k-1]: Different weights are assigned to the current category label probability matrix and the historical category label probability matrix. The formula is as follows: s=αCur+(1-α)His Among them, s is the smoothed category label probability matrix, and α represents the weight; The smoothed label L is the category label with the highest probability in the smoothed category label probability matrix. The specific formula is as follows: L=max(P Class ),P Class ∈s。 5. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images