Multi-target tracking method and device, storage medium and computer equipment

Abstract

According to the multi-target tracking method and device, the storage medium and the computer equipment provided by the invention, in the to-be-tracked video of the non-first frame, the to-be-tracked target and the historical detection frame thereof are determined from the previous frame, the target position of the current frame is further predicted, and the target position of the current frame is determined according to the confidence of each detection target in the current frame. The detection targets with the confidence coefficients lower than a threshold value serve as candidate detection targets, and the detection targets with the confidence coefficients not lower than the threshold value serve as main detection targets; on the basis of the historical detection frame and the current prediction frame of each to-be-tracked target and the current detection frame of each main detection target, performing multiple times of matching on each current prediction frame and the current detection frame of each main detection target; and matching the current prediction frame which is not matched with the current detection frame of each candidate detection target, and taking each main detection target and the successfully matched candidate detection target as a to-be-tracked target of the current frame. Therefore, the tracking error can be reduced to the greatest extent under the conditions of shielding and detection loss.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a multi-target tracking method, device, storage medium and computer equipment. Background Art

[0002] Multi-target tracking technology has important applications in autonomous driving, intelligent monitoring, drone navigation and other fields. Its main task is to predict the motion trajectories of multiple targets in real time based on the target position information provided by sensors, thereby achieving stable tracking of targets in complex scenarios.

[0003] Kalman filtering is a commonly used algorithm in multi-target tracking and is widely used in linear motion prediction scenarios. Kalman filtering is based on the system's state transition model and observation model, and gradually corrects the target position by minimizing the prediction error. However, this method assumes that the target motion is linear, which makes it difficult to deal with targets with nonlinear and complex trajectories. In addition, the Kalman filter has a large amount of calculation and has high requirements for real-time performance. In addition, Kalman filtering relies on continuous observation data. When the detector is lost or occluded, the prediction error tends to accumulate gradually, affecting the matching accuracy. Existing multi-target tracking also includes particle filtering and deep learning methods. Among them, when the target is occluded or the detector is lost, the particle weight update of the particle filter becomes unstable, resulting in an increase in tracking error; while the deep learning method is not robust enough to occlusion or partial target loss, which may lead to target loss or matching errors.

[0004] Based on this, existing multi-target tracking methods still have limitations in real-time and robustness, especially when the target is occluded or the detector is lost, which can easily lead to large errors. Summary of the invention

[0005] The purpose of the present application is to solve at least one of the above-mentioned technical defects, especially the technical defect in the prior art that large errors are easily generated when the target is blocked or the detector is lost.

[0006] In a first aspect, the present application provides a multi-target tracking method, the method comprising:

[0007] In the video to be tracked, if the current frame is not the first frame, each target to be tracked and the historical detection frame of each target to be tracked are determined in the previous frame, and the current prediction frame of each target to be tracked in the current frame is determined based on each historical detection frame;

[0008] Determine the current detection frame and confidence of each detection target detected in the current frame, and use the detection targets whose current confidence is lower than a preset confidence threshold as candidate detection targets, and use the detection targets whose current confidence is not lower than the confidence threshold as primary detection targets;

[0009] Based on the historical detection frame and current prediction frame of each target to be tracked, and the current detection frame of each main detection target, multiple matches are performed on the current prediction frame of each target to be tracked, and the current detection frame of each main detection target;

[0010] After multiple matches, the current prediction frame of the unmatched target to be tracked and the current detection frame of each candidate detection target are matched, and each main detection target and the successfully matched candidate detection target are used as the target to be tracked in the current frame.

[0011] In one embodiment, the step of determining the current prediction frame of each target to be tracked in the current frame according to each historical detection frame includes:

[0012] For each target to be tracked, the motion equation of the target to be tracked is determined, and the coordinates of the historical detection frame of the target to be tracked in the previous frame and the motion equation are used to calculate the current prediction frame in the current frame, wherein the motion equation is obtained by fitting according to the coordinates of the historical detection frames of the target to be tracked in multiple historical frames.

[0013] In one embodiment, based on the historical detection frame and the current prediction frame of each target to be tracked, and the current detection frame of each main detection target, the step of performing multiple matching on the current prediction frame of each target to be tracked, and the current detection frame of each main detection target includes:

[0014] Obtaining an embedded feature of each main detection target, and determining a current appearance feature of each main detection target based on the confidence and the embedded feature of each main detection target;

[0015] According to the historical detection frame and current prediction frame of each target to be tracked, as well as the current detection frame and current appearance features of each main detection target, the cost matrix corresponding to the current prediction frame of each target to be tracked and the current detection frame of each main detection target is calculated for initial matching;

[0016] After the initial matching, the current prediction box of the unmatched target to be tracked and the current detection box of the unmatched main detection target are matched again using the intersection-over-union matching method.

[0017] In one embodiment, the step of determining the current appearance feature of each main detection target based on the confidence and embedded features of each main detection target includes:

[0018] For each main detection target, if the confidence of the main detection target is equal to the confidence threshold, the appearance weight factor of the main detection target is taken as 1. If the confidence of the main detection target is equal to 1, the appearance weight factor of the main detection target is the preset weight factor value. If the confidence of the main detection target is greater than the confidence threshold and less than 1, when it is determined that the main detection target exists in the previous frame based on the target category of the main detection target obtained, the intersection-and-union ratio between the current detection frame of the main detection target in the current frame and the historical detection frame in the previous frame is calculated, and the corresponding appearance weight factor is determined according to the intersection-and-union ratio and confidence of the main detection target in two consecutive frames;

[0019] For each main detection target, the current appearance feature of the main detection target is calculated according to the appearance weight factor and the embedded feature of the main detection target.

[0020] In one embodiment, for each main detection target, the formula corresponding to the appearance weight factor of the main detection target is:

[0021]

[0022] In the formula, Indicates the preset weight factor value, Indicates the confidence of the main detection target, is the confidence threshold, It indicates the intersection-over-union ratio of the main detection target in two consecutive frames. Indicates a preset constant.

[0023] In one embodiment, the step of calculating the cost matrix corresponding to the current prediction frame of each target to be tracked and the current detection frame of each main detection target according to the historical detection frame and the current prediction frame of each target to be tracked, and the current detection frame and the current appearance feature of each main detection target, includes:

[0024] Calculate the first direction from the historical detection frame of each target to be tracked to the current prediction frame, and calculate the second direction from the historical detection frame of each target to be tracked to the current detection frame of each main detection target, and calculate the motion angle difference between each current prediction frame and each current detection frame according to each first direction and each second direction;

[0025] Obtain the historical appearance features of each target to be tracked in the previous frame, and calculate the appearance feature distance between each current prediction box and each current detection box based on the historical appearance features of each target to be tracked and the current appearance features of each main detection target;

[0026] Calculate the target intersection-over-union ratio between each current prediction box and each current detection box;

[0027] The cost matrix is ​​calculated based on the motion angle difference, appearance feature distance and target intersection-union ratio between each current prediction box and each current detection box.

[0028] In one embodiment, the formula corresponding to the cost matrix is:

[0029]

[0030] In the formula, represents the target intersection-over-union ratio, Indicates the difference in motion angle, represents the appearance feature distance, and They represent the appearance feature distance weight and angle difference weight respectively.

[0031] In a second aspect, the present application provides a multi-target tracking device, the device comprising:

[0032] A current prediction frame determination module is used to determine each target to be tracked and a historical detection frame of each target to be tracked in the previous frame if the current frame is not the first frame in the video to be tracked, and determine the current prediction frame of each target to be tracked in the current frame based on each historical detection frame;

[0033] A candidate detection target determination module is used to determine the current detection frame and confidence of each detection target detected in the current frame, and to take the detection target whose current confidence is lower than a preset confidence threshold as a candidate detection target, and to take the detection target whose current confidence is not lower than the confidence threshold as a primary detection target;

[0034] A multi-target matching module is used to perform multiple matches on the current prediction frames of each target to be tracked and the current detection frames of each main detection target based on the historical detection frames and current prediction frames of each target to be tracked and the current detection frames of each main detection target;

[0035] The candidate detection target matching module is used to match the current prediction frame of the unmatched target to be tracked and the current detection frame of each candidate detection target after multiple matches, and use each main detection target and the successfully matched candidate detection target as the target to be tracked in the current frame.

[0036] In a third aspect, the present application provides a storage medium: the storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of any multi-target tracking method in the above-mentioned embodiments.

[0037] In a fourth aspect, the present application provides a computer device, comprising: one or more processors, and a memory;

[0038] The memory stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the steps of any of the multi-target tracking methods in the above embodiments are performed.

[0039] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0040] In the multi-target tracking method, device, storage medium and computer equipment provided by the present application, in multi-target tracking, by taking the identified target and its historical detection frame in the previous frame as a reference, a prediction frame of each target in the current frame can be constructed, so that when the position of the target changes, a more accurate prediction can be made. The target detected in the current frame is divided into a main detection target and a candidate detection target according to the confidence threshold. By retaining the main detection target with higher confidence, the accuracy of the matching can be ensured. At the same time, the low-confidence target is used as a candidate target, which is convenient for flexible processing in subsequent matching and reduces the error caused by unstable target detection. The confidence screening mechanism can enhance the robustness of the system by flexibly processing confidence information in the case of occlusion or detection loss. Multiple matches are performed between the historical detection frame, the current prediction frame and the current detection frame of the main detection target, which improves the matching accuracy in complex scenes such as occlusion and motion changes. Additional matching is performed on unmatched targets and candidate detection targets, so that when the aforementioned matching process is unsuccessful, the candidate detection targets still have a chance to match. This process improves the tracking success rate when target detection is discontinuous or partially occluded, thereby enhancing the robustness of multi-target tracking and reducing the probability of target loss. Finally, the main detection target and the successfully matched candidate detection target are included in the set of targets to be tracked in the current frame, making the tracking results more comprehensive and stable, ensuring that multiple targets can still be accurately identified and tracked when the detection confidence fluctuates or the target position changes suddenly. Based on this, the multi-target tracking method of the present application enhances the real-time and robustness in complex scenes by introducing dynamic matching of historical detection frames and current prediction frames, as well as a multi-level screening and matching mechanism based on confidence, and through multiple matching and matching supplements of candidate targets, it can minimize tracking errors in the case of occlusion and detection loss, and improve the stability and accuracy of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1A flowchart of a multi-target tracking method provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of the structure of a multi-target tracking device provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] The present application provides a multi-target tracking method. The following embodiments are described by taking the method applied to a computer device as an example. It can be understood that the computer device can be any device with data processing functions, including but not limited to a single server, a server cluster, a personal laptop computer, a desktop computer, etc. Figure 1 As shown, the method may include the following steps:

[0047] S101: In a video to be tracked, if the current frame is not the first frame, each target to be tracked and a historical detection frame of each target to be tracked are determined in a previous frame, and a current prediction frame of each target to be tracked in the current frame is determined based on each historical detection frame.

[0048] The video to be tracked refers to video data containing multiple targets to be tracked. By processing the video frame by frame, the targets in each frame can be extracted and tracked. The current frame is a frame being processed in the video, which contains the target information at the current moment. The non-first frame is a frame other than the first frame in the video. In the non-first frame, the information of the previous frame can be used to predict and track the target of the current frame. The previous frame is a frame before the current frame, which is used to provide historical information and predict the target position of the current frame. The target to be tracked is the object to be identified and tracked in the video, and these targets can be identified by the detection algorithm. The historical detection box is a bounding box where the target to be tracked appears in the previous frame. The current prediction box is the target position of the current frame estimated based on the historical detection box, which is used to infer the possible position of the target in the new frame.

[0049] In this step, the target information in the previous frame can be obtained through historical records, including target category and detection frame information. Based on the historical detection frame of the previous frame, the motion prediction algorithm can be used to estimate the position of each target in the current frame. Among them, the estimation of the prediction frame can be combined with the position information of the target in the previous frames, and the reasonable position of the target in the current frame can be inferred through the state transition model, such as displacement, speed, etc. Furthermore, if the current frame is the first frame, the target detected in the current frame is used as the target to be tracked, the target information of the target to be tracked is saved, and the next frame is entered to perform multi-target tracking.

[0050] It can be understood that the historical detection frame of each target records the trajectory information of the target in the video frame sequence, ensuring that the historical motion path of the target can be accurately located. Through this historical path, a reasonable current prediction frame can be generated, which helps to make effective predictions when there is a sudden change in the target position. Using the historical detection frame to generate the current prediction frame helps the continuity and accuracy of tracking, especially when the target is occluded or the detector is lost. The prediction frame provides a redundant mechanism for tracking, which can complete tracking through estimation when there is no direct observation data, effectively reducing the impact of occlusion or detection loss, and improving the robustness of tracking.

[0051] S102: Determine the current detection frame and confidence of each detection target detected in the current frame, and use the detection targets whose current confidence is lower than a preset confidence threshold as candidate detection targets, and use the detection targets whose current confidence is not lower than the confidence threshold as primary detection targets.

[0052] Among them, the detection target is the object identified by the detection algorithm in the current frame, usually represented by a bounding box and a category label, which may be a moving object in the video such as a vehicle or pedestrian. The current detection box represents the rectangular bounding box of the detection target in the current frame. Confidence is the probability or credibility value that the detection target is correctly identified, usually between 0 and 1. The higher the value, the higher the confidence of the model in the detection. The preset confidence threshold is the threshold used to filter low-confidence detection targets. A fixed value such as 0.3 can be selected, or the optimal value can be obtained through testing. Detection targets above this threshold are considered reliable. Candidate detection targets are detection targets with a confidence lower than the preset threshold. Primary detection targets are detection targets with a confidence higher than or equal to the preset threshold. They have a high confidence and can usually be used directly for target tracking.

[0053] In this step, the target detection algorithm, such as YOLO, Faster R-CNN, etc., is used to identify the detection target in the current frame. The target detection model generates bounding box coordinates and confidence scores, that is, the output of each detection target includes the detection box and confidence. After that, the target is screened by comparing the confidence of each detection target with the preset confidence threshold. Targets with confidence higher than the threshold will be marked as "primary detection targets", and targets with confidence lower than the threshold will be marked as "candidate detection targets".

[0054] It can be understood that the determination of the current detection box and confidence can filter out unreliable targets. By marking low-confidence targets as candidates, it is possible to avoid directly using these uncertain targets for subsequent tracking, reducing the risk of false matching or mistracking. In addition, targets with high confidence can be directly included in the tracking, thereby ensuring that the main targets are tracked first in real-time processing. The confidence screening mechanism can improve the accuracy of detection during tracking. The candidate detection targets retain the detection information of low-confidence objects, which is convenient for re-evaluating these targets in the presence of occlusion, detector instability, etc. The main detection targets are directly tracked to avoid tracking interruptions caused by confidence fluctuations. In this way, the robustness of the system is effectively improved, ensuring the continuity and stability of tracking in complex scenarios.

[0055] S103: Based on the historical detection frame and the current prediction frame of each target to be tracked, and the current detection frame of each main detection target, multiple matches are performed on the current prediction frame of each target to be tracked, and the current detection frame of each main detection target.

[0056] In this step, first, after determining the historical detection frame and current prediction frame of the target to be tracked, as well as the current detection frame of the main detection target, the similarity between the prediction frame of each target to be tracked and the current detection frame can be calculated by using algorithms such as Euclidean distance or intersection ratio. In multiple matches, multiple pairing calculations are performed to evaluate the similarity between different frame pairs. Furthermore, the multiple matching process can use an optimization algorithm, such as the Hungarian algorithm or the greedy algorithm, to ensure that each prediction frame can find the best matching detection frame. After each round of matching, the matching results are recorded, and the matched and unmatched detection frames and prediction frame lists are updated until the matching conditions are met or the preset number of matches is reached.

[0057] It is understandable that in real-world scenarios, the movement and position of the target will be affected by factors such as occlusion and rapid movement, which may result in a single match error. Multiple matches allow different matching pairs to be tried in different rounds, minimizing mismatches and missed matches, which not only improves the accuracy of matching, but also enhances the robustness of the algorithm when occlusion or unstable target detection occurs.

[0058] S104: After multiple matches, the current prediction frame of the unmatched target to be tracked and the current detection frame of each candidate detection target are matched, and each main detection target and the successfully matched candidate detection target are used as the target to be tracked in the current frame.

[0059] Among them, the current prediction box of the unmatched target to be tracked is the predicted position of the current detection box that has not found a matching main detection target in multiple matching operations. The target to be tracked in the current frame is the tracking target finally confirmed in the current frame, including the main detection target that has passed the confidence screening and the candidate detection target that has successfully matched the prediction box. These targets will continue to be tracked in the next frame.

[0060] In this step, the unmatched prediction boxes and candidate detection boxes remaining after multiple matches are further compared. The similarity between the prediction box and the candidate detection box can be measured using distance metrics or intersection-over-union methods to determine whether they can be matched. A similarity threshold can be set in each round of comparison, and the prediction boxes and detection boxes whose matching degree exceeds the threshold will be recorded as a successful match.

[0061] Once the candidate detection targets that have been successfully matched are determined, the main detection targets and the candidate detection targets that have been successfully matched are included in the list of targets to be tracked in the current frame. This list will be used in the detection and matching process of subsequent frames.

[0062] It is understandable that in complex scenes in the video, the target may be temporarily occluded, causing the detector to lose the target or generate low-confidence candidate detection frames. By matching the unmatched prediction frames with the candidate detection frames, these uncertain targets can be captured and the continuity of tracking can be maintained. This process ensures that the historical trajectory of the target will not be discarded at will when the detector confidence fluctuates or is temporarily lost. By adding the successfully matched candidate detection targets to the list of targets to be tracked, these targets can continue to be tracked in the next frame, enhancing the robustness of the system. Ultimately, the multi-level matching mechanism ensures the continuity and accuracy of multi-target tracking, reduces the missed detection rate and false detection rate, and thus has better adaptability in complex dynamic scenes.

[0063] In the above embodiment, in multi-target tracking, by taking the identified target and its historical detection frame in the previous frame as a reference, the prediction frame of each target in the current frame can be constructed, so that when the position of the target changes, a more accurate prediction can be made. The target detected in the current frame is divided into a main detection target and a candidate detection target according to the confidence threshold. By retaining the main detection target with higher confidence, the accuracy of the matching can be ensured. At the same time, the low-confidence target is used as a candidate target, which is convenient for flexible processing in subsequent matching and reduces the error caused by unstable target detection. The confidence screening mechanism can enhance the robustness of the system by flexibly processing the confidence information in the case of occlusion or detection loss. Multiple matches are performed between the historical detection frame, the current prediction frame and the current detection frame of the main detection target, which improves the matching accuracy in complex scenes such as occlusion and motion changes. Additional matching is performed on the unmatched target and the candidate detection target, so that when the aforementioned matching process is unsuccessful, the candidate detection target still has the opportunity to match. This process improves the tracking success rate when the target detection is discontinuous or partially occluded, thereby enhancing the robustness of multi-target tracking and reducing the probability of target loss. Finally, the main detection target and the successfully matched candidate detection target are included in the set of targets to be tracked in the current frame, making the tracking results more comprehensive and stable, ensuring that multiple targets can still be accurately identified and tracked when the detection confidence fluctuates or the target position changes suddenly. Based on this, the multi-target tracking method of the present application enhances the real-time and robustness in complex scenes by introducing dynamic matching of historical detection frames and current prediction frames, as well as a multi-level screening and matching mechanism based on confidence. In addition, through multiple matching and matching supplements of candidate targets, the tracking error can be minimized in the case of occlusion and detection loss, thereby improving the stability and accuracy of target tracking.

[0064] In one embodiment, the step of determining the current prediction frame of each target to be tracked in the current frame according to each historical detection frame includes:

[0065] For each target to be tracked, the motion equation of the target to be tracked is determined, and the coordinates of the historical detection frame of the target to be tracked in the previous frame and the motion equation are used to calculate the current prediction frame in the current frame, wherein the motion equation is obtained by fitting according to the coordinates of the historical detection frames of the target to be tracked in multiple historical frames.

[0066] The motion equation is a mathematical formula used to describe the motion law of the target to be tracked, which can be obtained by fitting the position coordinates of the target in multiple historical frames. This equation can predict the next position of the target.

[0067] Specifically, the motion equation can be generated by collecting the coordinate data of the target to be tracked in multiple historical frames and using fitting methods, such as linear regression or curve fitting. When the computing device executes these fitting methods, it calculates the relationship between the coordinate data and generates equations that can represent the law of target motion. For example, consider the target coordinates of the nth frame as x and the target coordinates of the (n+1)th frame as y. Assume that the linear equation y=ax+b is used to fit the parameters a and b. The target coordinates of the (n+2)th frame are predicted by this equation, and the values ​​of a and b are continuously updated according to the detection results.

[0068] Based on the position coordinates of the historical detection frame in the previous frame, the possible position of the target in the current frame is predicted using the fitted motion equation. This process can substitute the position coordinates of the historical detection frame in the previous frame into the motion equation and solve the equation to obtain the coordinate position of the current prediction frame.

[0069] In this embodiment, by determining the motion equation, the motion law of the target to be tracked can be captured, so that the target position can still be estimated when there is occlusion, detector error or target motion change, and the prediction frame is used to assist the matching and position correction of the detection frame to avoid tracking interruption. The fitting of the motion equation uses the coordinate data of multiple historical frames, which helps to accurately capture the motion trend of the target, reduce the impact of single-frame data fluctuations, and achieve more accurate and continuous multi-target tracking, which helps to maintain a stable tracking effect in complex environments and improve the adaptability in dynamic scenes.

[0070] In one embodiment, based on the historical detection frame and the current prediction frame of each target to be tracked, and the current detection frame of each main detection target, the step of performing multiple matching on the current prediction frame of each target to be tracked, and the current detection frame of each main detection target includes:

[0071] Obtaining an embedded feature of each main detection target, and determining a current appearance feature of each main detection target based on the confidence and the embedded feature of each main detection target;

[0072] According to the historical detection frame and current prediction frame of each target to be tracked, as well as the current detection frame and current appearance features of each main detection target, the cost matrix corresponding to the current prediction frame of each target to be tracked and the current detection frame of each main detection target is calculated for initial matching;

[0073] After the initial matching, the current prediction box of the unmatched target to be tracked and the current detection box of the unmatched main detection target are matched again using the intersection-over-union matching method.

[0074] Among them, the embedded feature is a high-dimensional feature vector extracted from the target image by a deep learning model, which is used to describe the appearance and characteristics of the target, so that the same target can be recognized even when the posture, color or lighting changes. The current appearance feature is a descriptive feature generated based on the confidence and embedded features of the main detection target, which represents the appearance information of the target in the current frame and is used for tracking and matching. The cost matrix is ​​a matrix containing the similarity or difference between the target to be tracked and the main detection target. The smaller the value of the cost matrix, the more similar the two targets are as matching candidates. The intersection-over-union matching uses the intersection-over-union method to calculate the degree of overlap of two bounding boxes to measure their similarity.

[0075] Specifically, deep learning models, such as convolutional neural networks, can be used to generate embedded features from video frames, converting the image of the main detection target into a high-dimensional vector, so that the vector comparison method can be used to measure the appearance similarity between different targets. After that, the embedded features and confidence are combined to generate the current appearance features. Among them, the embedded features can be weighted, and the features with higher confidence are more significant, thereby generating more reliable appearance features for matching the target to be tracked. Then, based on the historical detection frame and current prediction frame of each target to be tracked, as well as the current detection frame and current appearance features of each main detection target, the similarity of the current prediction frame of each target to be tracked and the current detection frame of the main detection target is calculated, and the similarity is filled into the cost matrix. The values ​​in the matrix represent the degree of similarity between the target to be tracked and the main detection target.

[0076] In the first match, the cost matrix value is used to execute the Hungarian algorithm or other cost minimization algorithms. If the current prediction box of the target to be tracked and the current detection box of the main detection target are not successfully matched, they will enter the second match, calculate the intersection-over-union ratio in the second match, and take the candidate with the higher intersection-over-union ratio as the final matching pair.

[0077] In this embodiment, the embedded features can provide unique appearance information in complex scenes, and the appearance features generated in combination with the confidence level are helpful in identifying changing targets. The cost matrix provides a numerical matching basis, which is conducive to handling a large number of target matching tasks, and the intersection-over-union matching provides a more robust alternative when the embedded features are insufficient for matching. The first match ensures that most targets can be quickly and effectively matched, while the second match can compensate for the mismatch caused by insufficient embedded features, reducing the risk of tracking loss due to complex scenes or occlusions.

[0078] In one embodiment, the step of determining the current appearance feature of each main detection target based on the confidence and embedded features of each main detection target includes:

[0079] For each main detection target, if the confidence of the main detection target is equal to the confidence threshold, the appearance weight factor of the main detection target is taken as 1. If the confidence of the main detection target is equal to 1, the appearance weight factor of the main detection target is the preset weight factor value. If the confidence of the main detection target is greater than the confidence threshold and less than 1, when it is determined that the main detection target exists in the previous frame based on the target category of the main detection target obtained, the intersection-and-union ratio between the current detection frame of the main detection target in the current frame and the historical detection frame in the previous frame is calculated, and the corresponding appearance weight factor is determined according to the intersection-and-union ratio and confidence of the main detection target in two consecutive frames;

[0080] For each main detection target, the current appearance feature of the main detection target is calculated according to the appearance weight factor and the embedded feature of the main detection target.

[0081] Among them, the appearance weight factor is the weight value that determines the main detection target on the appearance feature. This factor combines the confidence and intersection-over-union information to identify the change or stability of the target, thereby affecting the calculation of the appearance feature. The preset weight factor value is a fixed weight value set when the confidence is higher than the threshold and reaches 1, which is used to identify the absolute reliability of the detection result. The target category is the type of target object, such as "person" or "car", which is used to determine whether it is the same target object in the previous frame and the current frame.

[0082] It is understandable that due to the problem of being blocked and moving too fast during the movement of the target, when it is blocked, the corresponding confidence is relatively low, and the representativeness of the target's appearance features in this state also decreases. In the process of updating the dynamic appearance, it is necessary to reduce the proportion of the embedded features of the target in this frame. At the same time, when the target moves too fast, there may be a large difference in the change of its target appearance, and the dynamic appearance calculation weight also needs to be adjusted.

[0083] Specifically, first check the confidence value of each main detection target. If the confidence is equal to the threshold, set the appearance weight factor to 1. If the confidence is 1, set the appearance weight factor to the preset weight factor value. When the confidence is between the threshold and 1, determine whether it exists in the previous frame based on the target category. If it exists, further calculate the intersection and union ratio of the target between the two frames. The intersection and union ratio result and the confidence together determine the final appearance weight factor. After obtaining the appearance weight factor, combine the appearance weight factor with the embedded feature to calculate the current appearance feature of the main detection target. Among them, the embedded features and weight factors can be combined by weighted averaging, feature fusion and other methods to generate a stable feature vector for the current frame.

[0084] In an example, the current appearance features of the main detection target can be calculated by the following expression:

[0085]

[0086] in, Indicates the current appearance characteristics, Represents the historical frame appearance features of the main detection target, Indicates the embedded features extracted by the deep neural network for the main detection target in the current frame. is the appearance weight factor.

[0087] In this embodiment, the appearance weight factor is determined by the confidence and intersection-over-union ratio to adapt to the changes and position stability of the target at different confidence levels. For targets with high confidence but changing positions, by dynamically adjusting the appearance weight factor, it is possible to adapt to slight changes in its appearance in the previous and next frames without easily changing the appearance features, thereby ensuring the accuracy and stability of tracking. By combining the current appearance features generated by the appearance weight factor and the embedded features, the appearance of the target in the current frame can be more accurately described, reducing the probability of tracking errors caused by target occlusion, posture changes or partial overlap. Ultimately, robust tracking of targets can be achieved in complex scenes, ensuring that the detected targets do not cause matching deviations due to short-term changes, effectively improving the reliability and accuracy of multi-target tracking.

[0088] In one embodiment, for each main detection target, the formula corresponding to the appearance weight factor of the main detection target is:

[0089]

[0090] In the formula, Indicates the preset weight factor value, Indicates the confidence of the main detection target, is the confidence threshold, It indicates the intersection-over-union ratio of the main detection target in two consecutive frames. Indicates a preset constant.

[0091] Specifically, is the difference between the calculated and preset weight factors, indicating the difference from the ideal weight To current weight degree of transformation. Used to normalize the confidence and limit it between 0 and 1. When the confidence is low, this value will decrease, thereby increasing the value of the final weight factor. A minimum value of 0.0001 is added to avoid zero values ​​during the calculation process and prevent instability in numerical calculations. The improvement of will positively affect the weight. A higher intersection-over-union ratio means that the position of the target in consecutive frames changes less, thereby increasing the effectiveness of the weight.

[0092] For example, is a fixed value of 0.85, is a fixed value of 2. hour, is approximately equal to 1, thus completely ignoring the new appearance embedding features. mean , will add new appearance features to the dynamic feature update to the maximum extent. hour, The value will also affect the final update. When the motion is slow, , which is approximately 1. Basically the same as the original, When , it means that the target has moved significantly, and the new appearance features will be added to the dynamic feature update to the maximum extent.

[0093] In this embodiment, the formula comprehensively considers the confidence, intersection-over-union ratio, and preset weights, and can more accurately reflect the stability of the target in the current frame. In particular, when the target confidence and intersection-over-union ratio change, the dynamic adjustment of the weight factor makes the model more flexible. When the confidence of the detected target is low or occlusion occurs, the formula will naturally reduce the weight factor, reduce the reliance on the target appearance features, and thus reduce the risk of false matching. By introducing The confidence level can effectively avoid tracking loss caused by short-term occlusion or appearance change, and improve the accuracy and robustness of the multi-target tracking system. The appearance features of the target will better reflect the real state of the target between different frames, thus achieving more accurate matching and tracking.

[0094] In one embodiment, the step of calculating the cost matrix corresponding to the current prediction frame of each target to be tracked and the current detection frame of each main detection target according to the historical detection frame and the current prediction frame of each target to be tracked, and the current detection frame and the current appearance feature of each main detection target, includes:

[0095] Calculate the first direction from the historical detection frame of each target to be tracked to the current prediction frame, and calculate the second direction from the historical detection frame of each target to be tracked to the current detection frame of each main detection target, and calculate the motion angle difference between each current prediction frame and each current detection frame according to each first direction and each second direction;

[0096] Obtain the historical appearance features of each target to be tracked in the previous frame, and calculate the appearance feature distance between each current prediction box and each current detection box based on the historical appearance features of each target to be tracked and the current appearance features of each main detection target;

[0097] Calculate the target intersection-over-union ratio between each current prediction box and each current detection box;

[0098] The cost matrix is ​​calculated based on the motion angle difference, appearance feature distance and target intersection-union ratio between each current prediction box and each current detection box.

[0099] Among them, the first direction is the vector pointing from the historical detection frame to the current prediction frame, which indicates the direction of movement of the target to be tracked. The second direction is the vector pointing from the historical detection frame to the current detection frame of each main detection target, which is used to evaluate the relationship between the target to be tracked and other detection targets. The motion angle difference is the angle difference between the first direction and the second direction, which reflects the deviation between the direction of movement of the target to be tracked and the direction of other main detection targets. The historical appearance feature is the appearance feature of the target to be tracked in the previous frame, which is used to describe the appearance information of the target. The appearance feature distance is used to measure the similarity between the historical appearance features of the target to be tracked and the current appearance features of the main detection target. The target intersection-union ratio is used to measure the degree of overlap between the current prediction frame and the current detection frame. The closer it is to 1, the more overlap it has.

[0100] Specifically, first, according to the coordinates of the historical detection frame and the current prediction frame of the current frame, the vector difference of the two frames is calculated, that is, the coordinates of the current prediction frame minus the coordinates of the historical detection frame, to obtain the first direction. For each main detection target, starting from the historical detection frame, the vector difference to the current detection frame of each main detection target is calculated to obtain the second direction. By calculating the angle between the first direction and the second direction, the motion angle difference is obtained. The historical appearance features of each target to be tracked in the previous frame are obtained from the historical records. The historical appearance features are compared with the current appearance features of each main detection target using an appropriate distance metric such as Euclidean distance or cosine distance to obtain the appearance feature distance. The intersection-union ratio is calculated using the area information of the two frames, that is, the ratio of the overlapping area of ​​the two frames to the area of ​​their union. The motion angle difference, appearance feature distance and target intersection-union ratio are combined according to predetermined weights to calculate the cost matrix.

[0101] In one example, the calculation expression for the first direction or the second direction may be:

[0102]

[0103] in, and are the coordinate values ​​of different detection boxes respectively.

[0104] In this embodiment, by calculating the motion angle difference and the appearance feature distance, the similarity and matching degree between the target to be tracked and the main detection target can be comprehensively evaluated. The cost matrix combines multiple influencing factors and provides a comprehensive metric for optimizing the target matching process. By comprehensively considering multiple directional information, appearance features and overlap, the accuracy of target matching can be improved, especially when the target similarity is high or partially occluded, the target can be effectively distinguished, the probability of false matching is reduced, and continuous and accurate target tracking is ensured.

[0105] In one embodiment, the formula corresponding to the cost matrix is:

[0106]

[0107] In the formula, represents the target intersection-over-union ratio, Indicates the difference in motion angle, represents the appearance feature distance, and They represent the appearance feature distance weight and angle difference weight respectively.

[0108] in, , Indicates the first direction from the historical detection frame of the target to be tracked to the current prediction frame. Indicates the second direction from the historical detection frame of the target to be tracked to the current detection frame of the main detection target.

[0109] In this embodiment, by introducing the appearance feature distance weight and angle difference weight , can flexibly adjust the influence of various factors in the cost calculation, so as to optimize the matching strategy in different scenarios and ensure the best matching results in different situations. The design of the cost matrix takes into account the appearance similarity and motion direction similarity between targets, which helps to effectively distinguish between target occlusion or similar targets, reduce the risk of mismatching, and improve the accuracy of multi-target tracking. In a dynamic environment, the target may be affected by factors such as occlusion and motion changes. Comprehensive consideration of the motion angle difference and appearance feature distance allows the matching stability to remain high when the target changes, thereby enhancing robustness.

[0110] The following describes a multi-target tracking device provided by an embodiment of the present application. The multi-target tracking device described below and the multi-target tracking method described above can be referred to in correspondence with each other. Figure 2 As shown, the present application provides a multi-target tracking device, the device comprising:

[0111] The current prediction frame determination module 201 is used to determine each target to be tracked and the historical detection frame of each target to be tracked in the previous frame if the current frame is not the first frame in the video to be tracked, and determine the current prediction frame of each target to be tracked in the current frame according to each historical detection frame;

[0112] The candidate detection target determination module 202 is used to determine the current detection frame and confidence of each detection target detected in the current frame, and to use the detection targets whose current confidence is lower than the preset confidence threshold as candidate detection targets, and to use the detection targets whose current confidence is not lower than the confidence threshold as primary detection targets;

[0113] A multi-target matching module 203 is used to perform multiple matches on the current prediction frames of each target to be tracked and the current detection frames of each main detection target based on the historical detection frames and the current prediction frames of each target to be tracked and the current detection frames of each main detection target;

[0114] The candidate detection target matching module 204 is used to match the current prediction frame of the unmatched target to be tracked and the current detection frame of each candidate detection target after multiple matches, and use each main detection target and the successfully matched candidate detection target as the target to be tracked in the current frame.

[0115] In one embodiment, the current prediction frame determination module 201 includes:

[0116] The current prediction frame determination unit is used to determine the motion equation of the target to be tracked for each target to be tracked, and use the coordinates of the historical detection frame of the target to be tracked in the previous frame and the motion equation to calculate the current prediction frame in the current frame, wherein the motion equation is obtained by fitting according to the coordinates of the historical detection frames of the target to be tracked in multiple historical frames.

[0117] In one embodiment, the multi-target matching module 203 includes:

[0118] a current appearance feature determination unit, used to obtain an embedded feature of each main detection target, and determine a current appearance feature of each main detection target based on the confidence and embedded feature of each main detection target;

[0119] The initial matching unit is used to calculate the cost matrix corresponding to the current prediction frame of each target to be tracked and the current detection frame of each main detection target based on the historical detection frame and current prediction frame of each target to be tracked, as well as the current detection frame and current appearance features of each main detection target, so as to perform initial matching;

[0120] The secondary matching unit is used to perform secondary matching on the current prediction box of the unmatched target to be tracked and the current detection box of the unmatched main detection target after the initial matching by using the intersection-over-union matching method.

[0121] In one embodiment, the current appearance feature determination unit includes:

[0122] An appearance weight factor determination subunit is used for, for each main detection target, if the confidence of the main detection target is equal to the confidence threshold, then the appearance weight factor of the main detection target is taken as 1, if the confidence of the main detection target is equal to 1, then the appearance weight factor of the main detection target is a preset weight factor value, if the confidence of the main detection target is greater than the confidence threshold, and less than 1, then when it is determined that the main detection target exists in the previous frame based on the target category of the main detection target obtained, the intersection-and-union ratio between the current detection frame of the main detection target in the current frame and the historical detection frame in the previous frame is calculated, and the corresponding appearance weight factor is determined based on the intersection-and-union ratio and confidence of the main detection target in two consecutive frames;

[0123] The current appearance feature calculation subunit is used to calculate the current appearance feature of each main detection target according to the appearance weight factor and embedded feature of the main detection target.

[0124] In one embodiment, for each main detection target, the formula corresponding to the appearance weight factor of the main detection target is:

[0125]

[0126] In the formula, Indicates the preset weight factor value, Indicates the confidence of the main detection target, is the confidence threshold, It indicates the intersection-over-union ratio of the main detection target in two consecutive frames. Indicates a preset constant.

[0127] In one embodiment, the initial matching unit includes:

[0128] a motion angle difference calculation subunit, used to calculate a first direction from a historical detection frame of each target to be tracked to a current prediction frame, and to calculate a second direction from a historical detection frame of each target to be tracked to a current detection frame of each main detection target, and to calculate a motion angle difference between each current prediction frame and each current detection frame according to each first direction and each second direction;

[0129] The appearance feature distance calculation subunit is used to obtain the historical appearance features of each target to be tracked in the previous frame, and calculate the appearance feature distance between each current prediction frame and each current detection frame according to the historical appearance features of each target to be tracked and the current appearance features of each main detection target;

[0130] The target intersection-over-union ratio calculation subunit is used to calculate the target intersection-over-union ratio between each current prediction box and each current detection box;

[0131] The cost matrix calculation subunit is used to calculate the cost matrix according to the motion angle difference, appearance feature distance and target intersection-union ratio between each current prediction box and each current detection box.

[0132] In one embodiment, the formula corresponding to the cost matrix is:

[0133]

[0134] In the formula, represents the target intersection-over-union ratio, Indicates the difference in motion angle, represents the appearance feature distance, and They represent the appearance feature distance weight and angle difference weight respectively.

[0135] In one embodiment, the present application also provides a storage medium, which stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the multi-target tracking method described in any of the above embodiments.

[0136] In one embodiment, the present application also provides a computer device, in which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the multi-target tracking method described in any of the above embodiments.

[0137] Indicatively, Figure 3 As shown, Figure 3 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 300 may be provided as a server. Figure 3 The computer device 300 includes a processing component 302, which further includes one or more processors, and a memory resource represented by a memory 301, for storing instructions executable by the processing component 302, such as an application. The application stored in the memory 301 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 302 is configured to execute instructions to perform the multi-target tracking method of any of the above embodiments.

[0138] The computer device 300 may further include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate based on an operating system stored in the memory 301, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0139] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0140] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. Herein, "one", "one", "said", "the" and "it" may also include plural forms, unless the context clearly indicates another way. A plurality refers to at least two cases, such as 2, 3, 5 or 8, etc. "And / or" includes any and all combinations of the relevant listed items.

[0141] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0142] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-target tracking method, characterized in that: The method comprises: In the video to be tracked, if the current frame is not the first frame, each target to be tracked and a historical detection frame of each target to be tracked are determined in the previous frame, and a current prediction frame of each target to be tracked in the current frame is determined according to each historical detection frame; Determine the current detection frame and confidence of each detection target detected in the current frame, and use the detection targets whose current confidence is lower than a preset confidence threshold as candidate detection targets, and use the detection targets whose current confidence is not lower than the confidence threshold as primary detection targets; Based on the historical detection frame and the current prediction frame of each of the targets to be tracked, and the current detection frame of each of the main detection targets, multiple matching is performed on the current prediction frame of each of the targets to be tracked, and the current detection frame of each of the main detection targets; After multiple matches, the current prediction frame of the unmatched target to be tracked and the current detection frame of each candidate detection target are matched, and each of the main detection targets and the successfully matched candidate detection targets are used as the target to be tracked in the current frame.

2. The multi-target tracking method according to claim 1, characterized in that: The step of determining the current prediction frame of each target to be tracked in the current frame according to each historical detection frame includes: For each of the targets to be tracked, the motion equation of the target to be tracked is determined, and the coordinates of the historical detection frame of the target to be tracked in the previous frame and the motion equation are used to calculate the current prediction frame in the current frame, wherein the motion equation is obtained by fitting according to the coordinates of the historical detection frames of the target to be tracked in multiple historical frames.

3. The multi-target tracking method according to claim 1, characterized in that: The step of performing multiple matching on the current prediction frames of each target to be tracked and the current detection frames of each main detection target based on the historical detection frames and the current prediction frames of each target to be tracked and the current detection frames of each main detection target comprises: Acquire the embedded features of each of the main detection targets, and determine the current appearance features of each of the main detection targets based on the confidence and the embedded features of each of the main detection targets; According to the historical detection frame and the current prediction frame of each target to be tracked, as well as the current detection frame and the current appearance feature of each main detection target, the cost matrix corresponding to the current prediction frame of each target to be tracked and the current detection frame of each main detection target is calculated to perform an initial matching; After the initial matching, the current prediction box of the unmatched target to be tracked and the current detection box of the unmatched main detection target are matched again using the intersection-over-union matching method.

4. The multi-target tracking method according to claim 3, characterized in that: The step of determining the current appearance feature of each of the main detection targets based on the confidence and embedded features of each of the main detection targets comprises: For each of the main detection targets, if the confidence of the main detection target is equal to the confidence threshold, the appearance weight factor of the main detection target is taken as 1, if the confidence of the main detection target is equal to 1, then the appearance weight factor of the main detection target is a preset weight factor value, if the confidence of the main detection target is greater than the confidence threshold, and less than 1, then when it is determined that the main detection target exists in the previous frame according to the target category of the main detection target obtained, the intersection-and-union ratio between the current detection frame of the main detection target in the current frame and the historical detection frame in the previous frame is calculated, and the corresponding appearance weight factor is determined according to the intersection-and-union ratio and confidence of the main detection target in two consecutive frames; For each of the main detection targets, the current appearance features of the main detection target are calculated according to the appearance weight factor and the embedded features of the main detection target.

5. The multi-target tracking method according to claim 4, characterized in that: For each of the main detection targets, the formula corresponding to the appearance weight factor of the main detection target is: In the formula, Indicates the preset weight factor value, Indicates the confidence of the main detection target, is the confidence threshold, It indicates the intersection-over-union ratio of the main detection target in two consecutive frames. Indicates a preset constant.

6. The multi-target tracking method according to claim 3, characterized in that: The step of calculating the current prediction frame of each target to be tracked and the cost matrix corresponding to the current detection frame of each main detection target according to the historical detection frame and the current prediction frame of each target to be tracked, as well as the current detection frame and the current appearance feature of each main detection target, comprises: Calculating a first direction from the historical detection frame of each target to be tracked to the current prediction frame, and calculating a second direction from the historical detection frame of each target to be tracked to the current detection frame of each main detection target, and calculating a motion angle difference between each current prediction frame and each current detection frame according to each of the first directions and each of the second directions; Obtaining the historical appearance features of each target to be tracked in the previous frame, and calculating the appearance feature distance between each current prediction frame and each current detection frame according to the historical appearance features of each target to be tracked and the current appearance features of each main detection target; Calculate the target intersection-union ratio between each current prediction box and each current detection box; The cost matrix is ​​calculated according to the motion angle difference, appearance feature distance and target intersection-union ratio between each current prediction box and each current detection box.

7. The multi-target tracking method according to claim 6, characterized in that: The formula corresponding to the cost matrix is: In the formula, represents the target intersection-over-union ratio, Indicates the difference in motion angle, represents the appearance feature distance, and They represent the appearance feature distance weight and angle difference weight respectively.

8. A multi-target tracking device, characterized in that: The device comprises: A current prediction frame determination module is used to determine each target to be tracked and a historical detection frame of each target to be tracked in a previous frame if the current frame is not the first frame in the video to be tracked, and determine a current prediction frame of each target to be tracked in the current frame according to each historical detection frame; A candidate detection target determination module is used to determine the current detection frame and confidence of each detection target detected in the current frame, and to take the detection targets whose current confidence is lower than a preset confidence threshold as candidate detection targets, and to take the detection targets whose current confidence is not lower than the confidence threshold as primary detection targets; A multi-target matching module, configured to perform multiple matches on the current prediction frames of the targets to be tracked and the current detection frames of the main detection targets based on the historical detection frames and the current prediction frames of the targets to be tracked and the current detection frames of the main detection targets; The candidate detection target matching module is used to match the current prediction frame of the unmatched target to be tracked and the current detection frame of each candidate detection target after multiple matches, and use each main detection target and the successfully matched candidate detection target as the target to be tracked in the current frame.

9. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the multi-target tracking method as described in any one of claims 1 to 7.

10. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the multi-target tracking method according to any one of claims 1 to 7 are performed.

Images