Self-adaptive multi-target tracking method based on stages

Abstract

The invention relates to a staged-based adaptive multi-target tracking method, and provides a staged association thought and an adaptive weight strategy, when a to-be-tracked target in a video sequence has complex conditions such as long-time shielding, deformation and illumination change, target trajectories are classified according to the target shielding degree, and the target tracking accuracy is improved. A reasonable track detection frame association sequence is set in combination with the actual situation, and meanwhile, different association strategies are designed for each association, so that the tracking accuracy is further improved; a confidence coefficient difference value and speed direction information are used as a weak clue to assist a strong clue in matching between a target track and a detection frame, so that the correlation success rate is further improved; and the appearance feature updating module designs different appearance feature updating strategies for the targets with different shielding degrees so as to improve the discrimination of the appearance features. The tracking method provided by the invention can still keep track continuity and tracking stability in a complex scene.

Description

Technical Field

[0001] The invention relates to the field of video target tracking, and in particular to a stage-based adaptive multi-target tracking method. Background Art

[0002] Visual target tracking is an important problem in the field of computer vision. Its main task is to automatically locate and track a given target in a video sequence. Video target tracking technology is widely used in video surveillance, remote sensing satellites, unmanned driving and other fields. Although visual target tracking has developed rapidly in the past few years, there are still many challenging factors to be solved in practical applications, such as occlusion, fast movement, target deformation, illumination, scale change and background clutter;

[0003] In multi-target tracking, occlusion refers to the situation where the target is partially or completely hidden by other objects or the environment during observation. Occlusion can be divided into long occlusion and short occlusion according to the length of time. When the target cannot be observed at all until the occluder is removed or the target changes position and reappears, we call it long occlusion. When part of the target is occluded, it is called short occlusion. In the multi-target tracking task, both long occlusion and short occlusion are challenging situations because they may cause target loss or trajectory interruption.

[0004] The detection-based target tracking task detects the position and appearance information of the target in each frame, and uses Kalman filtering to predict the position of the target in the next frame, and associates the predicted position information with the extracted appearance information to form a trajectory. However, in actual situations, due to the existence of occlusion and target appearance deformation, simply using appearance information and position information to associate targets is prone to association errors and trajectory breaks, which in turn affects the performance of multi-target tracking.

[0005] Therefore, in multi-target tracking, using only position information or appearance information to associate targets between adjacent frames is not the best choice in all scenarios. How to select the most appropriate association strategy under different target motion states is crucial to improving the performance of multi-target tracking. Summary of the invention

[0006] In view of the defects of the prior art, the present invention provides an adaptive multi-object tracking algorithm based on the stage-dividedcorrelation strategy (SDC-Track), which divides the target trajectory into four stages according to the movement situation: the target normal movement stage, the target about to enter the occlusion area stage, the target just leaving the occlusion area stage and the target, the permanent disappearance stage. According to the trajectory characteristics of different stages, different correlation strategies are designed to maximize the correlation accuracy and achieve better tracking performance.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A phase-based adaptive multi-target tracking method comprises the following steps:

[0009] S1. Send each frame of the entire video sequence to the pre-trained target detection network and appearance feature extraction network for target detection and appearance feature extraction. k The detection box information of the target in [B1,B2,B3,..,B n ] and the target appearance feature vector [e1,e2,e3,..,e n ];

[0010] S2, classify the target detection frame and trajectory, and obtain the target predicted position through Kalman filtering;

[0011] Trajectory set T (1) represents the normal motion trajectory, T (2) Indicates the target trajectory that is about to enter the occluded area, T (3) Represents the target trajectory that has just left the occluded area, and the detection box set D high represents a high confidence detection box, D low represents a low confidence detection box, D all Represents the union of high confidence detection boxes and low confidence detection boxes;

[0012] S3, for the trajectory set T (1) and the detection box set D all Make the first association;

[0013] S4, for the trajectory set T (2) and the detection box set D remain(1) Perform the second association, D remain(1) Indicates the detection box that was not successfully associated after the first association;

[0014] S5. For the trajectory set T (3)and the detection box set D remain(2) Perform the third association, D remain(2) Indicates the detection box that is still not successfully associated after the second association;

[0015] S6, updating the trajectory appearance features successfully associated in steps S3-S5;

[0016] S7, detection box set D remain(2) The unassociated detection boxes in are initialized as trajectories. For the trajectory set T (3) The tracks that were not successfully associated for 30 consecutive frames were removed;

[0017] Among them, the above steps S1-S2 are the target detection process, steps S3-S5 are the target association process, and steps S6-S7 are the subsequent processing process. By repeating steps S3-S7, a trajectory is gradually formed to complete the entire target tracking.

[0018] Furthermore, the target detection network described in step S1 is a YOLOX target detection network, through which the target location information and confidence value are obtained, and the appearance information of the target is obtained through the appearance feature extraction network. The target location information, appearance information, and target confidence are used as subsequent association clues to achieve association between targets, specifically:

[0019] S1.1. Each frame of the video is sent to the network for feature extraction. Darknet-53 is used as the backbone network. A multi-scale feature fusion mechanism is introduced. Features from different layers are fused through the feature pyramid network to enhance the detection capabilities of small and large targets. Finally, the output of the feature extraction network will be sent to the detection head for target classification and positioning to complete the target detection task.

[0020] S1.2. Use an independent ReID model to extract the target appearance feature vector. The input of the ReID model comes from the output results of the target detection network. The high-dimensional target appearance vector is extracted through the ReID model as an important clue in the association process.

[0021] Furthermore, the specific implementation steps in step S2 are:

[0022] S2.1. Set the confidence threshold τ for frame f k The target detection box with confidence threshold greater than τ is placed in D high In the set, for frame f k The target detection box with a confidence threshold smaller than τ is placed in D low In the set, set D high With set D low The union of is set to D all .

[0023] S2.2. Classify the trajectories according to the number of lost frames. The trajectories are divided into three categories according to the number of lost frames: Trajectories with 0 lost frames T (1) , Trajectory T with lost frames between 1 and 10 (2) , the trajectory T with the number of lost frames within 10-30 (3) .

[0024] S2.3. Use Kalman filtering to predict the position of the trajectory in the next frame of the image to help improve the accuracy of the multi-target tracking system.

[0025] S2.4. Calculate all trajectories at f k The prediction box in With frame f k The detection box in [B1,B2,B3,..,B n ]’s IOU distance C iou , cosine characteristic distance C appearance , confidence difference C conf , velocity direction difference C vel ;

[0026] Furthermore, the specific implementation steps of the first association in step S3 are:

[0027] S3.1. First time to track set T (1) and the detection box set D all To associate, we take the IOU distance C between the trajectory and the detection box iou , confidence difference C conf , velocity direction difference C vel The three and the cosine characteristic distance C appearance The final cost matrix is ​​obtained by weighting. The specific calculation formula is: λ1=0.75,λ2=0.25,the weight depends on the experimental results. The confidence difference of the target trajectory of normal motion is then calculated using the Hungarian algorithm. (1) The trajectory in D high Associating with the detection box in

[0028] S3.2. For successfully associated trajectories, Kalman filter updates are performed. For unsuccessfully associated trajectories, the number of lost frames is updated. For unsuccessfully associated detection frames, they are placed in set D. remain(1) middle.

[0029] Furthermore, the specific implementation steps of the second association in step S4 are:

[0030] S4.1. Second set of trajectories T (2) and the detection box set D remain(1)To associate, first convert the confidence difference C conf Mapped to the S(x) function, the function S(x) formula is: The function value S(C conf ) as the IOU distance C iou , velocity direction difference C vel The distance between the two and the cosine feature C appearance The weight between them is the IOU distance C iou , velocity direction difference C vel The distance between the two and the cosine feature C appearance After weighting, the second association cost matrix can be obtained. The specific calculation formula can be expressed as: in, Represents the confidence difference of the occluded target track.

[0031] S4.2. For the successfully associated trajectories, the Kalman filter is updated. For the unsuccessfully associated trajectories, the number of lost frames is updated. For the unsuccessfully associated detection boxes, they are put into the set D. remain(2) middle.

[0032] Furthermore, the specific implementation steps of the third association in step S5 are:

[0033] S5.1. The third time to track the set T (3) and the detection box set D remain(2) The number of lost frames m of the trajectory is multiplied by the coefficient α and then mapped to the function S(x). Experimental verification shows that when α is 0.2, the effect is optimal. The mapped function value S(m*α) is used as the IOU distance C iou , confidence difference C conf , velocity direction difference C vel The three and the cosine characteristic distance C appearance The weight between them is the IOU distance C iou , confidence difference C conf , velocity direction difference C vel The three and the cosine characteristic distance C appearance After weighting, the third correlation cost matrix can be obtained. This is the calculation formula for the third association cost matrix.

[0034] S5.2. For the successfully associated trajectories, the Kalman filter is updated. For the unsuccessfully associated trajectories, the number of lost frames is updated. For the unsuccessfully associated detection boxes, if their confidence is greater than the confidence threshold τ, they are initialized as new trajectories, otherwise they are regarded as background.

[0035] Furthermore, in step S6, in order to improve the discriminability of the target appearance features, the appearance features of all successfully associated trajectories need to be updated in real time. We designed a confidence-based appearance feature update module. For normal moving targets, the calculation formula of its appearance feature update strategy is: Where n is the number of consecutive successful associations of normal moving targets, emb n is the appearance feature vector of the currently matched detection frame, emb n-1 is the appearance feature vector of the detection frame that was last associated and matched by the normal moving target. is the updated trajectory appearance feature vector; for the target that was not successfully associated in the previous frame, if the current frame is successfully associated, the calculation formula of its appearance feature update strategy is: in is the updated trajectory appearance feature vector, is the appearance feature vector of the trajectory before updating, score is the confidence score of the detected target, emb n It is the appearance feature vector of the currently matched detection box.

[0036] Furthermore, in the last step S7, we need to update the Kalman filter parameters and trajectory appearance features of the trajectory after the association is completed. For the set T that is not successfully associated (3) If the track in the ,is not successfully associated within 30 frames, the track will be deleted.

[0037] Beneficial effects:

[0038] The video target tracking method provided by the present invention proposes a stage-based adaptive multi-target tracking method to solve the problem of association errors and track breakage caused by the target reappearing after being blocked for a long time or the target itself deforming. The method classifies the tracks according to the motion state, and uses different association strategies for different types of tracks according to their characteristics. Compared with the traditional multi-target tracker, the confidence difference and speed direction information are used as weak clues to assist the strong clues (position information, appearance information) in association, which further improves the tracking accuracy.

[0039] The appearance feature update module also uses different appearance feature update strategies according to different trajectory motion states. For targets in normal motion, we use the appearance features of adjacent frames as an important reference, select the appearance features of adjacent frames for fusion, and use them as the updated trajectory appearance feature vector. For targets in a lost state, the confidence of the associated detection frame is used as the update weight, and high-quality appearance features account for a larger proportion in the feature update to improve the discriminability of the appearance features. This module not only improves the tracking accuracy, but also meets the real-time running speed. The method of the present invention can quickly and accurately track any target appearing in the video sequence, improving the accuracy, rapidity and robustness of tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the overall method structure of the present invention;

[0041] Figure 2 It is a flow chart of the phase-by-phase related parts of the present invention;

[0042] Figure 3 It is a comparison chart of MOTA and IDF1 between the method of the present invention (SDC-Track) and other algorithms in the simulation test of MOT17 data set;

[0043] Figure 4 Comparison chart of MT and ML between the proposed method (SDC-Track) and other algorithms in the MOT17 dataset simulation experiment. DETAILED DESCRIPTION

[0044] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0001] An adaptive multi-target tracking method based on sub-trajectory includes a sub-trajectory association strategy and an appearance feature update strategy, and specifically includes the following steps S1 to S7. It should be noted that in the present invention, the trajectory set T (1) represents the normal motion trajectory, T (2) Indicates the target trajectory that is about to enter the occluded area, T (3) Represents the target trajectory that has just left the occluded area, and the detection box set D high represents a high confidence detection box, D low represents a low confidence detection box, D all represents the union of high confidence detection box and low confidence detection box, D remain(1) Denotes the detection box that was not successfully associated after the first association, D remain(2) Indicates the detection boxes that are not successfully associated after the second association.

[0002] like Figure 1-2 As shown, an adaptive multi-target tracking method based on sub-trajectory includes the following steps:

[0003] S1, each frame of the entire video sequence is sent to the YOLOX target detection network and appearance feature extraction network for target detection and appearance feature extraction, frame f k The detection box information of the target in [B1,B2,B3,..,B n ] and the target appearance feature vector [e1,e2,e3,..,e n ].

[0004] The target detection network described in step S1 is a YOLOX network, which uses Darknet-53 for feature extraction, adds an FPN feature pyramid structure for feature fusion, and uses an independent ReID model to extract the target appearance feature vector; specifically: each frame of the video is sent to the target detection network for feature extraction, and 3 prediction heads of different scales are obtained. The prediction head consists of several stacked convolutions, outputs the detection box position information, target confidence and target category information, and sends the target detection network result to the appearance feature extraction network to extract the target appearance feature vector.

[0005] S2. Preparation before association: classify the target detection frame and trajectory, and obtain the target predicted position through Kalman filtering. The specific steps are as follows:

[0006] S2.1. Set the confidence threshold τ for frame f k The target detection box with confidence threshold greater than τ is placed in D high In the set, for frame f k The target detection box with a confidence threshold less than τ is placed in D low In the set, D high With D low The union of is set to D all ;

[0007] S2.2. Classify the trajectories according to the number of lost frames, and regard the trajectories with 0 lost frames as normal motion trajectories T (1) , indicating that the target moves normally without occlusion; the trajectory with lost frames between 1 and 10 is regarded as the trajectory that is about to enter the occlusion area T (2) , indicating that part of the target area has been blocked and is gradually being completely blocked; the trajectory with lost frames between 11 and 30 is regarded as the trajectory that has just left the blocked area T (3) , indicating that the target is gradually leaving the occluded area, but may still be partially occluded; tracks with more than 30 lost frames are considered disappeared tracks, indicating that the target has disappeared from the video.

[0008] S2.3, use Kalman filtering to predict the position of the trajectory in the next frame of the picture, which helps improve the accuracy of the multi-target tracking system;

[0009] S2.4. Calculate all trajectories at f k The prediction box in With frame f k The detection box in [B1,B2,B3,..,θ n ]’s IOU distance C iou , confidence difference C conf , velocity direction difference C vel , cosine characteristic distance C appearance .

[0010] S3, perform the first association (trajectory set T (1) and the detection box set D all to associate).

[0011] S3.1, we take the IOU distance C between the trajectory and the detection box iou , confidence difference C conf , velocity direction difference C vel The three and the cosine characteristic distance C appearance The final cost matrix is ​​obtained by weighting, and the calculation formula is: Weight λ1=0.75,λ2=0.25,the weight depends on the experimental results. represents the confidence difference of the normal moving target trajectory, and then the Hungarian algorithm is used to calculate T (1) The trajectory in D high The detection boxes in are associated.

[0012] In the first association, the objects we target are normally moving targets, which are in a relatively ideal state. We sum the motion information, appearance information, and weak clues with fixed weights to obtain the cost matrix, and use the IOU (Intersection over Union) distance to represent the position information difference between targets, and the cosine feature distance to represent the appearance information difference between targets.

[0013] S3.2. For successfully associated trajectories, Kalman filter updates are performed. For unsuccessfully associated trajectories, the number of lost frames is updated. For unsuccessfully associated detection frames, they are placed in set D. remain(1) middle.

[0014] S4, perform the second association (trajectory set T (2) and the detection box set D remain(1) to associate).

[0015] S4.1. Second set of trajectories T (2) and the detection box set D remain(1) To associate, first convert the confidence difference C confMapped to the S(x) function, the function S(x) formula is: The function value S(C conf ) as the IOU distance C iou , velocity direction difference C vel The distance between the two and the cosine feature C appearance The weight between them is the IOU distance C iou , velocity direction difference C vel The distance between the two and the cosine feature C appearance After weighting, the second associated cost matrix can be obtained.

[0016] We fully consider the problem of appearance information distortion caused by slight occlusion. Once part of the target is occluded, the reliability of its appearance information in the association process will weaken as the occlusion range expands. Through observations of actual tracking situations, when the target is occluded, its confidence shows a linear downward trend. In view of this feature, we use the confidence difference based on linear prediction. The specific formula is:

[0017]

[0018]

[0019] in is the confidence of the target at frame t-1, is the confidence of the target at frame t-2, is the confidence prediction value when the target is in an occluded state in frame t, is the confidence difference when the target is in the occluded state in frame t (i.e., the confidence difference of the occluded target trajectory), Conf d Represents the confidence of the target in the detection box. We map the confidence difference to the S(x) function as an important weight factor in the second association cost matrix. As the confidence difference increases, the possibility of the target being occluded during movement also increases, resulting in a decrease in the importance of appearance information in the association process. In practical applications, the weights will change adaptively according to the situation to ensure that the proportion of different association information in the cost matrix reaches the optimal state. The final cost matrix calculation formula can be expressed as:

[0020]

[0021] S4.2. For the successfully associated trajectories, the Kalman filter is updated. For the unsuccessfully associated trajectories, the number of lost frames is updated. For the unsuccessfully associated detection boxes, they are put into the set D. remain(2) middle.

[0022] S5, perform the third association (trajectory set T (3) and the detection box set Dremain(2) ).

[0023] S5.1. For the trajectory set T (3) and the detection box set D remain(2) To associate, we multiply the number of trajectory loss frames m by the coefficient α and map it to the function S(x). Experimental verification shows that when α is 0.2, the effect is optimal. The mapped function value S(m*α) is used as the IOU distance C iou , confidence difference C conf , velocity direction difference C vel The three and the cosine characteristic distance C appearance The weight between them is the IOU distance C iou , confidence difference C conf , velocity direction difference C vel The three and the cosine characteristic distance C appearance After weighting, the third correlation cost matrix can be obtained.

[0024] We fully consider that the Kalman filter parameters cannot be updated in time due to long-term occlusion, resulting in inaccurate trajectory prediction results. To address this situation, our solution is to predict the accumulated error of the Kalman filter by judging the occlusion time of the target when it reappears after a long period of occlusion, and adaptively adjust the weight of the position information in the cost matrix during the association process. The final cost matrix calculation formula for the third association can be expressed as:

[0025]

[0026] S5.2. For the successfully associated trajectories, the Kalman filter is updated. For the unsuccessfully associated trajectories, the number of lost frames is updated. For the unsuccessfully associated detection boxes, if their confidence is greater than the confidence threshold τ, they are initialized as new trajectories, otherwise they are regarded as background.

[0027] S6. Use the appearance feature update module to update the appearance features of the trajectory successfully associated in steps S3-S5. If the trajectory is a normal moving target trajectory, the update strategy calculation formula is: Where n is the number of consecutive successful associations of normal moving targets, emb n is the appearance feature vector of the currently matched detection frame, emb n-1 is the appearance feature vector of the detection frame that was last associated and matched by the normal moving target. is the updated trajectory appearance feature vector. If the trajectory is a lost trajectory before successful association, the update strategy calculation formula is: in is the updated trajectory appearance feature vector, is the appearance feature vector of the trajectory before updating, score is the confidence score of the detected target, emb n It is the appearance feature vector of the currently matched detection box.

[0028] The appearance feature update module is designed to solve the problem of target appearance changes due to lighting, occlusion, camera angle, etc. during target tracking. In order to obtain reliable appearance information and maintain trajectory continuity, for normal moving target trajectories, we take into account that appearance features may be affected by external factors and change at any time. The target appearance features of adjacent frames can timely reflect the appearance status of the current target, so the appearance features of adjacent frames are used as an important reference. For targets that are successfully re-associated after being occluded, we take into account the historical appearance information of the target and perform a weighted sum based on the quality of the appearance features of each frame. This weighted summation method can make high-quality appearance features occupy a larger proportion in the update process. The appearance feature update module we proposed improves the discriminability of appearance features during the association process.

[0029] S7, detection box set D remain(2) If the confidence of the unassociated detection box in is greater than the confidence threshold τ, it is initialized as a new track, otherwise it is regarded as the background. (3) The tracks that were not successfully associated for 30 consecutive frames were removed;

[0030] The above S1-S2 is the target detection process, S3-S5 is the target association process, and steps S6-S7 are the subsequent processing process. The three are combined together to form a complete target tracking process. In the actual target tracking process, the entire target tracking is completed by repeating steps S3-S7. The bounding box information and appearance feature vector of the target tracking are obtained by step S1.

[0031] The effect of the present invention is verified by simulation experiments. The simulation experiments use the MOTchallenge17 dataset and compare with competitive open source methods in the field of target tracking.

[0032] Among them, ours refers to SDC-Track, i.e., the method of the present invention. The methods compared in the simulation experiment of the present invention include the following five methods:

[0033] 1. PermaTrack, reference [1]. Pavel Tokmakov, Jie Li, Wolfram Burgard, and Adrien Gaidon. Learning to track with object permanence. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 10860 - 10869, 2021.

[0034] 2. TransCenter, reference [2]. Yihong Xu, Yutong Ban, Guillaume Delorme, Chuang Gan, Daniela Rus, and Xavier Alameda - Pineda. Transcenter: Transformers with dense queries for multiple - object tracking. 2021.

[0035] 3. PixelGuide, reference [3]. Abhijeet Boragule, Hyunsung Jang, Namkoo Ha, and Moongu Jeon. Pixel - guided association for multi - object tracking. Sensors, 22(22):8922, 2022.

[0036] 4. NCT, reference [4]. Kai Zeng, Yujie You, Tao Shen, Qingwang Wang, Zhimin Tao, Zhifeng Wang, and Quanjun Liu. Nct: noise - control multi - object tracking. Complex & Intelligent Systems, 9(4):4331 - 4347, 2023.

[0037] 5. UTM, reference [5]. Sisi You, Hantao Yao, Bing-Kun Bao, and ChangshengXu. Utm: Aunified multiple object tracking model with identity-aware featureenhancement. In Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition, pages 21876-21886, 2023.

[0038] The simulation results are shown in the attached Figure 3 and attached Figure 4 , Figure 3 This is a comparison chart of the algorithm accuracy of this method and other competitive algorithms on the MOT17 dataset. Figure 3 The vertical axis represents the accuracy of the algorithm (MOTA). MOTA takes into account factors such as FP, FN, and IDswitch. It is calculated by normalizing the number of these factors to the total number of targets. FP refers to the number of backgrounds or other non-target objects that the tracker mistakenly marks as targets, FN refers to the number of real targets that the tracker fails to correctly mark as targets, and IDswitch refers to the number of target ID switches that occur during the tracking process. MOTA is used to measure the overall performance of the tracker. The horizontal axis reflects the target identity retention capability of the tracker (IDF1). The IDF1 indicator measures the accuracy and completeness of the tracker in correctly identifying the target identity. Figure 4 The ML metric on the horizontal axis indicates the proportion of targets that are lost most of the time during tracking. It measures the performance of the tracker in maintaining continuous visibility of the target. The MT metric on the vertical axis indicates the proportion of targets that are successfully tracked most of the time during tracking. Figure 3 and Figure 4 It can be seen that on the MOT17 dataset, the accuracy and precision of the proposed method (SDC-Track) are better than those of several other algorithms participating in the performance comparison.

[0039] The present invention proposes a phased association strategy, which can solve the problem of trajectory breakage caused by the target reappearing after being blocked for a long time, and the problem of association error caused by the decrease of target appearance discriminability in low light, blur, and bad weather. First, the trajectory is classified according to its characteristics: normal moving target trajectory, target movement trajectory about to enter the blocked area, target movement trajectory just leaving the blocked area, and removed target trajectory. A reasonable association order is arranged according to the characteristics of different trajectories. At the same time, the confidence difference is introduced, and the motion speed direction information is combined with the strong associated clues (target position information, appearance information). Different association strategies are designed for different trajectories to solve the problem of trajectory breakage caused by the target reappearing after being blocked for a long time to the greatest extent; an appearance feature update module is designed to improve the discriminability of the target's appearance features in bad environments, and different appearance feature update strategies are designed for normal movement trajectories and lost trajectories. The more reference appearance information improves the association accuracy, and also makes the method of the present invention more robust in complex environments. A comparative experiment was carried out on the MOTchallenge public dataset, which shows that the method of the present invention can still maintain trajectory continuity and tracking stability in complex scenes.

[0040] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A phase-based adaptive multi-target tracking method, characterized in that: The following steps are involved: S1, each frame in the entire video sequence is sent to the pre-trained target detection network and appearance feature extraction network for target detection and appearance feature extraction, and the frame f is obtained. k The detection box information of the target in [B1, B2, B3, .., B n ] and the target appearance feature vector [e1, e2, e3, .., e n ]; S2, classify the target detection frame and trajectory, and obtain the target predicted position through Kalman filtering; trajectory set T (1) represents the normal motion trajectory, T (2) Indicates the target trajectory that is about to enter the occluded area, T (3) Represents the target trajectory that has just left the occluded area, and the detection box set D high represents a high confidence detection box, D low represents a low confidence detection box, D all Represents the union of high confidence detection boxes and low confidence detection boxes; S3, for the trajectory set T (1) and the detection box set D all Make the first association; S4, for the trajectory set T (2) and the detection box set D remain(1) Perform the second association, D remain(1) Indicates the detection box that was not successfully associated after the first association; S5. For the trajectory set T (3) and the detection box set D remain(2) Perform the third association, D remain(2) Indicates the detection box that is still not successfully associated after the second association; S6, updating the trajectory appearance features successfully associated in steps S3-S5; S7, detection box set D remain(2) The unassociated detection boxes in are initialized as new trajectories. (3) The tracks that were not successfully associated for 30 consecutive frames were removed; Among them, the above steps S1-S2 are the target detection process, steps S3-S5 are the target association process, and steps S6-S7 are the subsequent processing process. By repeating steps S3-S7, a trajectory is gradually formed to complete the entire target tracking.

2. The method of adaptive multi-target tracking based on phases according to claim 1, characterized in that: The target detection network described in step S1 is a YOLOX network, which uses Darknet-53 for feature extraction, adds an FPN feature pyramid structure for feature fusion, and uses an independent ReID model to extract the target appearance feature vector.

3. The method of adaptive multi-target tracking based on phases according to claim 1, characterized in that: The specific implementation steps of step S2 are: S2.

1. Set the confidence threshold τ for frame f k The target detection box with confidence threshold greater than τ is placed in D high In the set, for frame f k The target detection box with a confidence threshold less than τ is placed in D low In the collection, set collection D high With set D low The union of is set D all ; S2.

2. Classify the trajectories according to the number of lost frames, and regard the trajectories with 0 lost frames as normal motion trajectories T (1) , the trajectory with lost frames between 1 and 10 is regarded as the trajectory that is about to enter the occlusion area T (2) , the trajectory with lost frames between 11 and 30 is regarded as the trajectory that just left the occluded area T (3) ; S2.3, use Kalman filtering to predict the position of the trajectory in the next frame of the image; S2.

4. Calculate all trajectories at f k The prediction box in With set D all The detection boxes in [B1, B2, B3, .., B n ]’s IOU distance C iou , confidence difference C conf , velocity direction difference C vel , and the cosine characteristic distance C appearance .

4. The method of adaptive multi-target tracking based on phases according to claim 3, characterized in that: In step S3, the trajectory set T (1) and the detection box set D all To associate, the specific steps are: S3.

1. The IOU distance C between the trajectory and the detection box iou , confidence difference C conf , velocity direction difference C vel The three and the cosine characteristic distance C appearance The first association cost matrix is ​​obtained by weighting, and the calculation formula is: Among them, weights λ1=0.75, λ2=0.25, represents the confidence difference of the normal moving target trajectory, and then the Hungarian algorithm is used to calculate T (1) The trajectory in D high Associating with the detection box in S3.

2. For the successfully associated trajectories, the Kalman filter is updated. For the unsuccessfully associated trajectories, the number of lost frames is updated. For the unsuccessfully associated detection boxes, they are put into the set D. remain(1) middle.

5. The method of adaptive multi-target tracking based on phases according to claim 4, characterized in that: In step S4, the trajectory set T (2) and the detection box set D remain(1) To associate, the specific steps are: S4.

1. First, the confidence difference C conf Mapped to the S(x) function, the function S(x) formula is: The function value S(C conf ) as the IOU distance C iou , velocity direction difference C vel The distance between the two and the cosine feature C appearance The weight between them is the IOU distance C iou , velocity direction difference C vel The distance between the two and the cosine feature C appearance After weighting, the second association cost matrix can be obtained. The specific calculation formula is expressed as: in Represents the confidence difference of the occluded target trajectory; S4.

2. For the successfully associated trajectories, the Kalman filter is updated. For the unsuccessfully associated trajectories, the number of lost frames is updated. For the unsuccessfully associated detection boxes, they are put into the set D. remain(2) middle.

6. The method of adaptive multi-target tracking based on phases according to claim 5, characterized in that: In step S5, the trajectory set T (3) and the detection box set D remain(2) To associate, the specific steps are: S5.

1. Multiply the number of lost frames m by the coefficient α and map it to the function S(x). The mapped function value S(m*α) is used as the IOU distance C iou , confidence difference C conf , velocity direction difference C vel The three and the cosine characteristic distance C appearance The weight between them is the IOU distance C iou , confidence difference C conf , velocity direction difference C vel The three and the cosine characteristic distance C appearance After weighting, the third association cost matrix can be obtained, and the calculation formula is: S5.

2. For the successfully associated trajectories, the Kalman filter is updated. For the unsuccessfully associated trajectories, the number of lost frames is updated. For the unsuccessfully associated detection boxes, if their confidence is greater than the confidence threshold τ, they are initialized as new trajectories, otherwise they are regarded as background.

7. The method of adaptive multi-target tracking based on phases according to claim 1, characterized in that: In step S6, the appearance features of all successfully associated trajectories are updated in real time using the confidence-based appearance feature update module: The calculation formula for the appearance feature update strategy for normal moving targets is: in is the updated trajectory appearance feature vector, n is the number of consecutive successful associations of normal moving targets, emb n is the appearance feature vector of the currently matched detection frame, emb n-1 It is the appearance feature vector of the detection frame that was last associated and matched by the normal moving target; For a target that was not successfully associated in the previous frame, if the current frame is successfully associated, the calculation formula for its appearance feature update strategy is: in is the updated trajectory appearance feature vector, is the appearance feature vector of the trajectory before updating, score is the confidence score of the detected target, emb n It is the appearance feature vector of the currently matched detection box.

8. The method for adaptive multi-target tracking based on phases according to claim 1, characterized in that: In step S7, the Kalman filter parameters and trajectory appearance features are updated for the associated trajectory, and the detection frame set D remain The unassociated detection boxes in (2) are initialized as new trajectories if their confidence is greater than the confidence threshold τ, otherwise they are considered as background. (3) If the track in the ,is not successfully associated within 30 frames, the track will be deleted.

Images