A cascaded drive multi-object tracking method and system

Abstract

The present invention provides a cascaded drive multi-object tracking method and system, belonging to the technical field of object tracking, to solve the problem that existing mainstream multi-object tracking algorithms usually use the appearance features of objects to assist filtering algorithms to complete tracking tasks. When the appearance of an object changes, it will lead to the problem of different identities for the same object, resulting in the failure of the tracking task. In the present invention, a lower body appearance feature matching module and a skeleton feature matching module are first proposed. Through the calculated relevant feature cost values, cascaded drive is performed in the direction from the original matching module to the lower body feature matching module and then to the skeleton feature matching module, weakening the utilization of the whole body appearance features.

Description

Technical Field

[0001] The present invention belongs to the technical field of target tracking, and particularly relates to a cascade-driven multi-target tracking method and system. Background Technique

[0002] Multi-target tracking algorithms are mainly dominated by the detection-tracking paradigm. This paradigm performs frame-by-frame detection and transforms the multi-target tracking task into a data association task. Mainstream tracking algorithms can achieve good results in the MOT challenge, but most of them rely too much on appearance features. If the appearance of the target changes when it disappears in the camera view, tracking failure will occur. And if the target repeatedly enters and exits the camera view, the error of the predicted value of the filtering algorithm will also increase, which will also lead to tracking failure.

[0003] Most traditional multi-target tracking algorithms are based on filtering algorithms. The MOSSE algorithm is the opening work of correlation filtering tracking, proposed by Bolme D S et al. MOSSE first constructs a response and then uses a two-dimensional Gaussian distribution to find the position with the maximum target response. Henriques J F et al. extended ridge regression, an approximate dense sampling method based on circular shift, and kernel methods to address the problem of sample redundancy caused by sparse sampling in the MOSSE algorithm, and proposed the CSK algorithm. Alex Bewley et al. proposed the SORT algorithm using the Kalman filtering algorithm and the Hungarian algorithm, which is a very simple, effective and practical multi-target tracking algorithm. Filter-based tracking algorithms have great advantages in speed, but they are extremely prone to ID conversion when the object moves fast and the tracking background is complex, and these algorithms do not utilize the appearance features of the target and cannot accurately confirm the target identity, resulting in tracking failure.

[0004] Nicolai Wojke et al. introduced a CNN network to extract target appearance features based on the SORT algorithm and proposed the DeepSORT algorithm, and assigned different weights to the appearance features and the filtering prediction results in the cascade matching. Yifu Zhang et al. proposed the FairMOT algorithm for the association problem between the target detection box and the extraction of ReID feature vectors. Yunhao Du et al. improved DeepSORT from multiple aspects such as target detection, feature embedding, and trajectory association, and proposed the StrongSORT algorithm. Although these algorithms can well maintain the consistency of the target identity by using appearance features and can complete conventional MOT tasks, most of them rely too much on the appearance features of the target. Once the appearance of the target changes when it disappears in the camera view, the matching error between the motion features of the filtering algorithm and the appearance features of the target will be large, resulting in the failure of the target tracking task.

[0005] Existing multi-object tracking algorithms have achieved good results in target localization. Mainstream multi-object tracking algorithms focus on the problem of how to maintain the consistency of target identities. However, these algorithms usually use the appearance features of targets to assist the filtering algorithm to complete the tracking task. When the appearance of the target changes, it will lead to the problem of different identities for the same target, resulting in the failure of the tracking task. Summary of the Invention

[0006] In view of this, the present invention provides a cascaded drive multi-object tracking method and system to solve the problem that existing multi-object tracking algorithms have achieved good results in target localization, and mainstream multi-object tracking algorithms focus on the problem of how to maintain the consistency of target identities. However, these algorithms usually use the appearance features of targets to assist the filtering algorithm to complete the tracking task. When the appearance of the target changes, it will lead to the problem of different identities for the same target, resulting in the failure of the tracking task.

[0007] The technical solution adopted by the present invention is as follows:

[0008] A cascaded drive multi-object tracking method includes:

[0009] Step 1: Extract the target to be detected based on the YOLOv7 detector, use the Kalman filter to establish an object corresponding to the target to be detected, prepare for the next prediction, then match the trajectory of the target to be detected based on IOU, and store the target to be detected in the feature repository to prepare for the localization and identity confirmation of the target in subsequent frames;

[0010] Step 2: Input the subsequent consecutive video frames to be processed into the YOLOv7 detector, and then input the current targets detected by the YOLOv7 detector into the filter, OSnet, and Alphapose respectively to extract motion features and appearance features;

[0011] In Step 2, the OSnet is used to extract the appearance features of the target, and the Alphapose is used to extract the skeleton features of the target.

[0012] Step 3: Input the motion features and appearance features of the current target obtained in Step 2 into the matching system, and the matching system compares and matches the motion features and appearance features of the current target with the motion features and appearance features of the target to be detected in the feature repository; the comparison and matching include lower body appearance feature matching, skeleton feature matching, original cascade matching, and IOU matching.

[0013] In Step 3, the process of the comparison and matching specifically includes the following steps:

[0014] Calculate the combined cost value of the motion feature and the full-body appearance feature. When the combined cost value of the motion feature and the full-body appearance feature is lower than the set threshold, the object detection result is obtained based on the Hungarian algorithm. When the combined cost value of the motion feature and the full-body appearance feature is higher than the set threshold, enter the lower-body appearance feature matching module for lower-body appearance feature matching; calculate the cost value of the lower-body appearance feature matching module through the following formula:

[0015] cost = γ * (1 - half) + (1 - γ)ori

[0016] where cost is the new cost value, γ can be considered as the adjustment weight, ori is the original cost value, and half is the value of the cosine similarity of the lower body appearance of the target in two adjacent frames.

[0017] When the cost value of the lower-body appearance feature matching module is lower than the set threshold, the object detection result is obtained based on the Hungarian algorithm. When the cost value of the lower-body appearance feature matching module is higher than the set threshold, enter the skeleton feature matching module for skeleton matching to match the targets that have not been successfully matched before, and send the remaining targets to the IOU matching module. The skeleton matching mainly includes comparing the cosine similarity between the current target skeleton and the skeletons of the targets to be detected in the repository.

[0018] The skeleton matching result is calculated through the following formula:

[0019]

[0020] where n represents the current frame, cost represents the calculated cost value, i represents the i-th frame, and p represents the skeleton feature.

[0021] Step 4: Save each feature of each current target to the feature repository, and update frame by frame to continue storing the current best features.

[0022] A cascaded drive multi-object tracking system, comprising:

[0023] The first module: Extract the targets to be detected based on the YOLOv7 detector, initialize the detected targets using Kalman filtering to prepare for parameter update for the next Kalman filtering prediction, and then assign an initial trajectory to each detected target through IOU matching to prepare for target localization in subsequent video frames;

[0024] Second Module: Input the subsequent consecutive video frames to be processed into the algorithm, perform object detection through the YOLOv7 detector, first send the detected objects into the Kalman filter to extract motion features, make predictions and update parameters, and then send the detected objects into OSnet and Alphapose to extract the appearance features and skeleton information of the objects, and save the extracted motion features, appearance features and skeleton features into the feature library;

[0025] Third Module: Input the motion features, appearance features and skeleton features of the current object obtained in the second module into the matching system for feature matching;

[0026] The matching system first matches the objects through the original cascaded matching. When the appearance of the upper body of the object changes, the features of the lower body of the object are matched;

[0027] When the appearance of the whole body of the object changes, the skeleton features of the object are matched;

[0028] In the third module, the matching system performs feature matching through the following method:

[0029] The matching system first calculates the combined cost value of the motion features and the whole body appearance features. When the combined cost value of the motion features and the whole body appearance features is lower than the set threshold, the object detection result is obtained based on the Hungarian algorithm. When the combined cost value of the motion features and the whole body appearance features is higher than the set threshold, it enters the lower body appearance feature matching module to perform lower body appearance feature matching;

[0030] When the cost value of the lower body appearance feature matching module is lower than the set threshold, the object detection result is obtained based on the Hungarian algorithm. When the cost value of the lower body appearance feature matching module is higher than the set threshold, it enters the skeleton feature matching module to perform skeleton matching to match the objects that have not been successfully matched before, and send the remaining objects into the IOU matching module;

[0031] Fourth Module: If there are still unmatched objects after the matching steps in the third module, input these unmatched objects into the IOU matching, establish a new matching trajectory, then continue to store the best-performing features during the matching process, and update frame by frame.

[0032] In summary, due to the adoption of the above technical solutions, the beneficial effects of the present invention are:

[0033] 1. In the present invention, the lower body appearance feature matching module and the skeleton feature matching module are first proposed. Through the calculated relevant feature cost values, the cascaded drive is carried out in the direction from the original matching module to the lower body feature matching module, and then to the skeleton feature matching module, weakening the utilization of the whole body appearance features;

[0034] 2. In the present invention, by replacing the detector with YOLOv7 and using OSnet as the re-identification network, the quality of the target appearance features is improved, and a long-term memory mechanism is designed to save the best features in the motion, appearance, and skeleton features of the target for a longer time, which has better robustness in the process of trajectory matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention will be described by way of examples with reference to the accompanying drawings, where:

[0036] Figure 1 is the overall structure diagram of the present invention;

[0037] Figure 2 is the simplified structure diagram of YOLOv7 of the present invention;

[0038] Figure 3 is the simplified structure diagram of OSnet of the present invention;

[0039] Figure 4 are targets with similar lower body appearances of the present invention;

[0040] Figure 5 is the simplified structure diagram of Alphapose of the present invention;

[0041] Figure 6 is the driving schematic diagram of the lower body appearance information matching module of the present invention;

[0042] Figure 7 is the simplified flowchart of trajectory matching of the present invention;

[0043] Figure 8 is the schematic diagram of skeleton information extraction of the present invention;

[0044] Figure 9 is the schematic diagram of the long-term memory mechanism of the present invention;

[0045] Figure 10 are some pictures of the self-made dataset of the present invention;

[0046] Figure 11 is the structural comparison diagram of single target visualization of the present invention;

[0047] Figure 12 is the algorithm verification diagram of the present invention;

[0048] Figure 13 is the schematic diagram of tracking failure when the lighting condition is poor of the present invention;

[0049] Figure 14 is the result display diagram after introducing skeleton information of the present invention;

[0050] Figure 15 is the continuous frame display diagram of the demonstration result of the present invention. Detailed implementation manners

[0051] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are only a part rather than all of the embodiments of the present invention. Usually, the components of the embodiments of the present invention described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations.

[0052] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0053] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0054] It should be noted that similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0055] In the present invention, unless otherwise clearly defined and limited, the first feature being "on" or "under" the second feature may include the direct contact between the first and second features, or may include the indirect contact between the first and second features through additional features therebetween. Moreover, the first feature being "above", "over" and "on top of" the second feature includes that the first feature is directly above and obliquely above the second feature, or merely means that the horizontal height of the first feature is higher than that of the second feature. The first feature being "under", "below" and "beneath" the second feature includes that the first feature is directly below and obliquely below the second feature, or merely means that the horizontal height of the first feature is lower than that of the second feature.

[0056] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0057] Embodiment 1

[0058] A cascade drive type multi-target tracking method is disclosed in an embodiment of the present invention, including:

[0059] Step 1: When the first frame is input into the algorithm, neither OSnet nor Alphapose is started, that is, the appearance features and skeleton features of the target are not extracted. First, the YOLOv7 detector is started to extract the targets to be detected, and Kalman filtering is used to establish the corresponding objects for the next prediction. Then, IOU matching is used, that is, the target trajectories are assigned through the intersection over union ratio to prepare for the subsequent frame to locate and identify the targets.

[0060] Step 2: The matching process of subsequent video frames is as shown by the black solid line in Figure 1 . First, the subsequent consecutive video frames to be processed are input into the detector, and then the targets detected by the detector are respectively input into the filter, OSnet, and Alphapose for feature extraction, as shown by the purple dashed box part in Figure 1 .

[0061] In this paper, the detector is replaced with YOLOv7 to provide more accurate target boxes for subsequent feature extraction, and the appearance features of the target are extracted through OSnet, showing better performance when comparing similar targets. In this paper, Alphapose is used to extract the skeleton features of the target to prepare for the skeleton feature matching module.

[0062] Since StrongSORT still follows the detection-tracking paradigm, a better tracker can improve the accuracy of the tracking task. Compared with the original YOLOX used, YOLOv7 can process more image frames under the same hardware conditions, thus meeting the requirements of real-time detection. And YOLOv7 supports multi-scale detection and can better detect targets farther from the camera. The schematic diagram of its network structure is as shown in Figure 2 .

[0063] OSnet can dynamically fuse multi-scale features, and this performance enables it to better identify similar targets. The network structure is as shown in Figure 3 .

[0064] Since the appearance similarity of pedestrians is relatively large, and the lower body appearance features are introduced in the cascade matching, and the lower body appearances of pedestrians are even more similar. As shown in Figure 4 , the red boxes frame two pedestrians with different identities. However, when only looking at their lower bodies, their appearance features are very similar. Therefore, replacing the re-identification network with OSnet can better enable the lower body appearance features to play a role in the matching and distinguish the identities of targets with similar lower body appearances.

[0065] And we extract the target bone key points through Alphapose, connect them into the complete skeleton of the target, and perform upsampling, and then use them as the skeleton features. The schematic diagram of its network structure is as shown in Figure 5 .

[0066] Step 3: Input the motion features and appearance features of the current target obtained in Step 2 into the matching system, and the matching system compares and matches the motion features and appearance features of the current target with those of the target to be detected in the feature repository; the comparison and matching include lower body appearance feature matching, skeleton feature matching, original cascade matching, and IOU matching.

[0067] After the target features are extracted, they will be input into the cascade matching. The lower body appearance feature matching module proposed in this paper is placed after the original cascade matching, as Figure 6 shown.

[0068] In Figure 6 , what is outside the red box is the original cascade matching. In this paper, the lower body appearance feature matching is placed at the position of the red dotted box and is driven by the cost value calculated at each level. In this paper, the lower body appearance features of the target are extracted separately, multiplied by a weight, and added to the original cascade matching, as shown in Formula 1.

[0069] cost = γ * (1 - half) + (1 - γ)ori Formula 1

[0070] Among them, cost is the new cost value, γ can be considered as the adjusted weight, ori is the original cost value, and half is the value of the appearance cosine similarity of the lower body of the target between two adjacent frames. By allocating the proportion of ori and half through γ, when ori has a large deviation, it will drive the lower body appearance feature matching module to calculate and obtain the new cost value. At this time, the proportion of ori is weakened, while the proportion of half is strengthened. In this way, while using the original features, new features are introduced, strengthening the robustness of the utilization of appearance features. Finally, if the cost value is less than the corresponding threshold, the cost matrix will be passed into the Hungarian algorithm for trajectory matching. Otherwise, a larger cost value will drive the skeleton feature matching module to receive the unmatched targets.

[0071] When the cost value calculated by the lower body appearance feature matching module is large, it means that neither relying on the full body appearance features nor the lower body appearance features is sufficient to complete the trajectory matching task. At this time, the appearance of the target has changed greatly, so the skeleton feature matching module will be driven. The skeleton feature matching module proposed in this paper is placed between the lower body appearance feature matching module and the IOU matching, and its flow diagram is as Figure 7 shown.

[0072] In this paper, the target skeleton features are extracted through Alphapose, and then the skeletons of each pedestrian are upsampled and stored in a separate repository. When the skeleton feature matching module is driven, the target skeleton of the current frame will be compared with the skeletons in the repository for cosine similarity. As Figure 8,Table 1 and Table 2, this paper selects three consecutive frames of two different targets to verify the feasibility of the matching module. It can be seen that the skeleton features of different pedestrians are different.

[0073] Table 1. Comparison of cosine similarity of the same pedestrians: P1-1 represents the first frame of the first person, and the rest are similar.

[0074]

[0075] Table 2 Comparison of pre-similarity of different pedestrians

[0076]

[0077] Tables 1 and 2 show that for consecutive frames of the same target walking, the cosine similarity values are all greater than or equal to 0.80. However, when comparing different targets, the cosine similarity values never exceed 0.8, demonstrating that skeletal feature matching is feasible. Equation 2 shows how skeletal feature matching is calculated.

[0078]

[0079] Where n represents the current frame, cost represents the calculated cost value, i represents the i-th frame, and p represents the skeleton feature.

[0080] The skeleton feature matching module proposed in this paper does not utilize the appearance features of the target, but uses the target skeleton features to fill the gap of matching only with appearance features. This module can be used to perform trajectory matching when the target appearance changes significantly.

[0081] Step 4: Save each feature of each current target into the feature repository, and update the best feature frame by frame to continue storing it.

[0082] The long memory mechanism proposed in this paper is to improve the accuracy of cascade matching when the target repeatedly enters and exits the camera. Figure 9 shown.

[0083] exist Figure 9 In the figure, the bars represent the storage of features from frames n-100 to n in the feature library. The feature with the highest cost in the cascade matching calculation is replaced with the feature at position n-99 to prevent it from being filtered out during feature updates. Position n-99 is updated for each frame. This operation allows the most costly features to be preserved for a longer period of time. If the target enters and exits the camera from different spatial locations, or repeatedly, the camera angle may be different. However, using the best-performing feature from the previous step can reduce the problem of high cost during the matching process caused by angle issues, thereby improving trajectory matching efficiency.

[0084] Experiment and Analysis

[0085] The experimental platform of this study is the Linux-Ubuntu 20.04 operating system with an NVIDIA GeForce GTX 3060 GPU of 12GB, and the CPU is an Intel i5 13400F. Four models were trained: YOLOv7 was trained using the WilderPerson data; the OSnet network was trained using the Market1501 data; 240 images were selected from the Market1501 dataset to crop the lower body and the OSnet was used for transfer training for the lower body matching in cascade matching; Alphapose was trained using the COCO2017 dataset for skeleton feature matching. The test tracking dataset is a self-made dataset.

[0086] Since the current datasets used for multi-object tracking tests, such as the MOT16, MOT17, and MOT20 datasets, are not specifically for the problems in this paper, a self-made dataset was created for the problems proposed in this paper. This dataset is made in the MOT17 format and is divided into 7 short videos with a video frame rate of 30 frames. It includes 1 video of 50s, 1 video of 40s, and 5 videos of 30s. And two of the videos were taken using a mobile platform, and five were taken using a fixed camera. In the seven datasets, two videos were for testing fairness without shooting the situation where the target repeatedly enters and exits the camera or the appearance of the target changes after disappearing. The dataset proposed in this paper has simulated factory scenes, hospital scenes, park scenes, and there is also a square with a large number of pedestrians, as Figure 10 shown.

[0087] Generally speaking, the self-made data in this paper includes indoor and outdoor scenes, shooting with a mobile platform and a fixed camera, and is not all for the problems proposed in this paper, which can be used for more comprehensive and fair test comparison.

[0088] First, a single target was verified. As Figure 11 (a) shows, it is the original StrongSORT. After the same target took off the black long coat, its ID changed from 1 to 2; Figure 11 (b) is the algorithm proposed in this paper, and the target ID always remains 1. It can be seen that for a single target, the algorithm in this paper has effectively solved the problems proposed in this paper.

[0089] Secondly, the multi-object situation was verified, as Figure 12 shown. Figure 12(a) shows the tracking results of the original StrongSORT. After the same target took off its gray top, its ID changed from 3 to 5, resulting in the failure of the multi-object tracking task due to appearance changes. For the algorithm proposed in this paper, as Figure 12 shown in (b), after the target with ID 3 walked out of the camera, took off its gray coat, and then returned to the camera, the ID of the target remained 3, indicating that the pedestrian's identity did not change. The algorithm proposed in this paper successfully solved this problem.

[0090] It can be seen by comparison that the method proposed in this paper can effectively solve the problems of targets repeatedly entering and exiting the camera and appearance changes after the targets disappear.

[0091] Since StrongSORT introduced target appearance features for matching in order to reduce the number of ID switches, but at night, especially when the lighting conditions are poor, the target appearance features are easily affected, resulting in tracking failure, as Figure 13 shown.

[0092] Through Figure 13 it can be seen that the initially detected targets with IDs 1 and 3, after overlapping, due to poor lighting conditions, the appearance features were affected, and ID 3 was wrongly assigned to the target that was originally ID 1, while the target that was originally ID 3 was not tracked anymore.

[0093] However, the algorithm in this paper solved the problem of over-reliance on appearance features by introducing skeleton feature matching. When the targets with IDs 1 and 3 overlapped, the targets could still be well tracked, as Figure 14 shown.

[0094] To better demonstrate this process, consecutive frames were intercepted for the targets with IDs 1 and 3 and shown, as Figure 15 shown.

[0095] Figure 15 In it, the numbers in the blue circles represent the order. From the start, to occlusion, and then to the end of occlusion of the two targets, it can be seen that the target IDs did not change, indicating that in this paper, the skeleton features weakened the influence of appearance features and enhanced the tracking performance.

[0096] First, it was tested on the MOT17 dataset. The main comparison metrics were MOTA, HOTA, IDF1, DetA, and IDs. Among them, except that the smaller the IDs, the better, the larger the others, the better. The final test results are shown in Table 3.

[0097] Table 3 MOT17 Test Results

[0098]

[0099] As can be seen from Table 3, although the method proposed in this paper does not achieve the highest scores in the MOTA and HOTA metrics, it also has little difference from the mainstream tracking algorithms. The method proposed in this paper can obtain better scores in IDF1 and DetA, and has the lowest IDs, indicating that the proposed method has better robustness in tracking performance.

[0100] The ablation experiment for the MOT17 dataset is shown in Table 4. Among them, Ours1 represents StrongSORT + OSne + YOLOv7t, Ours2 represents the introduction of target partial appearance feature matching on the basis of Ours1, and Ours3 represents the algorithm proposed in this paper.

[0101] Table 4 Ablation experiment for the MOT17 dataset

[0102]

[0103] Second, this paper conducts tests on the self-made dataset, and the test results are shown in Table 5.

[0104] Table 5 Test results of the self-made dataset

[0105]

[0106] Moreover, this paper conducts ablation experiments on the self-made data, and the test results are shown in Table 6.

[0107] Table 6 Ablation experiment for the self-made dataset

[0108]

[0109] As can be seen from the data in the table, the method proposed in this paper performs the best in several metrics. And for the sake of fairness, there are two videos in the self-made dataset that do not introduce the problems targeted by this paper, and still have good results. Generally speaking, the algorithm proposed in this paper can effectively solve the problems of targets repeatedly entering and leaving the camera and the appearance change of targets after disappearance.

[0110] Conclusion

[0111] The algorithm proposed in this paper focuses on solving the situation where targets repeatedly enter and leave the camera in the multi-object tracking task, or the appearance change of targets after disappearing in the camera. It introduces the lower-body appearance feature and the skeleton feature, and through cascade driving, improves the problem of tracking failure caused by the original algorithm's over-reliance on the whole-body appearance. And it introduces a long-term memory mechanism to keep the best features for a longer time. It has practical significance for the current vision-based security system. However, this algorithm can only be applied in the visible light environment, and night is an important event segment for security, so subsequent research will focus on the night scene and strive to construct an all-weather model.

[0112] Example 2

[0113] This embodiment proposes a cascaded drive multi - target tracking system, including:

[0114] The first module: Extract the target to be detected based on the YOLOv7 detector, initialize the detected target using Kalman filtering to prepare for the parameter update of the next Kalman filtering prediction, and then assign an initial trajectory to each detected target through IOU matching to prepare for the positioning of the target in subsequent video frames;

[0115] The second module: Input the subsequent consecutive video frames to be processed into the algorithm, perform target detection through the YOLOv7 detector, first send the detected target into the Kalman filter to extract motion features, perform prediction and parameter update, and then send the detected target into OSnet and Alphapose to extract the appearance features and skeleton information of the target, and save the extracted motion features, appearance features and skeleton features into the feature library;

[0116] The third module: Input the motion features, appearance features and skeleton features of the current target obtained by the second module into the matching system for feature matching;

[0117] The matching system first matches the target through the original cascaded matching. When the upper - body appearance of the target changes, the lower - body features of the target are matched;

[0118] When the whole - body appearance of the target changes, the skeleton features of the target are matched;

[0119] In the third module, the matching system performs feature matching through the following method:

[0120] The matching system first calculates the joint cost value of the motion features and the whole - body appearance features. When the joint cost value of the motion features and the whole - body appearance features is lower than the set threshold, the target detection result is obtained based on the Hungarian algorithm. When the joint cost value of the motion features and the whole - body appearance features is higher than the set threshold, it enters the lower - body appearance feature matching module for lower - body appearance feature matching;

[0121] When the cost value of the lower - body appearance feature matching module is lower than the set threshold, the target detection result is obtained based on the Hungarian algorithm. When the cost value of the lower - body appearance feature matching module is higher than the set threshold, it enters the skeleton feature matching module for skeleton matching to match the targets that have not been successfully matched before, and sends the remaining targets into the IOU matching module;

[0122] Fourth module: If there are still unmatched targets after the matching step of the third module, these unmatched targets will be input into the IOU matching to establish new matching trajectories, and then the features with the best performance during the matching process will be continuously stored and updated frame by frame.

[0123] The circuits, electronic components and modules involved are all prior arts and can be fully implemented by those skilled in the art without further elaboration. The content protected by the present invention does not involve improvements to software and methods either.

[0124] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other.

[0125] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A cascaded driving multi-target tracking method, characterized in that Including: Step 1: Extract the target to be detected based on the YOLOv7 detector, initialize the detected target using Kalman filtering to prepare for parameter update of the next Kalman filtering prediction, and assign an initial trajectory to each detected target through IOU matching to prepare for target localization in subsequent video frames; Step 2: Input the subsequent consecutive video frames to be processed into the algorithm, perform target detection through the YOLOv7 detector, first send the detected target into the Kalman filter to extract motion features, perform prediction and parameter update, and then send the detected target into OSnet and Alphapose to extract the appearance features and skeleton information of the target, and save the extracted motion features, appearance features and skeleton features into the feature library; Step 3: Input the motion features, appearance features and skeleton features of the current target obtained in Step 2 into the matching system for feature matching; The matching system first matches the target through the original cascade matching. When the upper body appearance of the target changes, the lower body features of the target are matched; When the whole body appearance of the target changes, the skeleton features of the target are matched; In Step 3, the matching system performs feature matching through the following method: The matching system first calculates the combined cost value of the motion feature and the whole body appearance feature. When the combined cost value of the motion feature and the whole body appearance feature is lower than the set threshold, the target detection result is obtained based on the Hungarian algorithm. When the combined cost value of the motion feature and the whole body appearance feature is higher than the set threshold, it enters the lower body appearance feature matching module for lower body appearance feature matching; When the cost value of the lower body appearance feature matching module is lower than the set threshold, the target detection result is obtained based on the Hungarian algorithm. When the cost value of the lower body appearance feature matching module is higher than the set threshold, it enters the skeleton feature matching module for skeleton matching to match the targets that have not been successfully matched before, and sends the remaining targets into the IOU matching module; Step 4: If there are still unmatched targets after the matching step in Step 3, input these unmatched targets into IOU matching to establish new matching trajectories, then continue to store the features in the matching process, and update frame by frame.

2. The cascade-driven multi-object tracking method according to claim 1, wherein In Step 2, the Osnet extracts the appearance features of the target through the following method: Step A1: Extract the appearance features of the target at different scales through a convolutional neural network, as shown in the following formula: Among them represents the finally obtained feature vector, AG represents the adaptive aggregation gate calculation, r represents the scale to be calculated, R represents the upper limit of the scale size, and F r represents the appearance feature with the scale size of r; Step A2: Fuse the appearance features at different scales through a channel adaptive aggregation gate.

3. The cascade drive type multi-target tracking method according to claim 1, characterized in that In Step 2, the Alphapose extracts the skeleton information of the target through the following method: Step B1: Calculate the corresponding feature vector of the skeleton through Faster-CNN, and then use symmetric integration to calculate the heat map of the skeleton key points, as shown in the following formula: hm = F(obj) In the formula, hm represents the heat map of the skeleton key points calculated, F represents the whole process of calculating the heat map, and obj represents the target input into Alphapose; Step B2: Use Pose NMS to eliminate redundant skeletal key points, and finally obtain the target skeleton, as shown in the following formula: P n = NMS(hm) n = 1...17, In the formula: Pn represents the nth skeletal key point processed by Pose NMS, NMS represents Pose NMS, a total of 17 skeletal key points are used, and the value of n ranges from 1 to 17.

4. A cascade-driven multi-object tracking method according to claim 1, characterized in that In Step 3, calculate the cost value of the lower body appearance feature matching module through the following formula: cost = γ * (1 - half) + (1 - γ)ori where cost is the new cost value, γ is the adjustment weight, ori is the original cost value, and half is the value of the appearance cosine similarity of the target lower body between two adjacent frames.

5. The cascade drive type multi-target tracking method according to claim 1, characterized in that The matching of the target skeleton features mainly includes: Step C1: Upsample the original skeleton map through upsampling, and the calculation formula is as follows: S = Ori ∩ R x , (and ) where S is the skeleton after upsampling, Ori is the original skeleton map, and Rx is the proportional coefficient of the upsampling effect; Step C2: Then extract its features through CNN and ELAN, and then compare them through pre-similarity, and calculate through the following formula: where n represents the current frame, cosθ represents the calculated cost value, i represents the ith frame, and p represents the skeleton feature.

6. A cascade-driven multi-target tracking system, characterized in that To implement a cascaded drive multi-target tracking method described in claims 1-5, including: The first module: Extract the target to be detected based on the YOLOv7 detector, use Kalman filtering to initialize the detected target, prepare for the parameter update of the next Kalman filtering prediction, and then assign an initial trajectory to each detected target through IOU matching to prepare for the positioning of the target in subsequent video frames; The second module: Input the subsequent continuous video frames to be processed into the algorithm, perform target detection through the YOLOv7 detector, first send the detected target into the Kalman filter to extract motion features, perform prediction and parameter update, and then input the detected target into OSnet and Alphapose to extract the appearance features and skeleton information of the target, and save the extracted motion features, appearance features and skeleton features into the feature library; The third module: Input the motion features, appearance features and skeleton features of the current target obtained in the second module into the matching system for feature matching; The matching system first performs matching on the target through the original cascaded matching. When the appearance of the target upper body changes, the lower body features of the target are matched; When the appearance of the target whole body changes, the target skeleton features are matched; In the third module, the matching system performs feature matching through the following method: The matching system first calculates the combined cost value of the motion feature and the whole body appearance feature. When the combined cost value of the motion feature and the whole body appearance feature is lower than the set threshold, the target detection result is obtained based on the Hungarian algorithm. When the combined cost value of the motion feature and the whole body appearance feature is higher than the set threshold, it enters the lower body appearance feature matching module for lower body appearance feature matching; When the cost value of the lower body appearance feature matching module is lower than the set threshold, the target detection result is obtained based on the Hungarian algorithm. When the cost value of the lower body appearance feature matching module is higher than the set threshold, the skeleton feature matching module is entered for skeleton matching to match the targets that were not successfully matched before, and the remaining targets are sent to the IOU matching module; Fourth module: If there are still unmatched targets after the matching steps of the third module, these unmatched targets are input into the IOU matching to establish new matching trajectories, and then the best-performing features during the matching process are continuously stored and updated frame by frame.

Images