A multi-pedestrian tracking method and system with high-quality trajectory generation

Through pseudo-deep sparse matching and local weighted interpolation, the problems of multi-peer tracking in crowded scenarios are solved, and the trajectory quality and computational efficiency are improved.

CN118072352BActive Publication Date: 2024-11-01SHANDONG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410253044.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-06
Publication Date
2024-11-01
Estimated Expiration
2044-03-06

AI Technical Summary

Technical Problem

The existing multi-peer tracking method is prone to error matching in crowded scenarios, and the distorted pedestrian detection box damages the performance of the Kalman filter, and the Gaussian smooth interpolation calculation is high.

Method used

The current frame is divided into multiple regions by pseudo-depth sparse matching method, matching the existing trajectory and pedestrian detection box of the same region, avoiding the distortion box to update the Kalman filter parameters, and recovering the missing trajectory through local weighted interpolation.

Benefits of technology

Improves the correlation performance and trajectory quality of multi-peer tracking, reduces the computational complexity, and ensures the accuracy and tracking performance of the Kalman filter.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118072352B_ABST
    Figure CN118072352B_ABST
Patent Text Reader

Abstract

The present invention relates to a multi-pedestrian tracking method and system for high-quality trajectory generation, including: Step 1: perform pedestrian detection on the current frame image and extract ReID features; Step 2: associate and match the pedestrian boundary box newly detected on the current frame with the existing pedestrian trajectory through pseudo-depth sparse matching; Step 3: restore the missing trajectory of pedestrians on certain frames through local weighted interpolation, and finally obtain the pedestrian trajectory. The present invention follows the detection-based tracking paradigm, combines the pseudo-depth sparse matching algorithm to achieve data association, and adopts a non-update strategy for the Kalman filter parameters of the distorted frame that is successfully matched, thereby effectively improving the data association matching performance. In addition, by introducing the local weighted interpolation method, the loss detection is reduced, a high-quality trajectory is formed, and the overall tracking performance is significantly improved, which has important research and application value in the fields of video surveillance and intelligent transportation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and signal processing technology, and in particular relates to a multi-pedestrian tracking method and system for high-quality trajectory generation. Background Art

[0002] Multi-Person Tracking (MPT) is an important task in the field of computer vision. Its purpose is to detect, track and identify multiple pedestrians in a video sequence and record their movement trajectories in time and space, that is, the bounding boxes and corresponding ID numbers of all pedestrians in the video frame sequence. MPT can analyze the activity information of pedestrians and plays an important role in many application fields. For example, it helps to respond quickly to abnormal events in video surveillance systems, improve traffic conditions in intelligent transportation systems, improve shopping experience in business intelligence, and create more realistic virtual environments in the field of virtual reality. Therefore, the progress and optimization of MPT is of great significance to promoting the application and development of computer vision technology in many fields.

[0003] Detection-based tracking is one of the most mainstream paradigms for multi-pedestrian tracking. It decomposes the multi-pedestrian tracking task into two stages: pedestrian detection and pedestrian association. In this paradigm, pedestrian detection algorithms are first used to detect and locate pedestrians appearing in video frames, and then the movement of pedestrians between different frames is tracked by association algorithms. Specifically, detection-based tracking methods usually include key steps such as pedestrian detection, feature extraction, pedestrian association, and trajectory update. Pedestrian detection algorithms usually use convolutional neural network technology based on deep learning to analyze video frames to identify and locate pedestrian objects. Feature extraction usually uses pedestrian re-identification networks (Re-Identification, ReID) to extract feature information, such as the appearance, shape, color, etc. of pedestrians, for subsequent association and tracking. Pedestrian association algorithms can match and associate pedestrians in adjacent frames, usually involving technologies such as Kalman filtering and Hungarian algorithm. Trajectory update is to continuously update the position of pedestrians in video frames as video frames are updated, thereby achieving pedestrian tracking.

[0004] The inventors have found that the existing detection-based tracking method for multiple pedestrians has the following defects:

[0005] Existing pedestrian association algorithms usually match existing trajectories with pedestrian detections in the entire space of the current frame. This method is prone to mismatching in crowded scenes. At the same time, when matching existing trajectories, high-score detection frames are usually considered first, and then low-score detection frames, which may cause existing trajectories that should have matched low-score detection frames to mistakenly match a high-score false detection frame. During the association matching process, when pedestrians are occluded, the detection frame is prone to distortion, which is specifically manifested in the lower confidence of the occluded pedestrian detection frame, the smaller the size of the detection frame, and the position of the detection frame can only frame a part of the occluded pedestrian. Although the existing pedestrian association algorithm can match the distorted pedestrian detection frame with the existing trajectory, it ignores the problem that inaccurate observations will damage the performance of the Kalman filter, that is, using the distorted pedestrian detection frame to update the Kalman filter parameters may make the motion estimation of the Kalman filter inaccurate. In addition, the existing method of restoring the position of the occluded pedestrian in certain frames through Gaussian smoothing interpolation has a high computational complexity. Summary of the invention

[0006] In order to solve at least one technical problem existing in the above-mentioned background technology, the present invention provides a multi-pedestrian tracking method and system for high-quality trajectory generation, which adopts a detection-based tracking paradigm. The core is to realize pedestrian association by pseudo-depth sparse matching, divide the current frame into multiple regions in space, and match the existing trajectory and pedestrian detection in the same region. This method decomposes the existing trajectory and pedestrian detection in the entire space into multiple sparse subsets, and matches them according to pseudo-depth, which can better solve the problem of dense crowds. When matching in the same area, the present invention allows the existing trajectory to be matched with the high-score detection frame and the low-score detection frame at the same time, and finds the best match from the global perspective of the entire area, thereby improving the association performance. At the same time, the distorted detection frame on the successful association will not be used as an observation value to update the Kalman filter parameters, thereby ensuring the accuracy of the Kalman filter motion estimation. In addition, the originally missing trajectory of the pedestrian can be calculated at a lower computational cost through local weighted interpolation. Through the above optimization strategy, the generation of high-quality pedestrian trajectories is guaranteed, which is of great value to the research and application of MPT.

[0007] Terminology explanation:

[0008] 1. CrowdHuman dataset: CrowdHuman is a human behavior dataset for target detection, which contains a large number of human images and annotation information. This dataset is mainly used to train and evaluate the performance of computer vision models in detecting humans. The images in the CrowdHuman dataset cover a variety of scenes and complex crowded situations, making the data challenging. The annotation information of this dataset includes the bounding box of the human body, the key points of the human body, and some attribute labels, such as gender, age, etc. The CrowdHuman dataset plays an important role in the research of human detection, behavior analysis and other fields.

[0009] 2. CityPersons dataset: CityPersons is a dataset for pedestrian detection and pose estimation, which aims to promote the research and development of pedestrian-related tasks in the field of computer vision. The dataset contains a large number of urban scene images, covering a variety of different environments and situations, such as sidewalks, roads, intersections, etc. The images in the CityPersons dataset are densely annotated, providing the bounding box and pose key point information of each pedestrian, as well as some attribute labels such as gender, age, etc. It is an important benchmark dataset in the fields of pedestrian detection and pose estimation.

[0010] 3. ETH dataset: The ETH dataset is a dataset used for pedestrian behavior analysis and computer vision research. The dataset includes video sequences of pedestrians in several scenes. These videos capture the natural behaviors of pedestrians in a campus environment, such as walking, gathering, etc. This dataset is mainly used for pedestrian detection and trajectory prediction research.

[0011] 4. MOT17 dataset: MOT17 is a commonly used dataset in the field of multiple object tracking. It is one of the public datasets in the field of object tracking, which aims to promote the research and evaluation of multiple object tracking (MOT) algorithms. The MOT17 dataset consists of several different sequences, each of which captures the multi-object tracking task in different scenarios. The dataset provides high-resolution video sequences and accurate object bounding box annotations.

[0012] 5. MOT20 dataset: Compared with MOT17, MOT20 focuses specifically on crowd tracking in crowded scenes, and puts higher requirements on the performance of the algorithm. This dataset contains dense crowds in different scenarios (such as streets and squares), which are more challenging for detection and tracking algorithms because they involve problems such as occlusion, dynamic changes in groups, and interactions between individuals in dense crowds. The MOT20 dataset provides high-quality video sequences, including a large number of manually annotated pedestrian bounding boxes and tracking data. Researchers and developers of multi-target tracking algorithms can use this data to train, test, and evaluate their own algorithms and compare them with other algorithms.

[0013] 6. YOLOX model: YOLOX is a real-time target detection algorithm that can detect the bounding box of the target of interest in the image. The popular multi-target tracking method ByteTrack first trained the YOLOX model with the above dataset and obtained a YOLOX model with good pedestrian detection performance, which was widely adopted by many subsequent multi-target tracking methods. Following the same settings as many methods, this method also uses the YOLOX model as a pedestrian detector.

[0014] 7. SBS-50 model: The SBS-50 model is a pedestrian re-identification model that can determine whether pedestrians in different images are the same person. The popular multi-target tracking method BoTSORT first trained the SBS-50 model with the MOT17 and MOT20 datasets to obtain the SBS-50 model with good ReID feature extraction capabilities, which was widely adopted by many subsequent multi-target tracking methods. Following the same settings as many methods, this method also uses the SBS-50 model as the ReID feature extractor.

[0015] In order to achieve the above object, the present invention adopts the following technical solution:

[0016] A first aspect of the present invention provides a multi-pedestrian tracking method for high-quality trajectory generation, comprising:

[0017] Step 1: Detect pedestrians in the current frame and extract corresponding ReID features;

[0018] Step 2: Use pseudo-depth sparse matching to associate and match the most recently detected pedestrian bounding box on the current frame with the existing pedestrian trajectory;

[0019] Step 3: After all video frame sequences are processed, the missing trajectories of pedestrians in some frames are restored through local weighted interpolation to finally obtain the pedestrian trajectory.

[0020] Preferably, according to the present invention, in step 1, pedestrian detection is performed on the current frame image, wherein the pedestrian detection algorithm adopts the YOLOX model pre-trained on the CrowdHuman, CityPerson, ETH, MOT17 and MOT20 data sets, and the ReID feature extraction adopts the SBS-50 model pre-trained on the MOT17 and MOT20 data sets.

[0021] Preferably, in step 2, the pedestrian boundary box most recently detected on the current frame is associated and matched with the existing pedestrian trajectory by pseudo-depth sparse matching; comprising:

[0022] The distance from the bottom edge of the pedestrian bounding box to the bottom edge of the image is defined as pseudo depth; the pedestrian bounding box refers to a rectangular box marked in the image to indicate the position of the pedestrian;

[0023] First, calculate the distance between the top detection box and the bottom edge of the bottom detection box on the current frame;

[0024] Then, the pseudo depth is evenly divided into K intervals from small to large, corresponding to K pseudo depth levels respectively. All pedestrian detection frames on the current frame are divided into K sub-detection sets according to their own pseudo depths and the intervals they fall into. The first sub-detection set includes the bottom detection frame, and the last sub-detection set includes the top detection frame.

[0025] Divide the prediction boxes of all existing trajectories on the current frame into K sub-prediction sets according to their pseudo-depths and the intervals in which they fall. The first sub-prediction set includes the bottom detection box, and the last sub-prediction set includes the top detection box.

[0026] Finally, after obtaining K sub-detection sets and K sub-prediction sets on the current frame, the sub-detection sets and sub-prediction sets of the same pseudo-depth level are matched, and the detections or predictions that were not successfully matched at the previous pseudo-depth level will participate in the associated matching process of the next pseudo-depth level.

[0027] More preferably, K=3.

[0028] Preferably, according to the present invention, matching the sub-detection set and the sub-prediction set at the same pseudo-depth level comprises:

[0029] First, the M detections at the current pseudo-depth level are divided into G high-score detections and MG low-score detections;

[0030] Then, the similarity matrix between the G high-score detections and the N existing trajectories is calculated to form a similarity matrix Ch with dimension (G, N), as shown in the following formula:

[0031]

[0032]

[0033] in, is the similarity matrix C h The (i,j)th element of is the IOU (Intersection Over Union) distance between the i-th pedestrian detection box and the j-th existing trajectory prediction box, is the cosine distance between the ReID features of the i-th pedestrian detection box and the j-th existing trajectory prediction box, is the newly obtained ReID feature similarity matrix, θ iou is the IOU threshold, θ $%& is the ReID feature threshold, θ h is the maximum matching distance allowed for high-score detection;

[0034] Finally, the IOU similarity matrix between the MG low-score detections and the N existing trajectories is calculated to form an IOU similarity matrix C' with a dimension of (MG, N), as shown in the following formula:

[0035]

[0036] in, is the similarity matrix C l The (i, j)th element of , A is the low-score detection box, B is the prediction box of the existing trajectory on the current frame, θ l is the maximum matching distance allowed for low-score detection;

[0037] Get the similarity matrix C h and C l After that, the two are concatenated to obtain a similarity matrix with dimension (M, N), as shown in the following formula:

[0038] C=(C h ,αC l )

[0039] Among them, C is the concatenated similarity matrix, and α is the scaling factor;

[0040] After obtaining the similarity matrix C, the Hungarian algorithm is used to achieve pairwise matching between pedestrian detection and existing trajectories. This includes:

[0041] More preferably, θ iou =0.9,θ emb =0.25,θ h =0.65,θ l =0.19,α=3.42.

[0042] Preferably, according to the present invention, for each successfully matched pedestrian detection frame, determining whether the pedestrian detection frame is a distorted frame comprises:

[0043] The IOU value between the pedestrian detection frame and the predicted frame obtained by using the Kalman filter on the current frame using the existing trajectory is calculated. If the IOU is less than the threshold R, the pedestrian detection frame is considered to be a distorted frame, otherwise the pedestrian detection frame is considered to be not a distorted frame.

[0044] Further preferably, the IOU value of the pedestrian detection frame and the prediction frame obtained by using the Kalman filter on the current frame using the existing trajectory is calculated, and the calculation of the IOU value IOU (A, B) is shown in the following formula:

[0045]

[0046] Among them, A is the low-score detection box, and B is the predicted box of the existing trajectory in the current frame.

[0047] More preferably, R=0.4.

[0048] Preferably, according to the present invention, the missing trajectory of pedestrians in certain frames is restored by local weighted interpolation; comprising:

[0049] Model the i-th trajectory as shown below:

[0050] p t =f (i) (t)

[0051] Where t represents the frame ID number, p t is the position coordinate [x, y, w, h] on the tth frame, where x and y represent the x and y values ​​of the center point of the pedestrian bounding box, and w and h represent the width and height of the pedestrian bounding box;

[0052] Given the trajectory S after linear interpolation (i) , by fitting the function f (i) To complete the local weighted interpolation modeling task;

[0053] With point x as the center, from S (i) In the forward or backward direction, a segment of data with a length of frac is intercepted respectively, and a weighted linear regression is performed on the segment of data using the weight function w to obtain the regression line, which is recorded as is the center value of the regression line, where f is the corresponding value of the fitted curve. If there are n data points and n regression lines are drawn respectively, the line connecting the center values ​​of each regression line is f (i) The curve of

[0054] Among them, the weight function w adopts the cubic function w(x), as shown in the following formula:

[0055]

[0056] Among them, x′ represents the center point of the data, and D is the data set within the frac range.

[0057] Further preferably, the length of frac is preferably 0.07 times the trajectory length.

[0058] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a multi-pedestrian tracking method for generating high-quality trajectories when executing the computer program.

[0059] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a multi-pedestrian tracking method for generating high-quality trajectories.

[0060] A second aspect of the present invention provides a multi-pedestrian tracking system for high-quality trajectory generation, comprising:

[0061] The pedestrian detection and ReID feature extraction module is configured to: perform pedestrian detection on the current frame image and extract corresponding ReID features;

[0062] The association matching module is configured to: associate and match the pedestrian bounding box most recently detected on the current frame with the existing pedestrian trajectory by pseudo-depth sparse matching;

[0063] The pedestrian trajectory acquisition module is configured as follows: after all video frame sequences are processed, the missing trajectories of pedestrians in certain frames are restored by local weighted interpolation, and finally the pedestrian trajectory is acquired.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] 1. This invention follows the detection-based tracking paradigm, combines the pseudo-depth sparse matching algorithm to achieve data association, and adopts a non-update strategy for the Kalman filter parameters of the successfully matched distorted frame, thereby effectively improving the data association matching performance. In addition, by introducing a local weighted interpolation method, the loss detection is reduced, and a high-quality trajectory is formed. Ultimately, the overall tracking performance is significantly improved on the basis of the original architecture, which has important research and application value in the fields of video surveillance, intelligent transportation, etc.

[0066] 2. The present invention replaces the original data association method with pseudo-depth sparse matching. By dividing all pedestrian detection frames and existing trajectory prediction frames on the current frame into multiple sparse subsets and matching them in sequence according to the pseudo-depth level, the problem of dense crowds can be effectively alleviated. At the same time, when matching at each pseudo-depth level, the high-score detection frame is considered to match the low-score detection frame and the existing trajectory at the same time, ensuring that each trajectory can find the optimal match in the current area, thereby improving the association matching performance.

[0067] 3. The present invention adopts a non-update strategy to replace the original normal update strategy for the Kalman filter parameters of the successfully matched distortion detection frame, thereby avoiding the unreliable observation values ​​of the distortion detection frame from updating the Kalman filter parameters, thereby damaging the performance of the Kalman filter. This non-update method can effectively improve the accuracy of the Kalman filter motion estimation, and the redundant calculation cost is negligible, with good robustness and portability, which is of great significance to the research and application of MPT.

[0068] 4. The present invention uses local weighted interpolation to reduce the detection lost on some frames and restore the blank part of the original trajectory, which can improve the trajectory quality and significantly improve the overall tracking performance of the tracker. Compared with the existing popular Gaussian smoothing interpolation method, local weighted interpolation can reduce the running time by 80%, greatly improving the algorithm operation efficiency. At the same time, this interpolation method can be applied to any occasion where missing trajectories need to be restored, and can be used as a reference for other MOT methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 Schematic diagram of the pseudo-depth level division process;

[0070] Figure 2 Schematic diagram of multi-pedestrian tracking method for high-quality trajectory generation;

[0071] Figure 3 A schematic diagram of the scene where the distortion frame appears;

[0072] Figure 4 Schematic diagram of the local weighted interpolation effect. DETAILED DESCRIPTION

[0073] The present invention will be further defined below in conjunction with the accompanying drawings and embodiments, but is not limited thereto.

[0074] Example 1

[0075] A multi-pedestrian tracking method with high-quality trajectory generation, such as Figure 2 As shown, including:

[0076] Step 1: Detect pedestrians in the current frame and extract corresponding ReID features;

[0077] ReID features are deep features extracted by the pre-trained deep learning model SBS-50. These deep features are discriminative. The more similar the ReID features extracted from different images of the same person are, the more different they are. ReID features are essentially a representation of the appearance features of pedestrians, including clothing color, height, body shape, etc. The extraction of these features makes it possible to determine whether pedestrians appearing in different video frames are the same person in the subsequent association matching step.

[0078] Step 2: Use pseudo-depth sparse matching to associate and match the most recently detected pedestrian bounding box on the current frame with the existing pedestrian trajectory;

[0079] Step 3: After all video frame sequences are processed, the missing trajectories of pedestrians in some frames are restored through local weighted interpolation to finally obtain the pedestrian trajectory.

[0080] Example 2

[0081] The difference between the multi-pedestrian tracking method for high-quality trajectory generation described in Example 1 is that:

[0082] In step 1, pedestrian detection is performed on the current frame image. The pedestrian detection algorithm uses the YOLOX model pre-trained on the CrowdHuman, CityPerson, ETH, MOT17 and MOT20 datasets, and the image input resolution is 1400*800. ReID feature extraction uses the SBS-50 model pre-trained on the MOT17 and MOT20 datasets.

[0083] In step 2, the newly detected pedestrian bounding box on the current frame is associated and matched with the existing pedestrian trajectory by pseudo-depth sparse matching; including:

[0084] The distance from the bottom edge of the pedestrian bounding box to the bottom edge of the image is defined as pseudo depth; the closer the distance from the bottom edge of the bounding box to the bottom edge of the image, the smaller the pseudo depth, and vice versa. The pedestrian bounding box refers to a rectangular box marked in the image to indicate the position of the pedestrian; usually, in pedestrian detection tasks, pedestrian bounding boxes are used to mark the positions of pedestrians in the image. These pedestrian bounding boxes usually consist of four rectangular boundary lines, which frame the approximate position of the pedestrian. The specific expression is [x, y, w, h], where x and y represent the x and y values ​​of the center point of the pedestrian bounding box, and w and h represent the width and height of the pedestrian bounding box. In pedestrian detection tasks, the algorithm usually outputs the bounding boxes of all pedestrians in the image.

[0085] First, calculate the distance between the top detection box and the bottom edge of the bottom detection box in the current frame. In the current frame, in addition to the pedestrian detection box, there is also a pedestrian prediction box. The pedestrian prediction box is a prediction box on the current frame from the existing trajectory in the past frame through the Kalman filter.

[0086] Then, if Figure 1 As shown in the figure, the pseudo depth is evenly divided into K intervals from small to large, corresponding to K pseudo depth levels respectively. All pedestrian detection frames on the current frame are divided into K sub-detection sets according to their own pseudo depths and the intervals they fall into. The first sub-detection set includes the bottom detection frame, and the last sub-detection set includes the top detection frame.

[0087] Divide the prediction boxes of all existing trajectories on the current frame into K sub-prediction sets according to their pseudo-depths and the intervals in which they fall. The first sub-prediction set includes the bottom detection box, and the last sub-prediction set includes the top detection box.

[0088] Finally, after obtaining K sub-detection sets and K sub-prediction sets on the current frame, the sub-detection sets and sub-prediction sets at the same pseudo-depth level are matched, and the detections or predictions that were not successfully matched at the previous pseudo-depth level will participate in the associated matching process of the next pseudo-depth level. After obtaining K sub-detection sets and K sub-prediction sets on the current frame, a total of K rounds of matching are performed, corresponding to K pseudo-depths. Each round of matching is mainly for matching sub-detection sets and sub-prediction sets at the same pseudo-depth level. There are three types of results after the matching is completed: successfully matched detections and predictions, unsuccessfully matched detections, and unsuccessfully matched predictions. Among them, unsuccessfully matched detections will be merged into the detection set that will participate in the next round of matching, and unsuccessfully matched predictions will be merged into the prediction set that will participate in the next round of matching, so that unsuccessfully matched detections or predictions can participate in the next round of matching.

[0089] Since the camera is above the ground, the closer the bottom edge of the pedestrian bounding box is to the camera, the closer it is to the bottom of the image. The pedestrian is larger and clearer in the picture and easier to detect and match. The same is true in reverse. Therefore, the order of pseudo-depth levels for matching is from small to large pseudo-depth. In the implementation process, K = 3.

[0090] Compared with the currently common method of calculating the pairwise similarity between pedestrian detection and existing trajectories in the entire space of the current frame for matching, the method of dividing the space into multiple sparse subsets for matching can effectively alleviate the problem of false matching that is prone to occur in densely crowded scenes with pedestrians.

[0091] The existing method often divides all detection frames on the current frame into high-score detection frames and low-score detection frames for matching respectively, and preferentially matches existing tracks with high-score detection frames, and matches existing tracks that have not been successfully matched with low-score detection frames. However, due to crowding, occlusion, etc., on the one hand, the score of the pedestrian detection frame may become low, and at the same time, high-score false detection frames may appear around, resulting in the low-score detection frame that should have been successfully matched with the existing track being matched with the high-score false detection frame, resulting in an incorrect match. When matching the sub-detection set and the sub-prediction set at the same pseudo-depth level, the present invention simultaneously considers the high-score matching frame and the low-score matching frame of the pedestrian, and realizes the global angle search for the optimal match at the pseudo-depth level, thereby improving the associated matching performance.

[0092] Match the sub-detection set with the sub-prediction set at the same pseudo-depth level, including:

[0093] First, the M detections at the current pseudo-depth level are divided into G high-score detections and MG low-score detections;

[0094] Then, calculate the similarity matrix between the G high-score detections and the N existing trajectories to form a similarity matrix C with dimension (G, N) h , as shown below:

[0095]

[0096]

[0097] in, is the similarity matrix C h The (i,j)th element of is the IOU (Intersection Over Union) distance between the i-th pedestrian detection box and the j-th existing trajectory prediction box, is the cosine distance between the ReID features of the i-th pedestrian detection box and the j-th existing trajectory prediction box, is the newly obtained ReID feature similarity matrix, θ iou is the IOU threshold, θ emb is the ReID feature threshold, θ h is the maximum matching distance allowed for high-score detection;

[0098] Finally, the IOU similarity matrix between the MG low-score detections and the N existing trajectories is calculated to form an IOU similarity matrix C with a dimension of (MG, N) l , as shown below:

[0099]

[0100] in, is the similarity matrix Cl The (i, j)th element of , A is the low-score detection box, B is the prediction box of the existing trajectory on the current frame, θ l is the maximum matching distance allowed for low-score detection;

[0101] Get the similarity matrix C h and C l After that, the two are concatenated to obtain a similarity matrix with dimension (M, N), as shown in the following formula:

[0102] C=(C h ,αC l )

[0103] Among them, C is the concatenated similarity matrix, and α is the scaling factor;

[0104] After obtaining the similarity matrix C, the Hungarian algorithm is used to achieve pairwise matching between pedestrian detection and existing trajectories. This includes:

[0105] Specifically, the basic principle of the Hungarian algorithm is to solve the optimal pairing problem by finding the best match in the graph. In multi-pedestrian tracking, the detected pedestrians are usually represented as a set of nodes, the existing trajectories are represented as another set of nodes, and the weights of the edges are used to represent the degree of association or matching cost between the nodes, that is, the similarity matrix C. Then, the Hungarian algorithm will find an optimal matching solution in this weighted bipartite graph to minimize the total matching cost.

[0106] 1) Construct a weighted bipartite graph: The detected pedestrians are represented as a set of nodes, the existing trajectories are represented as another set of nodes, and the matching cost between each pair of nodes (usually based on distance, appearance feature similarity, etc.) is calculated as the weight of the edge, that is, the similarity matrix C;

[0107] 2) Initialize matching schemes: Select an initial set of matching schemes from the graph.

[0108] 3) Continuously optimize the matching scheme: Iteratively try to adjust the matching scheme to reduce the total matching cost. This usually involves searching and expanding alternate paths.

[0109] 4) Find the optimal match: Through iterative optimization, find the matching solution that minimizes the total matching cost.

[0110] The advantage of the Hungarian algorithm is that it can find the optimal solution in polynomial time and has high efficiency and reliability. Therefore, it is widely used to achieve the pairwise matching of pedestrian detection and existing trajectories. Through the Hungarian algorithm, the detected pedestrians can be effectively associated with the existing trajectories, thereby achieving the pedestrian tracking task.

[0111] θiou =0.9,θ emb =0.25,θ h =0.65,θ l =0.19,α=3.42.

[0112] Experiments on the MOT17 validation set demonstrate the effectiveness of the pseudo-deep sparse matching method, and evaluate the tracking performance using MOTA, IDF1, and HOTA indicators. MOTA is calculated based on the number of FP, FN, and ID switches, and focuses more on detection performance. In contrast, IDF1 better measures the consistency of ID matching. HOTA is a comprehensive measure for evaluating the overall effectiveness of detection and association. The experiments are shown in Table 1:

[0113] Table 1

[0114] Method ReID Sparse Combine IDF1 MOTA HOTA ByteTrack × × × 79.5 76.4 68.09 BotTrack √ × × 80.9 77.1 68.86 Baseline(SparseTrack) × √ × 81.2 76.7 69.06 ReID+Combine √ × √ 81.8 76.9 69.16 Ours(ReID+Sparse+Combine) √ √ √ 82.2 76.9 69.4

[0115] Among them, ReID represents whether to use ReID information, Sparse represents whether to use the method of dividing sparse subsets, and Combine represents whether to consider both high-score frames and low-score frames. ByteTack, BotTrack and SparseTrack are the matching methods of existing popular methods. The present invention uses the matching method of SparseTrack as the baseline, and combines ReID information and the matching method of considering both high-score frames and low-score frames at the same time. Compared with the original Bytetrack matching method that does not use ReID, Sparse and Combine, it improves IDF1 by 2.7%, MOTA by 0.5% and HOTA by 1.31%. Compared with the Baseline, all performance indicators are improved. The experimental results show the effectiveness of ReID information. At the same time, the Sparse method can better alleviate the problem of crowd congestion, and the Combine method can find the global optimal match from the same area, thereby improving the associated matching performance.

[0116] For each successfully matched pedestrian detection frame, determine whether the pedestrian detection frame is a distorted frame; Figure 3 As shown, including:

[0117] Calculate the IOU value of the pedestrian detection frame and the prediction frame obtained by the Kalman filter on the current frame using the existing trajectory. If the IOU is less than the threshold R, it is considered that the pedestrian detection frame may be a distorted frame caused by occlusion and other reasons. At this time, the present invention does not use the pedestrian detection frame as an observation value to update the parameters of the Kalman filter to avoid inaccurate observations that damage the performance of the Kalman filter. Experiments show that this non-update strategy can effectively improve the performance of the tracker compared to the original update strategy. Otherwise, it is considered that the pedestrian detection frame is not a distorted frame.

[0118] Calculate the IOU value of the pedestrian detection frame and the prediction frame obtained by the Kalman filter on the current frame using the existing trajectory. The calculation of the IOU value IOU (A, B) is shown in the following formula:

[0119]

[0120] Among them, A is the low-score detection box, and B is the predicted box of the existing trajectory in the current frame.

[0121] R=0.4.

[0122] This part of the experiment uses the results obtained by pseudo-depth sparse matching as the baseline. On this basis, the strategy of not updating the Kalman filter parameters for the distortion frame is added. The experimental results are shown in Table 2:

[0123] Table 2

[0124] Method IDF1 MOTA HOTA Baseline 82.2 76.9 69.06 Nupdate1 82.6 77.8 69.49 Nupdate2 82.9 77.9 69.61

[0125] Among them, Nupdate1 represents the Kalman filter parameter non-update strategy for all distorted frames, and Nupdate2 represents the Kalman filter parameter non-update strategy for only high-score detection distorted frames. The experimental results show that the Notupdate2 method can improve 0.7% IDF1, 1% MOTA and 0.6% HOTA, which shows the effectiveness of this strategy and improves the tracker performance. According to the experiment, it can also be observed that the Not update2 method is more effective than the Not update1 method. This is because for low-score detection frames, although there is a certain degree of distortion, the effective information it carries is more important, so at this time, the parameter update of the Kalman filter is better than no update.

[0126] Restore the missing tracks of pedestrians in some frames through local weighted interpolation; including:

[0127] Model the i-th trajectory as shown below:

[0128] p 4 =f (i) (t)

[0129] Where t represents the frame ID number, p t is the position coordinate [x, y, w, h] on the tth frame, where x and y represent the x and y values ​​of the center point of the pedestrian bounding box, and w and h represent the width and height of the pedestrian bounding box;

[0130] Given the trajectory S after linear interpolation (i) , by fitting the function f (i) To complete the local weighted interpolation modeling task;

[0131] With point x as the center, from S (i)In the forward or backward direction, a segment of data with a length of frac is intercepted respectively, and a weighted linear regression is performed on the segment of data using the weight function w to obtain the regression line, which is recorded as is the center value of the regression line, where f is the corresponding value of the fitted curve. If there are n data points and n regression lines are drawn respectively, the line connecting the center values ​​of each regression line is f (i) The curve of

[0132] Among them, the weight function w adopts the cubic function w(x), as shown in the following formula:

[0133]

[0134] Among them, x′ represents the center point of the data, and D is the data set within the frac range.

[0135] The length of frac is preferably 0.07 times the trajectory length.

[0136] This part of the experiment uses the performance index results of the existing popular method SparseTrack as the baseline, and compares the linear interpolation LI, Gaussian smoothing interpolation GSI and local weighted interpolation LSI methods. The experimental results are shown in Table 3:

[0137] Table 3

[0138] Method IDF1 MOTA HOTA Time Baseline 81.2 76.7 69.0 LI 82.5 78.9 70.4 0.57 GSI 82.6 79.1 70.6 11.33 LSI 82.7 79.1 71.0 2.23

[0139] Experimental results show that local weighted interpolation can effectively improve the quality of trajectories generated by existing methods, improving IDF1 by 1.5%, MOTA by 2.4%, and HOTA by 2% compared to the baseline. While achieving comparable performance to the existing popular interpolation method GSI, the computational time is saved by about 80%, achieving a good balance between performance indicators and computational time. This is because the Gaussian process regression process needs to calculate the covariance matrix between data points, which has a higher computational complexity.

[0140] Finally, the three strategies proposed in this invention are implemented simultaneously with SparseTrack as the baseline. The ablation experiment results on the MOT17 validation set are shown in Table 4:

[0141] Table 4

[0142] Method Match Nupdate LSI IDF1 MOTA HOTA Baseline(SparseTrack) × × × 81.2 76.7 69.06 Baseline+Match √ × × 82.2 76.9 69.4 Baseline+Match+Nupdate √ √ × 82.9 77.9 69.61 Baseline+Match+Nupdate+LSI √ √ √ 84.2 80.2 71.66

[0143] Among them, Match represents the pseudo-depth sparse matching method, Update represents the distortion box Kalman filter parameter non-update strategy, LSI represents the local weighted interpolation method. The experimental results show that when the three methods are introduced at the same time, the present invention improves the MOTA by 3%, MOTA by 3.5% and HOTA by 2.6% compared with the Baseline, which effectively improves the overall tracking performance of the tracker.

[0144] Example 3

[0145] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the multi-pedestrian tracking method for generating high-quality trajectories described in embodiment 1 or 2 are implemented.

[0146] Example 4

[0147] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the multi-pedestrian tracking method for high-quality trajectory generation described in embodiment 1 or 2.

[0148] Example 5

[0149] A multi-pedestrian tracking system with high-quality trajectory generation, including:

[0150] The pedestrian detection and ReID feature extraction module is configured to: perform pedestrian detection on the current frame image and extract corresponding ReID features;

[0151] The association matching module is configured to: associate and match the pedestrian bounding box most recently detected on the current frame with the existing pedestrian trajectory by pseudo-depth sparse matching;

[0152] The pedestrian trajectory acquisition module is configured as follows: after all video frame sequences are processed, the missing trajectories of pedestrians in certain frames are restored by local weighted interpolation, and finally the pedestrian trajectory is acquired.

Claims

1. A multi-pedestrian tracking method with high-quality trajectory generation, characterized in that: include: Step 1: Detect pedestrians in the current frame and extract corresponding ReID features; Step 2: Use pseudo-depth sparse matching to associate and match the most recently detected pedestrian bounding box on the current frame with the existing pedestrian trajectory; Step 3: After all video frame sequences are processed, the missing trajectories of pedestrians in some frames are restored through local weighted interpolation to finally obtain the pedestrian trajectory; In step 2, the newly detected pedestrian bounding box on the current frame is associated and matched with the existing pedestrian trajectory through pseudo-depth sparse matching; include: The distance from the bottom edge of the pedestrian bounding box to the bottom edge of the image is defined as pseudo depth; the pedestrian bounding box refers to a rectangular box marked in the image to indicate the position of the pedestrian; First, calculate the distance between the top detection box and the bottom edge of the bottom detection box on the current frame; Then, the pseudo depth is evenly divided into K intervals from small to large, corresponding to K pseudo depth levels respectively. All pedestrian detection frames on the current frame are divided into K sub-detection sets according to their own pseudo depths and the intervals they fall into. The first sub-detection set includes the bottom detection frame, and the last sub-detection set includes the top detection frame. Divide the prediction boxes of all existing trajectories on the current frame into K sub-prediction sets according to their pseudo-depths and the intervals in which they fall. The first sub-prediction set includes the bottom detection box, and the last sub-prediction set includes the top detection box. Finally, after obtaining K sub-detection sets and K sub-prediction sets on the current frame, the sub-detection sets and sub-prediction sets of the same pseudo-depth level are matched, and the detection or prediction that was not successfully matched at the previous pseudo-depth level will participate in the association matching process of the next pseudo-depth level; Match the sub-detection set with the sub-prediction set at the same pseudo-depth level, including: First, the M detections at the current pseudo-depth level are divided into G high-score detections and MG low-score detections; Then, calculate the similarity matrix between the G high-score detections and the N existing trajectories to form a similarity matrix C with dimension (G, N) h , as shown below: in, is the similarity matrix C h The (i,j)th element of is the IOU distance between the i-th pedestrian detection box and the j-th existing trajectory prediction box, is the cosine distance between the ReID features of the i-th pedestrian detection box and the j-th existing trajectory prediction box, is the newly obtained ReID feature similarity matrix, θ iou is the IOU threshold, θemb is the ReID feature threshold, θ h is the maximum matching distance allowed for high-score detection; Finally, the IOU similarity matrix between the MG low-score detections and the N existing trajectories is calculated to form an IOU similarity matrix C' with a dimension of (MG, N), as shown in the following formula: in, is the (i, j)th element of the similarity matrix C', A is the low-score detection box, B is the prediction box of the existing trajectory in the current frame, θ l is the maximum matching distance allowed for low-score detection; Get the similarity matrix C h and C l After that, the two are concatenated to obtain a similarity matrix with dimension (M, N), as shown in the following formula: C=(C h ,αC l ) Among them, C is the concatenated similarity matrix, and α is the scaling factor; After obtaining the similarity matrix C, the Hungarian algorithm is used to achieve pairwise matching between pedestrian detection and existing trajectories.

2. The method for tracking multiple pedestrians with high-quality trajectory generation according to claim 1, characterized in that: In step 1, pedestrian detection is performed on the current frame image. The pedestrian detection algorithm uses the YOLOX model pre-trained on the CrowdHuman, CityPerson, ETH, MOT17 and MOT20 datasets, and the ReID feature extraction uses the SBS-50 model pre-trained on the MOT17 and MOT20 datasets.

3. The method for tracking multiple pedestrians with high-quality trajectory generation according to claim 1, characterized in that: K=3。 4. The method for tracking multiple pedestrians with high-quality trajectory generation according to claim 3, characterized in that: i iou =0.9,θemb=0.25,θ h =0.65,θ l =0.19, α=3.

42.

5. The method for tracking multiple pedestrians with high-quality trajectory generation according to claim 1, characterized in that: For each successfully matched pedestrian detection frame, determine whether the pedestrian detection frame is a distorted frame; including: The IOU value between the pedestrian detection frame and the predicted frame obtained by using the Kalman filter on the current frame using the existing trajectory is calculated. If the IOU is less than the threshold R, the pedestrian detection frame is considered to be a distorted frame, otherwise the pedestrian detection frame is considered to be not a distorted frame.

6. The method for tracking multiple pedestrians with high-quality trajectory generation according to claim 5, characterized in that: Calculate the IOU value of the pedestrian detection frame and the prediction frame obtained by the Kalman filter on the current frame using the existing trajectory. The calculation of the IOU value IOU (A, B) is shown in the following formula: Among them, A is the low-score detection box, and B is the prediction box of the existing trajectory on the current frame; R=0.4。 7. The method for tracking multiple pedestrians with high-quality trajectory generation according to claim 1, characterized in that: Restore the missing tracks of pedestrians in some frames through local weighted interpolation; including: Model the i-th trajectory as shown below: pt=f (i) (t) Where t represents the frame ID number, p4 is the position coordinate [x, y, w, h] on the tth frame, x and y represent the x and y values ​​of the center point of the pedestrian bounding box, and w and h represent the width and height of the pedestrian bounding box; Given the trajectory S after linear interpolation (i) , by fitting the function f (i) To complete the local weighted interpolation modeling task; With point x as the center, from S (i) In the forward or backward direction, a segment of data with a length of frac is intercepted respectively, and a weighted linear regression is performed on the segment of data using the weight function w to obtain the regression line, which is recorded as is the center value of the regression line, where f is the corresponding value of the fitted curve. If there are n data points and n regression lines are drawn respectively, the line connecting the center values ​​of each regression line is f (i) The curve of Among them, the weight function w adopts the cubic function w(x), as shown in the following formula: Among them, x' represents the data center point, and D is the data set within the frac range; The length of frac is 0.07 times the trajectory length.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the multi-pedestrian tracking method for generating high-quality trajectories described in any one of claims 1-7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-pedestrian tracking method for generating high-quality trajectories described in any one of claims 1 to 7 are implemented.

10. A multi-pedestrian tracking system with high-quality trajectory generation, characterized in that: include: The pedestrian detection and ReID feature extraction module is configured to: perform pedestrian detection on the current frame image and extract corresponding ReID features; The association matching module is configured to: associate and match the pedestrian bounding box most recently detected on the current frame with the existing pedestrian trajectory by pseudo-depth sparse matching; The pedestrian trajectory acquisition module is configured to: after all video frame sequences are processed, restore the missing trajectories of pedestrians in some frames through local weighted interpolation, and finally obtain the pedestrian trajectory; The newly detected pedestrian bounding box on the current frame is associated and matched with the existing pedestrian trajectory through pseudo-depth sparse matching; include: The distance from the bottom edge of the pedestrian bounding box to the bottom edge of the image is defined as pseudo depth; the pedestrian bounding box refers to a rectangular box marked in the image to indicate the position of the pedestrian; First, calculate the distance between the top detection box and the bottom edge of the bottom detection box on the current frame; Then, the pseudo depth is evenly divided into K intervals from small to large, corresponding to K pseudo depth levels respectively. All pedestrian detection frames on the current frame are divided into K sub-detection sets according to their own pseudo depths and the intervals they fall into. The first sub-detection set includes the bottom detection frame, and the last sub-detection set includes the top detection frame. Divide the prediction boxes of all existing trajectories on the current frame into K sub-prediction sets according to their pseudo-depths and the intervals in which they fall. The first sub-prediction set includes the bottom detection box, and the last sub-prediction set includes the top detection box. Finally, after obtaining K sub-detection sets and K sub-prediction sets on the current frame, the sub-detection sets and sub-prediction sets of the same pseudo-depth level are matched, and the detection or prediction that was not successfully matched at the previous pseudo-depth level will participate in the association matching process of the next pseudo-depth level; Match the sub-detection set with the sub-prediction set at the same pseudo-depth level, including: First, the M detections at the current pseudo-depth level are divided into G high-score detections and MG low-score detections; Then, calculate the similarity matrix between the G high-score detections and the N existing trajectories to form a similarity matrix C with dimension (G, N) h , as shown below: in, is the similarity matrix C h The (i,j)th element of is the IOU distance between the i-th pedestrian detection box and the j-th existing trajectory prediction box, is the cosine distance between the ReID features of the i-th pedestrian detection box and the j-th existing trajectory prediction box, is the newly obtained ReID feature similarity matrix, θ iou is the IOU threshold, θemb is the ReID feature threshold, θ h is the maximum matching distance allowed for high-score detection; Finally, the IOU similarity matrix between the MG low-score detections and the N existing trajectories is calculated to form an IOU similarity matrix C' with a dimension of (MG, N), as shown in the following formula: in, is the similarity matrix C l The (i, j)th element of , A is the low-score detection box, B is the prediction box of the existing trajectory on the current frame, θ l is the maximum matching distance allowed for low-score detection; Get the similarity matrix C h and C l After that, the two are concatenated to obtain a similarity matrix with dimension (M, N), as shown in the following formula: C=(C h ,αC l ) Among them, C is the concatenated similarity matrix, and α is the scaling factor; After obtaining the similarity matrix C, the Hungarian algorithm is used to achieve pairwise matching between pedestrian detection and existing trajectories.