A multi-target tracking algorithm based on spatial attributes

Abstract

This invention provides a multi-target tracking algorithm based on spatial attributes, belonging to the field of computer vision technology. It solves the problem of poor tracking performance in complex situations. The technical solution includes the following steps: S1: Strengthening the positional correlation between the same target by adding the positional features of the target from the previous frame to the current frame to improve anti-occlusion capability; S2: Enhancing detection capability with spatial information by proposing global and local positional feature extraction modules in the detector part to obtain detailed spatial positional information between different targets; S3: Proposing a new calculation standard in the target matching part of the tracker, utilizing the spatial relationship between targets to make the matching method between intersecting targets more accurate; S4: Matching the calculated target anchor boxes and combining them with a re-identification model to obtain the final tracking result. The beneficial effects of this invention are: This method strengthens spatial information and makes it more adaptable to targets with complex movements, resulting in more stable tracking.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a multi-target tracking algorithm based on spatial attributes. Background Technology

[0002] Multi-target tracking is a popular research area in computer vision. It involves identifying and locating independent moving targets in consecutive video frames and then correlating this information to determine the target's trajectory. Multi-target tracking has significant applications in surveillance and security. Since pedestrians are frequently the targets to be detected and tracked in videos, pedestrian multi-target detection has become a mainstream approach in this field.

[0003] With the maturation of target detection technology, target tracking has primarily shifted to detecting targets and performing correlation matching between adjacent frames. As detector performance improves, information matching based solely on detection boxes can achieve good tracking results. Target correlation between consecutive frames is a crucial step in target tracking. Since many tracking algorithms combine detection and feature extraction, most focus on processing detection boxes when calculating speed. However, the real-time performance of a tracking algorithm should consider both the target detection and feature extraction algorithms; relying solely on detection box pairing does not represent the overall tracking algorithm's computational speed. To address this issue with concatenated algorithms, a model combining target detection and feature extraction is proposed. This model simultaneously outputs target location information and pedestrian feature information for direct correlation in the next step. By combining these two tasks, it reduces the time spent extracting pedestrian features separately, significantly reducing the overall tracking runtime. Based on this model structure, Fairmot, based on an anchor-box-less detector, is proposed. They argue that anchor-box-based detectors increase network ambiguity during pedestrian feature extraction, leading to increased learning difficulty. Integrating an anchor-free detector into the model balances accuracy and speed, resulting in better performance. The detector's performance plays a crucial role in this type of model. However, current detection and tracking algorithms have not yet been further improved for complex situations. When similar or frequently overlapping targets appear, issues such as target switching may arise. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-target tracking algorithm based on spatial attributes, which solves the problem of poor tracking performance in complex situations. In the detector, the extraction of target position information is enhanced from both global and local perspectives, thereby improving detection accuracy. At the same time, the feature association of the same target is strengthened by reusing the target information of the previous frame, which improves accuracy and reduces the number of target switching. In the tracker, a distance calculation method between matching targets is proposed. This method enhances spatial information and makes it more adaptable to targets with complex motion, resulting in more stable tracking.

[0005] To achieve the aforementioned objectives, the present invention employs the following technical solution: a multi-target tracking algorithm based on spatial attributes, comprising the following steps:

[0006] Step S1: Strengthen the positional association between the same target by adding the positional features of the target from the previous frame to the current frame to improve the anti-occlusion capability;

[0007] Step S2: Enhance detection capability by leveraging spatial information. In the detector part, a global and local position feature extraction module is proposed to obtain detailed spatial position information between different targets;

[0008] Step S3: In the target matching part of the tracker, a new calculation standard is proposed, which utilizes the spatial relationship between targets to make the matching method between intersecting targets more accurate;

[0009] Step S4: Match the calculated target anchor boxes and combine them with the re-identification model to obtain the final tracking result.

[0010] Furthermore, the fusion of adjacent frame information in step S1 includes the following steps:

[0011] Step R1: Based on the information from the previous frame, use an additional convolutional layer with 128 kernels to extract and record the spatial features E∈R at each location. 128*H*W It can also extract object features centered at (x,y);

[0012] Step R2: Reuse the extracted feature information into the feature extraction network of the current frame.

[0013] Furthermore, the target detection with enhanced spatial features in step S2 includes the following steps:

[0014] Step T1: In local features, the location information is encoded using channel relationships and long-range dependencies in a coordinated manner. Coordination information is generated after embedding coordinate information. Figure 2Detailed location information features are extracted from the local feature part. To effectively capture the specific spatial information of the region of interest, we consider the aspect of collecting features from different directions. We use the function F1 to divide the feature map z into two one-dimensional features f in different directions. h f w Then use F w F h The corresponding one-dimensional features are convolutionally transformed to make the channels the same for merging in the next step, as shown in formula (1) below:

[0015] Out_L=In(z)*F h (F1(z h ))*F w (F1(z w )) (1)

[0016] Step T2: In the global space, a spatial attention map is generated using the spatial relationships of features. Unlike the extraction of local information, in order to make the obtained spatial information more global, information will be obtained from deeper features to ignore the influence of local information to a certain extent. Average pooling and global pooling are applied to the feature map respectively and then convolved to obtain the following formula (2):

[0017] Out_G=C[Avg(z),Max(z)] (2)

[0018] Where C[...] represents dimension merging and convolution. Since global spatial information is extracted at a deeper level than local spatial features, feature extraction also occurs through the residual structure R[...] during downsampling.

[0019] Step T3: Encapsulate the current overall spatial feature extraction module, recursively implement decoding and encoding. This module will be used to extract spatial features each time downsampling is performed. The recursive structure of the module itself is expressed by {...}. The final structure of the module is as shown in formula (3):

[0020] Out(z)=C[In(z)*F h (F1(z h ))*F w (F1(z w )),{R[In(z)]}C[Avg(z),Max(z)]] (3).

[0021] Furthermore, step S3, which combines spatial attribute tracking, specifically includes the following steps:

[0022] Step Y1: Obtain a similarity s between the anchor boxes paired with the previous and subsequent data. This parameter is used to calculate the size difference of the detection boxes in the two matches. Under a certain number of frames, even if the size of the target box differs between the two frames before and after the same target due to changes in the action, the difference between different targets will be greater. The factors affecting different targets are not only the difference caused by the difference in the size of the target itself, but also the difference caused by the perspective relationship of near objects being larger and far objects being smaller. Distant targets will be smaller. The similarity is measured by the ratio of the width of the detection box. Compared with the numerical side, the width is less affected by the action.

[0023] Step Y2: Obtain contrast, based on the target's own movement in the spatial depth direction, to Figure 3 For example, there are two object anchor boxes in different frames. The two objects are not necessarily the same tracking target. By... Figure 3 The analysis shows that the distance between the preceding and following detection boxes is d1, while d2 represents the distance between their center points. This distance cannot represent the spatial depth difference between the two targets. To more accurately measure the target, the vertical distances l2 and l1 between the two top edges are used for contrast calculation. Compared to l1, this value additionally includes the size difference between the two objects. The basic formula for this part is as follows:

[0024]

[0025] Step Y3: Calculate the contrast ratio. Since r is initially calculated based on two vertical distances, when l2 is fixed and l1 approaches 0, the overall value will also approach 0. However, since l2 is also variable and may also approach 0, this can cause significant fluctuations when the difference between the two objects is small. Adding 1 to each of these two values ​​ensures that r approaches 0.5 when both values ​​are sufficiently small. Between two frames that meet the tracking conditions, there are two scenarios: one is where the target's spatial depth movement is small, and the contrast ratio changes accordingly between 0 and 0.5 as the target's movement amplitude changes; the other is where the target's spatial depth movement amplitude is large, and the contrast ratio fluctuates significantly as the target's movement changes. Assume there exists a number θ, 0 < θ ≤ 0.5, which is the value that the two matching objects approach after calculation. The contrast ratio is calculated based on this value, so that a higher contrast ratio results in a higher confidence level for the cross-union ratio calculation. The specific contrast ratio calculation formula is as follows:

[0026]

[0027] Step Y4 applies similarity and contrast to the calculation of the intersection-union ratio (IU). Similarity and contrast are used as units of confidence, and since they are viewed from different perspectives, they are calculated separately with the confidence score. The root mean square (RMS) weights are used to weigh the values. The final calculation formula is as follows:

[0028]

[0029] Furthermore, in step S4, target features containing spatial information are obtained from the feature extraction network, the confidence level of re-identification is calculated with the help of the re-identification module, and the final matching result is obtained by combining the confidence level in step S3, thereby obtaining the multi-target tracking result and realizing multi-target tracking.

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0031] (1) In view of the various complex situations between targets, this invention designs a multi-target tracking model that focuses on position information and introduces spatial depth; the detector part extracts spatial position features and combines them with the feature information of the previous frame to improve the detection performance of targets with complex movements and occlusions; the tracker part proposes a calculation method that introduces spatial depth, which further utilizes spatial relationships to improve the tracking effect.

[0032] (2) The algorithm of the present invention has good performance in the complex dataset Dancetrack, and has better performance for spatial depth relationships under specific conditions. Compared with common scenes in reality, this dataset takes into account more special and complex situations, and the tracking is more robust. At the same time, the model speed can reach 22 frames. The present invention can achieve real-time performance while improving video tracking performance, which is of great significance in the field of computer vision.

[0033] (3) This invention proposes a detection model that enhances spatial information to improve the detection capability of targets in complex interactions between people, and outputs additional features to enhance the tracking capability. It also introduces a target matching calculation method that combines spatial attributes to enhance the tracking capability between intersecting targets by utilizing the spatial relationship between targets. Compared with the current mainstream methods, the tracking method using spatial attributes can achieve better tracking results for targets with complex movements without being affected by the appearance features of the targets. Attached Figure Description

[0034] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0035] Figure 1 This is a schematic diagram of the overall framework of the algorithm of this invention.

[0036] Figure 2 This is a schematic diagram of the specific structure of the global and local feature extraction modules in the algorithm structure of this invention.

[0037] Figure 3This is an example diagram of two closely spaced target boxes between two consecutive frames when calculating contrast in the algorithm of this invention.

[0038] Figure 4 This is a comparison of the tracking performance of the algorithm of this invention on targets under complex motion conditions.

[0039] Figure 5 This is a comparison of the tracking performance of the algorithm of this invention when the target is under occlusion.

[0040] Figure 6 This is a comparison of the tracking performance of the algorithm of this invention in the context of overlapping targets. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Of course, the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0042] Example 1

[0043] See Figures 1 to 6 The technical solution provided in this embodiment is a multi-target tracking algorithm based on spatial attributes, including the following:

[0044] (1) Introduction to the test dataset

[0045] The experimental results in this embodiment are evaluated based on the Dancetrack dataset. Dancetrack is a publicly available dataset in the field of multi-object tracking. Among current mainstream multi-object tracking datasets, the movement patterns of pedestrians are relatively simple, and pedestrian appearances vary considerably; detection combined with appearance matching already yields good results. However, the Dancetrack dataset is mostly derived from dance, kung fu, gymnastics, etc., where targets often have similar appearances, and the movement patterns between targets are highly diverse, accompanied by numerous intersecting limb movements and positions. This further increases the complexity of tracking and requires more comprehensive consideration.

[0046] (1) Parameter settings

[0047] In this experiment, the GPU used was an NVIDIA GeForce RTX 3090, and the memory was 32GB. The software configuration was Python 3.6.13, PyTorch 1.8.0, and CUDA 11.6. This embodiment used pre-trained weights on the COCO dataset as initial weights and continued training on the dancetrack dataset. A total of 30 epochs were trained, with an initial learning rate of le-4, which changed to le-5 after 20 epochs. The batch size was set to 12. The experimental results are based on the validation and test sets of dancetrack.

[0048] (2) Model Framework Process

[0049] Different situations during the tracking process can affect tracking. Target similarity and occlusion can cause tracking difficulties, while overlapping can cause target ID switching. To address these issues, this embodiment proposes a novel multi-target tracking algorithm based on spatial attributes. This method mainly consists of an adjacent frame information fusion module, a detection module that enhances spatial features, and a tracking module that integrates spatial attributes. The fusion module uses information from the previous frame to reduce the impact of occlusion, the detection module enhances the detection of complex targets through spatial information enhancement, and the tracking module proposes a new calculation method to make target tracking more stable. The overall structure diagram is shown below. Figure 1 As shown;

[0050] (3) Model performance comparison

[0051] For the detector part, this embodiment references Fairmot with dla as the backbone network. After improvement, the new detection model pays more attention to positional information and has a closer connection between two frames. This embodiment conducts experimental comparisons of the detector on the validation set. Since the results are biased towards detecting its performance, this embodiment additionally uses MOTA and IDF1 as performance evaluations for the tracking part. The tracker uses the same DeepSORT method. The comparison results are shown in Table 1. The Dancetrack dataset mainly targets overlapping targets. The targets are captured relatively clearly, so accuracy is not a major challenge. However, occluded targets are difficult to detect, making it difficult to improve the recall rate to some extent. After network improvement, this embodiment focuses on extracting the target's positional information and utilizes features from the previous frame to further enhance detection capabilities. This helps in detecting occluded targets, reduces false negatives, and improves the recall rate by 1.6%. The network also outputs additional feature information. The improved network includes positional features in the output information, which is further utilized in the DeepSORT tracker. This results in a 2.8% improvement in IDF1 and a 1.9% improvement in MOTA, demonstrating that positional information is helpful for occlusion and positional overlap issues.

[0052] Table 1. Results of detector comparison experiment

[0053]

[0054] Considering that spatial depth attributes are primarily for targets whose spatial differences can be observed through detectors, and that targets in Dancetrack are highly complex, with certain movements in dance and gymnastics further complicating tracking, this embodiment divides the dataset to more clearly demonstrate the role of spatial depth attributes. Data containing lying down or frequent staggered jumps—characters rarely seen in daily life—is filtered out, making the remaining dataset more generalizable to complex scenarios. This embodiment divides the test set and conducts comparative experiments on the divided data. The divided data and related experimental results are shown in Table 2.

[0055] Table 2 Comparative Experiments After Division

[0056]

[0057] In this dataset, the tracking method combining the basic dla model structure with DeepSORT did not perform well on the HOTA metric. The improved algorithm showed a general improvement in overall metrics, with the average IDF1 increasing by 14.5%. This indicates that in appropriate target environments, the method in this embodiment can significantly improve the accuracy of target tracking. DetA and Mota also improved by 3.8% and 3.7% respectively, demonstrating that improvements in both the detector and tracker bring positive benefits to multi-target tracking. In a comparison on HOTA, this embodiment showed an overall performance improvement of 7.7%, further proving that the method's utilization of spatial information can improve tracking performance.

[0058] Figure 4 The example demonstrates the impact of different algorithms on tracking under complex motion conditions. When targets are in complex interactions, poor detection can lead to tracking problems. For instance, in the baseline, targets 285 and 296 are detected as a single entity by the detector due to their close proximity and exaggerated movements. When they can be separated again, tracking of target 296 fails. This embodiment's improvements to the detector can optimize detection performance to some extent. Even when targets are in complex interactions, their overall positional relationship remains relatively clear. By enhancing the extraction of positional features, the two targets can be better separated, thereby improving tracking performance.

[0059] Figure 5The image demonstrates the performance of different algorithms under occlusion conditions. In the baseline, when occlusion occurs, the occluded target 302 becomes 305 after the occlusion ends, and the occlusion causes the tracking activity to end. Since the target appearance information is highly similar, it is difficult to re-match based on appearance features after occlusion. Target detection using positional features reduces the influence of appearance features, and information supplementation between adjacent frames can also improve performance under occlusion conditions.

[0060] Figure 6 When targets overlap, in the baseline, target number 115 is misaligned with target number 119 behind it, resulting in an ID swap between their bounding boxes. At this point, the two targets have a significant spatial distance. In this embodiment, the tracker introduces spatial depth to increase the distance between the two targets, reducing the likelihood of ID mixing between the two overlapping targets and thus minimizing ID switching when targets overlap.

[0061] Table 3 compares the algorithms from recent years. The detectors for Sort and Bytetrack are similar to the Centernet network used in the baseline of this embodiment. The table compares the results of multi-target tracking in recent years, showing that the method in this embodiment significantly improves performance on the HOTA standard, achieving a 6.2% improvement compared to the lowest-performing Sort method and a 1.5% improvement compared to the highest-performing Bytetrack. Centertrack performs best on DetA and MOTA, but it shows significant limitations in tracking on IDF1 datasets, struggling with complex datasets like Dancetrack with frequent occlusion. The method in this embodiment shows a significant improvement of 10.9% over Centertrack on IDF1. This embodiment's method performs well on complex datasets and has certain advantages over other methods.

[0062] Table 3 Algorithm Comparison

[0063]

[0064] (4) Detection speed

[0065] For video tracking tasks, detection speed is crucial. It is frequently used in surveillance recordings. If real-time performance cannot be achieved, the model will lack practical application value. This embodiment performs speed tests on the Dancetrack dataset, where images are pre-processed and sized to 1088*608 pixels. The detection speed is 22 frames per second, achieving real-time performance.

[0066] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-target tracking algorithm based on spatial attributes, characterized in that, Includes the following steps: Step S1: Strengthen the positional association between the same target by adding the positional features of the target from the previous frame to the current frame to improve the anti-occlusion capability; Step S2: Enhance detection capability by leveraging spatial information. In the detector part, a global and local position feature extraction module is proposed to obtain detailed spatial position information between different targets; Step S2, which enhances detection capabilities using spatial information, includes the following steps: Step T1: In local features, coordinate the use of location information encoding channel relationships and long-range dependencies. After embedding coordinate information, generate coordinated information and use a function. feature map One-dimensional features divided into two different directions , Then use , The corresponding one-dimensional features are convolutionally transformed to make the channels the same for merging in the next step, as shown in formula (1) below: (1)； Step T2: In the global space, a spatial attention map is generated using the spatial relationships of features. Unlike the extraction of local information, in order to make the obtained spatial information more global, information will be obtained from deeper features to ignore the influence of local information to a certain extent. Average pooling and global pooling are applied to the feature map respectively and then convolved to obtain formula (2) as follows: (2)； in, To merge dimensions and perform convolution, since global spatial information is extracted at a deeper level than local spatial features, it also passes through residual structures during downsampling. Perform feature extraction; Step T3: Encapsulate the current overall spatial feature extraction module, recursively implement decoding and encoding, and use this module to extract spatial features each time downsampling is performed. To express the recursive structure of the module itself, the final structure of the module is as shown in formula (3): (3)； Step S3: In the target matching part of the tracker, a new calculation standard is proposed, which utilizes the spatial relationship between targets to make the matching method between intersecting targets more accurate; The calculation process of the new calculation standard extracted in step S3 specifically includes the following steps: Step Y1: Obtain a similarity score between the anchor boxes paired with the preceding and following data. Similarity Used to calculate the size difference of the detection boxes in two matches. Under a certain number of frames, even if the size of the target box differs between two frames before and after the same target due to changes in motion, the difference between different targets will be greater. The factors affecting different targets are not only the difference caused by the difference in the size of the target itself, but also the difference caused by the perspective relationship of near objects being larger and far objects being smaller. Step Y2: Obtain the contrast ratio. Based on the target's movement in the spatial depth direction, there are two object anchor boxes in different frames. These two objects are not necessarily the same tracked target. Analyze the distance between the preceding and following detection boxes. ,and This represents the distance between the center points of the two objects. However, the distance between these two points cannot represent the difference in spatial depth between the two objects. To more reasonably measure the object, the vertical distance between the two top edges should be obtained. and Perform contrast calculations and compare. vertical distance The size difference between the two objects is also included. The initial contrast formula, based on the two perpendicular distances, is as follows: (4)； Step Y3, calculate the final contrast ratio, based on The initial contrast is calculated based on two vertical distances. The value is fixed. When the value approaches 0, its overall value will also approach 0, but because It is also an indefinite quantity and may tend to 0. This can cause large fluctuations when the difference between two objects is small. Adding 1 to each of these two numbers makes the two numbers sufficiently small. It will tend towards 0.

5. Between two frames that meet the tracking conditions, there are two situations: one is that the target's motion in spatial depth is small, and the initial contrast will change accordingly between 0 and 0.5 as the target's motion amplitude changes; the other situation is that when the target's motion amplitude is large, the initial contrast will fluctuate significantly with the target's motion. Assume there exists a number... ,0< ≤0.5 is the calculated value that two matching objects approach after calculation. The final contrast ratio is calculated based on this value. A higher final contrast ratio results in a higher confidence level for the crossover ratio (CVR) calculation. The specific formula for calculating the final contrast ratio is as follows: (5)； Step Y4: Similarity and final contrast are applied to the calculation of the intersection-union ratio (IU). Similarity and final contrast are used as units of confidence. Since they are viewed from different perspectives, they are calculated separately with the confidence and the values ​​are weighted by the root mean square. The final calculation formula is as follows: (6)； Step S4: Match the calculated target anchor boxes and combine them with the re-identification model to obtain the final tracking result.

2. The multi-target tracking algorithm based on spatial attributes according to claim 1, characterized in that, The fusion of adjacent frame information in step S1 includes the following steps: Step R1: Based on the information from the previous frame, use an additional convolutional layer with 128 kernels to extract and record the spatial features at each location. It can also extract object features centered at (x,y); Step R2: Reuse the extracted feature information into steps S1-S2 of the current frame.

3. The multi-target tracking algorithm based on spatial attributes according to claim 2, characterized in that, In step R2, target features containing spatial information in the previous frame are obtained, the confidence level of re-identification is calculated using the re-identification model, and the final matching result is obtained by combining the confidence level in step S3, thereby obtaining the multi-target tracking result and realizing multi-target tracking.

Images