Multi-target tracking method and system based on adaptive appearance motion fusion and hierarchical association

Abstract

The invention relates to the technical field of image analysis, in particular to a multi-target tracking method and system for adaptive appearance motion fusion and hierarchical association. According to the method, firstly, a detection frame is layered through a confidence coefficient threshold value, and differential processing is achieved to give consideration to precision and efficiency; then fusing the appearance and motion similarity of the high / middle-layer detection frame and adaptively allocating weights, obtaining an adaptive fusion similarity matrix, solving to obtain a first-stage matching result, and fully utilizing multi-dimensional features to improve the matching accuracy; acquiring an intersection-parallel ratio matrix of a low-layer detection frame of the current frame and an unmatched track, and solving to obtain a second-stage matching result; and finally, the matching results of the two stages are integrated to obtain a target tracking result, so that the probability of mismatching and mismatching is effectively reduced, and the tracking efficiency is optimized while the multi-target tracking precision is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image analysis, in particular to a multi-target tracking method and system based on adaptive appearance-motion fusion and hierarchical association. BACKGROUND

[0002] Multi-target tracking technology is one of the core problems in the field of computer vision, involving multiple disciplines such as computer vision, machine learning, sensor fusion, and optimization theory, and is widely used in intelligent security, autonomous driving, human-computer interaction, etc. Multi-target tracking technology assigns and maintains a unique identity to each detected target in a video sequence, and associates and fuses its motion trajectory in the time-space dimension, realizing the transition from independent "frame-level detection" to continuous "trajectory-level cognition".

[0003] Existing methods such as OC-SORT, StrongSORT, and FusionSORT, although to some extent, improve the tracking performance, but still have the following shortcomings: (1) the fusion mechanism cannot adapt to different scenes and target dynamic changes; (2) the traditional ReID feature extractor has limited ability to distinguish similar targets; (3) the standard IoU calculation cannot effectively handle complex motion patterns; (4) the processing of low-quality detection boxes is not robust enough. Especially in complex scenes such as crowded people, severe occlusion, and rapid motion, the tracking performance of existing methods decreases significantly. SUMMARY

[0004] The purpose of the present application is to provide a multi-target tracking method and system based on adaptive appearance-motion fusion and hierarchical association for pedestrian tracking in intelligent video surveillance scenarios.

[0005] The technical solution of the present application is as follows: A multi-target tracking method based on adaptive appearance-motion fusion and hierarchical association, comprising the following operations: S1: Obtain the detection box of the target in each frame of the video sequence through the target detector, and divide the detection box into three levels of high-level detection box, middle-level detection box and low-level detection box according to the pre-set confidence threshold; the higher the confidence, the higher the level; S2: For the high-level detection frame and / or the middle-level detection frame of the current frame in the video sequence, an appearance similarity matrix of the last frame detection frame and the current frame detection frame is obtained, and a motion similarity matrix of the current frame prediction track and the current frame detection frame is obtained; based on the appearance similarity matrix, the motion similarity matrix, the current frame detection target distribution, and the current frame detection frame motion information, a current frame adaptive fusion weight is obtained; based on the current frame adaptive fusion weight, the appearance similarity matrix and the motion similarity matrix are fused to obtain a current frame adaptive fusion similarity matrix; the current frame adaptive fusion similarity matrix is converted into a cost matrix, and the optimal matching frame is solved by using the Hungarian algorithm, and the current frame high-level detection frame and / or the middle-level detection frame form a first stage matching result of the current frame; S3: For the low-level detection frame of the current frame in the video sequence, an intersection over union matrix of the current frame low-level detection frame and the unmatched track is obtained, which is converted into a cost matrix, and the optimal matching frame is solved by using the Hungarian algorithm, and the current frame low-level detection frame forms a second stage matching result of the current frame. S4: The first stage matching result of the current frame and the second stage matching result of the current frame are merged to obtain a target tracking result of the current frame; all frame target tracking results form a multi-target tracking result.

[0006] The appearance similarity matrix is obtained by: inputting the feature map as the last frame detection frame or the current frame detection frame; the input feature map is subjected to difference perception attention processing and local block enhancement processing to obtain discriminative enhanced feature map and local feature enhanced feature map, and the fusion enhanced feature map is obtained after feature fusion based on the gating weight; the fusion enhanced feature maps corresponding to the last frame detection frame and the current frame detection frame are respectively subjected to full connection processing, and the similarity matrix is calculated as the appearance similarity matrix.

[0007] The operation of the local block enhancement processing is: the input feature map is divided into a plurality of grids to obtain a plurality of local feature blocks, each local feature block is subjected to multi-head self-attention mechanism processing, and then aggregated to obtain aggregated features; the aggregated features and the input feature map are subjected to residual connection to obtain the local feature enhanced feature map.

[0008] The motion similarity matrix is obtained by: based on the current frame prediction track center, the current frame detection frame center, and the prediction track historical speed, a motion difference is obtained; based on the motion difference, a scale normalization factor, and an age adjustment factor, a motion penalty term is obtained; based on the motion penalty term, and the intersection over union of the current frame prediction track and the current frame detection frame, a motion similarity is obtained.

[0009] The method for obtaining the adaptive fusion weight of the current frame comprises the following steps: obtaining a quality adjustment factor and a motion stability factor based on an appearance similarity matrix and a motion similarity matrix; obtaining a scene adaptation factor based on a detected target density and a detected target spatial uniformity of the current frame; obtaining a motion stability factor by acquiring a detected frame speed stability, a direction stability and an acceleration smoothness of the current frame; and performing weighted fusion on the quality adjustment factor and the scene adaptation factor and the motion stability factor to obtain the adaptive fusion weight of the current frame.

[0010] The unmatched trajectory is a predicted trajectory whose state prediction value or covariance prediction value is not within a corresponding prediction value range or whose predicted trajectory coordinates are not within a video frame coordinate range.

[0011] The current frame predicted trajectory is obtained by processing a last frame detection frame through a Kalman filter motion model.

[0012] The adaptive appearance-motion fusion and hierarchical association multi-target tracking system is used for implementing the adaptive appearance-motion fusion and hierarchical association multi-target tracking method, and comprises the following modules. The multi-level detection frame generation module is configured to acquire a detection frame of a target in each frame of a video sequence through a target detector, and divide the detection frame into three levels of high-level detection frames, middle-level detection frames and low-level detection frames according to a preset confidence threshold; the higher the confidence is, the higher the level is. The first-stage matching result generation module is configured to acquire an appearance similarity matrix of a last frame detection frame and a current frame detection frame, and a motion similarity matrix of a current frame predicted trajectory and the current frame detection frame for the high-level detection frame and / or the middle-level detection frame of the current frame of the video sequence; obtain a current frame adaptive fusion weight based on the appearance similarity matrix, the motion similarity matrix, a current frame detection target distribution and current frame detection frame motion information; fuse the appearance similarity matrix and the motion similarity matrix based on the current frame adaptive fusion weight to obtain a current frame adaptive fusion similarity matrix; and convert the current frame adaptive fusion similarity matrix into a cost matrix, and solve an optimal matching frame through a Hungarian algorithm to form a first-stage matching result of the current frame together with the current frame high-level detection frame and / or the current frame middle-level detection frame. The second-stage matching result generation module is configured to acquire an intersection-over-union matrix of the current frame low-level detection frame and an unmatched trajectory, convert the intersection-over-union matrix into a cost matrix, and solve an optimal matching frame through a Hungarian algorithm to form a second-stage matching result of the current frame together with the current frame low-level detection frame. The multi-target tracking result generation module is configured to merge the first-stage matching result of the current frame and the second-stage matching result of the current frame to obtain a target tracking result of the current frame; and form a multi-target tracking result by using all frame target tracking results.

[0013] An adaptive appearance motion fusion and hierarchical association multi-target tracking device includes a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the aforementioned adaptive appearance motion fusion and hierarchical association multi-target tracking method.

[0014] A computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned adaptive appearance motion fusion and hierarchical association multi-target tracking method.

[0015] The beneficial effects of this invention are as follows: This invention provides a multi-target tracking method that integrates adaptive appearance and motion fusion with hierarchical association. First, detection boxes are layered using a confidence threshold to achieve differentiated processing, balancing accuracy and efficiency. Then, appearance and motion similarity are fused between high / mid-level detection boxes, and weights are adaptively assigned. The resulting adaptive fusion similarity matrix is ​​transformed into a cost matrix, which is then used to solve for the first-stage matching result, fully utilizing multi-dimensional features to improve matching accuracy. Next, the intersection-union matrix (IUCN) of the current frame's low-level detection boxes and unmatched trajectories is obtained, transformed into a cost matrix, and used to solve for the second-stage matching result. Finally, the two-stage matching results are integrated to obtain the target tracking result, effectively reducing the probability of missed and incorrect matches, ultimately improving multi-target tracking accuracy while optimizing tracking efficiency. This invention provides an adaptive appearance-motion fusion and hierarchical association method for multi-target tracking. It utilizes hierarchical processing of detection box confidence, comparison of appearance and motion similarity, a differentiated association strategy, and a multi-dimensional scene adaptive fusion mechanism to construct an efficient and robust multi-target tracking framework. Experimental results show that on the MOT17 dataset, this method significantly improves upon the OC-SORT method in terms of HOTA, MOTA, and IDF1, reducing the number of ID switching by 60.5%, effectively improving tracking accuracy, efficiency, robustness, and real-time performance. This demonstrates that the method can achieve accurate and stable tracking of multiple moving targets in dense and complex scenes, exhibiting comprehensive advantages in tracking accuracy, efficiency, robustness, and real-time performance. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the exemplary embodiments of this application clearer, the technical solutions in the exemplary embodiments of this application are described clearly and completely below. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0017] Example 1 This embodiment provides a multi-target tracking method with adaptive appearance motion fusion and hierarchical association, including the following operations: S1: Obtain the target bounding box in each frame of the video sequence using the target detector, and classify the bounding boxes into three levels: high-level bounding box, mid-level bounding box, and low-level bounding box according to a preset confidence threshold; the higher the confidence, the higher the level. S2: For the high-level and / or mid-level detection boxes in the current frame of the video sequence, obtain the appearance similarity matrix between the detection boxes of the previous frame and the detection boxes of the current frame, as well as the motion similarity matrix between the predicted trajectory and the detection boxes of the current frame; based on the appearance similarity matrix, motion similarity matrix, target distribution in the current frame, and motion information of the detection boxes in the current frame, obtain the adaptive fusion weights for the current frame; based on the adaptive fusion weights for the current frame, fuse the appearance similarity matrix and motion similarity matrix to obtain the adaptive fusion similarity matrix for the current frame; convert the adaptive fusion similarity matrix of the current frame into a cost matrix, solve for the optimal matching box using the Hungarian algorithm, and form the first-stage matching result of the current frame with the high-level and / or mid-level detection boxes of the current frame; S3: For the low-level detection box of the current frame in the video sequence, obtain the intersection-union matrix of the low-level detection box of the current frame and the unmatched trajectory, convert it into a cost matrix, solve for the optimal matching box using the Hungarian algorithm, and form the second-stage matching result of the current frame with the low-level detection box of the current frame; S4: Merge the first-stage matching result and the second-stage matching result of the current frame to obtain the target tracking result of the current frame; the target tracking results of all frames are combined to form a multi-target tracking result.

[0018] The specific steps are detailed below.

[0019] S1: Obtain the target bounding box in each frame of the video sequence through the target detector, and divide the bounding boxes into three levels: high-level bounding box, mid-level bounding box, and low-level bounding box according to the preset confidence threshold; the higher the confidence, the higher the level.

[0020] Object detection is performed on each frame of the input video sequence using an object detector (including but not limited to the YOLOv8 model), obtaining the bounding box of the target in each frame. Each bounding box includes the bounding box coordinates and a confidence score. Based on preset confidence thresholds τ1=0.6, τ2=0.3, and τ3=0.1, the bounding boxes are divided into three levels: high-level, mid-level, and low-level. Higher confidence scores correspond to higher levels. This hierarchical processing mechanism achieves differentiated detection quality through confidence thresholds, providing a foundation for subsequent association strategies.

[0021] S2: For the high-level and / or mid-level detection boxes in the current frame of the video sequence, obtain the appearance similarity matrix between the detection boxes of the previous frame and the detection boxes of the current frame, as well as the motion similarity matrix between the predicted trajectory and the detection boxes of the current frame; based on the appearance similarity matrix, motion similarity matrix, the distribution of detected targets in the current frame, and the motion information of the detection boxes in the current frame, obtain the adaptive fusion weights for the current frame; based on the adaptive fusion weights for the current frame, fuse the appearance similarity matrix and motion similarity matrix to obtain the adaptive fusion similarity matrix for the current frame; convert the adaptive fusion similarity matrix for the current frame into a cost matrix, solve for the optimal matching box using the Hungarian algorithm, and form the first-stage matching result of the current frame with the high-level and / or mid-level detection boxes of the current frame.

[0022] Step 1: In the starting frame of the video sequence, create initial trajectories for each high-level and mid-level detection box: assign a unique identifier, initialize the Kalman filter state vector, extract initial appearance features (i.e., extract the fused enhanced feature map of the detection box), and add them to the trajectory set.

[0023] Specifically, for the starting frame of the video sequence, the system initializes an empty trajectory set; iterates through each high-level and mid-level detection box, performing the following initialization operations: assigning a globally unique identifier to the trajectory, and initializing the state vector x of the Kalman filter. k =[x,y,w,h,0,0,0,0] T Where (x,y) are the center coordinates of the detection box, and (w,h) are the width and height of the detection box; initialize the trajectory age. k =1; Use the feature extractor to extract the appearance features of the detection box. f k (i.e., extract the fused enhanced feature map of the detection box); set the trajectory not updated counter time_since_update=0; add the initial trajectory to the trajectory set; this initialization step establishes the starting point for the entire tracking process, ensuring that the system can build continuous trajectories from reliable detection results.

[0024] Step 2: Process the detection box of the previous frame (not the starting frame) using a Kalman filter motion model to obtain the predicted trajectory of the current frame.

[0025] In this embodiment, the state prediction is achieved through the Kalman filter motion model processing using the following formula: , This represents the trajectory state of the current frame. For the uniform motion model of the target, The state of the detection block in the previous frame is used; covariance prediction is achieved through the following formula: , The current frame trajectory covariance, The covariance of the detection boxes in the previous frame. Let be the process noise covariance matrix.

[0026] In addition, for predicted trajectories that fail (such as those with unstable values ​​or invalid bounding boxes), the predicted trajectories whose state prediction values ​​or covariance prediction values ​​are not within their respective prediction value ranges, or whose predicted trajectory coordinates are not within the range of video frame coordinates, are marked as unmatched trajectories and are to be deleted. They are then restored using the last valid observation value. This prediction step utilizes the target's motion priors, providing important spatiotemporal constraints for data association and significantly reducing the search space.

[0027] Step 3: Obtain the appearance similarity matrix between the detection box in the previous frame and the detection box in the current frame. The specific steps are detailed below.

[0028] Step 3.1: The input feature map is the detection box of the previous frame or the detection box of the current frame (the detection box is a high-level detection box or a middle-level detection box); the input feature map is processed by difference-aware attention processing and local block enhancement processing to obtain discriminative enhanced feature map and local feature enhanced feature map.

[0029] The differential awareness attention processing operation is as follows: a differential awareness map is generated by calculating the channel variance of the input feature map, an attention weight matrix is ​​generated by two convolutional layers, and the matrix is ​​multiplied element-wise with the input feature map to obtain a discriminative enhancement feature map.

[0030] The specific calculation formula is as follows: , , , , For channel variance, Here is the formula for calculating variance. For the input feature map, , These are the height and width of the input feature map, respectively. This represents the total number of channel indexes. For channel index, This is a difference perception map. This is the attention weight matrix. For ReLU activation function, For the sigmoid function, , These are the first learnable parameter and the second learnable parameter, respectively. This is a discriminative feature map.

[0031] The method for obtaining the local feature enhancement feature map is as follows: the input feature map is divided into multiple (G×G) grids to obtain multiple local feature blocks. After processing each local feature block with a multi-head self-attention mechanism, they are aggregated to obtain aggregated features. The aggregated features are then residually connected with the input feature map to obtain the local feature enhancement feature map that captures fine-grained appearance features.

[0032] Step 3.2: The discriminative enhanced feature map and the local enhanced feature map are fused together using a gating weight to obtain the fused enhanced feature map.

[0033] Specifically, the discriminative enhanced feature map is obtained by performing global average pooling on the discriminative enhanced feature map. The gate weights of the discriminative pooling feature map are then obtained (which can be implemented using the sigmoid function). Based on the gate weights, the discriminative enhanced feature map and the local enhanced feature map are fused to obtain a fused enhanced feature map that balances global and local features. The specific processing is as follows: ， ， , To distinguish pooling feature maps, This is a global average pooling process. The learnable coefficient, For gating weights, Enhance the feature map for local features.

[0034] Step 3.3: The fused and enhanced feature maps corresponding to the detection boxes in the previous frame and the current frame are processed by fully connected processing, compressed to obtain their respective feature vectors, and the similarity matrix is ​​calculated as the appearance similarity matrix.

[0035] The appearance similarity matrix is ​​calculated using the following formula: , This is the appearance similarity matrix between the detection boxes in the previous frame and the detection boxes in the current frame. , These are the feature vectors corresponding to the fused enhanced feature maps of the detection boxes in the previous frame and the current frame, respectively.

[0036] Simultaneously, a motion similarity matrix is ​​obtained based on the predicted trajectory and the detection box of the current frame. Specifically, motion difference is obtained based on the center of the predicted trajectory, the center of the detection box, and the historical velocity of the predicted trajectory; motion penalty term is obtained based on motion difference, scale normalization factor, and age adjustment factor set according to trajectory lifecycle; motion similarity is obtained based on motion penalty term and the intersection-union ratio of the predicted trajectory and the detection box of the current frame.

[0037] The specific formula for calculating motion similarity is as follows: , ， , Due to differences in movement, , These are the center of the detection box in the current frame and the center of the predicted trajectory in the current frame, respectively. To predict historical velocities, To predict the magnitude of the historical velocity of the trajectory, (the average velocity of the last 3 frames is taken, taking into account both stability and real-time performance). The time interval is used to calculate the position deviation after velocity compensation. Historical speeds are used to correct for positional deviations, reflecting the consistency of movement trends. As a weighting factor for velocity modulus, As the first compensation term, the formula for calculating motion difference can avoid the traditional motion difference calculation that only calculates the Euclidean distance at the current position, thus ignoring the correlation of motion trend; The adjustment coefficient is set to 0.6. The scale normalization factor, This is the age adjustment factor for the detection bounding box in the current frame. For sports penalty items, This is the intersection-union ratio (IoU) between the predicted trajectory (corresponding bounding box) of the current frame and the detected bounding box of the current frame.

[0038] Step 4: Based on the appearance similarity matrix, motion similarity matrix, current frame detected target, and current frame detection box motion information, obtain the current frame adaptive fusion weight.

[0039] The method for obtaining the adaptive fusion weights for the current frame is as follows: based on the appearance similarity matrix and motion similarity matrix, a quality adjustment factor is obtained; based on the detection target density and detection target spatial uniformity of the current frame, a scene adaptation factor is obtained; the velocity stability, orientation stability, and acceleration smoothness of the detection box in the current frame (all of which belong to the motion information of the detection box in the current frame) are obtained, a motion stability factor is calculated, and weighted and fused with the quality adjustment factor and the scene adaptation factor to obtain the adaptive fusion weights for the current frame.

[0040] The quality adjustment factor, scene adaptation factor, and motion stability factor are calculated using the following formulas: , , , For quality adjustment factor, , These are the feature vectors corresponding to the appearance similarity matrix and the motion similarity matrix, respectively. As the second compensation item, It is the sigmoid function; As a scene adaptation factor, , , These are the detection target density, detection target spatial uniformity, detection target density, and detection box normalization ratio for the current frame, respectively. As a motion stability factor, , , The current frame number k The detection frame velocity stability, directional stability, and acceleration smoothness of each detection frame. This represents the total number of detection boxes in the current frame.

[0041] Step 5: Based on the adaptive fusion weight of the current frame, fuse the appearance similarity matrix and the motion similarity matrix to obtain the adaptive fusion similarity matrix of the current frame.

[0042] Step 6: Convert the adaptive fusion similarity matrix of the current frame into a cost matrix, solve for the optimal matching box using the Hungarian algorithm, and combine it with the high-level detection box and / or the middle-level detection box of the current frame to form the first-stage matching result of the current frame.

[0043] S3: For the low-level detection bounding box of the current frame in the video sequence, obtain the intersection-union matrix (IUCN) between the current frame's low-level detection bounding box and the unmatched trajectory, convert it into a cost matrix, and use the Hungarian algorithm to solve for the optimal matching bounding box. This optimal matching bounding box, along with the current frame's low-level detection bounding box, forms the second-stage matching result for the current frame. Unmatched trajectories are those whose state prediction value or covariance prediction value is outside their respective prediction value range, or whose predicted trajectory coordinates are outside the video frame coordinate range.

[0044] S4: Merge (integrate) the first-stage matching results and the second-stage matching results of the current frame to obtain the target tracking result of the current frame; the target tracking results of all frames are combined to form a multi-target tracking result. This two-stage association strategy, through a hierarchical processing mechanism, ensures accurate matching of high-quality detections while making full use of the trajectory completion capabilities of low-quality detections, significantly improving the tracking recall rate.

[0045] Furthermore, to achieve trajectory updating and management, this embodiment maintains the entire lifecycle of the trajectory based on the association results of S4. For successfully matched trajectories, Kalman filtering is used to update their motion state, including Kalman gain calculation, state update, and covariance update. Appearance features are updated using an exponential smoothing strategy. Trajectories that have not been updated for 30 consecutive frames are terminated to release system resources. Simultaneously, the trajectory's age, hit count, and other state information are maintained to provide a basis for subsequent processing decisions. This trajectory management step, through a complete lifecycle management mechanism, ensures the stability, continuity, and resource utilization efficiency of the tracking system.

[0046] To verify the effectiveness of the method in this embodiment, the following experiment was conducted.

[0047] In the experiment, YOLOv8 was used as the detection model, and the method of this embodiment and existing state-of-the-art methods were used as the tracking models. Performance was compared on the MOT17 dataset (MOT17 is one of the MOT Challenge series datasets, which includes dense scenes with complex lighting changes, occlusion, and viewpoint changes. The dataset includes annotations of moving pedestrians, as well as stationary pedestrians, cyclists, or skateboarders, etc.). The experimental results are shown in Table 1. Evaluation metrics included HOTA, MOTA, IDF1, and IDs. HOTA aims to evaluate both detection and association performance in a balanced way; it assesses whether the target is accurately bounded and whether the bounded target is correctly identified. MOTA is a metric that measures overall detection accuracy. It evaluates performance by counting the number of errors (missed detections, false detections, ID switching) across all frames. IDF1 is a metric that measures identity association consistency, used to evaluate the tracker's ability to maintain the correct ID for the same target. IDs refers to the number of ID switchings; it calculates the number of times the same target is incorrectly assigned different IDs throughout the video.

[0048] As shown in Table 1, on the MOT17 dataset, the method in this embodiment improves HOTA by 7.1% compared to OC-SORT, MOTA by 74.8% compared to OC-SORT by 6.8%, IDF1 by 5.9%, and IDs by 60.5%. In comparison, the method in this embodiment significantly reduces the number of ID switching on the MOT17 dataset while maintaining high tracking accuracy, indicating that the method in this embodiment can achieve higher tracking accuracy and better identity preservation capabilities.

[0049] Table 1. Comparison of processing results of the method in this embodiment and OC-SORT on the MOT17 dataset.

[0050] This embodiment also provides an adaptive appearance motion fusion and hierarchical association multi-target tracking system to implement the above-mentioned adaptive appearance motion fusion and hierarchical association multi-target tracking method, including: The multi-level detection box generation module is used to obtain the detection box of the target in each frame of the video sequence through the target detector. The detection boxes are divided into three levels: high-level detection boxes, middle-level detection boxes and low-level detection boxes according to the preset confidence threshold; the higher the confidence, the higher the level. An adaptive fusion similarity matrix and intersection-over-union (IoU) matrix generation module is used to obtain, for each frame of a video sequence, the appearance similarity matrix between the previous frame detection box and the current frame detection box of the same target, as well as the motion similarity matrix between the current frame predicted trajectory and the current frame detection box, for high-level and / or mid-level detection boxes of the same target; based on the appearance similarity matrix, motion similarity matrix, current frame target distribution, and current frame detection box motion information, the current frame adaptive fusion weight is obtained; based on the current frame adaptive fusion weight, the appearance similarity matrix and motion similarity matrix are fused to obtain the current frame adaptive fusion similarity matrix; for each frame of a video sequence, the IoU matrix is ​​obtained between the current frame low-level detection box of the same target and the corresponding unmatched trajectory. The multi-target tracking result generation module converts the adaptive fusion similarity matrix and intersection-union matrix of the current frame into cost matrices, solves for the optimal matching box using the Hungarian algorithm, and forms the current frame detection matching set with the high-level detection boxes and / or mid-level detection boxes and / or low-level detection boxes of the current frame, which serves as the target tracking result of the current frame; the target tracking results of all frames form the multi-target tracking result.

[0051] This embodiment also provides an adaptive appearance motion fusion and hierarchical association multi-target tracking device, including a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the above-described adaptive appearance motion fusion and hierarchical association multi-target tracking method.

[0052] This embodiment also provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the above-described adaptive appearance motion fusion and hierarchical association multi-target tracking method.

[0053] This embodiment provides a multi-target tracking method that integrates adaptive appearance and motion fusion with hierarchical association. First, detection boxes are layered using a confidence threshold to achieve differentiated processing, balancing accuracy and efficiency. Then, appearance and motion similarity are fused between high / mid-level detection boxes, and weights are adaptively assigned. The resulting adaptive fusion similarity matrix is ​​converted into a cost matrix, which is then used to solve for the first-stage matching result, fully utilizing multi-dimensional features to improve matching accuracy. Next, the intersection-union matrix (IUCN) of the current frame's low-level detection boxes and unmatched trajectories is obtained, converted into a cost matrix, and used to solve for the second-stage matching result. Finally, the two-stage matching results are integrated to obtain the target tracking result, effectively reducing the probability of missed and incorrect matches, ultimately improving both multi-target tracking accuracy and tracking efficiency.

[0054] This embodiment provides an adaptive appearance-motion fusion and hierarchical association multi-target tracking method. It utilizes hierarchical processing of detection box confidence, comparison of appearance and motion similarity, a differentiated association strategy, and a multi-dimensional scene adaptive fusion mechanism to construct an efficient and robust multi-target tracking framework. Experimental results show that on the MOT17 dataset, the proposed method significantly improves upon the OC-SORT method in terms of HOTA, MOTA, and IDF1, reducing the number of ID switching by 60.5%, effectively improving tracking accuracy, efficiency, robustness, and real-time performance. This demonstrates that the proposed method can achieve accurate and stable tracking of multiple moving targets in dense and complex scenes, exhibiting comprehensive advantages in tracking accuracy, efficiency, robustness, and real-time performance.

[0055] While exemplary embodiments of the invention have been described herein, many other variations or modifications conforming to the principles of the invention can be directly determined or derived from the disclosure of this invention without departing from its spirit and scope. Therefore, the scope of the invention should be understood and recognized to cover all such other variations or modifications.

Claims

1. A multi-target tracking method with adaptive appearance motion fusion and hierarchical association, characterized in that, This includes the following operations: S1: Obtain the detection box of the target in each frame of the video sequence through the target detector, and divide the detection box into three levels: high-level detection box, middle-level detection box and low-level detection box according to the preset confidence threshold; The higher the confidence level, the higher the rating. S2: For the high-level detection boxes and / or mid-level detection boxes in the current frame of the video sequence, obtain the appearance similarity matrix between the detection boxes of the previous frame and the detection boxes of the current frame, as well as the motion similarity matrix between the predicted trajectory of the current frame and the detection boxes of the current frame. Based on the appearance similarity matrix, motion similarity matrix, target distribution in the current frame, and motion information of the detection box in the current frame, the adaptive fusion weights for the current frame are obtained. Based on the adaptive fusion weights of the current frame, the appearance similarity matrix and the motion similarity matrix are fused to obtain the adaptive fusion similarity matrix of the current frame; The adaptive fusion similarity matrix of the current frame is converted into a cost matrix, and the optimal matching box is solved by the Hungarian algorithm. This optimal matching box is then compared with the high-level detection box and / or the middle-level detection box of the current frame to form the first-stage matching result of the current frame. S3: Obtain the intersection-union matrix of the low-level detection box in the current frame and the corresponding unmatched trajectory, convert the intersection-union matrix of the current frame into a cost matrix, solve for the optimal matching box using the Hungarian algorithm, and form the second-stage matching result of the current frame with the low-level detection box in the current frame; S4: Merge the first-stage matching result and the second-stage matching result of the current frame to obtain the target tracking result of the current frame; The target tracking results of all frames are combined to form a multi-target tracking result.

2. The multi-target tracking method with adaptive appearance motion fusion and hierarchical association according to claim 1, characterized in that, In S2, the appearance similarity matrix is ​​obtained as follows: The input feature map is either the detection box of the previous frame or the detection box of the current frame; The input feature map is processed by difference-aware attention and local block enhancement to obtain discriminative enhanced feature map and local feature enhanced feature map. After feature fusion based on gating weight, a fused enhanced feature map is obtained. The fused and enhanced feature maps corresponding to the detection boxes in the previous frame and the current frame are processed by fully connected layers, and then a similarity matrix is ​​calculated to serve as the appearance similarity matrix.

3. The multi-target tracking method with adaptive appearance motion fusion and hierarchical association according to claim 1, characterized in that, In S2, the local block enhancement process is as follows: The input feature map is divided into multiple grids to obtain multiple local feature blocks. Each local feature block is processed by a multi-head self-attention mechanism and then aggregated to obtain aggregated features. The aggregated features and the input feature map are concatenated via residuals to obtain a local feature enhancement map.

4. The multi-target tracking method with adaptive appearance motion fusion and hierarchical association according to claim 1, characterized in that, In S2, the method for obtaining the motion similarity matrix is ​​as follows: Based on the predicted trajectory center in the current frame, the detection box center in the current frame, and the historical velocity of the predicted trajectory, the motion difference is obtained; Based on the differences in movement, the scale normalization factor, and the age adjustment factor, the movement penalty term was obtained. Motion similarity is obtained based on the motion penalty term and the intersection-union ratio of the predicted trajectory in the current frame and the detection box in the current frame.

5. The multi-target tracking method with adaptive appearance motion fusion and hierarchical association according to claim 1, characterized in that, In S2, the method for obtaining the adaptive fusion weights for the current frame is as follows: Based on the appearance similarity matrix and motion similarity matrix, the quality adjustment factor and motion stability factor are obtained; based on the detection target density and detection target spatial uniformity of the current frame, the scene adaptation factor is obtained. The velocity stability, orientation stability, and acceleration smoothness of the detection box in the current frame are obtained. The motion stability factor is calculated and weighted with the quality adjustment factor and the scene adaptation factor to obtain the adaptive fusion weight of the current frame.

6. The multi-target tracking method with adaptive appearance motion fusion and hierarchical association according to claim 1, characterized in that, Unmatched trajectories are predicted trajectories whose state prediction value or covariance prediction value is not within the range of their respective prediction values, or whose predicted trajectory coordinates are not within the range of video frame coordinates.

7. The multi-target tracking method with adaptive appearance motion fusion and hierarchical association according to claim 6, characterized in that, The predicted trajectory of the current frame is obtained by processing the detection box of the previous frame using a Kalman filter motion model.

8. A multi-target tracking system with adaptive appearance motion fusion and hierarchical association, used to implement the multi-target tracking method with adaptive appearance motion fusion and hierarchical association as described in claim 1, characterized in that, include: The multi-level detection box generation module is used to obtain the detection box of the target in each frame of the video sequence through the target detector, and divide the detection box into three levels: high-level detection box, middle-level detection box and low-level detection box according to the preset confidence threshold. The higher the confidence level, the higher the rating. The first-stage matching result generation module is used to obtain the appearance similarity matrix between the detection box of the previous frame and the detection box of the current frame, as well as the motion similarity matrix between the predicted trajectory of the current frame and the detection box of the current frame for the high-level detection box and / or the middle-level detection box of the current frame in the video sequence. Based on the appearance similarity matrix, motion similarity matrix, target distribution in the current frame, and motion information of the detection box in the current frame, the adaptive fusion weights for the current frame are obtained. Based on the adaptive fusion weights of the current frame, the appearance similarity matrix and the motion similarity matrix are fused to obtain the adaptive fusion similarity matrix of the current frame; The adaptive fusion similarity matrix of the current frame is converted into a cost matrix, and the optimal matching box is solved by the Hungarian algorithm. This optimal matching box is then compared with the high-level detection box and / or the middle-level detection box of the current frame to form the first-stage matching result of the current frame. The second-stage matching result generation module is used to obtain the intersection-union matrix of the low-level detection box of the current frame and the unmatched trajectory for the low-level detection box of the current frame in the video sequence, convert it into a cost matrix, solve for the optimal matching box through the Hungarian algorithm, and form the second-stage matching result of the current frame with the low-level detection box of the current frame. The multi-target tracking result generation module is used to merge the first-stage matching result and the second-stage matching result of the current frame to obtain the target tracking result of the current frame; The target tracking results of all frames are combined to form a multi-target tracking result.

9. A multi-target tracking device with adaptive appearance motion fusion and hierarchical association, characterized in that, It includes a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the multi-target tracking method of adaptive appearance motion fusion and hierarchical association as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the multi-target tracking method of adaptive appearance motion fusion and hierarchical association as described in any one of claims 1-7.

Images