Adaptive appearance motion fusion and hierarchical association based multi-target tracking method and system

Abstract

The present application relates to the technical field of image analysis, in particular to a multi-target tracking method and system based on adaptive appearance-motion fusion and hierarchical correlation, the method of the present application first divides the detection frame into layers through a confidence threshold, realizes differentiated processing to give consideration to precision and efficiency, then fuses appearance and motion similarity for high / medium layer detection frames and adaptively allocates weights, obtains an adaptive fusion similarity matrix, solves to obtain a first-stage matching result, fully utilizes multi-dimensional features to improve matching accuracy, then obtains an intersection-over-union matrix of the current frame low layer detection frame and unmatched trajectories, solves to obtain a second-stage matching result, finally integrates the two-stage matching results to obtain a target tracking result, effectively reduces the probability of missed matching and false matching, and finally improves the accuracy of multi-target tracking while optimizing the tracking efficiency.

Description

Technical Field

[0001] This invention relates to the field of image analysis technology, specifically to a multi-target tracking method and system with adaptive appearance motion fusion and hierarchical association. Background Technology

[0002] Multi-object tracking technology is one of the core problems in the field of computer vision, involving multiple disciplines such as computer vision, machine learning, sensor fusion, and optimization theory. It is widely used in fields such as intelligent security, autonomous driving, and human-computer interaction. Multi-object tracking technology achieves a leap from independent "frame-level detection" to continuous "trajectory-level cognition" by assigning and maintaining a unique identifier to each detected target in a video sequence and associating and fusing their motion trajectories in the spatiotemporal dimensions.

[0003] Existing methods such as OC-SORT, StrongSORT, and FusionSORT, while improving tracking performance to some extent, still have the following shortcomings: (1) the fusion mechanism cannot adapt to different scenarios and dynamic changes in targets; (2) traditional ReID feature extractors have limited ability to distinguish similar targets; (3) standard IoU calculation cannot effectively handle complex motion patterns; and (4) they are not robust enough to handling low-quality detection boxes. In particular, in complex scenarios such as dense crowds, severe occlusion, and rapid movement, the tracking performance of existing methods drops significantly. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-target tracking method and system for adaptive appearance motion fusion and hierarchical association of pedestrian tracking in intelligent video surveillance scenarios.

[0005] The technical solution of this invention is as follows:

[0006] A multi-target tracking method with adaptive appearance motion fusion and hierarchical association includes the following operations:

[0007] S1: Obtain the target bounding box in each frame of the video sequence using the target detector, and classify the bounding boxes into three levels: high-level bounding box, mid-level bounding box, and low-level bounding box according to a preset confidence threshold; the higher the confidence, the higher the level.

[0008] S2: For the high-level and / or mid-level detection boxes in the current frame of the video sequence, obtain the appearance similarity matrix between the detection boxes of the previous frame and the detection boxes of the current frame, as well as the motion similarity matrix between the predicted trajectory and the detection boxes of the current frame; based on the appearance similarity matrix, motion similarity matrix, target distribution in the current frame, and motion information of the detection boxes in the current frame, obtain the adaptive fusion weights for the current frame; based on the adaptive fusion weights for the current frame, fuse the appearance similarity matrix and motion similarity matrix to obtain the adaptive fusion similarity matrix for the current frame; convert the adaptive fusion similarity matrix for the current frame into a cost matrix, solve for the optimal matching box using the Hungarian algorithm, and form the first-stage matching result for the current frame with the high-level and / or mid-level detection boxes of the current frame;

[0009] S3: For the low-level detection box of the current frame in the video sequence, obtain the intersection-union matrix of the low-level detection box of the current frame and the unmatched trajectory, convert it into a cost matrix, solve for the optimal matching box using the Hungarian algorithm, and form the second-stage matching result of the current frame with the low-level detection box of the current frame;

[0010] S4: Merge the first-stage matching results and the second-stage matching results of the current frame to obtain the target tracking result of the current frame; the target tracking results of all frames are combined to form a multi-target tracking result.

[0011] The appearance similarity matrix is ​​obtained as follows: the input feature map is the detection box of the previous frame or the detection box of the current frame; the input feature map is processed by difference-aware attention processing and local block enhancement processing to obtain discriminative enhanced feature map and local feature enhanced feature map, and after feature fusion based on gate weight, a fused enhanced feature map is obtained; the fused enhanced feature maps corresponding to the detection boxes of the previous frame and the current frame are processed by fully connected processing, and the similarity matrix is ​​calculated as the appearance similarity matrix.

[0012] The local block enhancement process is as follows: the input feature map is divided into multiple grids to obtain multiple local feature blocks. Each local feature block is processed by a multi-head self-attention mechanism and then aggregated to obtain aggregated features. The aggregated features and the input feature map are connected by residuals to obtain a local feature enhancement feature map.

[0013] The motion similarity matrix is ​​obtained as follows: based on the center of the predicted trajectory in the current frame, the center of the detection box in the current frame, and the historical velocity of the predicted trajectory, the motion difference is obtained; based on the motion difference, the scale normalization factor, and the age adjustment factor, the motion penalty term is obtained; based on the motion penalty term and the intersection-union ratio of the predicted trajectory in the current frame and the detection box in the current frame, the motion similarity is obtained.

[0014] The method for obtaining the adaptive fusion weights of the current frame is as follows: based on the appearance similarity matrix and motion similarity matrix, the quality adjustment factor and motion stability factor are obtained; based on the detection target density and detection target spatial uniformity of the current frame, the scene adaptation factor is obtained; the velocity stability, orientation stability and acceleration smoothness of the detection box of the current frame are obtained, the motion stability factor is calculated, and weighted fusion is performed with the quality adjustment factor and the scene adaptation factor to obtain the adaptive fusion weights of the current frame.

[0015] Unmatched trajectories are predicted trajectories whose state prediction value or covariance prediction value is not within the range of their respective prediction values, or whose predicted trajectory coordinates are not within the range of video frame coordinates.

[0016] The predicted trajectory for the current frame is obtained by processing the detection box of the previous frame using a Kalman filter motion model.

[0017] An adaptive appearance motion fusion and hierarchical association multi-target tracking system, used to implement the above-mentioned adaptive appearance motion fusion and hierarchical association multi-target tracking method, includes:

[0018] The multi-level detection box generation module is used to obtain the detection box of the target in each frame of the video sequence through the target detector. The detection boxes are divided into three levels: high-level detection boxes, middle-level detection boxes and low-level detection boxes according to the preset confidence threshold; the higher the confidence, the higher the level.

[0019] The first-stage matching result generation module is used to obtain the appearance similarity matrix between the detection boxes of the previous frame and the detection boxes of the current frame, as well as the motion similarity matrix between the predicted trajectory of the current frame and the detection boxes of the current frame, for the high-level detection boxes and / or mid-level detection boxes in the current frame of the video sequence; based on the appearance similarity matrix, motion similarity matrix, current frame detection target distribution, and current frame detection box motion information, the current frame adaptive fusion weight is obtained; based on the current frame adaptive fusion weight, the appearance similarity matrix and motion similarity matrix are fused to obtain the current frame adaptive fusion similarity matrix; the current frame adaptive fusion similarity matrix is ​​converted into a cost matrix, and the optimal matching box is solved using the Hungarian algorithm, which, together with the current frame high-level detection boxes and / or mid-level detection boxes, forms the first-stage matching result of the current frame;

[0020] The second-stage matching result generation module is used to obtain the intersection-union matrix of the low-level detection box of the current frame and the unmatched trajectory for the low-level detection box of the current frame in the video sequence, convert it into a cost matrix, solve for the optimal matching box through the Hungarian algorithm, and form the second-stage matching result of the current frame with the low-level detection box of the current frame.

[0021] The multi-target tracking result generation module is used to merge the first-stage matching result and the second-stage matching result of the current frame to obtain the target tracking result of the current frame; the target tracking results of all frames are combined to form the multi-target tracking result.

[0022] An adaptive appearance motion fusion and hierarchical association multi-target tracking device includes a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the aforementioned adaptive appearance motion fusion and hierarchical association multi-target tracking method.

[0023] A computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned adaptive appearance motion fusion and hierarchical association multi-target tracking method.

[0024] The beneficial effects of this invention are as follows:

[0025] This invention provides a multi-target tracking method that integrates adaptive appearance and motion fusion with hierarchical association. First, detection boxes are layered using a confidence threshold to achieve differentiated processing, balancing accuracy and efficiency. Then, appearance and motion similarity are fused between high / mid-level detection boxes, and weights are adaptively assigned. The resulting adaptive fusion similarity matrix is ​​transformed into a cost matrix, which is then used to solve for the first-stage matching result, fully utilizing multi-dimensional features to improve matching accuracy. Next, the intersection-union matrix (IUCN) of the current frame's low-level detection boxes and unmatched trajectories is obtained, transformed into a cost matrix, and used to solve for the second-stage matching result. Finally, the two-stage matching results are integrated to obtain the target tracking result, effectively reducing the probability of missed and incorrect matches, ultimately improving multi-target tracking accuracy while optimizing tracking efficiency.

[0026] This invention provides an adaptive appearance-motion fusion and hierarchical association method for multi-target tracking. It utilizes hierarchical processing of detection box confidence, comparison of appearance and motion similarity, a differentiated association strategy, and a multi-dimensional scene adaptive fusion mechanism to construct an efficient and robust multi-target tracking framework. Experimental results show that on the MOT17 dataset, this method significantly improves upon the OC-SORT method in terms of HOTA, MOTA, and IDF1, reducing the number of ID switching by 60.5%, effectively improving tracking accuracy, efficiency, robustness, and real-time performance. This demonstrates that the method can achieve accurate and stable tracking of multiple moving targets in dense and complex scenes, exhibiting comprehensive advantages in tracking accuracy, efficiency, robustness, and real-time performance. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the exemplary embodiments of this application clearer, the technical solutions in the exemplary embodiments of this application are described clearly and completely below. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0028] Example 1

[0029] This embodiment provides a multi-target tracking method with adaptive appearance motion fusion and hierarchical association, including the following operations:

[0030] S1: Obtain the target bounding box in each frame of the video sequence using the target detector, and classify the bounding boxes into three levels: high-level bounding box, mid-level bounding box, and low-level bounding box according to a preset confidence threshold; the higher the confidence, the higher the level.

[0031] S2: For the high-level and / or mid-level detection boxes in the current frame of the video sequence, obtain the appearance similarity matrix between the detection boxes of the previous frame and the detection boxes of the current frame, as well as the motion similarity matrix between the predicted trajectory and the detection boxes of the current frame; based on the appearance similarity matrix, motion similarity matrix, target distribution in the current frame, and motion information of the detection boxes in the current frame, obtain the adaptive fusion weights for the current frame; based on the adaptive fusion weights for the current frame, fuse the appearance similarity matrix and motion similarity matrix to obtain the adaptive fusion similarity matrix for the current frame; convert the adaptive fusion similarity matrix of the current frame into a cost matrix, solve for the optimal matching box using the Hungarian algorithm, and form the first-stage matching result of the current frame with the high-level and / or mid-level detection boxes of the current frame;

[0032] S3: For the low-level detection box of the current frame in the video sequence, obtain the intersection-union matrix of the low-level detection box of the current frame and the unmatched trajectory, convert it into a cost matrix, solve for the optimal matching box using the Hungarian algorithm, and form the second-stage matching result of the current frame with the low-level detection box of the current frame;

[0033] S4: Merge the first-stage matching results and the second-stage matching results of the current frame to obtain the target tracking result of the current frame; the target tracking results of all frames are combined to form a multi-target tracking result.

[0034] The specific steps are detailed below.

[0035] S1: Obtain the target bounding box in each frame of the video sequence through the target detector, and divide the bounding boxes into three levels: high-level bounding box, mid-level bounding box, and low-level bounding box according to the preset confidence threshold; the higher the confidence, the higher the level.

[0036] Object detection is performed on each frame of the input video sequence using an object detector (including but not limited to the YOLOv8 model), obtaining the bounding box of the target in each frame. Each bounding box includes the bounding box coordinates and a confidence score. Based on preset confidence thresholds τ1=0.6, τ2=0.3, and τ3=0.1, the bounding boxes are divided into three levels: high-level, mid-level, and low-level. Higher confidence scores correspond to higher levels. This hierarchical processing mechanism achieves differentiated detection quality through confidence thresholds, providing a foundation for subsequent association strategies.

[0037] S2: For the high-level and / or mid-level detection boxes in the current frame of the video sequence, obtain the appearance similarity matrix between the detection boxes of the previous frame and the detection boxes of the current frame, as well as the motion similarity matrix between the predicted trajectory and the detection boxes of the current frame; based on the appearance similarity matrix, motion similarity matrix, the distribution of detected targets in the current frame, and the motion information of the detection boxes in the current frame, obtain the adaptive fusion weights for the current frame; based on the adaptive fusion weights for the current frame, fuse the appearance similarity matrix and motion similarity matrix to obtain the adaptive fusion similarity matrix for the current frame; convert the adaptive fusion similarity matrix for the current frame into a cost matrix, solve for the optimal matching box using the Hungarian algorithm, and form the first-stage matching result of the current frame with the high-level and / or mid-level detection boxes of the current frame.

[0038] Step 1: In the starting frame of the video sequence, create initial trajectories for each high-level and mid-level detection box: assign a unique identifier, initialize the Kalman filter state vector, extract initial appearance features (i.e., extract the fused enhanced feature map of the detection box), and add them to the trajectory set.

[0039] Specifically, for the starting frame of the video sequence, the system initializes an empty trajectory set; iterates through each high-level and mid-level detection box, performing the following initialization operations: assigning a globally unique identifier to the trajectory, and initializing the state vector x of the Kalman filter. k =[x,y,w,h,0,0,0,0] T Where (x,y) are the center coordinates of the detection box, and (w,h) are the width and height of the detection box; initialize the trajectory age. k =1; Use the feature extractor to extract the appearance features of the detection box. f k (i.e., extract the fused enhanced feature map of the detection box); set the trajectory not updated counter time_since_update=0; add the initial trajectory to the trajectory set; this initialization step establishes the starting point for the entire tracking process, ensuring that the system can build continuous trajectories from reliable detection results.

[0040] Step 2: Process the detection box of the previous frame (not the starting frame) using a Kalman filter motion model to obtain the predicted trajectory of the current frame.

[0041] In this embodiment, the state prediction is achieved through the Kalman filter motion model processing using the following formula: , This represents the trajectory state of the current frame. For the uniform motion model of the target, The state of the detection block in the previous frame is used; covariance prediction is achieved through the following formula: , The current frame trajectory covariance, The covariance of the detection boxes in the previous frame. Let be the process noise covariance matrix.

[0042] In addition, for predicted trajectories that fail (such as those with unstable values ​​or invalid bounding boxes), the predicted trajectories whose state prediction values ​​or covariance prediction values ​​are not within their respective prediction value ranges, or whose predicted trajectory coordinates are not within the range of video frame coordinates, are marked as unmatched trajectories and are to be deleted. They are then restored using the last valid observation value. This prediction step utilizes the target's motion priors, providing important spatiotemporal constraints for data association and significantly reducing the search space.

[0043] Step 3: Obtain the appearance similarity matrix between the detection box in the previous frame and the detection box in the current frame. The specific steps are detailed below.

[0044] Step 3.1: The input feature map is the detection box of the previous frame or the detection box of the current frame (the detection box is a high-level detection box or a middle-level detection box); the input feature map is processed by difference-aware attention processing and local block enhancement processing to obtain discriminative enhanced feature map and local feature enhanced feature map.

[0045] The differential awareness attention processing operation is as follows: a differential awareness map is generated by calculating the channel variance of the input feature map, an attention weight matrix is ​​generated by two convolutional layers, and the matrix is ​​multiplied element-wise with the input feature map to obtain a discriminative enhancement feature map.

[0046] The specific calculation formula is as follows:

[0047] ,

[0048] ,

[0049] ,

[0050] ,

[0051] For channel variance, Here is the formula for calculating variance. For the input feature map, , These are the height and width of the input feature map, respectively. This represents the total number of channel indexes. For channel index, This is a difference perception map. Here is the attention weight matrix. For ReLU activation function, For the sigmoid function, , These are the first learnable parameter and the second learnable parameter, respectively. This is a discriminative feature map.

[0052] The method for obtaining the local feature enhancement feature map is as follows: the input feature map is divided into multiple (G×G) grids to obtain multiple local feature blocks. After processing each local feature block with a multi-head self-attention mechanism, they are aggregated to obtain aggregated features. The aggregated features are then residually connected with the input feature map to obtain the local feature enhancement feature map that captures fine-grained appearance features.

[0053] Step 3.2: The discriminative enhanced feature map and the local enhanced feature map are fused together based on gating weights to obtain the fused enhanced feature map.

[0054] Specifically, the discriminative enhanced feature map is obtained by performing global average pooling on the discriminative enhanced feature map. The gate weights of the discriminative pooling feature map are then obtained (which can be implemented using the sigmoid function). Based on the gate weights, the discriminative enhanced feature map and the local enhanced feature map are fused to obtain a fused enhanced feature map that balances global and local features. The specific processing is as follows:

[0055] ，

[0056] ，

[0057] ,

[0058] To distinguish pooling feature maps, This is a global average pooling process. The learnable coefficient, For gating weights, Enhance the feature map for local features.

[0059] Step 3.3: The fused and enhanced feature maps corresponding to the detection boxes in the previous frame and the current frame are processed by fully connected processing, compressed to obtain their respective feature vectors, and the similarity matrix is ​​calculated as the appearance similarity matrix.

[0060] The appearance similarity matrix is ​​calculated using the following formula:

[0061] ,

[0062] This is the appearance similarity matrix between the detection boxes in the previous frame and the detection boxes in the current frame. , These are the feature vectors corresponding to the fused enhanced feature maps of the detection boxes in the previous frame and the current frame, respectively.

[0063] Simultaneously, a motion similarity matrix is ​​obtained based on the predicted trajectory and the detection box of the current frame. Specifically, motion difference is obtained based on the center of the predicted trajectory, the center of the detection box, and the historical velocity of the predicted trajectory; motion penalty term is obtained based on motion difference, scale normalization factor, and age adjustment factor set according to trajectory lifecycle; motion similarity is obtained based on motion penalty term and the intersection-union ratio of the predicted trajectory and the detection box of the current frame.

[0064] The specific formula for calculating motion similarity is as follows:

[0065] ,

[0066] ，

[0067] ,

[0068] Due to differences in movement, , These are the center of the detection box in the current frame and the center of the predicted trajectory in the current frame, respectively. To predict historical velocities, To predict the magnitude of the historical velocity of the trajectory, (the average velocity of the last 3 frames is taken, taking into account both stability and real-time performance). The time interval is used to calculate the position deviation after velocity compensation. Historical speeds are used to correct for positional deviations, reflecting the consistency of movement trends. As a weighting factor for velocity modulus, As the first compensation term, the formula for calculating motion difference can avoid the traditional motion difference calculation that only calculates the Euclidean distance at the current position, thus ignoring the correlation of motion trend; The adjustment coefficient is set to 0.6. The scale normalization factor, This is the age adjustment factor for the detection bounding box in the current frame. This is a sports penalty item. This is the intersection-union ratio (IoU) between the predicted trajectory (corresponding bounding box) of the current frame and the detected bounding box of the current frame.

[0069] Step 4: Based on the appearance similarity matrix, motion similarity matrix, current frame detected target, and current frame detection box motion information, obtain the current frame adaptive fusion weight.

[0070] The method for obtaining the adaptive fusion weights for the current frame is as follows: based on the appearance similarity matrix and motion similarity matrix, a quality adjustment factor is obtained; based on the detection target density and detection target spatial uniformity of the current frame, a scene adaptation factor is obtained; the velocity stability, orientation stability, and acceleration smoothness of the detection box in the current frame (all of which belong to the motion information of the detection box in the current frame) are obtained, a motion stability factor is calculated, and weighted and fused with the quality adjustment factor and the scene adaptation factor to obtain the adaptive fusion weights for the current frame.

[0071] The quality adjustment factor, scene adaptation factor, and motion stability factor are calculated using the following formulas:

[0072] ,

[0073] ,

[0074] ,

[0075] For quality adjustment factor, , These are the feature vectors corresponding to the appearance similarity matrix and the motion similarity matrix, respectively. As the second compensation item, It is the sigmoid function; As a scene adaptation factor, , , These are the detection target density, detection target spatial uniformity, detection target density, and detection box normalization ratio for the current frame, respectively. As a motion stability factor, , , The current frame number k The detection frame velocity stability, directional stability, and acceleration smoothness of each detection frame. This represents the total number of detection boxes in the current frame.

[0076] Step 5: Based on the adaptive fusion weight of the current frame, fuse the appearance similarity matrix and the motion similarity matrix to obtain the adaptive fusion similarity matrix of the current frame.

[0077] Step 6: Convert the adaptive fusion similarity matrix of the current frame into a cost matrix, solve for the optimal matching box using the Hungarian algorithm, and combine it with the high-level detection box and / or the middle-level detection box of the current frame to form the first-stage matching result of the current frame.

[0078] S3: For the low-level detection bounding box of the current frame in the video sequence, obtain the intersection-union matrix (IUCN) between the current frame's low-level detection bounding box and the unmatched trajectory, convert it into a cost matrix, and use the Hungarian algorithm to solve for the optimal matching bounding box. This optimal matching bounding box, along with the current frame's low-level detection bounding box, forms the second-stage matching result for the current frame. Unmatched trajectories are those whose state prediction value or covariance prediction value is outside their respective prediction value range, or whose predicted trajectory coordinates are outside the video frame coordinate range.

[0079] S4: Merge (integrate) the first-stage matching results and the second-stage matching results of the current frame to obtain the target tracking result of the current frame; the target tracking results of all frames are combined to form a multi-target tracking result. This two-stage association strategy, through a hierarchical processing mechanism, ensures accurate matching of high-quality detections while making full use of the trajectory completion capabilities of low-quality detections, significantly improving the tracking recall rate.

[0080] Furthermore, to achieve trajectory updating and management, this embodiment maintains the entire lifecycle of the trajectory based on the association results of S4. For successfully matched trajectories, Kalman filtering is used to update their motion state, including Kalman gain calculation, state update, and covariance update. Appearance features are updated using an exponential smoothing strategy. Trajectories that have not been updated for 30 consecutive frames are terminated to release system resources. Simultaneously, the trajectory's age, hit count, and other state information are maintained to provide a basis for subsequent processing decisions. This trajectory management step, through a complete lifecycle management mechanism, ensures the stability, continuity, and resource utilization efficiency of the tracking system.

[0081] To verify the effectiveness of the method in this embodiment, the following experiment was conducted.

[0082] In the experiment, YOLOv8 was used as the detection model, and the method of this embodiment and existing state-of-the-art methods were used as the tracking models. Performance was compared on the MOT17 dataset (MOT17 is one of the MOT Challenge series datasets, which includes dense scenes with complex lighting changes, occlusion, and viewpoint changes. The dataset includes annotations of moving pedestrians, as well as stationary pedestrians, cyclists, or skateboarders, etc.). The experimental results are shown in Table 1. Evaluation metrics included HOTA, MOTA, IDF1, and IDs. HOTA aims to evaluate both detection and association performance in a balanced way; it assesses whether the target is accurately bounded and whether the bounded target is correctly identified. MOTA is a metric that measures overall detection accuracy. It evaluates performance by counting the number of errors (missed detections, false detections, ID switching) across all frames. IDF1 is a metric that measures identity association consistency, used to evaluate the tracker's ability to maintain the correct ID for the same target. IDs refers to the number of ID switchings; it calculates the number of times the same target is incorrectly assigned different IDs throughout the video.

[0083] As shown in Table 1, on the MOT17 dataset, the method in this embodiment improves HOTA by 7.1% compared to OC-SORT, MOTA by 74.8% compared to OC-SORT by 6.8%, IDF1 by 5.9%, and IDs by 60.5%. In comparison, the method in this embodiment significantly reduces the number of ID switching on the MOT17 dataset while maintaining high tracking accuracy, indicating that the method in this embodiment can achieve higher tracking accuracy and better identity preservation capabilities.

[0084] Table 1. Comparison of processing results of the method in this embodiment and OC-SORT on the MOT17 dataset.

[0085]

[0086] This embodiment also provides an adaptive appearance motion fusion and hierarchical association multi-target tracking system to implement the above-mentioned adaptive appearance motion fusion and hierarchical association multi-target tracking method, including:

[0087] The multi-level detection box generation module is used to obtain the detection box of the target in each frame of the video sequence through the target detector. The detection boxes are divided into three levels: high-level detection boxes, middle-level detection boxes and low-level detection boxes according to the preset confidence threshold; the higher the confidence, the higher the level.

[0088] An adaptive fusion similarity matrix and intersection-over-union (IoU) matrix generation module is used to obtain, for each frame of a video sequence, the appearance similarity matrix between the previous frame detection box and the current frame detection box of the same target, as well as the motion similarity matrix between the current frame predicted trajectory and the current frame detection box, for high-level and / or mid-level detection boxes of the same target; based on the appearance similarity matrix, motion similarity matrix, current frame target distribution, and current frame detection box motion information, the current frame adaptive fusion weight is obtained; based on the current frame adaptive fusion weight, the appearance similarity matrix and motion similarity matrix are fused to obtain the current frame adaptive fusion similarity matrix; for each frame of a video sequence, the IoU matrix is ​​obtained between the current frame low-level detection box of the same target and the corresponding unmatched trajectory.

[0089] The multi-target tracking result generation module converts the adaptive fusion similarity matrix and intersection-union matrix of the current frame into cost matrices, solves for the optimal matching box using the Hungarian algorithm, and forms the current frame detection matching set with the high-level detection boxes and / or mid-level detection boxes and / or low-level detection boxes of the current frame, which serves as the target tracking result of the current frame; the target tracking results of all frames form the multi-target tracking result.

[0090] This embodiment also provides an adaptive appearance motion fusion and hierarchical association multi-target tracking device, including a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the above-described adaptive appearance motion fusion and hierarchical association multi-target tracking method.

[0091] This embodiment also provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the above-described adaptive appearance motion fusion and hierarchical association multi-target tracking method.

[0092] This embodiment provides a multi-target tracking method that integrates adaptive appearance and motion fusion with hierarchical association. First, detection boxes are layered using a confidence threshold to achieve differentiated processing, balancing accuracy and efficiency. Then, appearance and motion similarity are fused between high / mid-level detection boxes, and weights are adaptively assigned. The resulting adaptive fusion similarity matrix is ​​converted into a cost matrix, which is then used to solve for the first-stage matching result, fully utilizing multi-dimensional features to improve matching accuracy. Next, the intersection-union matrix (IUCN) of the current frame's low-level detection boxes and unmatched trajectories is obtained, converted into a cost matrix, and used to solve for the second-stage matching result. Finally, the two-stage matching results are integrated to obtain the target tracking result, effectively reducing the probability of missed and incorrect matches, ultimately improving both multi-target tracking accuracy and tracking efficiency.

[0093] This embodiment provides an adaptive appearance-motion fusion and hierarchical association multi-target tracking method. It utilizes hierarchical processing of detection box confidence, comparison of appearance and motion similarity, a differentiated association strategy, and a multi-dimensional scene adaptive fusion mechanism to construct an efficient and robust multi-target tracking framework. Experimental results show that on the MOT17 dataset, the proposed method significantly improves upon the OC-SORT method in terms of HOTA, MOTA, and IDF1, reducing the number of ID switching by 60.5%, effectively improving tracking accuracy, efficiency, robustness, and real-time performance. This demonstrates that the proposed method can achieve accurate and stable tracking of multiple moving targets in dense and complex scenes, exhibiting comprehensive advantages in tracking accuracy, efficiency, robustness, and real-time performance.

[0094] While exemplary embodiments of the invention have been described herein, many other variations or modifications conforming to the principles of the invention can be directly determined or derived from the disclosure of this invention without departing from its spirit and scope. Therefore, the scope of the invention should be understood and recognized to cover all such other variations or modifications.

Claims

1. A method for multi-target tracking with adaptive appearance motion fusion and hierarchical association, characterized in that, The method comprises the following steps of: S1: obtaining a detection box of a target in each frame of a video sequence by a target detector, and dividing the detection box into three levels of high-level detection box, middle-level detection box and low-level detection box according to a preset confidence threshold; The greater the confidence is, the higher the level is; S2: for the high-level detection box and / or the middle-level detection box of a current frame in the video sequence, obtaining an appearance similarity matrix of a previous frame detection box and the current frame detection box, and a motion similarity matrix of a current frame predicted trajectory and the current frame detection box; Based on the appearance similarity matrix, the motion similarity matrix, the current frame detection target distribution and the current frame detection box motion information, a current frame adaptive fusion weight is obtained, and the obtaining method is as follows: based on the appearance similarity matrix and the motion similarity matrix, a quality adjustment factor and a motion stability factor are obtained; Based on the detection target density and the detection target spatial uniformity of the current frame, a scene adaptation factor is obtained; The current frame detection box speed stability, direction stability and acceleration smoothness are obtained, the motion stability factor is calculated, and the quality adjustment factor and the scene adaptation factor are weighted and fused to obtain the current frame adaptive fusion weight; The quality adjustment factor, the scene adaptation factor and the motion stability factor are calculated by the following formulas respectively: ， ， ， For quality adjustment factor, , These are the feature vectors corresponding to the appearance similarity matrix and the motion similarity matrix, respectively. As the second compensation item, It is the sigmoid function; As a scene adaptation factor, , , These are the detection target density, detection target spatial uniformity, and detection box normalization ratio for the current frame, respectively. As a motion stability factor, , , The current frame number k The detection frame velocity stability, directional stability, and acceleration smoothness of each detection frame. This represents the total number of detection boxes in the current frame. Based on the current frame adaptive fusion weight, the appearance similarity matrix and the motion similarity matrix are fused to obtain a current frame adaptive fusion similarity matrix; The current frame adaptive fusion similarity matrix is converted into a cost matrix, and the optimal matching box is solved by the Hungarian algorithm, and the current frame high-level detection box and / or the middle-level detection box form a first stage matching result of the current frame; S3: obtaining an intersection-over-union matrix of the current frame low-level detection box and the corresponding unmatched trajectory, converting the intersection-over-union matrix of the current frame into a cost matrix, and solving the optimal matching box by the Hungarian algorithm, and the current frame low-level detection box form a second stage matching result of the current frame; S4: merging the first stage matching result of the current frame and the second stage matching result of the current frame to obtain a target tracking result of the current frame; All frame target tracking results form a multi-target tracking result.

2. The multi-target tracking method of claim 1, wherein, In S2, the appearance similarity matrix is obtained by the following method: The input feature map is the previous frame detection box or the current frame detection box; The input feature map is processed by difference perception attention and local block enhancement to obtain discriminative enhanced feature map and local feature enhanced feature map, and the fusion enhanced feature map is obtained after feature fusion based on the gate weight; The fusion enhanced feature maps corresponding to the previous frame detection box and the current frame detection box are respectively processed by full connection, and the similarity matrix is calculated as the appearance similarity matrix.

3. The multi-target tracking method of claim 1, wherein, In S2, the operation of local block enhancement processing is as follows: The input feature map is divided into multiple grids to obtain multiple local feature blocks, each local feature block is processed by multi-head self-attention mechanism, and then aggregated to obtain aggregated features; The aggregated features and the input feature map are connected by a residual connection to obtain a local feature enhanced feature map.

4. The multi-target tracking method of claim 1, wherein, In S2, the motion similarity matrix is obtained by the following method: Based on the current frame predicted trajectory center, the current frame detection box center and the predicted trajectory historical speed, a motion difference is obtained; The motion penalty term is obtained based on the motion difference, the scale normalization factor, and the age adjustment factor. The motion similarity is obtained based on the motion penalty term and the intersection over union of the current frame predicted trajectory and the current frame bounding box.

5. The multi-target tracking method of claim 1, wherein, The unmatched trajectory is a predicted trajectory whose state prediction value or covariance prediction value is not within the corresponding prediction value range, or whose predicted trajectory coordinates are not within the video frame coordinate range.

6. The multi-target tracking method of claim 5, wherein, The current frame predicted trajectory is obtained by processing the last frame bounding box through the Kalman filter motion model.

7. An adaptive appearance-motion-fusion and hierarchical association multi-target tracking system for implementing the adaptive appearance-motion-fusion and hierarchical association multi-target tracking method of claim 1, characterized by The method comprises the following steps: The multi-level bounding box generation module is configured to obtain the bounding box of the target in each frame of the video sequence through a target detector, and divide the bounding box into three levels, i.e., high-level bounding box, middle-level bounding box, and low-level bounding box, according to a preset confidence threshold. The higher the confidence is, the higher the level is. The first-stage matching result generation module is configured to obtain the appearance similarity matrix of the last frame bounding box and the current frame bounding box, and the motion similarity matrix of the current frame predicted trajectory and the current frame bounding box, for the high-level bounding box and / or the middle-level bounding box of the current frame. The current frame adaptive fusion weight is obtained based on the appearance similarity matrix, the motion similarity matrix, the current frame target distribution, and the current frame bounding box motion information. The current frame adaptive fusion similarity matrix is obtained by fusing the appearance similarity matrix and the motion similarity matrix based on the current frame adaptive fusion weight. The current frame adaptive fusion similarity matrix is converted into a cost matrix, and the optimal matching box is solved through the Hungarian algorithm, so as to form the first-stage matching result of the current frame together with the high-level bounding box and / or the middle-level bounding box of the current frame. The second-stage matching result generation module is configured to obtain the intersection over union matrix of the low-level bounding box of the current frame and the unmatched trajectory, convert the intersection over union matrix into a cost matrix, and solve the optimal matching box through the Hungarian algorithm, so as to form the second-stage matching result of the current frame together with the low-level bounding box of the current frame. The multi-target tracking result generation module is configured to combine the first-stage matching result of the current frame and the second-stage matching result of the current frame to obtain the target tracking result of the current frame. All the frame target tracking results form the multi-target tracking result.

8. A multi-target tracking device with adaptive appearance motion fusion and hierarchical association, characterized in that, The device comprises a processor and a memory, wherein the processor executes the computer program stored in the memory to realize the adaptive appearance-motion fusion and hierarchical association multi-target tracking method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The device comprises a processor and a memory, wherein the processor executes the computer program stored in the memory to realize the adaptive appearance-motion fusion and hierarchical association multi-target tracking method according to any one of claims 1-6.

Images