Multi-target tracking and segmentation method and system based on dynamic adaptive and enhanced features

Abstract

The present invention relates to the field of image processing technology, and proposes a multi-target tracking and segmentation method and system based on dynamic adaptation and enhanced features, the method comprising a pre-processing step, an appearance cost calculation step, a pre-matching step, a motion cost calculation step, a formal matching step, a post-processing step, a repetition step, and a visualization step. The method refines and integrates target features through data association with dynamic adaptation, and respectively enhances target appearance features and motion information by using a mask-based attention mechanism and a mask prediction based on four-triangle transformation, so as to achieve fast calculation and low-cost operation while maintaining high tracking accuracy, and is suitable for a wider range of application scenarios.

Description

Technical Field

[0001] The present invention relates to the technical fields of image processing and video signal processing, and in particular to a multi-target tracking and segmentation method and system based on dynamic self-adaptation and enhanced features. Background Art

[0002] Multi-target tracking and segmentation refers to the method of simultaneously identifying, tracking and distinguishing multiple moving targets in a video or image sequence. It is a systematic method that aims to evaluate the relationship between multiple moving targets at the pixel level in complex scenarios such as occlusion, similar targets, fast movement and perspective changes by comprehensively applying key steps such as target detection, feature extraction, data association and instance segmentation. The currently popular multi-target tracking and segmentation systems rely on deep learning technology to provide robust association information for data association during use. For example, a single instance segmentation model is trained to obtain high-quality instance segmentation masks, a deep learning model is relied on for state estimation, and a deep learning model is trained for association tracking in the image domain. At the same time, in resource-constrained or real-time environments, the high computational requirements of deep learning models may make it difficult to meet the requirements of efficient operation. In addition, the training and deployment costs of deep learning models are relatively high, and the complexity of the model may affect the scalability and maintainability of the system. At the same time, the current multi-target tracking and segmentation methods are still crude in the use of target features in videos. At the same time, the existing multi-target tracking technology is insufficient in handling dynamic environments, large-scale target scenes or extreme lighting conditions. In addition, the generalization ability, robustness and scalability of the algorithm are also issues that need to be further addressed in current research. The Chinese invention application with publication number CN116452631A discloses a multi-target tracking method that uses a strategy of first detecting and segmenting frames and then associating data, using traditional appearance and motion information for matching, and integrating pre-matching information and selecting a weighted fast association algorithm. However, this method is fixed when fusion weights are used and cannot dynamically adapt to the video environment. At the same time, the robustness of motion and appearance cues is limited. Summary of the invention

[0003] The purpose of the present invention is to solve the problems in the prior art.

[0004] The technical solution adopted by the present invention to solve the technical problem is: to provide a multi-target tracking and segmentation method based on dynamic adaptation and enhanced features, comprising the following steps:

[0005] Pre-processing step: read image or video information, perform target segmentation on each frame of image, and obtain target pixel-level information and target bounding box information;

[0006] The appearance cost calculation step uses the re-identification model to process the target pixel-level information and the target bounding box information to obtain the target mask and background features, and calculates the appearance feature vector based on the target mask and background features, thereby calculating the appearance cost matrix;

[0007] In the pre-matching step, the appearance cost matrix is ​​processed by the Hungarian algorithm to obtain matching candidate pairs and unmatched candidate pairs;

[0008] The motion cost calculation step uses the four vertices and the center point on the target bounding box as five key points, and obtains the speed of the five key points through the Kalman filter; maps the target mask of the previous frame image to the four triangles formed by the five key points and uses the key point speeds to perform motion simulation and calculate the motion cost matrix;

[0009] In the formal matching step, the motion cost is used as the final cost for the unmatched candidate pairs; the fusion weight is calculated for the matched candidate pairs based on the historical mask area, and the appearance cost and the motion cost are fused based on the fusion weight to obtain the final cost; the final cost of the unmatched candidate pairs and the final cost of the matched candidate pairs are integrated to obtain the final cost matrix, which is processed by the Hungarian algorithm to obtain the matched pairs and the unmatched pairs;

[0010] In the post-processing step, Kalman filter prediction is performed on all trajectories, and the position information is updated and corrected through the Kalman filter algorithm to obtain a new trajectory set. The trajectories and detection targets in the new trajectory set are divided into confirmed trajectories and unconfirmed trajectories according to the matching status. The confirmed trajectories are sent to the pre-matching step for the next pre-matching, and the unconfirmed trajectories are sent to the formal matching step for the next formal matching;

[0011] Repeat the steps, repeat the pre-matching step to the post-processing step until the matching is completed, and enter the visualization step;

[0012] The visualization step visualizes the trajectory of the new confirmation state and outputs a video with the detected target marked.

[0013] Preferably, the appearance cost calculation step adopts a mask-based attention mechanism, which specifically includes the following steps:

[0014] Pass the target pixel-level information and target bounding box information to the re-identification model to obtain the target mask M and background feature map f;

[0015] Give the mask M feature map Weight, giving the background feature map Weight, perform weighted fusion to obtain the appearance feature vector , expressed as:

[0016] ;

[0017] in, > ; (n,m) is the pixel coordinate of the background feature map f, ⊙ represents element multiplication operation; M(n,m)=1 represents ROI, M(n,m)=0 represents the background area;

[0018] The appearance feature vectors of the N matched targets in the t-1th frame of the video are taken as the trajectory T, and the feature vectors of the appearance features of the M targets in the tth frame of the video are taken as the trajectory S; the cosine distance of the appearance similarity between the i-th trajectory T[i] and the j-th detection S[j] is calculated. , get N*M appearance cost matrix :

[0019] ;

[0020] ;

[0021] in, represents the appearance feature vector of trajectory T[i], Represents the appearance feature vector of detection S[j]; and They represent the Euclidean norm of trajectory T[i] and detection S[j] respectively.

[0022] Preferably, in the pre-matching step, the N*M appearance cost matrix is ​​processed by the Hungarian algorithm to obtain the candidate pair with the minimum cost as the pre-matching candidate pair, and the other candidate pairs are used as unmatched candidate pairs.

[0023] Preferably, the motion cost calculation step includes the following steps:

[0024] Get the speed of the four points and the center point on the target bounding box through the Kalman filter ;

[0025] For the target mask of the t-1th frame, map it to the four triangles formed by these key points;

[0026] Using the velocities of these five points, we use affine-like transformation to simulate the motion of the mask to obtain the predicted mask of the tth frame and the motion information of the trajectory T[i];

[0027] By calculating the Mask-IoU score between the trajectory T[i] and the detection S[j], we get the N*M motion cost matrix :

[0028] ;

[0029] ;

[0030] in, represents the affine transformation function, is the key point predicted in the tth frame, Δ represents a triangle formed by the three points after the symbol as vertices; Represents the Mask-IoU score; represents a union, Indicates intersection.

[0031] Preferably, in the formal matching step, the final cost matrix obtained is C, where the matrix elements are expressed as:

[0032] ;

[0033] in, is the fusion weight calculated based on the historical mask area.

[0034] Preferably, the calculation process of the fusion weight includes the following steps:

[0035] Get the historical mask area, expressed as ,in represents the mask area of ​​the ith tracked object in the t-1th frame, and N represents the total number of tracked objects;

[0036] An exponential moving average mechanism is used to predict the predicted mask area of ​​the i-th object in the t-th frame. , expressed as:

[0037] ;

[0038] in, is the approximate mask region of the ith object in the t-1th frame, is the momentum term, Represents the detection mask area of ​​the i-th object in the t-th frame;

[0039] The Sigmoid function is applied to convert the rate of change into a continuous and smooth weight coefficient, namely the fusion weight, which is expressed as:

[0040] ;

[0041] in, Represents the approximate mask area of ​​the i-th object in the t-th frame The detection mask area of ​​the jth object in the tth frame The relative rate of change between .

[0042] Preferably, the post-processing step comprises:

[0043] In the first classification step, Kalman filter prediction is performed on all trajectories, and the position information is updated and corrected through the Kalman filter algorithm to obtain a new trajectory set; the trajectory that matches the detection target for three consecutive frames is identified as a confirmed trajectory and sent to the pre-matching step for the next pre-matching; the trajectory that fails to match the detection target for three consecutive frames is identified as an unconfirmed trajectory and sent to the formal matching step for the next formal matching; the trajectory that does not match the detection target is regarded as a failed matching trajectory and sent to the second classification step;

[0044] In the second classification step, the detected target is sent into the trajectory set as an unconfirmed trajectory and enters the first classification step; it is determined whether the trajectory is confirmed. If it is confirmed, it is determined whether its trajectory survival time a exceeds the threshold. The unconfirmed trajectories and the confirmed trajectories whose trajectory survival time a is greater than the threshold are deleted, and the trajectories that are not deleted are sent into the trajectory set and enter the first classification step.

[0045] The present invention also provides a multi-target tracking and segmentation system based on dynamic adaptation and enhanced features, comprising:

[0046] The pre-processing module reads the image or video information, performs target segmentation on each frame of the image, and obtains the target pixel-level information and target bounding box information;

[0047] The appearance cost calculation module uses the re-identification model to process the target pixel-level information and target bounding box information to obtain the target mask and background features, and calculates the appearance feature vector based on the target mask and background features, thereby calculating the appearance cost matrix;

[0048] The pre-matching module processes the appearance cost matrix through the Hungarian algorithm to obtain matching candidate pairs and unmatched candidate pairs;

[0049] The motion cost calculation module uses the four vertices and the center point on the target bounding box as five key points, and obtains the speed of the five key points through the Kalman filter; maps the target mask of the previous frame image to the four triangles formed by the five key points and uses the key point speeds to perform motion simulation and calculate the motion cost matrix;

[0050] In the formal matching module, the motion cost is used as the final cost for the unmatched candidate pairs; the fusion weight is calculated for the matched candidate pairs based on the historical mask area, and the appearance cost and the motion cost are fused based on the fusion weight to obtain the final cost; the final cost of the unmatched candidate pairs and the final cost of the matched candidate pairs are integrated to obtain the final cost matrix, which is processed by the Hungarian algorithm to obtain the matched pairs and the unmatched pairs;

[0051] The post-processing module performs Kalman filter prediction on all trajectories, updates and corrects the position information through the Kalman filter algorithm, and obtains a new trajectory set. The trajectories and detection targets in the new trajectory set are divided into confirmed trajectories and unconfirmed trajectories according to the matching status. The confirmed trajectories are sent to the pre-matching step for the next pre-matching, and the unconfirmed trajectories are sent to the formal matching step for the next formal matching;

[0052] Repeat module, repeat the pre-matching step to the post-processing step until the matching is completed, and enter the visualization module;

[0053] The visualization module visualizes the trajectory of the new confirmed state and outputs a video with the detected target marked.

[0054] The present invention has the following beneficial effects:

[0055] (1) The present invention proposes a multi-target tracking and segmentation method based on dynamic adaptation and enhanced features, which can achieve fast calculation and low-cost operation while maintaining high tracking accuracy, and is suitable for a wider range of application scenarios;

[0056] (2) The present invention improves the robustness of target appearance and motion features through a mask-based attention mechanism and a mask shape prediction enhancement based on four-triangle transformation.

[0057] (3) The present invention uses the law of mask area change to calculate dynamic weights to integrate target appearance and motion features, and realizes tracking of multiple targets through dynamic adaptive data association;

[0058] (4) The present invention optimizes data association and realizes dynamic adaptive utilization of appearance features and motion information from a dynamic perspective, integrating motion and appearance costs through dynamic weights; at the same time, the mask-based attention mechanism and the mask prediction based on four-triangle transformation are used to strengthen the target appearance features and motion information respectively.

[0059] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A method step diagram of an embodiment of the present invention;

[0061] Figure 2 A schematic diagram of a flow chart of an embodiment of the present invention;

[0062] Figure 3 Schematic diagram of a mask-based attention mechanism according to an embodiment of the present invention;

[0063] Figure 4 A schematic diagram of calculating motion cost based on mask prediction of four-triangle transformation according to an embodiment of the present invention;

[0064] Figure 5 A schematic diagram of calculating a dynamic weight based on a mask area change rate according to an embodiment of the present invention;

[0065] Figure 6 4 is a system structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0066] See also Figure 1 and Figure 2 The method step diagram and flow chart of an embodiment of the present invention are shown, which include the following steps:

[0067] S101, a pre-processing step, reading image or video information, performing target segmentation on each frame of the image, and obtaining target pixel-level information and target bounding box information;

[0068] S102, an appearance cost calculation step, using the re-identification model to process the target pixel-level information and the target bounding box information, obtain the target mask and background features, calculate the appearance feature vector based on the target mask and background features, and thus calculate the appearance cost matrix;

[0069] S103, a pre-matching step, processing the appearance cost matrix by the Hungarian algorithm to obtain matching candidate pairs and unmatched candidate pairs;

[0070] S104, a motion cost calculation step, using four vertices and a center point on the target bounding box as five key points, obtaining the speeds of the five key points through a Kalman filter; mapping the target mask of the previous frame image to four triangles formed by the five key points and using the key point speeds to perform motion simulation and calculate a motion cost matrix;

[0071] S105, formal matching step, the motion cost of the unmatched candidate pairs is used as the final cost; the matching candidate pairs calculate the fusion weight based on the historical mask area, and the appearance cost and the motion cost are fused based on the fusion weight to obtain the final cost; the final cost of the unmatched candidate pairs and the final cost of the matched candidate pairs are integrated to obtain the final cost matrix, and the final cost matrix is ​​processed by the Hungarian algorithm to obtain the matched pairs and the unmatched pairs;

[0072] S106, post-processing step, Kalman filter prediction is performed on all trajectories, and the corrected position information is updated by the Kalman filter algorithm to obtain a new trajectory set. The trajectories and detection targets in the new trajectory set are divided into unconfirmed state trajectories and unconfirmed state trajectories according to the matching status. The confirmed state trajectories are sent to the pre-matching step for the next pre-matching, and the unconfirmed state trajectories are sent to the formal matching step for the next formal matching;

[0073] S107, repeat the steps, set the matching end condition, repeat the pre-matching step to the post-processing step until the matching is completed, and enter the visualization step;

[0074] S108, a visualization step, performs a visualization operation on the trajectory of the new confirmation state and outputs a video marking the detection target.

[0075] Specifically, in the S101 of the embodiment of the present invention, the image or video information is read through open-cv; the confidence threshold is set, and the Mask R-CNN is used to segment the target for each frame of the image to obtain the target pixel-level information and the bounding box information;

[0076] Specifically, the schematic diagram of S102 is shown in Figure 3 As shown, the target pixel-level information and bounding box information are passed to the re-identification model to obtain the target mask M and background feature map f in the middle layer of the re-identification model. The embodiment of the present invention adopts the ResNet50 re-identification model, which generates the target mask and background feature map f in its middle layer when processing the input. These feature maps contain high-level feature representations of the target and background; then the mask M feature map is given The background feature map gives (relatively small) weights, and then fused to obtain the final appearance feature information, expressed as:

[0077] ;

[0078] Where (n,m) is the pixel coordinate of the feature map f, ⊙ represents element-wise multiplication; M(n,m)=1 represents ROI (region of interest), and M(n,m)=0 represents the background area;

[0079] The tracking system obtains the feature vectors of the N target appearance features that have been matched in the t-1th frame of the video as the trajectory T, and the feature vectors of the M target appearance features in the tth frame of the video as the trajectory S. The cosine distance of the appearance similarity of the feature vectors between the trajectory T[i] and the detection S[j] is calculated. , get N*M appearance cost matrix :

[0080] ;

[0081] ;

[0082] represents the appearance feature vector of trajectory T[i], Represents the appearance feature vector of detection S[j]. and Represents vectors and The Euclidean norm (i.e. length) of .

[0083] Specifically, S103 processes the cost matrix by the Hungarian algorithm to obtain other candidate pairs and the candidate pair with the minimum cost, wherein the minimum cost The candidate pairs of are high in appearance similarity, indicating low dynamics, and are considered as matching candidate pairs; the remaining candidate pairs are high in cost due to high dynamics or different targets, that is, low in appearance similarity, and are considered as unmatched candidate pairs. At the same time, the appearance cost matrix is ​​retained. .

[0084] Specifically, the schematic diagram of S104 is shown in Figure 4 As shown, the Kalman filter is used to obtain the speed of the four points on the target bounding box and the center point. . For the target mask of the t-1th frame, map it to the four triangles formed by these key points. Then, by using the speed of these five points and using affine-like transformation, perform motion simulation on the mask to obtain the predicted mask of the tth frame and the motion information of the trajectory T[i]. By calculating the Mask-IoU between the trajectory T[i] and the detection S[j], we get the N*M motion cost matrix :

[0085] ;

[0086] in, is the key point predicted in the tth frame, Represents an affine transformation function.

[0087] ;

[0088] Represents the Mask-IoU score, i.e., the motion cost.

[0089] Specifically, in S105, the pre-matching result is obtained, and for the unmatched candidate pairs, the Mask-IoU score between the trajectory T[i] and the detection S[j] is directly used. As the final cost; for the matching candidate pairs in the pre-matching, obtain the historical mask area of ​​the trajectory T[i] of the t-1th frame before t-1, and update the approximate mask area of ​​the i-th trajectory at time step t through the exponential moving average (EMA) mechanism, and then calculate the mask change rate of the detection S[j] in the t-th frame , and uses the sigmoid function to map it to [0,1] as a dynamic weight that integrates appearance features and motion information; for the specific process, see Figure 5 As shown, firstly, the mask area is expressed as ,in represents the mask area of ​​the ith tracked object in the t-1th frame, and N represents the total number of tracked objects. Next, the exponential moving average (EMA) mechanism is used to predict the approximate mask area of ​​the ith object in the tth frame:

[0090] ;

[0091] in, is the average mask area in the t-1th frame =0.9 is the momentum term. Finally, the Sigmoid function is applied to convert the rate of change into a continuous and smooth weight coefficient to ensure the rationality and stability of the weight:

[0092] ;

[0093] here, Represents the approximate mask region of the i-th object in the t-th frame The detection mask region of the jth object The relative rate of change between is the fusion weight between the i-th and j-th matching pairs in the t-th frame. When the mask sequence contains only one mask, the fusion weight is set to a fixed value of 0.7, and the final cost of the matching candidate pair is calculated based on the following formula:

[0094] = * + (1- )* ;

[0095] Integrate the final cost of the matching candidate pairs and the Mask-IoU score I[i] between the unmatched candidate pairs, i=(0,…,n*m-1), to get the final cost matrix C, expressed as:

[0096] ;

[0097] Specifically, in S106, the tracking system performs Kalman filtering updates on the trajectory of the previous frame, and the trajectory that matches the detection target for three consecutive frames is considered as a confirmed trajectory, and the other trajectories that do not match the detection target are considered as failed matching trajectories; for failed matching trajectories and detection targets, different treatments are performed according to different types, and they are divided into confirmed trajectories and unconfirmed trajectories, and enter the next round of pre-matching or the next round of formal matching respectively. Specifically including:

[0098] In the first classification step, Kalman filter prediction is performed on all trajectories, and the corrected position information is updated through the Kalman filter algorithm to obtain a new trajectory set; the trajectory that matches the detection target for three consecutive frames is identified as a confirmed trajectory and sent to the pre-matching step for the next pre-matching; the trajectory that fails to match the detection target for three consecutive frames is identified as an unconfirmed trajectory and sent to the formal matching step for the next formal matching; the trajectory that does not match the detection target is regarded as a failed matching trajectory and sent to the second classification step;

[0099] In the second classification step, the detected target is sent into the trajectory set as an unconfirmed trajectory and enters the first classification step; it is determined whether the trajectory is confirmed. If it is confirmed, it is determined whether its trajectory survival time a exceeds the threshold. The unconfirmed trajectories and the confirmed trajectories whose trajectory survival time a is greater than the threshold are deleted, and the trajectories that are not deleted are sent into the trajectory set and enter the first classification step.

[0100] Specifically, in S108 of the embodiment of the present invention, the visualization is performed using open-cv.

[0101] This embodiment adopts the typical pedestrian re-identification model ResNet50 (its variants, such as ResNet50-Embedding, may also be used). This model is based on the deep residual network (ResNet) architecture and is specifically used to extract pedestrian features from different camera perspectives and achieve pedestrian re-identification tasks by matching these features. Different methods are used to track and segment the two datasets MOTS20 and KITTI MOTS. The results are shown in Tables 1 and 2, respectively, where HD-Track in the table is the method of the embodiment of the present invention.

[0102] Table 1 MOTS20 recognition results:

[0103]

[0104] Table 2 KITTI MOTS recognition results:

[0105]

[0106] It can be seen from Table 1 and Table 2 that when the embodiment of the present invention is applied to different data sets and different objectives, each evaluation index is at the top of all methods, among which many indexes are much higher than other methods.

[0107] See also Figure 6 FIG. 1 is a system structure diagram of an embodiment of the present invention, including:

[0108] The pre-processing module 601 reads the image or video information, performs target segmentation on each frame of the image, and obtains target pixel-level information and target bounding box information;

[0109] The appearance cost calculation module 602 processes the target pixel level information and the target bounding box information using the re-identification model to obtain the target mask and background features, calculates the appearance feature vector based on the target mask and background features, and thus calculates the appearance cost matrix;

[0110] A pre-matching module 603 processes the N*M appearance cost matrix by using the Hungarian algorithm to obtain matching candidate pairs and unmatched candidate pairs;

[0111] The motion cost calculation module 604 uses four vertices and a center point on the target bounding box as five key points, obtains the speeds of the five key points through a Kalman filter, maps the target mask of the t-1th frame to the four triangles formed by the five key points, performs motion simulation using the key point speeds, and calculates the motion cost matrix;

[0112] In the formal matching module 605, the motion cost is used as the final cost for the unmatched candidate pairs; the fusion weight is calculated for the matched candidate pairs based on the historical mask area before t-1 of the t-1 frame trajectory T[i], and the appearance cost and the motion cost are fused based on the fusion weight to obtain the final cost; the final cost of the unmatched candidate pairs and the final cost of the matched candidate pairs are integrated to obtain the final cost matrix, and the final cost matrix is ​​processed by the Hungarian algorithm to obtain the matched pairs and the unmatched pairs;

[0113] The post-processing module 606 performs Kalman filter prediction on all trajectories, updates and corrects the position information through the Kalman filter algorithm, and obtains a new trajectory set. The trajectories and detection targets in the new trajectory set are divided into unconfirmed state trajectories and unconfirmed state trajectories according to the matching status. The confirmed state trajectories are sent to the pre-matching step for the next pre-matching, and the unconfirmed state trajectories are sent to the formal matching step 6 for the next formal matching;

[0114] Repeating module 607, setting the matching end condition, repeating the pre-matching step to the post-processing step until the matching is completed, and entering the visualization step;

[0115] The visualization module 608 performs visualization operations on the trajectory of the new confirmation state.

[0116] It can be seen that the present invention proposes a multi-target tracking and segmentation method based on dynamic adaptation and enhanced features. The method refines and integrates target features through data association with dynamic adaptation, and utilizes a mask-based attention mechanism and a mask prediction based on four-triangle transformation to respectively enhance the target appearance features and motion information, so as to achieve fast calculation and low-cost operation while maintaining high tracking accuracy, and is suitable for a wider range of application scenarios.

[0117] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-target tracking and segmentation method based on dynamic adaptive and enhanced features, characterized in that: The following steps are involved: Pre-processing step: read image or video information, perform target segmentation on each frame of image, and obtain target pixel-level information and target bounding box information; The appearance cost calculation step uses the re-identification model to process the target pixel-level information and the target bounding box information to obtain the target mask and background features, and calculates the appearance feature vector based on the target mask and background features, thereby calculating the appearance cost matrix; In the pre-matching step, the appearance cost matrix is ​​processed by the Hungarian algorithm to obtain matching candidate pairs and unmatched candidate pairs; In the motion cost calculation step, the four vertices and the center point of the target bounding box are used as five key points, and the speed of the five key points is obtained through the Kalman filter; Map the target mask of the previous frame image to the four triangles formed by the five key points and use the key point speed to perform motion simulation and calculate the motion cost matrix; In the formal matching step, the motion cost of the unmatched candidate pairs is used as the final cost; The matching candidate pairs calculate the fusion weight based on the historical mask area, and the appearance cost and motion cost are fused based on the fusion weight to obtain the final cost; the final cost of the unmatched candidate pairs and the final cost of the matched candidate pairs are integrated to obtain the final cost matrix, and the final cost matrix is ​​processed by the Hungarian algorithm to obtain the matched pairs and the unmatched pairs; In the post-processing step, Kalman filter prediction is performed on all trajectories, and the position information is updated and corrected through the Kalman filter algorithm to obtain a new trajectory set. The trajectories and detection targets in the new trajectory set are divided into confirmed trajectories and unconfirmed trajectories according to the matching status. The confirmed trajectories are sent to the pre-matching step for the next pre-matching, and the unconfirmed trajectories are sent to the formal matching step for the next formal matching; Repeat the steps, repeat the pre-matching step to the post-processing step until the matching is completed, and enter the visualization step; Visualization step: visualize the trajectory of the new confirmation state and output a video with the detected target marked; The appearance cost calculation step adopts a mask-based attention mechanism and specifically includes the following steps: Pass the target pixel-level information and target bounding box information to the re-identification model to obtain the target mask M and background feature map f; Give w to the mask M feature map in Weight, giving w to the background feature map out Weight, perform weighted fusion to obtain the appearance feature vector F final (n,m), expressed as: F final (n,m)=w in ·f(n,m)⊙M(n,m)+w out ·f(n,m)⊙(1-M(n,m)); Among them, w in >w out ; (n,m) is the pixel coordinate of the background feature map f, ⊙ represents element multiplication operation; M(n,m)=1 represents ROI, M(n,m)=0 represents the background area; The appearance feature vectors of the N matched targets in the t-1th frame of the video are taken as the trajectory T, and the feature vectors of the appearance features of the M targets in the tth frame of the video are taken as the trajectory S; the cosine distance of the appearance similarity between the i-th trajectory T[i] and the j-th detection S[j] is calculated. Get the N*M appearance cost matrix C 外观 : Among them, V i Represents the appearance feature vector of trajectory T[i], V j Represents the appearance feature vector of detection S[j]; ||V i || and ||V j || represents the Euclidean norm of trajectory T[i] and detection S[j] respectively.

2. The multi-target tracking and segmentation method based on dynamic adaptation and enhanced features according to claim 1 is characterized in that: In the pre-matching step, the N*M appearance cost matrix is ​​processed by the Hungarian algorithm to obtain the candidate pair with the minimum cost as the pre-matching candidate pair, and the other candidate pairs are used as unmatched candidate pairs.

3. The multi-target tracking and segmentation method based on dynamic adaptation and enhanced features according to claim 1 is characterized in that: The motion cost calculation step includes the following steps: The Kalman filter is used to obtain the speeds of the four points on the target bounding box and the center point {P1, P2, P3, P4, P c }; For the target mask of the t-1th frame, map it to the four triangles formed by these key points; Using the velocities of these five points, we use affine-like transformation to simulate the motion of the mask to obtain the predicted mask of the tth frame and the motion information of the trajectory T[i]; By calculating the Mask-IoU score between the trajectory T[i] and the detection S[j], we get the N*M motion cost matrix C 运动 : Where τ represents the affine transformation function, is the key point predicted in the tth frame, Δ represents a triangle formed by the three points after the symbol as vertices; Represents the Mask-IoU score; ∪ represents the union, and ∩ represents the intersection.

4. The multi-target tracking and segmentation method based on dynamic adaptation and enhanced features according to claim 1, characterized in that: In the formal matching step, the final cost matrix obtained is C, where the matrix elements are expressed as: Among them, w i,j is the fusion weight calculated based on the historical mask area.

5. The multi-target tracking and segmentation method based on dynamic adaptation and enhanced features according to claim 1, characterized in that: The calculation process of the fusion weight includes the following steps: Get the historical mask area, expressed as in represents the mask area of ​​the ith tracked object in the t-1th frame, and N represents the total number of tracked objects; An exponential moving average mechanism is used to predict the predicted mask area of ​​the i-th object in the t-th frame. It is expressed as: in, is the approximate mask region of the ith object in the t-1th frame, λ is the momentum term, Represents the detection mask area of ​​the i-th object in the t-th frame; The Sigmoid function is applied to convert the rate of change into a continuous and smooth weight coefficient, namely the fusion weight, which is expressed as: in, Represents the approximate mask area of ​​the i-th object in the t-th frame The detection mask area of ​​the jth object in the tth frame The relative rate of change between .

6. The multi-target tracking and segmentation method based on dynamic adaptation and enhanced features according to claim 1, characterized in that: The post-processing step comprises: In the first classification step, Kalman filter prediction is performed on all trajectories, and the position information is updated and corrected through the Kalman filter algorithm to obtain a new trajectory set; the trajectory that matches the detection target for three consecutive frames is identified as a confirmed trajectory and sent to the pre-matching step for the next pre-matching; the trajectory that fails to match the detection target for three consecutive frames is identified as an unconfirmed trajectory and sent to the formal matching step for the next formal matching; the trajectory that does not match the detection target is regarded as a failed matching trajectory and sent to the second classification step; In the second classification step, the detected target is sent into the trajectory set as an unconfirmed trajectory and enters the first classification step; it is determined whether the trajectory is confirmed. If it is confirmed, it is determined whether its trajectory survival time a exceeds the threshold. The unconfirmed trajectories and the confirmed trajectories whose trajectory survival time a is greater than the threshold are deleted, and the trajectories that are not deleted are sent into the trajectory set and enter the first classification step.

7. A multi-target tracking and segmentation system based on dynamic adaptive and enhanced features, characterized in that: include: The pre-processing module reads the image or video information, performs target segmentation on each frame of the image, and obtains the target pixel-level information and target bounding box information; The appearance cost calculation module uses the re-identification model to process the target pixel-level information and target bounding box information to obtain the target mask and background features, and calculates the appearance feature vector based on the target mask and background features, thereby calculating the appearance cost matrix; The pre-matching module processes the appearance cost matrix through the Hungarian algorithm to obtain matching candidate pairs and unmatched candidate pairs; The motion cost calculation module uses the four vertices and the center point of the target bounding box as five key points, and obtains the speed of the five key points through the Kalman filter; Map the target mask of the previous frame image to the four triangles formed by the five key points and use the key point speed to perform motion simulation and calculate the motion cost matrix; In the formal matching module, the motion cost of unmatched candidate pairs is used as the final cost; The matching candidate pairs calculate the fusion weight based on the historical mask area, and the appearance cost and motion cost are fused based on the fusion weight to obtain the final cost; the final cost of the unmatched candidate pairs and the final cost of the matched candidate pairs are integrated to obtain the final cost matrix, and the final cost matrix is ​​processed by the Hungarian algorithm to obtain the matched pairs and the unmatched pairs; The post-processing module performs Kalman filter prediction on all trajectories, updates and corrects the position information through the Kalman filter algorithm, and obtains a new trajectory set. The trajectories and detection targets in the new trajectory set are divided into confirmed trajectories and unconfirmed trajectories according to the matching status. The confirmed trajectories are sent to the pre-matching step for the next pre-matching, and the unconfirmed trajectories are sent to the formal matching step for the next formal matching; Repeat module, repeat the pre-matching step to the post-processing step until the matching is completed, and enter the visualization module; The visualization module visualizes the trajectory of the new confirmed state and outputs a video with the detected target marked; The appearance cost calculation step adopts a mask-based attention mechanism and specifically includes the following steps: Pass the target pixel-level information and target bounding box information to the re-identification model to obtain the target mask M and background feature map f; Give w to the mask M feature map in Weight, giving w to the background feature map out Weight, perform weighted fusion to obtain the appearance feature vector F final (n,m), expressed as: F final (n,m)=w in ·f(n,m)⊙M(n,m)+w out ·f(n,m)⊙(1-M(n,m)); Among them, w in >w out ; (n,m) is the pixel coordinate of the background feature map f, ⊙ represents element multiplication operation; M(n,m)=1 represents ROI, M(n,m)=0 represents the background area; The appearance feature vectors of the N matched targets in the t-1th frame of the video are taken as the trajectory T, and the feature vectors of the appearance features of the M targets in the tth frame of the video are taken as the trajectory S; the cosine distance of the appearance similarity between the i-th trajectory T[i] and the j-th detection S[j] is calculated. Get the N*M appearance cost matrix C 外观 : Among them, V i Represents the appearance feature vector of trajectory T[i], V j Represents the appearance feature vector of detection S[j]; ||V i || and ||V j || represents the Euclidean norm of trajectory T[i] and detection S[j] respectively.

Images