Multi-target tracking and segmentation method and system based on feature fusion and space-time memory

Abstract

This invention belongs to the field of computer vision and artificial intelligence technology, and relates to a multi-target tracking and segmentation method and system based on feature fusion and spatiotemporal memory. The multi-target tracking and segmentation method includes: acquiring the original state features and predicted bounding boxes of each target in each frame of the image, and constructing a spatiotemporal memory database containing a short-term buffer and a long-term storage area; for the first detected target in the multi-target video sequence, instantiating an independent long-term trajectory prototype for it, and establishing a corresponding index in the spatiotemporal memory database; performing hierarchical matching of the original state features and predicted bounding boxes of each target in the frame of the image based on the spatiotemporal memory database; updating the Kalman filter parameters and the spatiotemporal memory database based on the matching results; and continuing the tracking and segmentation of the next frame of the image using the updated Kalman filter and spatiotemporal memory database until each frame of the image has been traversed to obtain the multi-target tracking and segmentation result. This invention can effectively solve the problem of re-identification after long-term target occlusion.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a method and system for multi-target tracking and segmentation based on feature fusion and spatiotemporal memory. Background Technology

[0002] Multi-Object Tracking and Segmentation (MOTS) is an important research direction in the field of computer vision. Its core task is to simultaneously detect multiple objects in a video sequence, segment instances, and associate identities across frames. This technology has wide application value in scenarios such as autonomous driving, intelligent surveillance, and motion analysis.

[0003] Current mainstream methods generally follow a "detection-segmentation-association" technical framework, with key challenges concentrated in two stages: data association and feature utilization. Typically, in the data association stage, most methods rely on two types of information for matching: first, predicting target positions based on motion models (such as Kalman filtering) and associating them using the intersection-over-union (IoU) ratio between bounding boxes; second, extracting the apparent features of the targets using convolutional neural networks (CNNs) and using feature similarity metrics (such as cosine distance) to help determine if they are the same target. In trajectory management, a fixed time threshold is usually set; if a trajectory fails to match a detection result within that time, it is terminated and removed from the current tracking set.

[0004] However, the existing solutions have several obvious drawbacks. First, the location-based and simple appearance-based association methods lack robustness in complex scenes: methods relying solely on IoU are prone to geometric matching failure when the target undergoes rapid deformation, non-uniform motion, or violent camera movement; while methods relying solely on the appearance features of the current frame are easily affected by changes in lighting, pose, and local occlusion, leading to feature drift and difficulty in effectively distinguishing different targets with similar appearances, thus causing mismatches and identity switching problems. Second, the system lacks an effective long-term memory and re-identification mechanism: existing methods mostly use short-term or medium-term memory models. Once a target disappears for a long time due to severe occlusion or leaving the field of view, its corresponding trajectory and feature information are often directly discarded, making it impossible to associate the target with historical trajectories when it reappears, resulting in trajectory breaks. In addition, existing methods fail to fully utilize the discriminative shape information contained in the segmentation mask: most methods only use the instance segmentation result as the final output, and fail to effectively utilize the fine contour and temporal shape features contained in the mask during the tracking and association stage, losing the association basis that remains stable under occlusion and deformation. Existing MOTS methods rely excessively on easily degraded geometric or appearance features in their association strategies, lack long-term memory mechanisms in trajectory management, and fail to deeply integrate the shape information provided by the segmentation stage. These shortcomings significantly limit their performance when handling complex scenarios such as long-term occlusion, drastic target deformation, camera motion, and interference from similar-looking targets. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a multi-target tracking and segmentation method and system based on feature fusion and spatiotemporal memory. This invention compensates for the limitations of single features under deformation or illumination changes by fusing semantic, shape, and appearance features. Combined with camera motion compensation and dynamic noise Kalman filtering, it significantly improves tracking stability in moving camera scenarios. Furthermore, this invention introduces an enhanced spatiotemporal memory library containing trajectory prototypes, dynamically adjusting computational complexity at different matching stages. This effectively solves the problem of re-identification after prolonged target occlusion, ensuring both high recall rate for difficult samples and efficient system operation.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a multi-target tracking and segmentation method based on feature fusion and spatiotemporal memory, comprising: S1. Configure the multi-target video sequence to be detected and obtain the original state features of each target in each frame of the image; S2. Calculate the first... Frame image compared to the first Camera motion compensation matrix of a frame image. For each target, obtain its value on the 1st... The state vector of the Kalman filter in the frame image is transformed into a coordinate system based on the camera motion compensation matrix to obtain the compensated state vector. S3. Based on the compensated state vector, a Kalman filter is used to obtain the state vector of each target in the first position. Predicted bounding boxes in a frame image; S4. Construct a spatiotemporal memory library containing a short-term buffer and a long-term storage area; for the first detected target in a multi-target video sequence, instantiate an independent long-term trajectory prototype for it and establish a corresponding index in the spatiotemporal memory library; S5. Based on the spatiotemporal memory bank, the first... The original state features and predicted bounding boxes of each target in the frame image are hierarchically matched, and the Kalman filter parameters and spatiotemporal memory are updated based on the matching results. S6. Order The updated Kalman filter and spatiotemporal memory are used to return to the execution steps S3 to S5 until each frame of the multi-target video sequence is traversed to obtain the multi-target tracking and segmentation results. The multi-target tracking and segmentation results are then post-processed to obtain the final tracking and segmentation results.

[0007] As one possible implementation, the original state features of each target in each frame of the image are obtained using the following method: S10. Use an object detection and segmentation network to extract the detection box and segmentation mask for each object in each frame of the image; S11. Configure a shape encoder and a lightweight re-identification network, crop and scale the target image region corresponding to the detection box, and input them into the backbone networks of the lightweight re-identification network and the target detection and segmentation network respectively to extract semantic features and appearance features; at the same time, input the segmentation mask into the shape encoder for multi-scale encoding to obtain shape features. S12. Concatenate the semantic and shape features of each target, and perform fusion encoding and dimensionality reduction on the concatenated features to obtain memory-encoded features; S13. Based on memory encoding features and appearance features, obtain the original state features of each target.

[0008] As one possible implementation, hierarchical matching includes: First-level matching: Configure the overlap threshold. For each target, obtain its last observed segmentation mask as the reference mask, and calculate the first-level matching. The geometric intersection-over-union ratio (GOU) of the detected bounding box and the predicted bounding box in the frame image, and the mask intersection-over-union ratio of the segmentation mask and the reference mask are weighted and summed to obtain the mixed overlap. It is determined whether the mixed overlap is greater than the overlap threshold. If it is greater, it is determined that the original state feature and the predicted bounding box come from the same target. Otherwise, perform the second-level matching: configure the matching cost threshold, retrieve historical trajectory state features from the short-term buffer, retrieve trajectory prototype features from the long-term trajectory prototype area, calculate the fused features based on the current original state features as the query vector and the historical trajectory state features as the key vector using a lightweight cross-attention fusion mechanism; calculate the appearance similarity based on the trajectory prototype features and construct the cost matrix, calculate the minimum total cost matching using the Hungarian algorithm, and determine whether the minimum total cost matching is less than the matching cost threshold. If it is less, it is determined that the original state feature and the predicted box come from the same target. For targets remaining after the first two levels of matching, if the number of consecutive unmatched frames is within the maximum allowed number of retained frames, then the third level of matching is executed; otherwise, the first level of matching continues. The third level of matching is as follows: configure a similarity threshold and a geometric intersection-union (GUU) threshold that are less than the matching cost threshold; retrieve key trajectory state features from the long-term storage area; obtain trajectory prototypes from the long-term trajectory prototype area; filter targets whose GUU of the predicted bounding box and the detection box is greater than the geometric GUU threshold; obtain deep fusion state features based on the deep cross-attention fusion mechanism; calculate the first similarity between the trajectory prototype and the current appearance features, and the second similarity between the deep fusion state features and the current original state features; perform a weighted summation of the first and second similarities; and select the target with the largest summation result that is greater than the similarity threshold, which is the target that is successfully matched. Otherwise, a new target creation threshold is preset, and the confidence level of the original state feature is calculated. If the confidence level is higher than the new target creation threshold, the trajectory is initialized as a new target trajectory; otherwise, the target is discarded.

[0009] As one possible implementation, the cost matrix is ​​as follows: in, Indicates the first The trajectory and the first The matching cost between targets , , These represent the weight coefficients for geometric overlap, appearance feature similarity, and memory fusion similarity, respectively. Indicates the first The predicted bounding box or prediction mask of the first trajectory and the second trajectory The degree of overlap between the detection bounding boxes or segmentation masks of each target; Indicates the first The trajectory prototype features of the first trajectory are similar to those of the second trajectory. The similarity of the appearance features of the targets; Indicates the current frame number The memory encoding features of each target Indicates the first A set of historical trajectory state features retrieved from a short-term buffer for each trajectory; It represents the cosine similarity of the features between the memory-encoded features and the set of historical trajectory state features after lightweight cross-attention fusion.

[0010] As one possible implementation, the deep fusion state characteristics are as follows: in, Indicates the characteristics of deep fusion state. This represents the intermediate fused feature vector after the attention residual. The query vector is composed of the memory-encoded features of the current target. These are the key vector and the value vector, respectively, both composed of key trajectory state features retrieved from the long-term storage area; This indicates a multi-head cross-attention operation. This indicates a normalization operation. This represents the nonlinear mapping of a feedforward neural network.

[0011] As one possible implementation, for each target, the spatiotemporal memory is updated after a successful match using the following method: Configure quality gating conditions, velocity gating conditions, and keyframe determination conditions, and update the short-term buffer using the following method: Configure the capacity of the short-term buffer, store the frame data containing the original state characteristics of the target that simultaneously meets the quality gating conditions and velocity gating conditions into the short-term buffer, and if the short-term buffer is full, remove the frame data of the earliest frame. The long-term trajectory prototype region is updated using the following method: the smooth prototype features of the long-term trajectory prototype region are updated using the exponential moving average algorithm; the capacity of the prototype feature set of the long-term trajectory prototype region is configured, and the original state features of the current frame are stored in the prototype feature set. If the upper limit of the prototype feature set capacity is reached, the quality score of each prototype feature in the prototype feature set is calculated, and only the top-K features with the highest quality scores are retained. The long-term storage area is updated using the following method: Configure the capacity of the long-term storage area, filter frame data that simultaneously meet the quality gating condition, speed gating condition, and keyframe determination condition, and store them in the long-term storage area. If the long-term storage area is full, sort the frame data in the storage area based on the importance score, and remove the frame data with the lowest importance score.

[0012] As one possible implementation, the smoothed prototype feature is as follows: in, No. Smooth prototype features after frame update The momentum coefficient represents the exponential moving average, and its value ranges from 0 to 1. Indicates the first Smooth prototype features of frames, Indicates the first The original state features of the current target whose frame was successfully matched.

[0013] As one possible implementation, the object detection and segmentation network is YOLO11-seg, the lightweight re-identification network is OSNet, and the shape encoder is a single-channel multi-scale convolutional neural network based on ResNet18.

[0014] As one possible implementation, post-processing includes: filling in the missing position and mask for the missing trajectory segments using linear interpolation; performing morphological smoothing on the segmented mask to remove noise at the mask edges; the missing trajectory segments are target trajectories whose consecutive unmatched frames are within the maximum allowed number of retained frames and which were not successfully matched in the previous frame.

[0015] Secondly, the present invention provides a multi-target tracking and segmentation system based on feature fusion and spatiotemporal memory, comprising: The original state feature acquisition unit is configured with a multi-target video sequence to be detected and acquires the original state features of each target in each frame of the image. Prediction box acquisition unit, calculate the first Frame image compared to the first Camera motion compensation matrix of a frame image. For each target, obtain its value on the 1st... The Kalman filter state vector in the frame image is transformed in coordinate system based on the camera motion compensation matrix to obtain the compensated state vector. Based on the compensated state vector, the Kalman filter is used to obtain the state vector of each target in the [frame image]. Predicted bounding boxes in a frame image; A hierarchical matching and update unit is constructed, which includes a spatiotemporal memory bank containing a short-term buffer and a long-term storage area. For the first detected target in a multi-target video sequence, an independent long-term trajectory prototype is instantiated for it, and a corresponding index is established in the spatiotemporal memory bank. Based on the spatiotemporal memory bank, the first target is matched and updated. The original state features and predicted bounding boxes of each target in the frame image are hierarchically matched, and the Kalman filter parameters and spatiotemporal memory are updated based on the matching results. The result output unit, let The updated Kalman filter and spatiotemporal memory are used to return the prediction box acquisition unit and the hierarchical matching and update unit until every frame of the multi-target video sequence is traversed to obtain the multi-target tracking and segmentation results. The multi-target tracking and segmentation results are then post-processed to obtain the final tracking and segmentation results.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The multi-target tracking and segmentation method and system based on feature fusion and spatiotemporal memory proposed in this invention compensates for the shortcomings of single features under deformation or illumination changes by fusing semantic features, shape features and appearance features, thus providing a foundation for the accuracy of target recognition.

[0017] 2. The multi-target tracking and segmentation method and system based on feature fusion and spatiotemporal memory proposed in this invention introduces an enhanced spatiotemporal memory library containing trajectory prototypes. The trajectory prototypes are maintained through a dual mechanism of EMA update and quality gating, which effectively solves the problem of re-identification after long-term target occlusion.

[0018] 3. The multi-target tracking and segmentation method and system based on feature fusion and spatiotemporal memory proposed in this invention adopts a three-level matching strategy and uses a lightweight cross-attention fusion mechanism and a deep cross-attention fusion mechanism to dynamically adjust the computational complexity at different matching stages, so as to ensure the recall rate of high-difficulty samples while taking into account the system's operating efficiency.

[0019] 4. The multi-target tracking and segmentation method and system based on feature fusion and spatiotemporal memory proposed in this invention, combined with camera motion compensation and dynamic noise Kalman filtering, significantly improves the tracking stability in camera motion scenarios. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of the multi-target tracking and segmentation method based on feature fusion and spatiotemporal memory proposed in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating multimodal feature extraction and encoding based on a target segmentation network in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the process of obtaining deep fusion state features based on a deep cross-attention fusion mechanism and calculating fusion features based on a lightweight cross-attention fusion mechanism in an embodiment of the present invention. Figure 4 This is a schematic diagram of the spatiotemporal memory bank update process in an embodiment of the present invention. Detailed Implementation

[0021] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, the first threshold and the second threshold are merely used to distinguish different thresholds and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0022] It should be noted that in this invention, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0023] In this invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one" or similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, "at least one of a, b, or c" can represent: a, b, c, a combination of a and b, a combination of a and c, a combination of b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0024] This invention aims to provide a multi-target tracking and segmentation method and system based on feature fusion and spatiotemporal memory. By fusing semantic, shape, and appearance features, it compensates for the shortcomings of single features under deformation or illumination changes. Combined with camera motion compensation and dynamic noise Kalman filtering, it significantly improves tracking stability in moving camera scenarios. Furthermore, this invention introduces an enhanced spatiotemporal memory library containing trajectory prototypes, dynamically adjusting computational complexity at different matching stages. This effectively solves the problem of re-identification after long-term target occlusion, ensuring both high recall rate for difficult samples and efficient system operation.

[0025] Firstly, embodiments of the present invention provide a multi-target tracking and segmentation method based on feature fusion and spatiotemporal memory, see [link to previous section]. Figure 1 It includes the following steps: S1. Configure the multi-target video sequence to be detected and obtain the original state features of each target in each frame of the image; As one possible implementation, the original state features of each target in each frame of the image are obtained using the following method: S10. Use an object detection and segmentation network to extract the detection box and segmentation mask for each object in each frame of the image; S11. Configure a shape encoder and a lightweight re-identification network, crop and scale the target image region corresponding to the detection box, and input them into the backbone networks of the lightweight re-identification network and the target detection and segmentation network respectively to extract semantic features and appearance features; at the same time, input the segmentation mask into the shape encoder for multi-scale encoding to obtain shape features. S12. Concatenate the semantic and shape features of each target, and perform fusion encoding and dimensionality reduction on the concatenated features to obtain memory-encoded features; S13. Based on memory encoding features and appearance features, obtain the original state features of each target.

[0026] As one possible implementation, the object detection and segmentation network is YOLO11-seg, the lightweight re-identification network is OSNet, and the shape encoder is a single-channel multi-scale convolutional neural network based on ResNet18.

[0027] See Figure 2 As an example, for each frame of a multi-target video sequence, the YOLO11-seg target segmentation network is used for forward inference. Based on the detected target regions, RoIs are cropped, scaled to a fixed size, and then input back into the YOLO Backbone. Features from layers P3 and P4 are extracted and averaged to obtain semantic features. The detected bounding boxes are then cropped and scaled before being input into OSNet to extract appearance feature vectors that are robust to illumination and viewpoint. A shape encoder based on the ResNet18 architecture is configured, and its first convolutional layer is modified to use a single-channel input to adapt to the binary mask. The segmentation mask output from the segmentation process is then input into this encoder to extract shape feature vectors containing fine contour information. as follows: in, This represents the input single-channel binary segmentation mask. This represents the multi-layer feature extraction of the YOLO11-seg object segmentation network (which can be compared to the convolutional and residual layers of ResNet18). Indicates global average pooling. For linear projection layers, For L2 normalization.

[0028] To unify the representation of the target state, semantic feature vectors and shape feature vector The data is concatenated and input into a feature encoding module based on a Transformer Encoder structure. A self-attention mechanism is used to mine the complementarity between semantics and shape, and dimensionality reduction is achieved through a fully connected layer, outputting fixed-dimensional memorized encoded features. At this point, each target is represented as a tuple: .

[0029] This invention integrates semantic features, shape features, and appearance features, making up for the shortcomings of single features under deformation or illumination changes, and providing a foundation for the accuracy of target recognition.

[0030] S2. Calculate the first... Frame image compared to the first Camera motion compensation matrix of a frame image. For each target, obtain its value on the 1st... The state vector of the Kalman filter in the frame image is transformed into a coordinate system based on the camera motion compensation matrix to obtain the compensated state vector. S3. Based on the compensated state vector, a Kalman filter is used to obtain the state vector of each target in the first position. Predicted bounding boxes in a frame image; As an example, the sparse optical flow method is used to extract the first... Frame and the Calculate the camera motion compensation matrix from feature points in the frame image. To characterize the global motion of the camera, the camera motion compensation matrix Including rotation matrix Translation vector For each objective, obtain its value at the 1st... The Kalman filter state vector in the frame image is as follows: in, Represents the positional component. Indicates width and height, Represents the velocity component. Let represent the rates of change of the target bounding box width and height, respectively. The position and velocity components are updated as follows after compensation using the camera motion compensation matrix: in, and These represent the compensated position and velocity, respectively.

[0031] This invention combines camera motion compensation with dynamic noise Kalman filtering, which significantly improves the tracking stability of cameras in motion scenarios.

[0032] S4. Construct a spatiotemporal memory library containing a short-term buffer and a long-term storage area; for the first detected target in a multi-target video sequence, instantiate an independent long-term trajectory prototype for it and establish a corresponding index in the spatiotemporal memory library; S5. Based on the spatiotemporal memory bank, the first... The original state features and predicted bounding boxes of each target in the frame image are hierarchically matched, and the Kalman filter parameters and spatiotemporal memory are updated based on the matching results. As one possible implementation, hierarchical matching includes: First-level matching: Configure the overlap threshold. For each target, obtain its last observed segmentation mask as the reference mask, and calculate the first-level matching. The geometric intersection-over-union ratio (GOU) of the detected bounding box and the predicted bounding box in the frame image, and the mask intersection-over-union ratio of the segmentation mask and the reference mask are weighted and summed to obtain the mixed overlap. It is determined whether the mixed overlap is greater than the overlap threshold. If it is greater, it is determined that the original state feature and the predicted bounding box come from the same target. Otherwise, perform the second-level matching: configure the matching cost threshold, retrieve historical trajectory state features from the short-term buffer, retrieve trajectory prototype features from the long-term trajectory prototype area, calculate the fused features based on the current original state features as the query vector and the historical trajectory state features as the key vector using a lightweight cross-attention fusion mechanism; calculate the appearance similarity based on the trajectory prototype features and construct the cost matrix, calculate the minimum total cost matching using the Hungarian algorithm, and determine whether the minimum total cost matching is less than the matching cost threshold. If it is less, it is determined that the original state feature and the predicted box come from the same target. As an example, in first-level matching, the mixed overlap is represented as: in, Represents the mixing coefficient. These represent the predicted bounding box and the detected bounding box, respectively. This represents the geometric intersection-union ratio (GUC) between the detection box and the predicted box. These represent the prediction mask and the detection mask, respectively. This represents the crossover ratio (COR) of the segmentation mask and the prediction mask.

[0033] For targets remaining after the first two levels of matching, if the number of consecutive unmatched frames is within the maximum allowed number of retained frames, a third level of matching is performed; otherwise, the first level of matching continues. The third level of matching is as follows: configure a similarity threshold and a geometric intersection-union (GUU) threshold that are less than the matching cost threshold; retrieve key trajectory state features from the long-term storage area; obtain trajectory prototypes from the long-term trajectory prototype area; filter targets whose GUU of the predicted bounding box and the detection box is greater than the geometric GUU threshold; obtain deep fusion state features based on the deep cross-attention fusion mechanism; calculate the first similarity between the trajectory prototype and the current appearance features, and the second similarity between the deep fusion state features and the current original state features; perform a weighted summation of the first and second similarities; and select the target with the largest summation result that is greater than the similarity threshold, which is the target that is successfully matched. Otherwise, a new target creation threshold is preset, and the confidence level of the original state feature is calculated. If the confidence level is higher than the new target creation threshold, the trajectory is initialized as a new target trajectory; otherwise, the target is discarded.

[0034] For example, the processes of obtaining deep fusion state features based on the deep cross-attention fusion mechanism and calculating fusion features based on the lightweight cross-attention fusion mechanism are respectively as follows: Figure 3 (a) and Figure 3 As shown in (b).

[0035] As one possible implementation, the cost matrix is ​​as follows: in, Indicates the first The trajectory and the first The matching cost between targets , , These represent the weight coefficients for geometric overlap, appearance feature similarity, and memory fusion similarity, respectively. Indicates the first The predicted bounding box or prediction mask of the first trajectory and the second trajectory The degree of overlap between the detection bounding boxes or segmentation masks of each target; Indicates the first The trajectory prototype features of the first trajectory are similar to those of the second trajectory. The similarity of the appearance features of the targets; Indicates the current frame number The memory encoding features of each target Indicates the first A set of historical trajectory state features retrieved from a short-term buffer for each trajectory; It represents the cosine similarity of the features between the memory-encoded features and the set of historical trajectory state features after lightweight cross-attention fusion.

[0036] As one possible implementation, the deep fusion state characteristics are as follows: in, Indicates the characteristics of deep fusion state. This represents the intermediate fused feature vector after the attention residual. The query vector is composed of the memory-encoded features of the current target. These are the key vector and the value vector, respectively, both composed of key trajectory state features retrieved from the long-term storage area; This indicates a multi-head cross-attention operation. This indicates a normalization operation. This represents the nonlinear mapping of a feedforward neural network.

[0037] This invention employs a three-level matching strategy, using a lightweight cross-attention fusion mechanism and a deep cross-attention fusion mechanism to dynamically adjust computational complexity at different matching stages, ensuring both high recall rate for difficult samples and efficient system operation.

[0038] See Figure 4 As one possible implementation, for each target, the spatiotemporal memory is updated after a successful match using the following method: Configure quality gating conditions, velocity gating conditions, and keyframe determination conditions, and update the short-term buffer using the following method: Configure the capacity of the short-term buffer, store the frame data containing the original state characteristics of the target that simultaneously meets the quality gating conditions and velocity gating conditions into the short-term buffer, and if the short-term buffer is full, remove the frame data of the earliest frame. The long-term trajectory prototype region is updated using the following method: the smooth prototype features of the long-term trajectory prototype region are updated using the exponential moving average algorithm; the capacity of the prototype feature set of the long-term trajectory prototype region is configured, and the original state features of the current frame are stored in the prototype feature set. If the upper limit of the prototype feature set capacity is reached, the quality score of each prototype feature in the prototype feature set is calculated, and only the top-K features with the highest quality scores are retained. As one possible implementation, the smoothed prototype feature is as follows: in, No. Smooth prototype features after frame update The momentum coefficient represents the exponential moving average, and its value ranges from 0 to 1. Indicates the first Smooth prototype features of frames, Indicates the first The original state features of the current target whose frame was successfully matched.

[0039] The long-term storage area is updated using the following method: Configure the capacity of the long-term storage area, filter frame data that simultaneously meet the quality gating condition, speed gating condition, and keyframe determination condition, and store them in the long-term storage area. If the long-term storage area is full, sort the frame data in the storage area based on the importance score, and remove the frame data with the lowest importance score.

[0040] As an example, the importance score is calculated using the following method: in, This represents the total area of ​​the foreground pixels in the segmentation mask. This represents the total size of the segmented mask image. Indicates the percentage of the segmented mask area. For multi-target gain coefficients, Indicates the number of targets in the segmentation mask.

[0041] S6. Order The updated Kalman filter and spatiotemporal memory are used to return to the execution steps S3 to S5 until each frame of the multi-target video sequence is traversed to obtain the multi-target tracking and segmentation results. The multi-target tracking and segmentation results are then post-processed to obtain the final tracking and segmentation results.

[0042] As one possible implementation, post-processing includes: filling in the missing position and mask for the missing trajectory segments using linear interpolation; performing morphological smoothing on the segmented mask to remove noise at the mask edges; the missing trajectory segments are target trajectories whose consecutive unmatched frames are within the maximum allowed number of retained frames and which were not successfully matched in the previous frame.

[0043] This invention introduces an enhanced spatiotemporal memory library containing trajectory prototypes, and maintains trajectory prototypes through a dual mechanism of EMA update and quality gating, effectively solving the problem of re-identification after long-term target occlusion.

[0044] In one possible implementation, assuming the multi-target video sequence contains mixed targets such as multiple cars, multiple pedestrians, and multiple bicycles, a unified spatiotemporal memory is constructed to manage all targets. It is assumed that the short-term buffer capacity is 7 frames, the long-term trajectory prototype region has a prototype feature set capacity of 10 features, and the long-term storage region has a capacity of 20 keyframes. The momentum coefficient of the exponential moving average is used. The threshold is set to 0.9. The overlap threshold for the first-level matching is set to 0.65, the matching cost threshold for the second-level matching is set to 0.75, the similarity threshold for the third-level matching is set to 0.6, and the new target creation threshold is set to 0.7. The keyframe determination criteria are set as follows: original feature confidence greater than 0.85, and target area as a percentage of image area greater than 0.01.

[0045] Suppose that targets are detected in the first frame of the image: a red car A, a pedestrian B, and a blue truck C. The target IDs, confidence scores, detection boxes, and segmentation masks for the three targets are shown in Table 1. Table 1 Initial Target Information Trajectory prototypes are established for the three targets in the spatiotemporal memory, and their respective Kalman filters are initialized to record their position, velocity, and other motion states. For each frame of the image, the corresponding original state features of the three targets are extracted. If there is camera motion displacement, the Kalman filter state vector is compensated, and the predicted bounding boxes of the red car A, pedestrian B, and blue truck C in the second frame image are predicted based on the compensated state vector. Taking red car A as an example, assuming the predicted bounding box is (112,203,192,263), and mapped to the current frame coordinate system by the camera motion compensation matrix, it becomes (110,205,190,265). The calculated geometric intersection-over-union ratio is 0.85, and the mask intersection-over-union ratio is 0.82. Based on empirical values, the geometric intersection-over-union ratio weight is set to 0.7. The calculated mixed overlap is as follows: 0.7×0.85+0.3×0.82=0.84, which is greater than the overlap threshold of 0.65. Therefore, the match is successful, and red car A is successfully matched. At this time, its corresponding Kalman filter is updated. Assuming the quality gate condition is 0.7, its confidence is greater than 0.7. Therefore, the second frame image data containing the original state features of red car A is stored in the short-term buffer, and the long-term trajectory prototype area is updated. It is determined that the key frame determination condition is not met, so it is not added to the long-term storage area for the time being. Assuming that pedestrian B and blue truck C are both successfully matched in the second frame, and a new target motorcycle D appears in the second frame, then motorcycle D is assigned a target ID of D01, its corresponding trajectory prototype is initialized in the spatiotemporal memory, and a Kalman filter is assigned to it.

[0046] Suppose that starting from frame 10, red car A is gradually obscured by a large truck, becoming completely invisible by frame 15. The detection confidence of red car A then gradually decreases, and it becomes completely undetectable by frame 15. Assuming that red car A is still not detected between frames 16 and 25, until frame 26 when a white car E appears in the frame, its position close to that of red car A before it was obscured, the following matching process is performed in frame 26: First, a first-level matching is performed. If the matching fails, a second-level matching is performed. If this also fails, key trajectory state features are retrieved from the long-term storage area, and trajectory prototypes are obtained from the long-term trajectory prototype area. The first and second similarities are calculated, and a weighted sum of the first and second similarities is calculated. If multiple weighted sums are greater than the similarity threshold, the target with the largest sum is selected, indicating a successful match. Otherwise, the original feature confidence of white car E is calculated. If it is higher than the new target creation threshold, it is initialized as a new target trajectory.

[0047] Secondly, embodiments of the present invention provide a multi-target tracking and segmentation system based on feature fusion and spatiotemporal memory, comprising: The original state feature acquisition unit is configured with a multi-target video sequence to be detected and acquires the original state features of each target in each frame of the image. Prediction box acquisition unit, calculate the first Frame image compared to the first Camera motion compensation matrix of a frame image. For each target, obtain its value on the 1st... The Kalman filter state vector in the frame image is transformed in coordinate system based on the camera motion compensation matrix to obtain the compensated state vector. Based on the compensated state vector, the Kalman filter is used to obtain the state vector of each target in the [frame image]. Predicted bounding boxes in a frame image; A hierarchical matching and update unit is constructed, which includes a spatiotemporal memory bank containing a short-term buffer and a long-term storage area. For the first detected target in a multi-target video sequence, an independent long-term trajectory prototype is instantiated for it, and a corresponding index is established in the spatiotemporal memory bank. Based on the spatiotemporal memory bank, the first target is matched and updated. The original state features and predicted bounding boxes of each target in the frame image are hierarchically matched, and the Kalman filter parameters and spatiotemporal memory are updated based on the matching results. The result output unit, let The updated Kalman filter and spatiotemporal memory are used to return the prediction box acquisition unit and the hierarchical matching and update unit until every frame of the multi-target video sequence is traversed to obtain the multi-target tracking and segmentation results. The multi-target tracking and segmentation results are then post-processed to obtain the final tracking and segmentation results.

[0048] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, the disclosure, and the description of the drawings, in carrying out the claimed invention. In this specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several of the functions listed in the specification. While certain measures are described in different embodiments, this does not mean that these measures cannot be combined to produce good results.

[0049] Although the invention has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made therein without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely illustrative of the invention and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if such modifications and modifications fall within the scope of the invention and its equivalents, the invention is also intended to include such modifications and modifications.

Claims

1. A multi-target tracking and segmentation method based on feature fusion and spatiotemporal memory, characterized in that, include: S1. Configure the multi-target video sequence to be detected and obtain the original state features of each target in each frame of the image; S2. Calculate the first... Frame image compared to the first Camera motion compensation matrix of a frame image. ; For each objective, obtain its value at the [missing value] in the [missing value]. The state vector of the Kalman filter in the frame image is transformed into a coordinate system based on the camera motion compensation matrix to obtain the compensated state vector. S3. Based on the compensated state vector, a Kalman filter is used to obtain the state vector of each target in the first position. Predicted bounding boxes in a frame image; S4. Construct a spatiotemporal memory library containing a short-term buffer and a long-term storage area; for the first detected target in a multi-target video sequence, instantiate an independent long-term trajectory prototype for it and establish a corresponding index in the spatiotemporal memory library; S5. Based on the spatiotemporal memory bank, the first... The original state features and predicted bounding boxes of each target in the frame image are hierarchically matched, and the Kalman filter parameters and spatiotemporal memory are updated based on the matching results. S6. Order The updated Kalman filter and spatiotemporal memory are used to return to the execution steps S3 to S5 until each frame of the multi-target video sequence is traversed to obtain the multi-target tracking and segmentation results. The multi-target tracking and segmentation results are then post-processed to obtain the final tracking and segmentation results.

2. The multi-target tracking and segmentation method based on feature fusion and spatiotemporal memory according to claim 1, characterized in that, The original state features of each target in each frame of the image are obtained using the following method: S10. Use an object detection and segmentation network to extract the detection box and segmentation mask for each object in each frame of the image; S11. Configure a shape encoder and a lightweight re-identification network, crop and scale the target image region corresponding to the detection box, and input them into the backbone networks of the lightweight re-identification network and the target detection and segmentation network respectively to extract semantic features and appearance features; at the same time, input the segmentation mask into the shape encoder for multi-scale encoding to obtain shape features. S12. Concatenate the semantic and shape features of each target, and perform fusion encoding and dimensionality reduction on the concatenated features to obtain memory-encoded features; S13. Based on memory encoding features and appearance features, obtain the original state features of each target.

3. The multi-target tracking and segmentation method based on feature fusion and spatiotemporal memory according to claim 2, characterized in that, The hierarchical matching includes: First-level matching: Configure the overlap threshold. For each target, obtain its last observed segmentation mask as the reference mask, and calculate the first-level matching. The geometric intersection-over-union ratio (GOU) of the detected bounding box and the predicted bounding box in the frame image, and the mask intersection-over-union ratio of the segmentation mask and the reference mask are weighted and summed to obtain the mixed overlap. It is determined whether the mixed overlap is greater than the overlap threshold. If it is greater, it is determined that the original state feature and the predicted bounding box come from the same target. Otherwise, perform the second-level matching: configure the matching cost threshold, retrieve historical trajectory state features from the short-term buffer, retrieve trajectory prototype features from the long-term trajectory prototype area, calculate the fused features based on the current original state features as the query vector and the historical trajectory state features as the key vector using a lightweight cross-attention fusion mechanism; calculate the appearance similarity based on the trajectory prototype features and construct the cost matrix, calculate the minimum total cost matching using the Hungarian algorithm, and determine whether the minimum total cost matching is less than the matching cost threshold. If it is less, it is determined that the original state feature and the predicted box come from the same target. For targets remaining after the first two levels of matching, if the number of consecutive unmatched frames is within the maximum allowed number of retained frames, then the third level of matching is executed; otherwise, the first level of matching continues. The third level of matching is as follows: configure a similarity threshold and a geometric intersection-union (GUU) threshold that are less than the matching cost threshold; retrieve key trajectory state features from the long-term storage area; obtain trajectory prototypes from the long-term trajectory prototype area; filter targets whose GUU of the predicted bounding box and the detection box is greater than the geometric GUU threshold; obtain deep fusion state features based on the deep cross-attention fusion mechanism; calculate the first similarity between the trajectory prototype and the current appearance features, and the second similarity between the deep fusion state features and the current original state features; perform a weighted summation of the first and second similarities; and select the target with the largest summation result that is greater than the similarity threshold, which is the target that is successfully matched. Otherwise, a new target creation threshold is preset, and the confidence level of the original state feature is calculated. If the confidence level is higher than the new target creation threshold, the trajectory is initialized as a new target trajectory; otherwise, the target is discarded.

4. The multi-target tracking and segmentation method based on feature fusion and spatiotemporal memory according to claim 3, characterized in that, The cost matrix is ​​as follows: in, Indicates the first The trajectory and the first The cost of matching between targets , , These represent the weighting coefficients for geometric overlap, appearance feature similarity, and memory fusion similarity, respectively. Indicates the first The predicted bounding box or prediction mask of the first trajectory and the second trajectory The degree of overlap between the detection bounding boxes or segmentation masks of each target; Indicates the first The trajectory prototype features of the first trajectory are similar to those of the second trajectory. The similarity of the appearance features of the targets; Indicates the current frame number The memory encoding features of each target Indicates the first A set of historical trajectory state features retrieved from the spatiotemporal memory bank for each trajectory; It represents the cosine similarity of the features between the memory-encoded features and the set of historical trajectory state features after lightweight cross-attention fusion.

5. The multi-target tracking and segmentation method based on feature fusion and spatiotemporal memory according to claim 3, characterized in that, The deep fusion state characteristics are as follows: in, Indicates the characteristics of deep fusion state. This represents the intermediate fused feature vector after the attention residual. The query vector is composed of the memory-encoded features of the current target. These are the key vector and the value vector, respectively, both composed of key trajectory state features retrieved from the long-term storage area; This indicates a multi-headed cross-attention operation. This indicates a normalization operation. This represents the nonlinear mapping of a feedforward neural network.

6. The multi-target tracking and segmentation method based on feature fusion and spatiotemporal memory according to claim 3, characterized in that, For each target, after a successful match, the spatiotemporal memory is updated using the following method: Configure quality gating conditions, velocity gating conditions, and keyframe determination conditions, and update the short-term buffer using the following method: Configure the capacity of the short-term buffer, store the frame data containing the original state characteristics of the target that simultaneously meets the quality gating conditions and velocity gating conditions into the short-term buffer, and remove the earliest frame data if the short-term buffer is full. The long-term trajectory prototype region is updated using the following method: the smooth prototype features of the long-term trajectory prototype region are updated using the exponential moving average algorithm; the capacity of the prototype feature set of the long-term trajectory prototype region is configured, and the original state features of the current frame are stored in the prototype feature set. If the upper limit of the prototype feature set capacity is reached, the quality score of each prototype feature in the prototype feature set is calculated, and only the top-K features with the highest quality scores are retained. The long-term storage area is updated using the following method: Configure the capacity of the long-term storage area, filter frame data that simultaneously meet the quality gating condition, speed gating condition, and keyframe determination condition, and store them in the long-term storage area. If the long-term storage area is full, sort the frame data in the storage area based on the importance score, and remove the frame data with the lowest importance score.

7. The multi-target tracking and segmentation method based on feature fusion and spatiotemporal memory according to claim 3, characterized in that, The smooth prototype features are as follows: in, No. Smooth prototype features after frame update The momentum coefficient represents the exponential moving average, and its value ranges from 0 to 1. Indicates the first Smooth prototype features of frames, Indicates the first The original state features of the current target whose frame was successfully matched.

8. The multi-target tracking and segmentation method based on feature fusion and spatiotemporal memory according to claim 2, characterized in that, The target detection and segmentation network is YOLO11-seg, the lightweight re-identification network is OSNet, and the shape encoder is a single-channel multi-scale convolutional neural network based on ResNet18.

9. The multi-target tracking and segmentation method based on feature fusion and spatiotemporal memory according to claim 3, characterized in that, The post-processing includes: For trajectory segments in a lost state, linear interpolation is used to fill in their position and mask missingness; morphological smoothing is performed on the segmentation mask to remove noise at the mask edges; trajectory segments in a lost state are target trajectories whose consecutive unmatched frames are within the maximum allowed number of retained frames and which were not successfully matched in the previous frame.

10. A multi-target tracking and segmentation system based on feature fusion and spatiotemporal memory, characterized in that, include: The original state feature acquisition unit is configured with a multi-target video sequence to be detected and acquires the original state features of each target in each frame of the image. Prediction box acquisition unit, calculate the first Frame image compared to the first Camera motion compensation matrix of a frame image. ; For each objective, obtain its value at the [missing value] in the [missing value]. The Kalman filter state vector in the frame image is transformed in coordinate system based on the camera motion compensation matrix to obtain the compensated state vector. Based on the compensated state vector, the Kalman filter is used to obtain the state vector of each target in the [frame image]. Predicted bounding boxes in a frame image; A hierarchical matching and update unit is constructed, which includes a spatiotemporal memory bank containing a short-term buffer and a long-term storage area. For the first detected target in a multi-target video sequence, an independent long-term trajectory prototype is instantiated for it, and a corresponding index is established in the spatiotemporal memory bank. Based on the spatiotemporal memory bank, the first target is matched and updated. The original state features and predicted bounding boxes of each target in the frame image are hierarchically matched, and the Kalman filter parameters and spatiotemporal memory are updated based on the matching results. The result output unit, let The updated Kalman filter and spatiotemporal memory are used to return the prediction box acquisition unit and the hierarchical matching and update unit until every frame of the multi-target video sequence is traversed to obtain the multi-target tracking and segmentation results. The multi-target tracking and segmentation results are then post-processed to obtain the final tracking and segmentation results.

Images