Multi-target tracking and segmentation system and method

Abstract

This application relates to a multi-target tracking and segmentation system and method. The system includes: a feature extraction module to extract image features from target video; multiple detection and segmentation modules, each including a segmentation head and a detector connected to the segmentation head, with each detector connected to the next detector and each segmentation head connected to the next segmentation head; in the first detection and segmentation module and each intermediate detection and segmentation module: the detector performs detection to obtain a second feature and image extraction features; the segmentation head fuses the intermediate stage segmentation reference features predicted based on the second feature with the second feature to obtain a third feature; in the last detection and segmentation module: the detector performs detection to obtain a first feature, a second feature, and a tracking embedding feature; the detection head detects the first feature to obtain a target detection result; the segmentation head predicts the segmentation result based on the third feature and the second feature; and the tracker determines the tracking result based on the tracking embedding feature. The system exhibits high prediction accuracy, good precision, and robustness.

Description

Technical Field

[0001] This application relates to the field of vision technology, and in particular to a multi-target tracking and segmentation system and method. Background Technology

[0002] Multi-object tracking and segmentation (MOTS) is a fundamental task in video analysis within computer vision. Its goal is to enable models to accurately predict bounding boxes and segmentation masks for each object in a video, and to match object IDs over time, performing an ID match between each frame – meaning the same object needs to be assigned the same ID. Compared to multi-object tracking (MOT), which focuses on detecting bounding boxes, MOTS requires predicting instance-level segmentation masks, i.e., the object's outline. With the development of deep learning technology, deep neural networks have been successfully applied to MOT and MOTS tasks, resulting in methods such as DeepSORT (Simple Online Realtime Tracking with a Deep Association Metric), FairMOT (Fair Multiple Object Tracking), QDTrack (Quasi-Dense Tracking), and PCAN (Prototypical Cross-Attention Networks). Previous MOTS methods were based on the MOT method with the addition of a segmentation branch, and the segmentation results depended entirely on the detection results in the spatial dimension. Therefore, the prediction of segmentation was limited, and the segmentation accuracy was also affected.

[0003] The technical solution proposed in this invention aims to solve the above-mentioned problems. It performs a spatial and temporal optimization process for the segmentation branches. This invention proposes a novel end-to-end MOTS framework that, while refining the spatial features of the segmentation, also performs temporal fusion of the segmentation features, making MOTS more accurate and robust. Summary of the Invention

[0004] In view of this, a multi-target tracking and segmentation system and method are proposed, which improves the accuracy and robustness of multi-target tracking and segmentation, and improves the precision of the results.

[0005] In a first aspect, embodiments of this application provide a multi-target tracking and segmentation system, including: a feature extraction module, multiple detection and segmentation modules, and a tracker. Each detection and segmentation module includes a segmentation head and a detector connected to the segmentation head. Each detector is sequentially connected to the detector of the next detection and segmentation module, and each segmentation head is connected to the segmentation head of the next detection and segmentation module. The multiple detection and segmentation modules are divided into a first detection and segmentation module, one or more intermediate detection and segmentation modules, and a last detection and segmentation module. The last detection and segmentation module further includes a detection head connected to the detector.

[0006] The feature extraction module is used to extract features from each video frame of the target video and obtain the image features of each video frame.

[0007] In the first detection and segmentation module: the detector is used to detect the image features to obtain the second feature and the image extraction feature; the segmentation head is used to predict the intermediate stage segmentation reference feature based on the received second feature, and process the intermediate stage segmentation reference feature and fuse it with the second feature to obtain the third feature;

[0008] In each of the intermediate detection and segmentation modules: the detector is used to detect the features extracted from the received image to obtain the second feature and the image extracted feature; the segmentation head is used to predict the intermediate stage segmentation reference feature based on the received third feature and the second feature, and then process the intermediate stage segmentation reference feature and fuse it with the second feature to obtain the third feature;

[0009] In the last detection and segmentation module: the detector is used to extract features from the received image and detect them to obtain a first feature, a second feature, and a tracking embedding feature; the detection head is used to perform target detection based on the received first feature to obtain the target detection result; and the segmentation head is used to predict the segmentation result based on the received third feature and the second feature.

[0010] The tracker is used to determine the tracking result based on the tracking embedding features and the target detection result.

[0011] The multi-object tracking and segmentation system provided in the first aspect can enrich the representation of objects, thereby improving the overall prediction accuracy, precision, and robustness.

[0012] In one possible implementation, the first detection and segmentation module further includes a processing submodule, in which:

[0013] The detector is used to send the second feature to the processing submodule and the segmentation head, respectively;

[0014] The processing submodule is used to refine the received second feature and then send the refined second feature to the segmentation head.

[0015] The segmentation head is used to predict intermediate stage segmentation reference features based on the received second feature, including: the segmentation head is used to predict intermediate stage segmentation reference features based on the received second feature and the refined second feature.

[0016] In this way, the segmentation head can improve the accuracy and precision of the intermediate stage segmentation reference features by making predictions based on the second feature and the refined second feature.

[0017] In one possible implementation, one or more of the intermediate detection and segmentation modules and the last detection and segmentation module further include processing submodules. Each processing submodule is connected between the detector and the segmentation head of the intermediate detection and segmentation module or the last detection and segmentation module. Then, in the intermediate detection and segmentation module or the last detection and segmentation module equipped with the processing submodule:

[0018] The detector is used to send the second feature to the processing submodule;

[0019] The processing submodule is used to refine the received second feature and then send the refined second feature to the connected segmentation head.

[0020] The segmentation head is used to predict intermediate stage segmentation reference features based on the received third feature and the second feature, including: the segmentation head predicting intermediate stage segmentation reference features based on the received third feature and the refined second feature; and / or

[0021] The segmentation head is used to predict the segmentation result based on the received third feature and the second feature, including: the segmentation head is used to predict the segmentation result based on the received third feature and the refined second feature.

[0022] In this way, the segmentation head can improve the accuracy and precision of the intermediate-stage segmentation reference features and segmentation results by making predictions based on the third feature and the refined second feature.

[0023] In one possible implementation, each of the intermediate detection and segmentation modules and the first detection and segmentation module is provided with a detection head, and each detection head is connected to the detector of the next detection and segmentation module.

[0024] In the first detection and segmentation module: the detector is further used to detect the received image features to obtain a first feature, and the detection head is used to detect the first feature to obtain an intermediate target detection result, and send the intermediate target detection result to the connected next detector;

[0025] In each of the intermediate detection and segmentation modules: the detector is used to detect the received image extracted features to obtain a second feature and an image extracted feature, including: the detector is used to detect the received image extracted features and the target detection intermediate result to obtain a first feature, a second feature and an image extracted feature; the detection head is used to detect the first feature to obtain a target detection intermediate result, and send the target detection intermediate result to the connected next detector;

[0026] In the last detection and segmentation module: the detector is used to detect the features extracted from the received image to obtain the first feature, the second feature and the tracking embedding feature, including: the detector is used to detect the features extracted from the received image and the intermediate results of target detection to obtain the first feature, the second feature and the tracking embedding feature.

[0027] In this way, the detectors in each intermediate detection and segmentation module and the last detection and segmentation module perform detection and feature extraction based on image-extracted features and intermediate target detection results, which can improve the accuracy of the detector's detection results, and thus improve the accuracy and precision of the prediction results of the entire system.

[0028] In one possible implementation, the intermediate stage segmentation reference feature is processed and then fused with the second feature to obtain a third feature, including:

[0029] The intermediate stage segmentation reference feature is linearly transformed and then summed with the second feature to obtain the third feature.

[0030] In one possible implementation, the system further includes: at least one time-domain fusion module.

[0031] Each of the temporal fusion modules is connected between two adjacent segmentation heads. It is used to determine the corresponding temporal information based on the third feature from the previous segmentation head and the third feature of the reference video frame corresponding to the current video frame through inter-frame attention calculation and differential attention calculation, and fuse it into the third feature to obtain the fused third feature. The reference video frame is the video frame before the current video frame.

[0032] The segmentation head is used to predict intermediate stage segmentation reference features based on the received third feature and the second feature, including: the segmentation head predicting intermediate stage segmentation reference features based on the received fused third feature and the second feature; and / or

[0033] The segmentation head is used to predict the segmentation result based on the received third feature and the second feature, including: the segmentation head is used to predict the segmentation result based on the received fused third feature and the second feature.

[0034] In this way, since the computational complexity of inter-frame difference is low, and it can accurately represent the motion information of consecutive frames, the combination of dual attention mechanism (i.e., inter-frame attention calculation and differential attention calculation) can mine continuous features in the temporal domain and obtain temporal feature changes, which is suitable for temporal feature modeling in MOTS scenarios. After obtaining the temporal information between frames, the temporal information between frames can be effectively fused into the third feature, so that the segmentation head can predict more accurate and higher precision results based on the fused third feature.

[0035] In one possible implementation, based on the third feature from the previous segmentation head and the third feature of the reference video frame corresponding to the current video frame, the corresponding temporal information is determined through inter-frame attention calculation and differential attention calculation and fused into the third feature to obtain the fused third feature, including:

[0036] Perform a first convolution transformation on the third feature of the current video frame and the third feature of the reference video frame respectively to obtain the first transformed feature of the current video frame and the second transformed feature of the reference video frame.

[0037] The first transformed feature and the second transformed feature are processed to obtain the difference features of the current video frame relative to the reference frame;

[0038] The difference features and the first transformed features are multiplied by a matrix to obtain the difference similarity features of the current video frame relative to the reference frame;

[0039] The first transformed feature and the second transformed feature are multiplied by a matrix to obtain the two-frame similarity feature of the current video frame relative to the reference frame;

[0040] After performing activation operations on the differential similarity features, they are multiplied by matrix with the third transformed features obtained by performing a second convolution transformation on the third features of the current video frame to obtain differential attention features.

[0041] After performing activation operations on the dual-frame similarity features, they are multiplied by a matrix with the third transformed features obtained by performing a second convolution transformation on the third features of the current video frame to obtain dual-frame attention features.

[0042] The fused third feature is calculated by summing the third feature, the differential attention feature, and the dual-frame attention feature.

[0043] In one possible implementation, during the training process of the system,

[0044] In the first detection and segmentation module, the segmentation head is also used to predict intermediate segmentation results based on the received second feature;

[0045] In each of the intermediate detection and segmentation modules, the segmentation head is further used to predict intermediate segmentation results based on the received second and third features;

[0046] In the last detection and segmentation module, the segmentation head is also used to predict intermediate stage segmentation reference features based on the received third feature and the second feature.

[0047] The output results of each detection head and each segmentation head are used as the basis for calculating the system loss function during the system training process. The system loss function is the basis for updating the parameters of the system.

[0048] In this way, during the training process, the intermediate target detection results or target detection results output by the detection head, the intermediate segmentation results or segmentation results output by the segmentation head, the intermediate stage segmentation reference features, the tracking results output by the tracker, and the image extraction features output by each detector can be used to supervise the training process of the system and ensure the optimal overall performance of the system.

[0049] In systems comprising multiple detection and segmentation modules and one or more temporal fusion modules, a tracking-by-detection framework based on MOT is employed to fully leverage the powerful expressive capabilities of the detectors for temporal matching. Multiple detection and segmentation modules fuse features with spatial information, while the temporal fusion module fuses features with temporal information, resulting in richer object representations and thus improving the prediction accuracy of MOTS.

[0050] Secondly, embodiments of this application provide a multi-target tracking and segmentation system, which includes: a feature extraction module, multiple detection and segmentation modules, a tracker, and a temporal fusion module. Each detection and segmentation module includes a detector. The multiple detection and segmentation modules are divided into a first detection and segmentation module, one or more intermediate detection and segmentation modules, and a last detection and segmentation module. The last detection and segmentation module further includes a detection head and a segmentation head. The temporal fusion module is connected between the detector and the segmentation head of the last detection and segmentation module.

[0051] The feature extraction module is used to extract features from each video frame of the target video and obtain the image features of each video frame.

[0052] In the first detection and segmentation module, the detector is used to detect the received image features, extract image features, and send them to the next connected detector;

[0053] In each of the intermediate detection and segmentation modules, the detector is used to detect the features extracted from the received image, obtain the image extracted features, and send them to the next connected detector;

[0054] In the last detection and segmentation module, the detector is used to extract features from the received image and detect them to obtain a first feature, a second feature, and a tracking embedding feature; the detection head is used to perform target detection based on the received first feature to obtain the target detection result.

[0055] The temporal fusion module is used to, after receiving the second feature, determine the corresponding temporal information based on the second feature and the second feature of the reference video frame corresponding to the current video frame through inter-frame attention calculation and differential attention calculation, and fuse it into the second feature to obtain the fused second feature, and send the fused second feature to the segmentation head in the last detection and segmentation module, wherein the reference video frame is the previous video frame of the current video frame;

[0056] The segmentation head in the last detection and segmentation module is used to predict the segmentation result based on the received fused second feature;

[0057] The tracker is used to determine the tracking result based on the tracking embedding features and the target detection result.

[0058] In this way, the end-to-end MOTS framework based on the Query mechanism can naturally incorporate the fusion of temporal and spatial features to ensure real-time prediction. The combined effect of the fusion of spatial and temporal dimensions can enrich the representation of objects, thereby improving overall prediction accuracy, robustness, and precision.

[0059] In one possible implementation, the fused second feature, obtained by determining corresponding temporal information through inter-frame attention calculation and differential attention calculation based on the second feature and the second feature of a reference video frame corresponding to the current video frame, and fusing it into the second feature, includes:

[0060] Perform a first convolution transformation on the second feature of the current video frame and the second feature of the reference video frame respectively to obtain the first transformed feature of the current video frame and the second transformed feature of the reference video frame;

[0061] The first transformed feature and the second transformed feature are processed to obtain the difference features of the current video frame relative to the reference frame;

[0062] The difference features and the first transformed features are multiplied by a matrix to obtain the difference similarity features of the current video frame relative to the reference frame;

[0063] The first transformed feature and the second transformed feature are multiplied by a matrix to obtain the two-frame similarity feature of the current video frame relative to the reference frame;

[0064] After performing activation operations on the differential similarity features, they are multiplied by a matrix with the third transformed features obtained by performing a second convolution transformation on the second features of the current video frame to obtain differential attention features.

[0065] After performing activation operations on the dual-frame similarity features, they are multiplied by a matrix with the third transformed features obtained by performing a second convolution transformation on the second features of the current video frame to obtain dual-frame attention features.

[0066] The fused second feature is calculated by summing the second feature, the differential attention feature, and the dual-frame attention feature.

[0067] In one possible implementation, each of the intermediate detection and segmentation modules and the first detection and segmentation module is provided with a detection head, and each detection head is connected to the detector of the next detection and segmentation module.

[0068] In the first detection and segmentation module: the detector is further used to detect the received image features to obtain a first feature, the detection head is used to detect the first feature to obtain an intermediate target detection result, and the intermediate target detection result is sent to the detector in the next detection and segmentation module;

[0069] In each of the intermediate detection and segmentation modules: the detector is used to detect the received image extracted features to obtain new image extracted features, including: the detector is used to detect the received image extracted features and the target detection intermediate results to obtain a first feature and the image extracted features; the detection head is used to detect the first feature to obtain the target detection intermediate results, and send the target detection intermediate results to the detector in the next detection and segmentation module;

[0070] In the last detection and segmentation module: the detector is used to detect the features extracted from the received image to obtain the first feature, the second feature and the tracking embedding feature, including: the detector is used to detect the features extracted from the received image and the intermediate results of target detection to obtain the first feature, the second feature and the tracking embedding feature.

[0071] In this way, the detectors in each intermediate detection and segmentation module and the last detection and segmentation module perform detection and feature extraction based on image-extracted features and intermediate target detection results, which can improve the accuracy of the detector's detection results, and thus improve the accuracy and precision of the prediction results of the entire system.

[0072] In one possible implementation, each of the intermediate detection and segmentation modules and the first detection and segmentation module is equipped with a segmentation head during the training process of the system.

[0073] In the first detection and segmentation module: the detector is further used to detect the image features to obtain a second feature, and send the second feature to the segmentation head of the first detection and segmentation module, the segmentation head is used to predict the intermediate segmentation result based on the second feature;

[0074] In each of the intermediate detection and segmentation modules: the detector is further configured to detect the features extracted from the image to obtain a second feature, and send the second feature to the segmentation head of the intermediate detection and segmentation module; the segmentation head is configured to predict the intermediate segmentation result based on the received second feature;

[0075] The output results of each detection head and each segmentation head are used as the basis for calculating the system loss function during the system training process. The system loss function is the basis for updating the parameters of the system.

[0076] In this way, during the training process, the training process of the system can be supervised by the intermediate target detection results or target detection results output by the detection head, the intermediate segmentation results or segmentation results output by the segmentation head, the tracking results output by the tracker, and the results of image extraction features output by each detector, so as to ensure the optimal overall performance of the system.

[0077] In one possible implementation, the last detection and segmentation module further includes a processing submodule.

[0078] The time-domain fusion module is also used to send the fused second feature to the processing submodule;

[0079] The processing submodule is used to refine the received fused second feature and then send the refined second feature to the connected segmentation head.

[0080] The segmentation head in the last detection and segmentation module is used to predict the segmentation result based on the received fused second feature, including: the segmentation head in the last detection and segmentation module is used to predict the segmentation result based on the received fused second feature and the refined second feature.

[0081] In this way, the segmentation head can improve the accuracy and precision of the segmentation results by making predictions based on the fused second feature and the refined second feature.

[0082] In a system comprising multiple detection and segmentation modules as well as a temporal fusion module, a tracking-by-detection framework based on MOT is employed to fully leverage the powerful expressive capabilities of the detectors for temporal matching. The temporal fusion module integrates features with temporal information, resulting in richer object representations and thus improving the prediction accuracy of MOTS.

[0083] Thirdly, embodiments of this application provide a multi-target tracking and segmentation method, which includes: an image feature extraction stage, a first detection and segmentation stage, one or more intermediate detection and segmentation stages, a final detection and segmentation stage, and a tracking result determination stage.

[0084] Image feature extraction stage: Feature extraction is performed on each video frame of the target video to obtain the image features of each video frame;

[0085] First detection and segmentation stage: The image features are detected to obtain second features and image extraction features; intermediate stage segmentation reference features are predicted based on the second features, and the intermediate stage segmentation reference features are processed and fused with the second features to obtain third features; the image extraction features and third features obtained in the current stage are used as inputs for the first intermediate detection and segmentation stage.

[0086] Each intermediate detection and segmentation stage: Detect the received image features to obtain the second feature and the image extraction feature; predict the intermediate stage segmentation reference feature based on the received third feature and the second feature, and process the intermediate stage segmentation reference feature and fuse it with the second feature to obtain the third feature; use the image extraction feature and the third feature obtained in the current stage as the input for the next intermediate detection and segmentation stage or the final detection and segmentation stage;

[0087] The final detection and segmentation stage involves: extracting features from the received image to obtain the first feature, the second feature, and the tracking embedding feature; performing target detection based on the received first feature to obtain the target detection result; and predicting the segmentation result based on the received third feature and the second feature.

[0088] Tracking result determination stage: Based on the tracking embedding features and the target detection results, the tracking result is determined.

[0089] In one possible implementation, predicting intermediate stage segmentation reference features based on the second feature includes:

[0090] Based on the second feature and the refined second feature obtained by refining the second feature, the intermediate stage segmentation reference feature is predicted.

[0091] In one possible implementation, predicting intermediate stage segmentation reference features based on the received third feature and the second feature includes: predicting intermediate stage segmentation reference features based on the received third feature and a refined second feature obtained by refining the second feature; and / or

[0092] The segmentation result is predicted based on the received third feature and the second feature, including: predicting the segmentation result based on the received third feature and the refined second feature obtained by refining the second feature.

[0093] In one possible implementation, the first detection and segmentation stage further includes: detecting the received image features to obtain a first feature, detecting the first feature to obtain an intermediate target detection result, and using the intermediate target detection result obtained in the previous stage as the input to the subsequent first intermediate detection and segmentation stage;

[0094] In each intermediate detection and segmentation stage, the received image features are detected to obtain the second feature and the image extracted features, including: detecting the received image extracted features and the intermediate target detection results to obtain the first feature, the second feature and the image extracted features; each intermediate detection and segmentation stage also includes: detecting the first feature of the current stage to obtain the intermediate target detection result, and using the intermediate target detection result obtained in the current stage as the input of the next intermediate detection and segmentation stage or the final detection and segmentation stage;

[0095] In the final detection and segmentation stage, the features extracted from the received image are detected to obtain the first feature, the second feature, and the tracking embedding feature. This includes detecting the extracted features from the received image and the intermediate results of target detection to obtain the first feature, the second feature, and the tracking embedding feature.

[0096] In one possible implementation, the intermediate stage segmentation reference feature is processed and then fused with the second feature to obtain a third feature, including:

[0097] The intermediate stage segmentation reference feature is linearly transformed and then summed with the second feature to obtain the third feature.

[0098] In one possible implementation, the method further includes:

[0099] After obtaining the third feature of the current stage in the first detection and segmentation stage and each of the intermediate detection and segmentation stages, the corresponding temporal information is determined by inter-frame attention calculation and differential attention calculation based on the third feature obtained in the current stage and the third feature of the reference video frame corresponding to the current video frame, and then fused into the third feature to obtain the fused third feature. The reference video frame is the previous video frame of the current video frame.

[0100] The step of predicting intermediate stage segmentation reference features based on the received third feature and the second feature includes: predicting intermediate stage segmentation reference features based on the received fused third feature and the second feature; and / or

[0101] The segmentation result is predicted based on the received third feature and the second feature, including: predicting the segmentation result based on the received fused third feature and the second feature.

[0102] In one possible implementation, based on the third feature from the previous segmentation head and the third feature of the reference video frame corresponding to the current video frame, the corresponding temporal information is determined through inter-frame attention calculation and differential attention calculation and fused into the third feature to obtain the fused third feature, including:

[0103] Perform a first convolution transformation on the third feature of the current video frame and the third feature of the reference video frame respectively to obtain the first transformed feature of the current video frame and the second transformed feature of the reference video frame.

[0104] The first transformed feature and the second transformed feature are processed to obtain the difference features of the current video frame relative to the reference frame;

[0105] The difference features and the first transformed features are multiplied by a matrix to obtain the difference similarity features of the current video frame relative to the reference frame;

[0106] The first transformed feature and the second transformed feature are multiplied by a matrix to obtain the two-frame similarity feature of the current video frame relative to the reference frame;

[0107] After performing activation operations on the differential similarity features, they are multiplied by matrix with the third transformed features obtained by performing a second convolution transformation on the third features of the current video frame to obtain differential attention features.

[0108] After performing activation operations on the dual-frame similarity features, they are multiplied by a matrix with the third transformed features obtained by performing a second convolution transformation on the third features of the current video frame to obtain dual-frame attention features.

[0109] The fused third feature is calculated by summing the third feature, the differential attention feature, and the dual-frame attention feature.

[0110] In one possible implementation, the first detection segmentation stage during training also includes: predicting intermediate segmentation results based on the received second feature;

[0111] Each intermediate detection and segmentation stage also includes: predicting intermediate segmentation results based on the received second and third features;

[0112] The final detection and segmentation stage also includes: predicting intermediate stage segmentation reference features based on the received third feature and the second feature;

[0113] The output results of each stage are used as the basis for calculating the system loss function during the training process, and the system loss function is the basis for updating the parameters of the system.

[0114] The beneficial effects of the multi-target tracking and segmentation methods provided in the third aspect and its various possible implementations are the same as those in the first aspect and its various possible implementations, and will not be elaborated here to avoid redundancy.

[0115] Fourthly, embodiments of this application provide a multi-target tracking and segmentation method, which includes: an image feature extraction stage, a first detection and segmentation stage, one or more intermediate detection and segmentation stages, a final detection and segmentation stage, a temporal fusion stage, and a tracking result determination stage.

[0116] Image feature extraction stage: Feature extraction is performed on each video frame of the target video to obtain the image features of each video frame;

[0117] First detection and segmentation stage: The received image features are detected to obtain image extraction features, and the image extraction features obtained in the current stage are used as the input for the first intermediate detection and segmentation stage.

[0118] Each intermediate detection and segmentation stage: Detect the features extracted from the received image to obtain new image features, and use the image features obtained in the current stage as input for the next intermediate detection and segmentation stage or the final detection and segmentation stage;

[0119] Finally, in the segmentation and detection stage: features are extracted from the received image and detected to obtain the first feature, the second feature, and the tracking embedding feature; target detection is performed based on the received first feature to obtain the target detection result.

[0120] Temporal fusion stage: Based on the second feature of the current video frame extracted in the final detection and segmentation stage and the second feature of the reference video frame corresponding to the current video frame, the corresponding temporal information is determined by inter-frame attention calculation and differential attention calculation and fused into the second feature to obtain the fused second feature, and the fused second feature is used as the input of the final detection and segmentation stage;

[0121] The final detection and segmentation stage also includes: predicting the segmentation result based on the received fused second feature;

[0122] Tracking result determination stage: Based on the tracking embedding features and the target detection results, the tracking result is determined.

[0123] In one possible implementation, based on the second feature of the current video frame extracted in the final detection and segmentation stage and the second feature of the reference video frame corresponding to the current video frame, the corresponding temporal information is determined through inter-frame attention calculation and differential attention calculation and fused into the second feature to obtain the fused second feature, including:

[0124] The second features of the current video frame and the second features of the reference video frame extracted in the final detection and segmentation stage are respectively subjected to the first convolution transformation to obtain the first transformed features of the current video frame and the second transformed features of the reference video frame.

[0125] The first transformed feature and the second transformed feature are processed to obtain the difference features of the current video frame relative to the reference frame;

[0126] The difference features and the first transformed features are multiplied by a matrix to obtain the difference similarity features of the current video frame relative to the reference frame;

[0127] The first transformed feature and the second transformed feature are multiplied by a matrix to obtain the two-frame similarity feature of the current video frame relative to the reference frame;

[0128] After performing activation operations on the differential similarity features, they are multiplied by matrix with the third transformed features obtained by performing a second convolution transformation on the second features of the current video frame extracted in the final detection and segmentation stage to obtain differential attention features.

[0129] After performing activation operations on the dual-frame similarity features, they are multiplied by a matrix with the third transformed features obtained by performing a second convolution transformation on the second features of the current video frame to obtain dual-frame attention features.

[0130] The fused second feature is calculated by summing the second feature extracted in the final detection and segmentation stage, the differential attention feature, and the dual-frame attention feature.

[0131] In one possible implementation, the first detection and segmentation stage further includes: detecting the received image features to obtain a first feature, detecting the first feature to obtain an intermediate target detection result, and using the intermediate target detection result obtained in the current stage as the input to the next first intermediate detection and segmentation stage;

[0132] In each intermediate detection and segmentation stage, the received image extraction features are detected to obtain new image extraction features, including: detecting the received image extraction features and the target detection intermediate results to obtain a first feature and the image extraction features; each intermediate detection and segmentation stage also includes: detecting the first feature to obtain a target detection intermediate result, and using the target detection intermediate result obtained in the current stage as the input of the next intermediate detection and segmentation stage or the final detection and segmentation stage;

[0133] In the final detection and segmentation stage, the features extracted from the received image are detected to obtain the first feature, the second feature, and the tracking embedding feature. This includes detecting the extracted features from the received image and the intermediate results of target detection to obtain the first feature, the second feature, and the tracking embedding feature.

[0134] In one possible implementation, during the training process, the first detection and segmentation stage further includes: detecting the image features to obtain a second feature, and predicting an intermediate segmentation result based on the second feature;

[0135] Each intermediate detection and segmentation stage further includes: detecting the features extracted from the image to obtain a second feature, and predicting the intermediate segmentation result based on the second feature;

[0136] The output results of each stage are used as the basis for calculating the system loss function during the training process, and the system loss function is the basis for updating the parameters of the system.

[0137] In one possible implementation, predicting the segmentation result based on the received fused second feature includes:

[0138] The received fused second feature is refined to obtain a refined second feature, and the segmentation result is predicted based on the fused second feature and the refined second feature.

[0139] The beneficial effects of the multi-target tracking and segmentation methods provided in the fourth aspect and its various possible implementations are the same as those in the second aspect and its various possible implementations, and will not be elaborated here to avoid redundancy.

[0140] Fifthly, embodiments of this application provide a target tracking and segmentation apparatus, comprising:

[0141] processor;

[0142] Memory used to store processor-executable instructions;

[0143] The processor is configured to implement, when executing the instructions, one or more of the third aspect or multiple possible implementations of the third aspect, or one or more of the fourth aspect or multiple possible implementations of the fourth aspect, a multi-target tracking and segmentation method.

[0144] Sixthly, embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement one or more of the multi-target tracking and segmentation methods described in the third and fourth aspects or various possible implementations of the third and fourth aspects.

[0145] In a seventh aspect, embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in an electronic device, a processor in the electronic device executes one or more of the third aspect or multiple possible implementations of the third aspect, or one or more of the fourth aspect or multiple possible implementations of the fourth aspect, a multi-target tracking and segmentation method.

[0146] These and other aspects of this application will become more apparent in the description of the following embodiments(s). Attached Figure Description

[0147] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this application together with the specification and serve to explain the principles of this application.

[0148] Figure 1 A block diagram of a multi-target tracking and segmentation system according to an embodiment of this application is shown.

[0149] Figure 2 This diagram illustrates an application scenario of a multi-target tracking and segmentation system according to an embodiment of this application.

[0150] Figure 3 A block diagram of a multi-target tracking and segmentation system according to an embodiment of this application is shown.

[0151] Figure 4 This diagram illustrates spatial fusion in a multi-target tracking and segmentation system according to an embodiment of the present application.

[0152] Figure 5 A schematic diagram showing the fusion of timing information in a third feature according to an embodiment of this application is shown.

[0153] Figure 6 A schematic diagram illustrating the working process of a multi-target tracking and segmentation system according to an embodiment of this application is shown.

[0154] Figure 7 A block diagram of a multi-target tracking and segmentation system according to an embodiment of this application is shown.

[0155] Figure 8 A schematic diagram showing the fusion of timing information in a second feature according to an embodiment of this application is illustrated. Detailed Implementation

[0156] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0157] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0158] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0159] To address the issues of inaccurate segmentation prediction accuracy and lack of temporal feature fusion in MOTS tasks, this application proposes a multi-object tracking and segmentation system and method. The end-to-end MOTS framework based on the Query mechanism naturally incorporates temporal and spatial feature fusion, ensuring real-time prediction. The combined effect of spatial and temporal dimension fusion enriches the object representation, thereby improving overall prediction accuracy and robustness.

[0160] Figure 1 A block diagram of a multi-target tracking and segmentation system according to an embodiment of this application is shown. Figure 1 As shown, the multi-target tracking and segmentation system provided in this application includes: a feature extraction module 10, and multiple detection and segmentation modules (such as...). Figure 1 (as shown in 21, 22, 23) and tracker 30. Each of the detection and segmentation modules includes a segmentation head (e.g., ...). Figure 1(as shown in 213, 223, 233) and detectors connected to the segmentation head (such as...) Figure 1 As shown in diagrams 211, 221, and 231), each detector is sequentially connected to the detector of the next detection and segmentation module, and each segmentation head is connected to the segmentation head of the next detection and segmentation module. The plurality of detection and segmentation modules are divided into a first detection and segmentation module 21, one or more intermediate detection and segmentation modules 22, and a last detection and segmentation module 23. The last detection and segmentation module 23 further includes a detection head 232 connected to the detector 231. The system may include one or more intermediate detection and segmentation modules 22. For simplicity... Figure 1 The other figures in this application only schematically show one intermediate detection and segmentation module 22. The number of intermediate detection and segmentation modules 22 can be set according to actual needs, and this application does not limit this.

[0161] The feature extraction module 10 is used to extract features from each video frame of the target video to obtain the image features of each video frame. In some embodiments, the feature extraction module can use a pre-trained backbone network to extract features from each video frame, which is not limited in this application. The target video can be a video actually captured by an image acquisition device such as a vehicle-mounted camera.

[0162] In the first detection and segmentation module 21: the detector 211 is used to detect the image features to obtain the second feature and the image extraction feature, and then sends the second feature to the segmentation head 213, and sends the image extraction feature to the first intermediate detection and segmentation module (i.e., Figure 1 The detector 221 in the intermediate detection and segmentation module 22) is used to predict intermediate stage segmentation reference features based on the second feature received from the detector 211, process the intermediate stage segmentation reference features and fuse them with the second feature to obtain a third feature, and then send the obtained third feature to the first intermediate detection and segmentation module (i.e., Figure 1 The segmentation head in the intermediate detection and segmentation module 22).

[0163] In each of the intermediate detection and segmentation modules 22: detector 221 is used to detect the received image extraction features (the image extraction features come from detector 211 or detector 221 in the previous intermediate detection and segmentation module 22) to obtain a second feature and an image extraction feature, and then send the second feature to the segmentation head 223 and the image extraction feature to the next connected detector (i.e., detector 221 in the next intermediate detection and segmentation module 22 or detector 231 in the last detection and segmentation module 23). Segmentation head 223 is used to predict intermediate stage segmentation reference features based on the received third feature (the third feature comes from the previous segmentation head, where the previous segmentation head can be the first detection and segmentation module or the segmentation head in the previous intermediate detection and segmentation module) and the second feature from detector 221 in the current detection and segmentation module, and then process the intermediate stage segmentation reference features and fuse them with the second feature to obtain a third feature, and then send the obtained third feature to the segmentation head in the next intermediate detection and segmentation module or the last detection and segmentation module.

[0164] In the last detection and segmentation module 23: detector 231 is used to extract features from the received image (from detector 221 of the last intermediate detection and segmentation module 22) to obtain a first feature, a second feature, and a tracking embedding feature. Then, the first feature is sent to the detection head 232, the second feature is sent to the segmentation head 233, and the tracking embedding feature is sent to the tracker 30. Detection head 232 is used to perform target detection based on the received first feature to obtain a target detection result. Segmentation head 233 is used to predict the segmentation result based on the received third feature (from the segmentation head 223 of the last intermediate detection and segmentation module 22) and the second feature.

[0165] The tracker 30 is used to determine the tracking result based on the tracking embedding features and the target detection result.

[0166] In this embodiment, after determining the target detection result, segmentation result, and tracking result, the system can output all three as a prediction result. This prediction result can be sent to the object that needs further processing based on the prediction result. The multi-target tracking and segmentation system can be applied to scenarios requiring the tracking and detection of multiple targets, such as autonomous driving, driver assistance systems, and new retail. For example, Figure 2 This diagram illustrates an application scenario of a multi-target tracking and segmentation system according to an embodiment of this application. Figure 2As shown, the system provided in this application can be applied to vehicles. The vehicle-mounted camera acquires continuous multi-frame images and sends these as target videos to a multi-target tracking and segmentation system. The multi-target tracking and segmentation system then sends the predicted results (including target detection results, segmentation results, and tracking results) to the vehicle-mounted chip based on the target videos. This allows the vehicle-mounted chip to determine the actual positions and timing information of pedestrians, vehicles, traffic signs, cyclists, and other targets on the road where the vehicle is currently located based on the prediction results during the autonomous driving process. Based on these conditions, the chip makes decisions to control the autonomous driving of the vehicle to further control its movement.

[0167] In this embodiment, the prediction result can be the result obtained by tracking, detecting, and segmenting the target in the video frame. The target detection result can include the target bounding box, category, and confidence score of the target contained in each video frame. The segmentation result can include the mask result of the target segmentation. The tracking result can include the target ID and other identifiers obtained by tracking the target in the video frame.

[0168] In some embodiments, the first feature detected by the detector can characterize target-related information such as the target's position in the video frame, and may be a mask feature or other feature used for target bounding box detection. The second feature detected by the detector may be information such as a mask feature or other feature that can characterize the target segmentation in the video frame. The intermediate-stage segmentation reference feature obtained by the segmentation head may be a mask feature that can be used as a reference for target segmentation. The algorithms for bounding box detection by each detection head, tracking detection by each tracker, and tracking embedding feature detection by the segmentation head can be set according to actual needs, and this application does not impose any limitations on these settings.

[0169] By sending the third feature obtained from the previous segmentation head to the next segmentation head, the outputs of each segmentation head are fused. This not only fully utilizes the spatial information of the features but also fully integrates the unique feature representation of the mask, while reducing the computational load. It achieves the fusion of spatial information and is suitable for multi-stage mask prediction in spatial dimensions with high computational load in MOTS.

[0170] Figure 3 A block diagram of a multi-target tracking and segmentation system according to an embodiment of this application is shown. In one possible implementation, such as... Figure 3As shown, the first detection and segmentation module 21 may further include a processing submodule 214. In the first detection and segmentation module 21: the detector 211 is used to send the second feature to the processing submodule 214 and the segmentation head 213 respectively. The processing submodule 214 is used to refine the received second feature and then send the refined second feature to the segmentation head 213. The segmentation head 213 is used to predict intermediate stage segmentation reference features based on the received second feature, which may include: the segmentation head 213 predicting intermediate stage segmentation reference features based on the received second feature and the refined second feature. Thus, the segmentation head 213's prediction based on the second feature and the refined second feature can improve the accuracy and precision of the intermediate stage segmentation reference features. The refined second feature may be a feature with more channels than the second feature obtained by processing the second feature.

[0171] In one possible implementation, such as Figure 3 As shown, one or more modules in the intermediate detection and segmentation module 22 may further include a processing submodule 224. Each processing submodule 224 is connected between the detector 221 and the segmentation head 223 of the intermediate detection and segmentation module 22. In the intermediate detection and segmentation module 22 with the processing submodule 224: the detector 221 is used to send the second feature to the processing submodule 224. The processing submodule 224 is used to refine the received second feature and then send the refined second feature to the connected segmentation head 223. The segmentation head 223 is used to predict intermediate stage segmentation reference features based on the received third feature and the second feature, which may include: the segmentation head 223 predicting intermediate stage segmentation reference features based on the received third feature and the refined second feature. Thus, the segmentation head 223's prediction based on the third feature and the refined second feature can improve the accuracy and precision of the intermediate stage segmentation reference features. The refined second feature may be a feature with more channels than the second feature obtained by processing the second feature.

[0172] In one possible implementation, such as Figure 3As shown, the last detection and segmentation module 23 may further include a processing submodule 234, which is connected between the detector 231 and the segmentation head 233 of the last detection and segmentation module 23. In the last detection and segmentation module 23: the detector 232 is used to send the second feature to the processing submodule 234. The processing submodule 234 is used to refine the received second feature and then send the refined second feature to the connected segmentation head 233. The segmentation head 233 is used to predict the segmentation result based on the received third feature and the second feature, which may include: the segmentation head 233 predicting the segmentation result based on the received third feature and the refined second feature. Thus, the segmentation head 233's prediction based on the third feature and the refined second feature can improve the accuracy and precision of the segmentation result. The refined second feature can be a feature with more channels than the second feature obtained by processing the second feature.

[0173] In one possible implementation, fusing the intermediate stage segmentation reference feature with the second feature to obtain the third feature may include: performing a linear transformation on the intermediate stage segmentation reference feature and summing it with the second feature to obtain the third feature. In some embodiments, the feature obtained after the linear transformation of the intermediate stage segmentation reference feature and the second feature may be summed using a weighted summation method to obtain the third feature. The third feature can be calculated based on the intermediate stage segmentation reference feature and the second feature, and this application does not impose any limitations on this.

[0174] For example, Figure 4 This diagram illustrates spatial fusion in a multi-target tracking and segmentation system according to an embodiment of this application. (Combined with...) Figure 4 It can be determined that the third feature M s It can be calculated using the following formula (1):

[0175] M s =mask_head(refine(F s ), fuse(F s ,trans(R s-1 ))) (1)

[0176] Among them, refine(F s ) is the refined second feature, R s-1 It is a reference feature for intermediate stage segmentation. F s This is the second characteristic.

[0177] In one possible implementation, such as Figure 3As shown, each intermediate detection and segmentation module 22 is equipped with a detection head 222, and the first detection and segmentation module 21 is equipped with a detection head 212. Each detection head is connected to the detector of the next detection and segmentation module. For example, detection head 212 is connected to detector 221, and detection head 222 is connected to detector 231 (not shown in the figure). In the first detection and segmentation module 21: detector 211 is also used to detect the received image features to obtain a first feature, and detection head 212 is used to detect the first feature to obtain an intermediate target detection result, and send the intermediate target detection result to the connected next detector (i.e., the detector in the first intermediate detection and segmentation module). In each intermediate detection and segmentation module 22: detector 221 is used to detect the received image extracted features to obtain a second feature and an image extracted feature, which may include: detector 221 is used to detect the received image extracted features and the intermediate target detection result to obtain a first feature, a second feature, and an image extracted feature, and also sends the first feature to detection head 212. The detection head 212 is used to detect the first feature to obtain an intermediate target detection result, and sends the intermediate target detection result to the connected next detector (i.e., the detector in the next intermediate detection segmentation module or the detector in the last detection segmentation module). In the last detection segmentation module 23: the detector 231 is used to detect features extracted from the received image to obtain a first feature, a second feature, and a tracking embedding feature. This may include: the detector 231 is used to detect the received image extracted features and the intermediate target detection result to obtain the first feature, the second feature, and the tracking embedding feature. In this way, the detectors in each intermediate detection segmentation module and the last detection segmentation module perform detection and feature extraction based on image extracted features and the intermediate target detection result, which can improve the accuracy of the detector's detection results, and thus improve the accuracy and precision of the prediction results of the entire system. The intermediate target detection result can refer to the result obtained by each detection head after performing target detection based on the received first feature.

[0178] In one possible implementation, such as Figure 3As shown, the system may further include at least one temporal fusion module 40. Each temporal fusion module 40 is connected between two adjacent segmentation heads and is used to determine the corresponding temporal information based on the third feature from the previous segmentation head and the third feature of a reference video frame corresponding to the current video frame, through inter-frame attention calculation and differential attention calculation, and fuse it into the third feature to obtain the fused third feature. The reference video frame is the previous video frame of the current video frame. The segmentation head 223 is used to predict intermediate stage segmentation reference features based on the received third feature and the second feature, and may include: the segmentation head 223 predicting intermediate stage segmentation reference features based on the received fused third feature and the second feature. And / or, the segmentation head 233 is used to predict the segmentation result based on the received third feature and the second feature, and may include: the segmentation head predicting the segmentation result based on the received fused third feature and the second feature. In this way, since the computational complexity of inter-frame difference is low, and it can accurately represent the motion information of consecutive frames, the combination of dual attention mechanism (i.e., inter-frame attention calculation and differential attention calculation) can mine continuous features in the temporal domain and obtain temporal feature changes, which is suitable for temporal feature modeling in MOTS scenarios. After obtaining the temporal information between frames, the temporal information between frames can be effectively fused into the third feature, so that the segmentation head can predict more accurate and higher precision results based on the fused third feature.

[0179] In this implementation, the system can be as follows: Figure 3 The system includes multiple temporal fusion modules 40, each used to fuse the third feature output by a segmentation head with temporal information. Alternatively, the system may include only one temporal fusion module connected to each segmentation head, used to fuse the third feature output by each segmentation head with temporal information, and then send the fused third feature to the corresponding next segmentation head. In some embodiments, the temporal fusion module in the system may also fuse only the third feature and temporal information input to the segmentation head 233 in the last detection and segmentation module 23, which can improve the overall system performance and reduce the computational load.

[0180] Figure 5 This diagram illustrates the fusion of timing information in a third feature according to an embodiment of this application. In one possible implementation, such as... Figure 5 As shown, based on the third feature from the previous segmentation head and the third feature of the reference video frame corresponding to the current video frame, the corresponding temporal information is determined through inter-frame attention calculation and differential attention calculation and fused into the third feature to obtain the fused third feature, which may include:

[0181] The third feature of the current video frame and the third feature of the reference video frame are subjected to a first convolution transformation to obtain a first transformed feature of the current video frame and a second transformed feature of the reference video frame. The first transformed feature and the second transformed feature are processed to obtain a difference feature of the current video frame relative to the reference frame. The difference feature and the first transformed feature are multiplied by a matrix to obtain a difference similarity feature of the current video frame relative to the reference frame. The first transformed feature and the second transformed feature are multiplied by a matrix to obtain a two-frame similarity feature of the current video frame relative to the reference frame. The difference similarity feature is activated and then multiplied by a matrix with the third transformed feature obtained by the second convolution transformation of the third feature of the current video frame to obtain a difference attention feature. The two-frame similarity feature is activated and then multiplied by a matrix with the third transformed feature obtained by the second convolution transformation of the third feature of the current video frame to obtain a two-frame attention feature. The fused third feature is calculated by summing the third feature, the difference attention feature, and the two-frame attention feature.

[0182] In this implementation, the first and second convolution transformations can be the same or different, and the activation operation can utilize a non-linear activation function (such as...). Figure 3 The activation functions (such as softmax) shown are used, but this application does not impose any restrictions on them.

[0183] In some embodiments, the fused third feature M of the T-th video frame is calculated. T It can be done using the following formula (2).

[0184]

[0185] in, It is a dual-frame attention feature. For differential attention features, λ1 and λ2 are the third feature of the T-th video frame.

[0186] In one possible implementation, such as Figure 4 As shown, during the training process of the system, in the first detection and segmentation module 21, the segmentation head 213 is further used to predict intermediate segmentation results based on the received second feature. In each of the intermediate detection and segmentation modules 22, the segmentation head 223 is further used to predict intermediate segmentation results based on the received second and third features. In the last detection and segmentation module 23, the segmentation head 233 is further used to predict intermediate stage segmentation reference features based on the received third feature and the second feature.

[0187] The outputs of each detection head and each segmentation head are used as the basis for calculating the system loss function during system training. This system loss function is the basis for updating the system parameters. Thus, during system training, the training process can be supervised using the outputs of various modules, including intermediate target detection results from the detection heads, intermediate segmentation results from the segmentation heads, intermediate-stage segmentation reference features, tracking results from the tracker, and image extraction features from each detector, ensuring optimal overall system performance.

[0188] Figure 6 A schematic diagram illustrating the operation of a multi-target tracking and segmentation system according to an embodiment of this application is shown. Figure 6 As shown, in a system comprising multiple detection and segmentation modules and one or more temporal fusion modules, a tracking-by-detection framework based on MOT is employed to fully utilize the powerful expressive capabilities of the detectors for temporal matching. The fusion of features and spatial information is achieved through multiple detection and segmentation modules, and the fusion of features and temporal information is achieved through the temporal fusion module, resulting in richer object representations and thus improving the prediction accuracy of MOTS.

[0189] Furthermore, to demonstrate that the system provided in this application can improve task performance, a backbone network pre-trained on ImageNet is used to evaluate PCAN (the optimal implementation of multi-object tracking and segmentation in related technologies). Figure 1 , Figure 3 The system shown was validated on the BDD100K dataset, and the validation results are shown in Table 1. Referring to Table 1, it can be seen that this application... Figure 1 , Figure 3 The provided system shows improved performance compared to PCAN, particularly in the mMOTA metric. Figure 1 , Figure 3 The system shown is 3.5 and 3.9 higher than PCAN respectively, which is obviously superior to the multi-target tracking and segmentation technology solutions in related technologies.

[0190] Table 1 System Validation Results

[0191] System Solution mMOTA mMOTP FN FP IDs MT ML IDF1 PCAN (baseline) 27.1 67.0 14870 13765 912 1744 868 45.7 Figure 1 The system shown 30.6 67.8 17249 10427 773 1566 1006 46.8 Figure 3 The system shown 31.0 68.9 18912 10035 921 1396 1120 45.1

[0192] This application also provides a multi-target tracking and segmentation method, applicable to... Figure 1 or Figure 3 The system shown includes a method comprising: an image feature extraction stage, a first detection and segmentation stage, one or more intermediate detection and segmentation stages, a final detection and segmentation stage, and a tracking result determination stage.

[0193] Image feature extraction stage: Feature extraction is performed on each video frame of the target video to obtain the image features of each video frame.

[0194] First detection and segmentation stage: The image features are detected to obtain the second feature and the image extraction feature; the intermediate stage segmentation reference feature is predicted based on the second feature, and the intermediate stage segmentation reference feature is processed and fused with the second feature to obtain the third feature; the image extraction feature and the third feature obtained in the current stage are used as the input for the first intermediate detection and segmentation stage.

[0195] Each intermediate detection and segmentation stage: Detect the received image extraction features to obtain the second feature and the image extraction features; predict the intermediate stage segmentation reference features based on the received third feature and the second feature, and process the intermediate stage segmentation reference features and fuse them with the second feature to obtain the third feature; use the image extraction features and the third feature obtained in the current stage as inputs for the next intermediate detection and segmentation stage or the final detection and segmentation stage.

[0196] Finally, in the segmentation detection stage: features are extracted from the received image and detected to obtain the first feature, the second feature, and the tracking embedding feature; target detection is performed based on the received first feature to obtain the target detection result; and the segmentation result is predicted based on the received third feature and the second feature.

[0197] Tracking result determination stage: Based on the tracking embedding features and the target detection results, the tracking result is determined.

[0198] In one possible implementation, predicting intermediate stage segmentation reference features based on the second feature may include:

[0199] Based on the second feature and the refined second feature obtained by refining the second feature, the intermediate stage segmentation reference feature is predicted.

[0200] In one possible implementation, predicting intermediate-stage segmentation reference features based on the received third feature and the second feature may include: predicting intermediate-stage segmentation reference features based on the received third feature and a refined second feature obtained by refining the second feature. And / or, predicting the segmentation result based on the received third feature and the second feature may include: predicting the segmentation result based on the received third feature and a refined second feature obtained by refining the second feature.

[0201] In one possible implementation, the first detection and segmentation stage may further include: detecting the received image features to obtain a first feature; detecting the first feature to obtain an intermediate target detection result; and using the intermediate target detection result obtained in the previous stage as the input to the next intermediate detection and segmentation stage. In each intermediate detection and segmentation stage, detecting the received image-extracted features to obtain a second feature and an image-extracted feature may include: detecting the received image-extracted features and the intermediate target detection result to obtain the first feature, the second feature, and the image-extracted feature; each intermediate detection and segmentation stage further includes: detecting the first feature of the current stage to obtain an intermediate target detection result; and using the intermediate target detection result obtained in the current stage as the input to the next intermediate detection and segmentation stage or the final detection and segmentation stage. In the final detection and segmentation stage, detecting the received image-extracted features to obtain the first feature, the second feature, and the tracking embedding feature may include: detecting the received image-extracted features and the intermediate target detection result to obtain the first feature, the second feature, and the tracking embedding feature.

[0202] In one possible implementation, processing the intermediate stage segmentation reference feature and fusing it with the second feature to obtain the third feature may include:

[0203] The intermediate stage segmentation reference feature is linearly transformed and then summed with the second feature to obtain the third feature.

[0204] In one possible implementation, the method may further include:

[0205] After obtaining the third feature of the current stage in the first detection and segmentation stage and each intermediate detection and segmentation stage, the corresponding temporal information is determined by inter-frame attention calculation and differential attention calculation based on the third feature obtained in the current stage and the third feature of the reference video frame corresponding to the current video frame, and then fused into the third feature to obtain the fused third feature. The reference video frame is the previous video frame of the current video frame.

[0206] The step of predicting intermediate stage segmentation reference features based on the received third feature and the second feature may include: predicting intermediate stage segmentation reference features based on the received fused third feature and the second feature. And / or, predicting the segmentation result based on the received third feature and the second feature may include: predicting the segmentation result based on the received fused third feature and the second feature.

[0207] In one possible implementation, based on the third feature from the previous segmentation head and the third feature of the reference video frame corresponding to the current video frame, the corresponding temporal information is determined through inter-frame attention calculation and differential attention calculation and fused into the third feature to obtain the fused third feature, which may include:

[0208] Perform a first convolution transformation on the third feature of the current video frame and the third feature of the reference video frame respectively to obtain the first transformed feature of the current video frame and the second transformed feature of the reference video frame.

[0209] The first transformed feature and the second transformed feature are processed to obtain the difference features of the current video frame relative to the reference frame;

[0210] The difference features and the first transformed features are multiplied by a matrix to obtain the difference similarity features of the current video frame relative to the reference frame;

[0211] The first transformed feature and the second transformed feature are multiplied by a matrix to obtain the two-frame similarity feature of the current video frame relative to the reference frame;

[0212] After performing activation operations on the differential similarity features, they are multiplied by matrix with the third transformed features obtained by performing a second convolution transformation on the third features of the current video frame to obtain differential attention features.

[0213] After performing activation operations on the dual-frame similarity features, they are multiplied by a matrix with the third transformed features obtained by performing a second convolution transformation on the third features of the current video frame to obtain dual-frame attention features.

[0214] The fused third feature is calculated by summing the third feature, the differential attention feature, and the dual-frame attention feature.

[0215] In one possible implementation, during the training process,

[0216] The first detection and segmentation stage may also include: predicting intermediate segmentation results based on the received second feature;

[0217] Each intermediate detection and segmentation stage also includes: predicting intermediate segmentation results based on the received second and third features;

[0218] The final detection and segmentation stage may further include: predicting intermediate stage segmentation reference features based on the received third feature and the second feature;

[0219] The output results of each stage are used as the basis for calculating the system loss function during the training process, and the system loss function is the basis for updating the parameters of the system.

[0220] The implementation methods and beneficial effects of each step in the multi-target tracking and segmentation method of this application can be found in the relevant descriptions of the corresponding modules and sub-modules in the multi-target tracking and segmentation system mentioned above. To avoid redundancy, they will not be repeated here.

[0221] Figure 7 A block diagram of a multi-target tracking and segmentation system according to an embodiment of this application is shown. Figure 7 As shown, the multi-target tracking and segmentation system provided in this application includes: a feature extraction module 10, and multiple detection and segmentation modules (such as...). Figure 7 As shown in 51, 52, and 53), tracker 30, and temporal fusion module 40. Each of the detection and segmentation modules includes a detector (such as...). Figure 7 As shown in diagrams 511, 521, and 531, the plurality of detection and segmentation modules are divided into a first detection and segmentation module 51, one or more intermediate detection and segmentation modules 52, and a last detection and segmentation module 53. The last detection and segmentation module 53 further includes a detection head 532 and a segmentation head 533. The temporal fusion module 40 is connected 533 between the detector 531 and the segmentation head of the last detection and segmentation module 53. The system may include one or more intermediate detection and segmentation modules 52. For simplicity... Figure 7 The other figures in this application only schematically show one intermediate detection and segmentation module 52. The number of intermediate detection and segmentation modules 52 can be set according to actual needs, and this application does not limit this.

[0222] The feature extraction module 10 is used to extract features from each video frame of the target video to obtain the image features of each video frame. An embodiment of the feature extraction module 10 can be found in the description of the feature extraction module 10 in the previous multi-target tracking and segmentation system; to avoid redundancy, it will not be repeated here.

[0223] In the first detection and segmentation module 51, the detector 511 is used to detect the received image features, extract image features, and send them to the connected next detector (i.e., Figure 7 (Detector 521 in the intermediate detection and segmentation module 52).

[0224] In each of the intermediate detection and segmentation modules 52, the detector 521 is used to detect the features extracted from the received image, obtain the image extracted features, and send them to the next connected detector (i.e., Figure 7 The detector 521 in the next intermediate detection and segmentation module 52 or the detector 531 in the last detection and segmentation module 53).

[0225] In the last detection and segmentation module 53, the detector 531 is used to extract features from the received image (from the detector 521 of the last detection and segmentation module 52) and detect them to obtain a first feature, a second feature and a tracking embedding feature. Then, the second feature is sent to the temporal fusion module 40, the first feature is sent to the detection head 532 and the tracking embedding feature is sent to the tracker 30. The detection head 532 is used to perform target detection based on the received first feature to obtain the target detection result.

[0226] The temporal fusion module 40, upon receiving the second feature (from detector 531 in the last detection and segmentation module 53), determines the corresponding temporal information based on the second feature and the second feature of the reference video frame corresponding to the current video frame through inter-frame attention calculation and differential attention calculation, and fuses it into the second feature to obtain the fused second feature. The fused second feature is then sent to the segmentation head 533 in the last detection and segmentation module 53, where the reference video frame is the previous video frame of the current video frame. Thus, due to the low complexity of inter-frame differential calculation and its ability to accurately represent the motion information of consecutive frames, the temporal fusion module 40, combined with a dual attention mechanism (i.e., inter-frame attention calculation and differential attention calculation), can mine continuous temporal features and obtain temporal feature changes, making it suitable for temporal feature modeling in MOTS scenarios. It obtains inter-frame temporal information and can effectively fuse this information into the second feature, allowing the segmentation head to predict more accurate and precise results based on the fused second feature.

[0227] The segmentation head 533 in the last detection and segmentation module 53 is used to predict the segmentation result based on the received fused second feature (from the temporal fusion module 40).

[0228] The tracker 30 is used to determine the tracking result based on the tracking embedding features and the target detection result.

[0229] In this embodiment, after determining the target detection result, segmentation result, and tracking result, the system can output all three as a prediction result. This prediction result can be sent to the object that needs further processing based on the prediction result. The information indicated by the target detection result, segmentation result, and tracking result can be found above and will not be repeated here to avoid redundancy. The multi-target tracking and segmentation system can be applied to scenarios requiring the tracking and detection of multiple targets, such as autonomous driving, driver assistance systems, and new retail. Figure 7 The system shown can also be applied to Figure 2 In the application scenarios shown.

[0230] In some embodiments, the first feature detected by the detector can characterize target-related information such as the target's position in the video frame, and may be a mask feature or other feature used for target bounding box detection. The second feature detected by the detector may be information such as a mask feature or other feature that can characterize the target segmentation in the video frame. The intermediate-stage segmentation reference feature obtained by the segmentation head may be a mask feature that can be used as a reference for target segmentation. The algorithms for bounding box detection by each detection head, tracking detection by each tracker, and tracking embedding feature detection by the segmentation head can be set according to actual needs, and this application does not impose any limitations on these settings.

[0231] The system can be like Figure 7 The system includes a temporal fusion module connected between the detector 531 and the segmentation head 533 of the last detection and segmentation module 53. This module fuses the second feature output by the detector 531 with temporal information and then sends the fused second feature to the segmentation head 533. Alternatively, the system can include multiple temporal fusion modules 40. Each module 40 can fuse the second feature of a specific detector with temporal information and then send the fused second feature to the corresponding segmentation head. This allows the segmentation head to predict the segmentation result based on the fused second feature with temporal information, improving overall system performance and reducing computational load.

[0232] Figure 8 A schematic diagram illustrating the fusion of timing information in a second feature according to an embodiment of this application is shown. In one possible implementation, such as... Figure 8 As shown, the fused second feature, obtained by determining the corresponding temporal information through inter-frame attention calculation and differential attention calculation based on the second feature and the second feature of the reference video frame corresponding to the current video frame, and fusing it into the second feature, may include:

[0233] The first convolution transformation is performed on the second feature of the current video frame and the second feature of the reference video frame respectively to obtain the first transformed feature of the current video frame and the second transformed feature of the reference video frame. The first transformed feature and the second transformed feature are processed to obtain the difference feature of the current video frame relative to the reference frame. The difference feature and the first transformed feature are multiplied by a matrix to obtain the difference similarity feature of the current video frame relative to the reference frame. The first transformed feature and the second transformed feature are multiplied by a matrix to obtain the two-frame similarity feature of the current video frame relative to the reference frame. The difference similarity feature is activated and then multiplied by a matrix with the third transformed feature obtained by the second convolution transformation of the second feature of the current video frame to obtain the difference attention feature. The two-frame similarity feature is activated and then multiplied by a matrix with the third transformed feature obtained by the second convolution transformation of the second feature of the current video frame to obtain the two-frame attention feature. The fused second feature is calculated by summing the second feature, the difference attention feature, and the two-frame attention feature.

[0234] In this implementation, the first and second convolution transformations can be the same or different, and the activation operation can utilize a non-linear activation function (such as...). Figure 8 The calculation of the second feature after fusion (using the softmax method shown) is performed, and this application does not impose any restrictions on this. The calculation process and formula used for the second feature after fusion are similar to those for the third feature after fusion described above. The calculation process and formula for the second feature after fusion can be set according to the calculation process and formula for the third feature after fusion described above. To avoid redundancy, they will not be elaborated here.

[0235] In one possible implementation, such as Figure 7 As shown, each of the intermediate detection and segmentation modules 52 and the first detection and segmentation module 51 is equipped with a detection head (e.g., Figure 7As shown in 522 and 512, each of the detection heads is connected to the detector of the next detection and segmentation module. For example, detection head 512 is connected to detector 521, and detection head 522 is connected to detector 531 (not shown in the figure). In the first detection and segmentation module 51: detector 511 is further used to detect the received image features to obtain a first feature, and detection head 512 is used to detect the first feature to obtain an intermediate target detection result, and send the intermediate target detection result to the detector in the next detection and segmentation module (i.e., the detector in the first intermediate detection and segmentation module). In each of the intermediate detection and segmentation modules 52: detector 521 is used to detect the received image extracted features to obtain new image extracted features, which may include: detector 521 is used to detect the received image extracted features and the intermediate target detection result to obtain a first feature and the image extracted feature, and also sends the first feature to detection head 512. Detection head 512 is used to detect the first feature to obtain an intermediate target detection result, and sends the intermediate target detection result to the detector in the next detection and segmentation module (i.e., the next intermediate detection and segmentation module or the last detection and segmentation module). In the last detection and segmentation module 53: the detector 531 is used to detect features extracted from the received image to obtain a first feature, a second feature, and a tracking embedding feature. This includes: the detector is used to detect the received image extracted features and intermediate target detection results to obtain the first feature, the second feature, and the tracking embedding feature. In this way, the detectors in each intermediate detection and segmentation module and the last detection and segmentation module perform detection and feature extraction based on image extracted features and intermediate target detection results, which can improve the accuracy of the detector's detection results, and thus improve the accuracy and precision of the prediction results of the entire system. The intermediate target detection results can refer to the results obtained by each detection head after performing target detection based on the received first feature.

[0236] In one possible implementation, each of the intermediate detection and segmentation modules 52 and the first detection and segmentation module 51 is provided with a segmentation head (e.g., ...). Figure 7As shown in 523 and 513), during the training process of the system, the detector 511 in the first detection and segmentation module 51 is further used to detect the image features to obtain a second feature, and send the second feature to the segmentation head 513 of the first detection and segmentation module 51. The segmentation head 513 is used to predict the intermediate segmentation result based on the second feature. Similarly, the detector 521 in each intermediate detection and segmentation module 52 is further used to detect the image extracted features to obtain a second feature, and send the second feature to the segmentation head 523 of the corresponding intermediate detection and segmentation module 52. The segmentation head 523 is used to predict the intermediate segmentation result based on the received second feature. The output results of each detection head and each segmentation head are used as the basis for calculating the system loss function during the system training process. The system loss function is the basis for updating the system parameters. In this way, during the system training process, the intermediate target detection results or target detection results output by the detection heads, the intermediate segmentation results or segmentation results output by the segmentation heads, the tracking results output by the tracker, and the image extracted features output by each detector, etc., can be used to supervise the system training process and ensure optimal overall system performance.

[0237] In one possible way, such as Figure 7 As shown, the last detection and segmentation module 53 further includes a processing submodule 534. The temporal fusion module 40 is further configured to send the fused second feature to the processing submodule 534. The processing submodule 534 is configured to refine the received fused second feature and then send the refined second feature to the connected segmentation head 533. The segmentation head 533 in the last detection and segmentation module 53 is configured to predict the segmentation result based on the received fused second feature, which may include: the segmentation head 533 in the last detection and segmentation module 53 is configured to predict the segmentation result based on the received fused second feature and the refined second feature. Thus, the segmentation head 533's prediction based on the fused second feature and the refined second feature can improve the accuracy and precision of the segmentation result. The refined second feature can be a feature with more channels obtained by processing the fused second feature.

[0238] In some embodiments, such as Figure 7 As shown, the first detection and segmentation module 51 and / or the intermediate detection and segmentation module 52 may further include a processing submodule (such as...). Figure 7(See diagrams 514 and 524). In the first detection and segmentation module 51 or the intermediate detection and segmentation module 52, which includes a processing submodule: the detector is used to send the second feature to the processing submodule and the segmentation head, respectively. The processing submodule is used to refine the received second feature and then send the refined second feature to the segmentation head of the detection and segmentation module. This allows the segmentation head to make predictions based on the second feature and the refined second feature during training, obtaining intermediate segmentation results. This can improve the accuracy and precision of system training and the overall accuracy and precision of the system's output prediction results.

[0239] In this embodiment, Figure 7 The working process of the system shown can be combined with Figure 6 Understanding, such as Figure 6 As shown, in a system comprising multiple detection and segmentation modules and a temporal fusion module, a tracking-by-detection framework based on MOT is employed to fully utilize the powerful expressive capabilities of the detector for temporal matching. The temporal fusion module integrates features and temporal information, resulting in richer object representations and thus improving the prediction accuracy of MOTS.

[0240] This application also provides a multi-target tracking and segmentation method, applicable to... Figure 7 The system shown includes the following methods: an image feature extraction stage, a first detection and segmentation stage, one or more intermediate detection and segmentation stages, a final detection and segmentation stage, a temporal fusion stage, and a tracking result determination stage.

[0241] Image feature extraction stage: Feature extraction is performed on each video frame of the target video to obtain the image features of each video frame.

[0242] First detection and segmentation stage: The received image features are detected to obtain image extraction features, and the image extraction features obtained in the current stage are used as the input for the first intermediate detection and segmentation stage.

[0243] Each intermediate detection and segmentation stage: Detect the features extracted from the received image to obtain new image features, and use the image features obtained in the current stage as input for the next intermediate detection and segmentation stage or the final detection and segmentation stage.

[0244] Finally, in the segmentation and detection stage: features are extracted from the received image and detected to obtain the first feature, the second feature, and the tracking embedding feature; target detection is performed based on the received first feature to obtain the target detection result.

[0245] Temporal fusion stage: Based on the second feature of the current video frame extracted in the final detection and segmentation stage and the second feature of the reference video frame corresponding to the current video frame, the corresponding temporal information is determined by inter-frame attention calculation and differential attention calculation and fused into the second feature to obtain the fused second feature, and the fused second feature is used as the input of the final detection and segmentation stage.

[0246] The final detection and segmentation stage also includes: predicting the segmentation result based on the received fused second feature.

[0247] Tracking result determination stage: Based on the tracking embedding features and the target detection results, the tracking result is determined.

[0248] In one possible implementation, the fused second feature, obtained by determining the corresponding temporal information through inter-frame attention calculation and differential attention calculation based on the second feature of the current video frame extracted in the final detection and segmentation stage and the second feature of the reference video frame corresponding to the current video frame, and fusing it into the second feature, may include:

[0249] The second features of the current video frame and the second features of the reference video frame extracted in the final detection and segmentation stage are respectively subjected to the first convolution transformation to obtain the first transformed features of the current video frame and the second transformed features of the reference video frame.

[0250] The first transformed feature and the second transformed feature are processed to obtain the difference features of the current video frame relative to the reference frame;

[0251] The difference features and the first transformed features are multiplied by a matrix to obtain the difference similarity features of the current video frame relative to the reference frame;

[0252] The first transformed feature and the second transformed feature are multiplied by a matrix to obtain the two-frame similarity feature of the current video frame relative to the reference frame;

[0253] After performing activation operations on the differential similarity features, they are multiplied by matrix with the third transformed features obtained by performing a second convolution transformation on the second features of the current video frame extracted in the final detection and segmentation stage to obtain differential attention features.

[0254] After performing activation operations on the dual-frame similarity features, they are multiplied by a matrix with the third transformed features obtained by performing a second convolution transformation on the second features of the current video frame to obtain dual-frame attention features.

[0255] The fused second feature is calculated by summing the second feature extracted in the final detection and segmentation stage, the differential attention feature, and the dual-frame attention feature.

[0256] In one possible implementation, the first detection and segmentation stage may further include: detecting the received image features to obtain a first feature, detecting the first feature to obtain an intermediate target detection result, and using the intermediate target detection result obtained in the current stage as the input to the next first intermediate detection and segmentation stage;

[0257] In each intermediate detection and segmentation stage, the received image extracted features are detected to obtain new image extracted features. This may include: detecting the received image extracted features and the target detection intermediate results to obtain a first feature and the image extracted features; each intermediate detection and segmentation stage may also include: detecting the first feature to obtain a target detection intermediate result, and using the target detection intermediate result obtained in the current stage as the input of the next intermediate detection and segmentation stage or the final detection and segmentation stage.

[0258] In the final detection and segmentation stage, the features extracted from the received image are detected to obtain the first feature, the second feature, and the tracking embedding feature. This may include: detecting the extracted features from the received image and the intermediate results of target detection to obtain the first feature, the second feature, and the tracking embedding feature.

[0259] In one possible implementation, during the training process,

[0260] The first detection and segmentation stage may further include: detecting the image features to obtain a second feature, and predicting an intermediate segmentation result based on the second feature;

[0261] Each intermediate detection and segmentation stage may further include: detecting the features extracted from the image to obtain a second feature, and predicting the intermediate segmentation result based on the second feature;

[0262] The output results of each stage are used as the basis for calculating the system loss function during the training process, and the system loss function is the basis for updating the parameters of the system.

[0263] In one possible implementation, predicting the segmentation result based on the received fused second feature may include:

[0264] The received fused second feature is refined to obtain a refined second feature, and the segmentation result is predicted based on the fused second feature and the refined second feature.

[0265] The implementation methods and beneficial effects of each step in the multi-target tracking and segmentation method of this application can be found in the relevant descriptions of the corresponding modules and sub-modules in the above-mentioned multi-target tracking and segmentation system. To avoid redundancy, they will not be elaborated here.

[0266] Embodiments of this application provide a multi-target tracking and segmentation apparatus, including: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing the instructions.

[0267] Embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0268] Embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0269] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital video disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing.

[0270] The computer-readable program instructions or code described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0271] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this application.

[0272] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0273] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0274] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0275] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0276] It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented using hardware (such as circuits or ASICs (Application Specific Integrated Circuits)) that performs the corresponding function or action, or using a combination of hardware and software, such as firmware.

[0277] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0278] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A multi-target tracking and segmentation system, characterized by, include: The system includes a feature extraction module, multiple detection and segmentation modules, and a tracker. Each detection and segmentation module includes a segmentation head and a detector connected to the segmentation head. Each detector is sequentially connected to the detector of the next detection and segmentation module, and each segmentation head is connected to the segmentation head of the next detection and segmentation module. The multiple detection and segmentation modules are divided into a first detection and segmentation module, one or more intermediate detection and segmentation modules, and a last detection and segmentation module. The last detection and segmentation module also includes a detection head connected to the detector. The feature extraction module is used to extract features from each video frame of the target video and obtain the image features of each video frame. In the first detection and segmentation module: the detector is used to detect the image features to obtain the second feature and the image extraction feature; The segmentation head is used to predict intermediate stage segmentation reference features based on the received second feature, and then process the intermediate stage segmentation reference features and fuse them with the second feature to obtain a third feature; In each of the intermediate detection and segmentation modules: the detector is used to detect the features extracted from the received image to obtain the second feature and the image extracted feature; The segmentation head is used to predict intermediate stage segmentation reference features based on the received third feature and the second feature, and then process the intermediate stage segmentation reference features and fuse them with the second feature to obtain the third feature; In the last detection and segmentation module: the detector is used to extract features from the received image and detect them to obtain a first feature, a second feature, and a tracking embedding feature; the detection head is used to perform target detection based on the received first feature to obtain the target detection result; and the segmentation head is used to predict the segmentation result based on the received third feature and the second feature. The tracker is used to determine the tracking result based on the tracking embedding features and the target detection result.

2. The system according to claim 1, characterized in that, The first detection and segmentation module further includes a processing submodule, in which: The detector is used to send the second feature to the processing submodule and the segmentation head, respectively; The processing submodule is used to refine the received second feature and then send the refined second feature to the segmentation head. The segmentation head is used to predict intermediate stage segmentation reference features based on the received second feature, including: the segmentation head is used to predict intermediate stage segmentation reference features based on the received second feature and the refined second feature.

3. The system according to claim 1 or 2, characterized in that, One or more of the intermediate detection and segmentation modules and the last detection and segmentation module further include processing sub-modules. Each processing sub-module is connected between the detector and the segmentation head of the intermediate detection and segmentation module or the last detection and segmentation module. Therefore, in the intermediate detection and segmentation module or the last detection and segmentation module equipped with a processing sub-module: The detector is used to send the second feature to the processing submodule; The processing submodule is used to refine the received second feature and then send the refined second feature to the connected segmentation head. The segmentation head is used to predict intermediate stage segmentation reference features based on the received third feature and the second feature, including: the segmentation head predicting intermediate stage segmentation reference features based on the received third feature and the refined second feature; and / or The segmentation head is used to predict the segmentation result based on the received third feature and the second feature, including: the segmentation head is used to predict the segmentation result based on the received third feature and the refined second feature.

4. The system according to claim 1, characterized in that, Each of the intermediate detection and segmentation modules and the first detection and segmentation module is equipped with a detection head, and each detection head is connected to the detector of the next detection and segmentation module. In the first detection and segmentation module: the detector is further used to detect the received image features to obtain a first feature, and the detection head is used to detect the first feature to obtain an intermediate target detection result, and send the intermediate target detection result to the connected next detector; In each of the intermediate detection and segmentation modules: the detector is used to detect the received image extracted features to obtain a second feature and an image extracted feature, including: the detector is used to detect the received image extracted features and the target detection intermediate result to obtain a first feature, a second feature and an image extracted feature; the detection head is used to detect the first feature to obtain a target detection intermediate result, and send the target detection intermediate result to the connected next detector; In the last detection and segmentation module: the detector is used to detect the features extracted from the received image to obtain the first feature, the second feature and the tracking embedding feature, including: the detector is used to detect the features extracted from the received image and the intermediate results of target detection to obtain the first feature, the second feature and the tracking embedding feature.

5. The system according to claim 1, characterized in that, The intermediate stage segmentation reference features are processed and then fused with the second feature to obtain the third feature, including: The intermediate stage segmentation reference feature is linearly transformed and then summed with the second feature to obtain the third feature.

6. The system according to claim 1, characterized in that, Also includes: At least one time-domain fusion module, Each of the temporal fusion modules is connected between two adjacent segmentation heads. It is used to determine the corresponding temporal information based on the third feature from the previous segmentation head and the third feature of the reference video frame corresponding to the current video frame through inter-frame attention calculation and differential attention calculation, and fuse it into the third feature to obtain the fused third feature. The reference video frame is the video frame before the current video frame. The segmentation head is used to predict intermediate stage segmentation reference features based on the received third feature and the second feature, including: the segmentation head predicts intermediate stage segmentation reference features based on the received fused third feature and the second feature; and / or The segmentation head is used to predict the segmentation result based on the received third feature and the second feature, including: the segmentation head is used to predict the segmentation result based on the received fused third feature and the second feature.

7. The system according to claim 1, characterized in that, Based on the third feature from the previous segmentation head and the third feature of the reference video frame corresponding to the current video frame, the corresponding temporal information is determined through inter-frame attention calculation and differential attention calculation and fused into the third feature to obtain the fused third feature, including: Perform a first convolution transformation on the third feature of the current video frame and the third feature of the reference video frame respectively to obtain the first transformed feature of the current video frame and the second transformed feature of the reference video frame. The first transformed feature and the second transformed feature are processed to obtain the difference features of the current video frame relative to the reference video frame; The difference features and the first transformed features are multiplied by a matrix to obtain the difference similarity features between the current video frame and the reference video frame. The first transformed feature and the second transformed feature are multiplied by a matrix to obtain the two-frame similarity feature of the current video frame relative to the reference video frame; After performing activation operations on the differential similarity features, they are multiplied by matrix with the third transformed features obtained by performing a second convolution transformation on the third features of the current video frame to obtain differential attention features. After performing activation operations on the dual-frame similarity features, they are multiplied by a matrix with the third transformed features obtained by performing a second convolution transformation on the third features of the current video frame to obtain dual-frame attention features. The fused third feature is calculated by summing the third feature, the differential attention feature, and the dual-frame attention feature.

8. The system according to claim 1, characterized in that, During the training process of the system, In the first detection and segmentation module, the segmentation head is also used to predict intermediate segmentation results based on the received second feature; In each of the intermediate detection and segmentation modules, the segmentation head is further used to predict intermediate segmentation results based on the received second and third features; In the last detection and segmentation module, the segmentation head is also used to predict intermediate stage segmentation reference features based on the received third feature and the second feature. The output results of each detection head and each segmentation head are used as the basis for calculating the system loss function during the system training process. The system loss function is the basis for updating the parameters of the system.

9. A multi-target tracking and segmentation system, characterized in that, include: The system comprises a feature extraction module, multiple detection and segmentation modules, a tracker, and a temporal fusion module. Each detection and segmentation module includes a detector. The multiple detection and segmentation modules are divided into a first detection and segmentation module, one or more intermediate detection and segmentation modules, and a last detection and segmentation module. The last detection and segmentation module further includes a detection head and a segmentation head. The temporal fusion module is connected between the detector and the segmentation head of the last detection and segmentation module. The feature extraction module is used to extract features from each video frame of the target video and obtain the image features of each video frame. In the first detection and segmentation module, the detector is used to detect the received image features, extract image features, and send them to the next connected detector; In each of the intermediate detection and segmentation modules, the detector is used to detect the features extracted from the received image, obtain the image extracted features, and send them to the next connected detector; In the last detection and segmentation module, the detector is used to extract features from the received image and detect them to obtain the first feature, the second feature, and the tracking embedding feature; The detection head is used to perform target detection based on the received first feature to obtain the target detection result; The temporal fusion module is used to, after receiving the second feature, determine the corresponding temporal information based on the second feature and the second feature of the reference video frame corresponding to the current video frame through inter-frame attention calculation and differential attention calculation, and fuse it into the second feature to obtain the fused second feature, and send the fused second feature to the segmentation head in the last detection and segmentation module, wherein the reference video frame is the previous video frame of the current video frame; The segmentation head in the last detection and segmentation module is used to predict the segmentation result based on the received fused second feature; The tracker is used to determine the tracking result based on the tracking embedding features and the target detection result.

10. The system according to claim 9, characterized in that, Based on the second feature and the second feature of the reference video frame corresponding to the current video frame, the corresponding temporal information is determined through inter-frame attention calculation and differential attention calculation, and then fused into the second feature to obtain the fused second feature, including: Perform a first convolution transformation on the second feature of the current video frame and the second feature of the reference video frame respectively to obtain the first transformed feature of the current video frame and the second transformed feature of the reference video frame; The first transformed feature and the second transformed feature are processed to obtain the difference features of the current video frame relative to the reference video frame; The difference features and the first transformed features are multiplied by a matrix to obtain the difference similarity features between the current video frame and the reference video frame. The first transformed feature and the second transformed feature are multiplied by a matrix to obtain the two-frame similarity feature of the current video frame relative to the reference video frame; After performing activation operations on the differential similarity features, they are multiplied by a matrix with the third transformed features obtained by performing a second convolution transformation on the second features of the current video frame to obtain differential attention features. After performing activation operations on the dual-frame similarity features, they are multiplied by a matrix with the third transformed features obtained by performing a second convolution transformation on the second features of the current video frame to obtain dual-frame attention features. The fused second feature is calculated by summing the second feature, the differential attention feature, and the dual-frame attention feature.

11. The system according to claim 9 or 10, characterized in that, Each of the intermediate detection and segmentation modules and the first detection and segmentation module is equipped with a detection head, and each detection head is connected to the detector of the next detection and segmentation module. In the first detection and segmentation module: the detector is further used to detect the received image features to obtain a first feature, the detection head is used to detect the first feature to obtain an intermediate target detection result, and send the intermediate target detection result to the detector in the next detection and segmentation module; In each of the intermediate detection and segmentation modules: the detector is used to detect the received image extracted features to obtain new image extracted features, including: the detector is used to detect the received image extracted features and the target detection intermediate results to obtain a first feature and the image extracted features; the detection head is used to detect the first feature to obtain the target detection intermediate results, and send the target detection intermediate results to the detector in the next detection and segmentation module; In the last detection and segmentation module: the detector is used to detect the features extracted from the received image to obtain the first feature, the second feature and the tracking embedding feature, including: the detector is used to detect the features extracted from the received image and the intermediate results of target detection to obtain the first feature, the second feature and the tracking embedding feature.

12. The system according to claim 11, characterized in that, Each of the intermediate detection and segmentation modules and the first detection and segmentation module is equipped with a segmentation head. During the training process of the system, In the first detection and segmentation module: the detector is further used to detect the image features to obtain a second feature, and send the second feature to the segmentation head of the first detection and segmentation module, the segmentation head is used to predict the intermediate segmentation result based on the second feature; In each of the intermediate detection and segmentation modules: the detector is further configured to detect the features extracted from the image to obtain a second feature, and send the second feature to the segmentation head of the intermediate detection and segmentation module; the segmentation head is configured to predict the intermediate segmentation result based on the received second feature; The output results of each detection head and each segmentation head are used as the basis for calculating the system loss function during the system training process. The system loss function is the basis for updating the parameters of the system.

13. The system according to claim 9, characterized in that, The last detection and segmentation module also includes a processing submodule. The time-domain fusion module is also used to send the fused second feature to the processing submodule; The processing submodule is used to refine the received fused second feature and then send the refined second feature to the connected segmentation head. The segmentation head in the last detection and segmentation module is used to predict the segmentation result based on the received fused second feature, including: the segmentation head in the last detection and segmentation module is used to predict the segmentation result based on the received fused second feature and the refined second feature.

14. A multi-target tracking and segmentation method, characterized in that, The method includes: an image feature extraction stage, a first detection and segmentation stage, one or more intermediate detection and segmentation stages, a final detection and segmentation stage, and a tracking result determination stage. Image feature extraction stage: Feature extraction is performed on each video frame of the target video to obtain the image features of each video frame; First detection and segmentation stage: Detect the image features to obtain second features and image extraction features; predict intermediate stage segmentation reference features based on the second features, and process the intermediate stage segmentation reference features and fuse them with the second features to obtain third features; use the image extraction features and third features obtained in the current stage as inputs for the first intermediate detection and segmentation stage. Each intermediate detection and segmentation stage: Detect the received image features to obtain the second feature and the image extraction feature; predict the intermediate stage segmentation reference feature based on the received third feature and the second feature, and process the intermediate stage segmentation reference feature and fuse it with the second feature to obtain the third feature; use the image extraction feature and the third feature obtained in the current stage as the input for the next intermediate detection and segmentation stage or the final detection and segmentation stage; The final detection and segmentation stage involves: extracting features from the received image to obtain the first feature, the second feature, and the tracking embedding feature; performing target detection based on the received first feature to obtain the target detection result; and predicting the segmentation result based on the received third feature and the second feature. Tracking result determination stage: Based on the tracking embedding features and the target detection results, the tracking result is determined.

15. The method according to claim 14, characterized in that, Based on the second feature, intermediate stage segmentation reference features are predicted, including: Based on the second feature and the refined second feature obtained by refining the second feature, the intermediate stage segmentation reference feature is predicted.

16. The method according to claim 14 or 15, characterized in that, Predicting intermediate stage segmentation reference features based on the received third feature and the second feature includes: predicting intermediate stage segmentation reference features based on the received third feature and a refined second feature obtained by refining the second feature; and / or The segmentation result is predicted based on the received third feature and the second feature, including: predicting the segmentation result based on the received third feature and the refined second feature obtained by refining the second feature.

17. The method according to claim 14, characterized in that, The first detection and segmentation stage also includes: detecting the received image features to obtain a first feature, detecting the first feature to obtain an intermediate target detection result, and using the intermediate target detection result obtained in the previous stage as the input for the next first intermediate detection and segmentation stage; In each intermediate detection and segmentation stage, the received image features are detected to obtain the second feature and the image extracted features, including: detecting the received image extracted features and the intermediate target detection results to obtain the first feature, the second feature and the image extracted features; each intermediate detection and segmentation stage also includes: detecting the first feature of the current stage to obtain the intermediate target detection result, and using the intermediate target detection result obtained in the current stage as the input of the next intermediate detection and segmentation stage or the final detection and segmentation stage; In the final detection and segmentation stage, the features extracted from the received image are detected to obtain the first feature, the second feature, and the tracking embedding feature. This includes detecting the extracted features from the received image and the intermediate results of target detection to obtain the first feature, the second feature, and the tracking embedding feature.

18. The method according to claim 14, characterized in that, The intermediate stage segmentation reference features are processed and then fused with the second feature to obtain the third feature, including: The intermediate stage segmentation reference feature is linearly transformed and then summed with the second feature to obtain the third feature.

19. The method according to claim 14, characterized in that, The method further includes: After obtaining the third feature of the current stage in the first detection and segmentation stage and each of the intermediate detection and segmentation stages, the corresponding temporal information is determined by inter-frame attention calculation and differential attention calculation based on the third feature obtained in the current stage and the third feature of the reference video frame corresponding to the current video frame, and then fused into the third feature to obtain the fused third feature. The reference video frame is the previous video frame of the current video frame. The step of predicting intermediate stage segmentation reference features based on the received third feature and the second feature includes: predicting intermediate stage segmentation reference features based on the received fused third feature and the second feature; and / or The segmentation result is predicted based on the received third feature and the second feature, including: predicting the segmentation result based on the received fused third feature and the second feature.

20. The method according to claim 14, characterized in that, Based on the third feature from the previous segmentation head and the third feature of the reference video frame corresponding to the current video frame, the corresponding temporal information is determined through inter-frame attention calculation and differential attention calculation and fused into the third feature to obtain the fused third feature, including: Perform a first convolution transformation on the third feature of the current video frame and the third feature of the reference video frame respectively to obtain the first transformed feature of the current video frame and the second transformed feature of the reference video frame. The first transformed feature and the second transformed feature are processed to obtain the difference features of the current video frame relative to the reference video frame; The difference features and the first transformed features are multiplied by a matrix to obtain the difference similarity features between the current video frame and the reference video frame. The first transformed feature and the second transformed feature are multiplied by a matrix to obtain the two-frame similarity feature of the current video frame relative to the reference video frame; After performing activation operations on the differential similarity features, they are multiplied by matrix with the third transformed features obtained by performing a second convolution transformation on the third features of the current video frame to obtain differential attention features. After performing activation operations on the dual-frame similarity features, they are multiplied by a matrix with the third transformed features obtained by performing a second convolution transformation on the third features of the current video frame to obtain dual-frame attention features. The fused third feature is calculated by summing the third feature, the differential attention feature, and the dual-frame attention feature.

21. The method according to claim 14, characterized in that, During the training process, The first detection and segmentation stage also includes: predicting intermediate segmentation results based on the received second feature; Each intermediate detection and segmentation stage also includes: predicting intermediate segmentation results based on the received second and third features; The final detection and segmentation stage also includes: predicting intermediate stage segmentation reference features based on the received third feature and the second feature; The output results of each stage are used as the basis for calculating the system loss function during the training process, and the system loss function is the basis for updating the parameters of the system.

22. A multi-target tracking and segmentation method, characterized in that, The method includes: an image feature extraction stage, a first detection and segmentation stage, one or more intermediate detection and segmentation stages, a final detection and segmentation stage, a temporal fusion stage, and a tracking result determination stage. Image feature extraction stage: Feature extraction is performed on each video frame of the target video to obtain the image features of each video frame; First detection and segmentation stage: The received image features are detected to obtain image extraction features, and the image extraction features obtained in the current stage are used as the input for the first intermediate detection and segmentation stage. Each intermediate detection and segmentation stage: Detect the features extracted from the received image to obtain new image features, and use the image features obtained in the current stage as input for the next intermediate detection and segmentation stage or the final detection and segmentation stage; Finally, in the segmentation and detection stage: features are extracted from the received image and detected to obtain the first feature, the second feature, and the tracking embedding feature; target detection is performed based on the received first feature to obtain the target detection result. Temporal fusion stage: Based on the second feature of the current video frame extracted in the final detection and segmentation stage and the second feature of the reference video frame corresponding to the current video frame, the corresponding temporal information is determined by inter-frame attention calculation and differential attention calculation and fused into the second feature to obtain the fused second feature, and the fused second feature is used as the input of the final detection and segmentation stage; The final detection and segmentation stage also includes: predicting the segmentation result based on the received fused second feature; Tracking result determination stage: Based on the tracking embedding features and the target detection results, the tracking result is determined.

23. The method according to claim 22, characterized in that, Based on the second feature of the current video frame extracted in the final detection and segmentation stage and the second feature of the reference video frame corresponding to the current video frame, the corresponding temporal information is determined through inter-frame attention calculation and differential attention calculation and fused into the second feature to obtain the fused second feature, including: The second features of the current video frame and the second features of the reference video frame extracted in the final detection and segmentation stage are respectively subjected to the first convolution transformation to obtain the first transformed features of the current video frame and the second transformed features of the reference video frame. The first transformed feature and the second transformed feature are processed to obtain the difference features of the current video frame relative to the reference video frame; The difference features and the first transformed features are multiplied by a matrix to obtain the difference similarity features between the current video frame and the reference video frame. The first transformed feature and the second transformed feature are multiplied by a matrix to obtain the two-frame similarity feature of the current video frame relative to the reference video frame; After performing activation operations on the differential similarity features, they are multiplied by matrix with the third transformed features obtained by performing a second convolution transformation on the second features of the current video frame extracted in the final detection and segmentation stage to obtain differential attention features. After performing activation operations on the dual-frame similarity features, they are multiplied by a matrix with the third transformed features obtained by performing a second convolution transformation on the second features of the current video frame to obtain dual-frame attention features. The fused second feature is calculated by summing the second feature extracted in the final detection and segmentation stage, the differential attention feature, and the dual-frame attention feature.

24. The method according to claim 22 or 23, characterized in that, The first detection and segmentation stage also includes: detecting the received image features to obtain a first feature, detecting the first feature to obtain an intermediate target detection result, and using the intermediate target detection result obtained in the current stage as the input for the next first intermediate detection and segmentation stage; In each intermediate detection and segmentation stage, the received image extraction features are detected to obtain new image extraction features, including: detecting the received image extraction features and the target detection intermediate results to obtain a first feature and the image extraction features; each intermediate detection and segmentation stage also includes: detecting the first feature to obtain a target detection intermediate result, and using the target detection intermediate result obtained in the current stage as the input of the next intermediate detection and segmentation stage or the final detection and segmentation stage; In the final detection and segmentation stage, the features extracted from the received image are detected to obtain the first feature, the second feature, and the tracking embedding feature. This includes detecting the extracted features from the received image and the intermediate results of target detection to obtain the first feature, the second feature, and the tracking embedding feature.

25. The method according to claim 24, characterized in that, During the training process, The first detection and segmentation stage also includes: detecting the image features to obtain second features, and predicting intermediate segmentation results based on the second features; Each intermediate detection and segmentation stage further includes: detecting the features extracted from the image to obtain a second feature, and predicting the intermediate segmentation result based on the second feature; The output results of each stage are used as the basis for calculating the system loss function during the training process, and the system loss function is the basis for updating the parameters of the system.

26. The method according to claim 22, characterized in that, The segmentation result is predicted based on the received fused second feature, including: The received fused second feature is refined to obtain a refined second feature, and the segmentation result is predicted based on the fused second feature and the refined second feature.

27. A target tracking and segmentation device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 14-21, or the method of any one of claims 22-26, when executing the instructions.

28. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 14-21, or the method described in any one of claims 22-26.

29. A computer program product comprising computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, characterized in that, When the computer-readable code is executed in an electronic device, the processor in the electronic device performs the method according to any one of claims 14-21 or the method according to any one of claims 22-26.

Images