A video instance segmentation method based on cross-frame instance association

Abstract

This invention discloses a video instance segmentation method based on cross-frame instance association. The method inputs a sequence of video frames to be segmented into a multi-scale feature extractor to extract feature maps at different scales. A transformer encoder extracts spatiotemporal features, which are then fused using a pixel decoder. Finally, the fused spatiotemporal features are obtained through the transformer decoder, resulting in a final embedding vector. This embedding vector is then multiplied by the high-resolution spatiotemporal features to obtain the instance segmentation result. This invention learns the spatiotemporal correlation of dynamic instances and constructs more stable cross-frame instance associations through a multi-scale method oriented towards spatiotemporal features. This establishes reliable cross-frame instance associations, improving the accuracy of video instance segmentation tasks and achieving state-of-the-art performance on two popular datasets compared to recent methods.

Description

Technical Field

[0001] This application belongs to the field of video instance segmentation technology, and in particular relates to a video instance segmentation method based on cross-frame instance association. Background Technology

[0002] Video instance segmentation aims to simultaneously detect, segment, and track target instances in video, which is helpful for many downstream tasks, including autonomous driving, video surveillance, and video understanding. Compared to image instance segmentation, video instance segmentation is more challenging because factors such as appearance deformation, rapid motion, and occlusion make accurate segmentation and tracking of target instances in video more difficult.

[0003] With the introduction of DETR and deformable DETR frameworks, Transformer-based end-to-end video instance segmentation methods have recently become mainstream. Following the video input, video output paradigm, VisTR was the first to apply Transformer to solve the video instance segmentation problem, using instance queries to obtain instance sequences from the video. However, this method learns an embedding for each instance in each frame, making it difficult to handle variable-length or long-duration video sequences. To reduce the explosive computational cost of VisTR and to build cross-frame instance associations, subsequent research has utilized target queries and proposed novel variants: constructing memory tokens with contextual temporal relevance and query separation mechanisms for building cross-frame instance associations. These methods essentially focus on single-frame features and detect instances, then perform cross-frame instance matching; however, this deliberate distinction between images and videos irreversibly ignores the rich spatiotemporal contextual information present in the video.

[0004] Furthermore, existing methods primarily focus on network improvement, but lack attention to the datasets required for training and testing. Research has revealed that commonly used datasets at present are prone to overfitting during training due to insufficient training data. Summary of the Invention

[0005] The purpose of this application is to provide a video instance segmentation method based on cross-frame instance association to overcome the problems mentioned in the background art. In this application, this method is also referred to as IAST.

[0006] To achieve the above objectives, the technical solution of this application is as follows:

[0007] A video instance segmentation method based on cross-frame instance association includes:

[0008] A video instance segmentation network is constructed and trained, the video instance segmentation network including a multi-scale feature extractor, a transformer encoder, a pixel decoder and a transformer decoder;

[0009] The video frame sequence to be segmented is input into a multi-scale feature extractor to extract feature maps at different scales, which are designated as feature maps C2, C3, C4 and C5 according to their scale size.

[0010] The extracted feature maps C3, C4, and C5 are input into the transformer encoder to extract spatiotemporal features.

[0011] Combine feature map C2 and spatiotemporal features Input to the pixel decoder and spatiotemporal features Separate features into features corresponding to scales C3, C4, and C5 of the feature map. and Then, progressive upsampling and cross-fusion are performed to obtain the fused spatiotemporal features. and the spatiotemporal characteristics after fusion and the spatiotemporal characteristics after fusion

[0012] Features and The input is fed into the transformer decoder to obtain the final embedding vector;

[0013] Integrating embedding vectors with spatiotemporal features Perform a dot product operation to obtain the instance segmentation result.

[0014] Furthermore, the training video instance segmentation network includes preprocessing the acquired video frame sequence to generate training sample data, including:

[0015] Take two frame sequences with the same number of frames from the collected video frame sequence dataset, use one of them as the target set and the other as the source set;

[0016] Establish a one-to-one correspondence between image frames in the target set and the source set according to the time sequence;

[0017] Then, the instances in the source set of images are copied and pasted onto the corresponding image frames in the target set to generate a new frame sequence, which is then placed into the video frame sequence dataset.

[0018] Furthermore, the extracted feature maps C3, C4, and C5 are input into the transformer encoder to extract spatiotemporal features. include:

[0019] The feature maps C3, C4, and C5 are positionally encoded, and then tensor flattening is performed on each of them before being input into the deformable attention module to generate the basic feature F.

[0020] Feature maps C3, C4, and C5 are positionally encoded and then input into the S2S attention module to generate basic spatiotemporal features. The S2S attention module includes an intra-scale temporal attention module and an inter-scale spatiotemporal attention module. The intra-scale temporal attention module adopts a temporal attention mechanism, and the inter-scale spatiotemporal attention module adopts a deformable attention mechanism.

[0021] Fusing two features F of the same dimension Finally, the spatiotemporal characteristics were obtained.

[0022] Furthermore, the feature map C2 and the spatiotemporal features Input to the pixel decoder and spatiotemporal features Separate features into features corresponding to scales C3, C4, and C5 of the feature map. and Then, progressive upsampling and cross-fusion are performed to obtain the fused spatiotemporal features. and the spatiotemporal characteristics after fusion and the spatiotemporal characteristics after fusion include:

[0023] spatiotemporal features Separate features into features corresponding to scales C3, C4, and C5 of the feature map. and

[0024] Features Upsampling, adjusted to match features At the same scale, then using bilinear interpolation, it is compared with... Cross-fusion generates fused spatiotemporal features

[0025] spatiotemporal features after fusion Upsampling, adjusted to match features At the same scale, then bilinear interpolation is used to correlate it with the features. Cross-fusion generates fused spatiotemporal features

[0026] spatiotemporal features after fusion Upsampled to the same scale as feature map C2, and then bilinear interpolation was used to cross-fuse it with feature map C2 to generate fused spatiotemporal features.

[0027] Furthermore, the transformer decoder includes three decoder units corresponding to different scales connected in series and an MLP module, which will then convert the features... and The input is fed into the transformer decoder to obtain the final embedding vector, which includes:

[0028] Features and The input features are fed into the corresponding scale decoder unit, and the input features are used as the attention mask, key and value of the decoder unit. The query features of the first decoder unit are the initial query features, and the query features of subsequent decoder units are the features output by the previous decoder unit.

[0029] In each decoder unit, cross-attention operation is performed first, followed by self-attention operation;

[0030] The query features output by the last decoder unit are processed by the MLP module to generate the final embedding vector.

[0031] Furthermore, the three decoder units of different scales iterate cyclically a preset number of times.

[0032] This application proposes a video instance segmentation method based on cross-frame instance association, addressing the problem of existing methods neglecting spatiotemporal context information in the video. Specifically, based on the deformable DETR framework, this application proposes a multi-scale method oriented towards spatiotemporal features to learn the spatiotemporal correlations of dynamic instances and construct more stable cross-frame instance associations. Notably, compared to previous methods, our method can establish reliable cross-frame instance associations without requiring complex frame-by-frame processing. Furthermore, we propose a data augmentation method called temporal copy-paste to effectively alleviate the overfitting problem caused by insufficient training data and improve the robustness of the model. This application improves the accuracy of video instance segmentation tasks, achieving state-of-the-art performance on two popular datasets. Attached Figure Description

[0033] Figure 1 This is a flowchart of the video instance segmentation method based on cross-frame instance association in this application;

[0034] Figure 2 This is a schematic diagram of the video instance segmentation network framework for this application;

[0035] Figure 3 This is a schematic diagram of the decoder unit. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0037] In one embodiment, such as Figure 1 As shown, a video instance segmentation method based on cross-frame instance association is proposed, including:

[0038] Step S1: Construct and train a video instance segmentation network, which includes a multi-scale feature extractor, a transformer encoder, a pixel decoder, and a transformer decoder.

[0039] like Figure 2 As shown, the video instance segmentation network constructed in this application includes a multi-scale feature extractor, a transformer encoder, a pixel decoder, and a transformer decoder. The multi-scale feature extractor extracts multi-scale features from the input data; the transformer encoder captures multi-scale spatiotemporal features; the pixel decoder cross-fuses the multi-scale features to generate high-resolution features for mask prediction; and the transformer decoder iteratively updates the query features.

[0040] Then, video frame sequences are acquired, preprocessed, and used to generate training sample data to train the constructed video instance segmentation network. The training of the network model is a relatively mature technique in this field and will not be elaborated upon here.

[0041] During training, to enhance the robustness of the model and alleviate overfitting during training, this application preprocesses the acquired video frame sequences to generate training sample data, including:

[0042] Take two frame sequences with the same number of frames from the collected video frame sequence dataset, use one of them as the target set and the other as the source set;

[0043] Establish a one-to-one correspondence between image frames in the target set and the source set according to the time sequence;

[0044] Then, the instances in the source set of images are copied and pasted onto the corresponding image frames in the target set to generate a new frame sequence, which is then placed into the video frame sequence dataset.

[0045] For example, first, take a frame sequence and create a target set Tgt. Then, randomly select another video frame sequence of the same T frames from the dataset and create a source set Src. Next, sort the source set according to the time order t. and target set Establish a one-to-one connection between them, and then transfer the instance from Copy and paste to Where t∈[1,T], thus generating a new frame sequence.

[0046] Finally, the annotations (also known as ground truth) of the new frame sequence are updated, including the mask, bounding box, and category of partially occluded objects, while objects that are completely occluded are removed.

[0047] It should be noted that the image frame preprocessing in this application also includes scaling the image frame to a certain size and normalizing the pixel values ​​of the image frame. These are common image preprocessing methods and will not be elaborated here. This application achieves data augmentation through temporal copy and paste, which can effectively enhance the robustness of the model and alleviate the overfitting problem during training.

[0048] Step S2: Input the video frame sequence to be segmented into the multi-scale feature extractor to extract feature maps of different scales, which are C2, C3, C4 and C5 according to the scale size.

[0049] This application uses ResNet50, which is more popular in the field of vision, as a multi-scale feature extractor, requiring the input frame sequence to be... Where T represents the number of frames, 3 represents the number of RGB channels, H and W are the height and width of the image, respectively, and the multi-scale feature extractor extracts multi-scale features. Its output is a series of feature maps of different sizes, where 256 represents the number of channels, H i and W i These represent the height and width of the feature map, respectively. This embodiment yields four feature maps at different scales, corresponding to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original video frame image size, respectively denoted as C2, C3, C4, and C5.

[0050] Step S3: Input the extracted feature maps C3, C4, and C5 into the transformer encoder to extract spatiotemporal features.

[0051] This application utilizes a Transformer encoder to capture rich multi-scale spatiotemporal features, including:

[0052] Step S301: Encode the feature maps C3, C4 and C5 at their positions, then perform tensor flattening operations on each and input them into the deformable attention module to generate the basic feature F.

[0053] In the Transformer encoder, selecting {C3,C4,C5} for position encoding yields the following result. Where A i =H i ×W i The features {P3, P4, P5} are flattened using tensors and then input into the deformable attention module. The deformable attention module then calculates and generates the basic features F of the standard deformable attention output. The deformable attention module is a mature technology in Transformer encoders and will not be elaborated upon here.

[0054] Step S302: Encode the feature maps C3, C4, and C5 at their positions, and then input them into the S2S attention module to generate basic spatiotemporal features.

[0055] The difference between the Transformer encoder in this embodiment and the traditional Transformer encoder is that the Transformer encoder in this embodiment adds an S2S attention module to the traditional Transformer encoder.

[0056] like Figure 2 As shown, the S2S attention module consists of an intra-scale temporal attention module and an inter-scale spatiotemporal attention module. The intra-scale temporal attention block uses a temporal attention mechanism (TA) that leverages the interdependence of cross-frame features to model the temporal correlation of cross-frame features.

[0057] use This represents the input of the temporal attention block at different scales, where i∈[3,5] represents different scales; p∈[0,A] represents the input of the temporal attention block at different scales. i [-1] represents the pixel position at scale i; t∈[0,T-1] represents the frame number; C is the feature dimension.

[0058] The process of calculating intra-scale temporal attention is as follows:

[0059]

[0060]

[0061]

[0062] The S here i This indicates a feature with global temporal correlation at scale i; W q W k W v These are the weight matrices used to generate the query, key, and value at pixel position p, respectively. LN represents the normalized layer.

[0063] Next, the inter-scale spatiotemporal attention block uses a deformable attention mechanism to learn the spatiotemporal features and similarities across frames in adjacent scale spaces.

[0064] Specifically, use The output of the time attention block within the scale. This serves as the input to the inter-scale spatiotemporal attention block. First, low-resolution scale features are upsampled using bilinear interpolation, then concatenated with high-resolution scale features along the feature dimension, and finally... Projecting it to the C dimension is as follows:

[0065]

[0066] Will Adjusted to Subsequently, a flexible deformable attention mechanism (STDeformAttn) was chosen to reduce the expensive computational cost required to compute spatiotemporal features.

[0067] Then, the inter-scale spatiotemporal attention calculation process is as follows:

[0068]

[0069]

[0070]

[0071] Here, STDeformAttn changes the key and value based on position interpolation; It is by This was obtained after a location shift occurred. It incorporates features with spatiotemporal correlation. Connecting along the spatial dimension to generate basic spatiotemporal features Then Remodeling

[0072] In summary, the S2S attention module implicitly constructs cross-frame instance associations by modeling the spatiotemporal correlation of pixel-by-pixel features across frames.

[0073] Step S303: Fuse basic feature F and fundamental spatiotemporal features Finally, the spatiotemporal characteristics were obtained.

[0074] In this way, the Transformer encoder produces the basic features F of the standard deformable attention output and the basic spatiotemporal features of the S2S attention module output. Finally, the features F and F of the two identical dimensions are fused. Finally, the spatiotemporal characteristics were obtained. And send it to the pixel decoder.

[0075] Step S4: Combine feature map C2 and spatiotemporal features Input to the pixel decoder and spatiotemporal features Separate features into features corresponding to scales C3, C4, and C5 of the feature map. and Then, progressive upsampling and cross-fusion are performed to obtain the fused spatiotemporal features. and the spatiotemporal characteristics after fusion and the spatiotemporal characteristics after fusion

[0076] The spatiotemporal features captured in the Transformer encoder are input into the pixel decoder, and low-resolution multi-scale spatiotemporal features are progressively upsampled and cross-fused to generate high-resolution features. The generated high-resolution feature map is used for the final mask prediction.

[0077] Specifically, the corresponding original obtained in step S2 Figure 1 / 4 scale features C2 and spatiotemporal features captured in the Transformer encoder Input to the pixel decoder and perform the following operations:

[0078] Step S401: Spatiotemporal characteristics Separate features into features corresponding to scales C3, C4, and C5 of the feature map. and

[0079] This embodiment uses the resolution of the scale (i.e., width multiplied by height) as a condition to separate and obtain multi-scale spatiotemporal features corresponding to 1 / 8, 1 / 16, and 1 / 32 scales of the original image. and

[0080] Step S402, for features Upsampling, adjusted to match features At the same scale, then using bilinear interpolation, it is compared with... Cross-fusion generates fused spatiotemporal features

[0081] Step S403: Analyze the fused spatiotemporal features Upsampling, adjusted to match features At the same scale, then bilinear interpolation is used to correlate it with the features. Cross-fusion generates fused spatiotemporal features

[0082] Step S404: Analyze the fused spatiotemporal features Upsampled to the same scale as feature map C2, and then bilinear interpolation was used to cross-fuse it with feature map C2 to generate fused spatiotemporal features.

[0083] This embodiment integrates low-resolution multi-scale spatiotemporal features to generate high-resolution features, and the final generated high-resolution feature map (Scale is original) Figure 1 / 4 of the features are used for the final mask prediction.

[0084] Step S5: Add features and The input is fed into the transformer decoder to obtain the final embedding vector.

[0085] This step involves low-resolution multi-scale spatiotemporal features (scale is the original). Figure 1 Features at 1 / 32, 1 / 16, and 1 / 8 scales are sequentially input into Transformer decoders at different corresponding scales. The keys and values ​​used in the Transformer decoder computation are generated from low-resolution multi-scale spatiotemporal features. In this embodiment, the Transformer decoder consists of three decoders corresponding to different scales. The specific details of each decoder at different scales are as follows: Figure 3 As shown.

[0086] In one embodiment, the feature and The input is fed into the transformer decoder to obtain the final embedding vector, which includes:

[0087] Features and The input features are fed into the corresponding scale decoder unit, and the input features are used as the attention mask, key and value of the decoder unit. The query features of the first decoder unit are the initial query features, and the query features of subsequent decoder units are the features output by the previous decoder unit.

[0088] In each decoder unit, cross-attention operation is performed first, followed by self-attention operation;

[0089] The last decoder unit outputs the final embedding vector.

[0090] Specifically, the pixel decoder has a scale of original. Figure 1 Features at resolutions of 32, 1 / 16, and 1 / 8 are input into the decoders at each scale in ascending order of resolution. The input features are used as attention masks, keys, and values ​​and are input into the decoders in sequence.

[0091] The attention calculation process of the Transformer decoder is as follows:

[0092] X l =softmax(M l-1 +W l k′ l V l +X l-1 ,

[0093] Here, l is the level index; It is the Query Features in the C dimension of layer l; f K (X l-1 ) and fV (X l-1 Spatiotemporal characteristics under transformation, where T is the number of frames, H l and W l For spatial resolution; f Q f K and f V It is a linear function. Furthermore, the three-dimensional attention mask M at the feature location (t,x,y)... l-1 for:

[0094]

[0095] Here It is the binarized output of the resized 3D mask prediction of the (l-1)th transformer decoder layer (threshold is 0.5).

[0096] The first query of the Transformer decoder is an initial set of learnable query features called Init Query. The query is updated iteratively as the decoder loops, and the query features of subsequent decoder units are the features output by the previous decoder unit.

[0097] In each decoder unit, cross attention is performed first, followed by self attention. This embodiment introduces an attention mask during the cross attention calculation process. Figure 3 This is called masked attention. The query features output by the last decoder unit are processed by the MLP module to generate the final embedding vector.

[0098] In one embodiment, the three decoder units of different scales iterate cyclically a preset number of times.

[0099] That is, after the three decoder operations, a loop iteration is performed, and the three decoders perform a total of 3L decoding loops to generate the final query features. Then, after the query features pass through the MLP module, a set of n query embedding vectors are generated. The input features of the MLP module pass through a linear layer, a BatchNorm layer and a ReLU activation function from the input side to the output side.

[0100] Step S6: Combine the embedding vector with the spatiotemporal features Perform a dot product operation to obtain the instance segmentation result.

[0101] Finally, by performing a simple dot product operation between this set of embedded vectors and the high-resolution features obtained from the pixel decoder, we can obtain n query 3D masks, which are the masks of each frame of the entire video. The n obtained 3D masks are the result of instance segmentation.

[0102] Meanwhile, the embedded vectors are passed through a linear layer with the feature dimension as input and the number of categories as output to obtain the category prediction results for all instances in the entire video.

[0103] Table 1 shows the performance comparison between our method and previous state-of-the-art methods on the convolutional neural network-based backbone and YouTube-VIS 2019 datasets. On ResNet-50, our method achieves a mask AP of 47.4%, demonstrating state-of-the-art performance. Without training with additional data, our method outperforms Seqformer by 2.3% in mask AP. Compared to the baseline (Mask2Former), we achieve an absolute gain of 1.0%. Similarly, on ResNet-101, our method consistently outperforms all previous methods, with an overall mask AP of 49.5%. These results demonstrate that our method can more effectively utilize multi-scale spatiotemporal features to construct more stable cross-frame instance associations.

[0104] Experimental data from this application demonstrate that the method described in this application achieves better segmentation accuracy compared to other existing methods. The experimental data are shown in Tables 1 and 2 below:

[0105] Method Data AP <![CDATA[AP 50 ]]> <![CDATA[AP 75 ]]> <![CDATA[AR1]]> <![CDATA[AR 10 ]]> MaskTrack R-CNN V 30.3 51.1 32.6 31.0 35.5 CrossVIS V 36.3 56.8 38.9 35.6 40.7 VisTR V 35.6 56.8 37.0 35.2 40.2 IFC V 42.8 65.8 43.8 41.1 49.7 SeqFormer V 45.1 66.9 50.5 45.6 54.6 SeqFormer V+C80k 47.4 69.8 51.8 45.5 54.8 Mask2Former V 46.4 68.0 50.0 - - Technical solution of this application V 47.4 71.0 53.0 46.1 58.1

[0106] Table 1

[0107] Table 1 shows the performance comparison between the proposed method and other methods on the YouTube-VIS 2019 dataset. "V" indicates that only the YouTube-VIS training set was used, and "V+C80K" means that synthetic videos of overlapping categories from MS-COCO were also used for joint training. AP represents the average accuracy of the segmentation mask prediction results. 50 AP represents the accuracy when the IOU threshold is 0.5. 75 This represents the precision when the IOU threshold is 0.75. AR represents the average recall rate. AR with a subscript of 1 indicates that each image was detected once, and AR with a subscript of 10 indicates that each image was detected 10 times.

[0108] Method Data AP <![CDATA[AP 50 ]]> <![CDATA[AP 75 ]]> <![CDATA[AR1]]> <![CDATA[AR 10 ]]> MaskTrack R-CNN V 28.6 48.9 29.6 26.5 33.8 CrossVIS V 34.2 54.4 37.9 30.4 38.2 IFC V 36.6 57.9 39.3 - - SeqFormer V+C80k 40.5 62.4 43.7 36.1 48.1 Mask2Former V 40.6 60.9 41.8 - - Technical solution of this application V 41.6 64.4 44.8 38.2 50.9

[0109] Table 2

[0110] As shown in Table 2, on the recently released YouTube-VIS 2021 dataset, the method in this application achieves a mask AP (accuracy of segmentation mask prediction), which is better than the previous state-of-the-art offline video instance segmentation method by at least 1.0% mask AP.

[0111] The embodiments described above are merely illustrative of implementation methods of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A video instance segmentation method based on cross-frame instance association, characterized in that, The video instance segmentation method based on cross-frame instance association includes: A video instance segmentation network is constructed and trained, the video instance segmentation network including a multi-scale feature extractor, a transformer encoder, a pixel decoder and a transformer decoder; The video frame sequence to be segmented is input into a multi-scale feature extractor to extract feature maps at different scales. The feature maps are then classified according to their scale. , , and ; Extracted feature maps , and Spatiotemporal features were extracted from the input to the transformer encoder. ; feature map and spatiotemporal characteristics Input to the pixel decoder and spatiotemporal features Separation to feature map scale , and Corresponding features , and Then, progressive upsampling and cross-fusion are performed to obtain the fused spatiotemporal features. and the spatiotemporal characteristics after fusion and the spatiotemporal characteristics after fusion ; Features , and The input is fed into the transformer decoder to obtain the final embedding vector; Integrating embedding vectors with spatiotemporal features Perform a dot product operation to obtain the instance segmentation result; Among them, the extracted feature map , and Spatiotemporal features were extracted from the input to the transformer encoder. ,include: feature map , and Position encoding is performed, followed by tensor flattening operations and input into a deformable attention module to generate basic features. ; feature map , and Position encoding is performed and then input into the S2S attention module to generate basic spatiotemporal features. The S2S attention module includes an intra-scale temporal attention module and an inter-scale spatiotemporal attention module. The intra-scale temporal attention module adopts a temporal attention mechanism, and the inter-scale spatiotemporal attention module adopts a deformable attention mechanism. Fusion of features from two identical dimensions and Finally, the spatiotemporal characteristics were obtained. .

2. The video instance segmentation method based on cross-frame instance association according to claim 1, characterized in that, The training video instance segmentation network includes preprocessing the acquired video frame sequences to generate training sample data, including: Take two frame sequences with the same number of frames from the collected video frame sequence dataset, use one of them as the target set and the other as the source set; Establish a one-to-one correspondence between image frames in the target set and the source set according to the time sequence; Then, the instances in the source set of images are copied and pasted onto the corresponding image frames in the target set to generate a new frame sequence, which is then placed into the video frame sequence dataset.

3. The video instance segmentation method based on cross-frame instance association according to claim 1, characterized in that, The feature map and spatiotemporal characteristics Input to the pixel decoder and spatiotemporal features Separate into feature maps , and Features corresponding to scale , and Then, progressive upsampling and cross-fusion are performed to obtain the fused spatiotemporal features. and the spatiotemporal characteristics after fusion and the spatiotemporal characteristics after fusion ,include: spatiotemporal features Separate into feature maps , and Features corresponding to scale , and ; Features Upsampling, adjusted to match features At the same scale, then using bilinear interpolation, it is compared with... Cross-fusion generates fused spatiotemporal features ; spatiotemporal features after fusion Upsampling, adjusted to match features At the same scale, then bilinear interpolation is used to correlate it with the features. Cross-fusion generates fused spatiotemporal features ; spatiotemporal features after fusion Upsampling, adjusted to match the feature map At the same scale, bilinear interpolation is then used to correlate it with the feature map. Cross-fusion generates fused spatiotemporal features .

4. The video instance segmentation method based on cross-frame instance association according to claim 1, characterized in that, The transformer decoder includes three decoder units of different scales connected in series and an MLP module, which will feature... , and The input is fed into the transformer decoder to obtain the final embedding vector. include: Features , and The input features are fed into the corresponding scale decoder unit, and the input features are used as the attention mask, key and value of the decoder unit. The query features of the first decoder unit are the initial query features, and the query features of subsequent decoder units are the features output by the previous decoder unit. In each decoder unit, cross-attention operation is performed first, followed by self-attention operation; The query features output by the last decoder unit are processed by the MLP module to generate the final embedding vector.

5. The video instance segmentation method based on cross-frame instance association according to claim 4, characterized in that, The three decoder units of different scales are cyclically iterated a preset number of times.

Images