A method, apparatus, device, and readable storage medium for video target segmentation.

Abstract

This application relates to a method, apparatus, device, and readable storage medium for segmenting a target in a video. The method includes: extracting features from a target video sequence and a target language description to obtain image features, sentence-level text features, and word-level text features; extracting features and frame-level spatial information from the above features to obtain image frame features and frame-level instance embeddings; combining the frame-level instance embeddings into spatiotemporal dimensions and performing spatiotemporal consistency modeling and spatiotemporal representation learning to generate target instance features; aggregating these features with the video-level query embeddings to generate video-level instance embeddings; performing a linear transformation on the video-level instance embeddings to obtain classification probabilities and conditional convolution kernels; performing convolution operations on the conditional convolution kernels and image frame features to obtain multiple mask prediction sequences; and selecting the target mask prediction sequence from the multiple mask prediction sequences based on the classification probabilities as the video target segmentation result, effectively improving spatiotemporal consistency and accuracy.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, apparatus, device, and readable storage medium for video target segmentation. Background Technology

[0002] Referencing Video Object Segmentation (RVOS) aims to segment a video sequence into objects described by a textual description. It provides a more natural and user-friendly approach to human-computer interaction and is a fundamental technology for researching large-scale cross-modal video segmentation models. It is also one of the key technologies for achieving "general artificial intelligence." RVOS has a wide range of applications, including text-driven video editing, video object retrieval, smart security, and robotics, playing a positive role in promoting social development.

[0003] In recent years, video target segmentation methods based on multimodal Transformer models (a type of neural network that learns context and thus meaning by tracking relationships in sequence data) have demonstrated the superiority of treating video target segmentation as a query-based sequence prediction problem. However, current methods for video target segmentation often employ frame-independent cross-modal interactions within the Transformer model to reduce computational overhead. The lack of inter-frame information exchange results in generated instance embeddings lacking spatiotemporal representation information of the target, leading to spatiotemporal consistency defects in the final target mask prediction. Consequently, both the spatiotemporal consistency and accuracy of the video target segmentation results are poor. Summary of the Invention

[0004] This application provides a method, apparatus, device, and readable storage medium for segmenting video targets, in order to solve the problem that the spatiotemporal consistency and accuracy of video target segmentation results are poor when the segmentation is achieved solely by using independent cross-modal interactions between frames.

[0005] Firstly, a method for segmenting referential video targets is provided, including the following steps:

[0006] Feature extraction is performed on the target video sequence and the target language description content to obtain image features, sentence-level text features, and word-level text features;

[0007] Feature extraction is performed on image features and word-level text features to obtain image frame features, and frame-level spatial information is extracted from image features and sentence-level text features to generate frame-level instance embeddings;

[0008] The spatiotemporal dimensions of the frame-level instance embeddings are merged, and the spatiotemporal consistency modeling and spatiotemporal representation learning are performed on the merged frame-level instance embeddings to generate target instance features;

[0009] Spatiotemporal information aggregation processing is performed on the target instance features and the preset video-level query embedding to generate video-level instance embeddings;

[0010] A linear transformation is performed on the video-level instance embedding to obtain the classification probability and conditional convolution kernel. Then, a convolution operation is performed on the conditional convolution kernel and the image frame features to generate multiple mask prediction sequences.

[0011] The target mask prediction sequence is selected from multiple mask prediction sequences based on classification probability as the video target segmentation result.

[0012] In some embodiments, the step of performing spatiotemporal consistency modeling and spatiotemporal representation learning on the merged frame-level instance embeddings to generate target instance features includes:

[0013] Add position encoding to the merged frame-level instance embedding to form a new frame-level instance embedding;

[0014] Based on the instance encoder, spatiotemporal consistency modeling and spatiotemporal representation learning are performed on the new frame-level instance embedding to generate target instance features. The instance encoder includes a multi-head self-attention module, an addition module, a normalization module, and a feedforward network.

[0015] In some embodiments, the step of performing spatiotemporal information aggregation processing on the target instance features and the preset video-level query embedding to generate a video-level instance embedding includes:

[0016] A video-level query embedding is generated through random initialization;

[0017] The instance decoder performs spatiotemporal information aggregation processing on target instance features and video-level query embeddings. The instance decoder includes a multi-head mutual attention module, an addition module, a normalization module, a multi-head self-attention module, and a feedforward network.

[0018] In some embodiments, the linear transformation of the video-level instance embedding to obtain the classification probability and conditional convolution kernel includes:

[0019] The classification head network performs a linear transformation on the video-level instance embedding to obtain the classification probability. The classification head network includes a single-layer linear transformation layer.

[0020] The video-level instance embedding is linearly transformed by a masking terminal network to obtain a conditional convolution kernel. The masking terminal network includes multiple linear transformation layers.

[0021] In some embodiments, the step of selecting the target mask prediction sequence from multiple mask prediction sequences based on classification probability as the video target segmentation result includes:

[0022] For each mask prediction sequence, the average classification probability of the mask prediction sequence is calculated based on the classification probability corresponding to each frame;

[0023] The mask prediction sequence with the highest average classification probability is selected from multiple mask prediction sequences as the target mask prediction sequence;

[0024] The target mask prediction sequence is used as the video target segmentation result.

[0025] In some embodiments, prior to the step of feature extraction of image features and word-level text features, the method further includes:

[0026] After extracting features from the image features using a Transformer encoder, new image features are generated. Based on the new image features, the step of extracting features from the image features and word-level text features is performed. The Transformer encoder includes a multi-scale deformable self-attention module, an addition module, a normalization module, and a feedforward network.

[0027] In some embodiments, the step of extracting features from the target video sequence and the target language description content to obtain image features, sentence-level text features, and word-level text features includes:

[0028] Image features are obtained by performing multi-scale feature extraction on the target video sequence using a visual encoder;

[0029] Text features are extracted from the target language description by a text encoder to obtain sentence-level and word-level text features.

[0030] Secondly, a device for segmenting video targets is provided, comprising:

[0031] The feature extraction unit is used to extract features from the target video sequence and the target language description content, respectively, to obtain image features, sentence-level text features and word-level text features;

[0032] The frame-level spatial information extraction unit is used to extract features from image features and word-level text features to obtain image frame features, and to extract frame-level spatial information from image features and sentence-level text features to generate frame-level instance embeddings.

[0033] The spatiotemporal information aggregation unit is used to merge the spatiotemporal dimensions of frame-level instance embeddings, and to perform spatiotemporal consistency modeling and spatiotemporal representation learning on the merged frame-level instance embeddings to generate target instance features; and to perform spatiotemporal information aggregation processing on the target instance features and the preset video-level query embeddings to generate video-level instance embeddings.

[0034] The instance sequence segmentation unit performs a linear transformation on the video-level instance embedding to obtain classification probabilities and conditional convolution kernels. It then performs convolution operations on the conditional convolution kernels and image frame features to generate multiple mask prediction sequences. Based on the classification probabilities, it selects the target mask prediction sequence from the multiple mask prediction sequences as the video target segmentation result.

[0035] Thirdly, a device for segmenting a video target is provided, comprising: a memory and a processor, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the aforementioned method for segmenting a video target.

[0036] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method for segmenting a referred video target.

[0037] This application provides a method, apparatus, device, and readable storage medium for segmenting a target video. The method includes: extracting features from the target video sequence and the target language description content to obtain image features, sentence-level text features, and word-level text features; extracting features from the image features and word-level text features to obtain image frame features, and extracting frame-level spatial information from the image features and sentence-level text features to generate frame-level instance embeddings; merging the spatiotemporal dimensions of the frame-level instance embeddings, and performing spatiotemporal consistency modeling and spatiotemporal representation learning on the merged frame-level instance embeddings to generate target instance features; aggregating the spatiotemporal information of the target instance features and a preset video-level query embedding to generate video-level instance embeddings; performing a linear transformation on the video-level instance embeddings to obtain classification probabilities and conditional convolution kernels, and performing convolution operations on the conditional convolution kernels and image frame features to generate multiple mask prediction sequences; and selecting the target mask prediction sequence from the multiple mask prediction sequences based on the classification probabilities as the video target segmentation result. This application employs a two-tiered strategy—first spatial, then spatiotemporal—to progressively model the spatiotemporal consistency of the target and learn its spatiotemporal representation, thereby providing a more comprehensive interpretation of target information in video sequences and improving the spatiotemporal consistency and accuracy of video target segmentation results. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A flowchart illustrating a method for segmenting a video target, provided as an embodiment of this application;

[0040] Figure 2 A schematic diagram illustrating the framework of the video target segmentation method based on spatiotemporal hierarchical query provided in this application embodiment;

[0041] Figure 3 A schematic diagram of the first-stage Transformer encoder-decoder structure provided for embodiments of this application;

[0042] Figure 4 A schematic diagram of the structure of the second-stage example encoder-decoder provided in the embodiments of this application;

[0043] Figure 5 This is a schematic diagram of a video target segmentation device provided in an embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] This application provides a method, apparatus, device, and readable storage medium for segmenting video targets, which can solve the problem that the spatiotemporal consistency and accuracy of video target segmentation results are poor when the segmentation is achieved solely by using independent cross-modal interactions between frames.

[0046] See Figure 1 and Figure 2 As shown, this application provides a method for segmenting a referred video target, including the following steps:

[0047] Step S10: Extract features from the target video sequence and the target language description content to obtain image features, sentence-level text features, and word-level text features;

[0048] As an example, in this embodiment, a video sequence containing T frames is given to form a target video sequence, and a language description containing L words describing the target is given to form the target language description content (e.g., Figure 2 The phrase "aredfox that runs and jumps in another fox" appears in the text. Feature extraction is performed on the target video sequence to obtain multi-scale backbone features (i.e., image features F) for each frame. v The spatial step size of the multi-scale backbone features can be [8, 16, 32]. Then, feature extraction is performed on the input target language description content to generate sentence-level text features and word-level text features respectively.

[0049] Furthermore, the feature extraction of the target video sequence and the target language description content, respectively, yields image features, sentence-level text features, and word-level text features, including:

[0050] Image features are obtained by performing multi-scale feature extraction on the target video sequence using a visual encoder;

[0051] Text features are extracted from the target language description by a text encoder to obtain sentence-level and word-level text features.

[0052] As an example, in this embodiment, image features and text features will be extracted through a feature extraction unit. Specifically, the feature extraction unit includes a visual encoder and a text encoder; wherein, the visual encoder extracts multi-scale backbone features of image frames in the target video sequence, and the text encoder extracts sentence-level and word-level text features of the target language description content. It should be noted that the visual encoder can be a 2D spatial encoder, such as ResNet and SwinTransformer, or a 3D spatiotemporal encoder, such as VideoswinTransformer; while the text encoder can preferably be a text encoding model RoBERTa.

[0053] Furthermore, prior to the feature extraction steps for image features and word-level text features, the following steps are also included:

[0054] After extracting features from the image features using a Transformer encoder, new image features are generated. Based on the new image features, the step of extracting features from the image features and word-level text features is performed. The Transformer encoder includes a multi-scale deformable self-attention module, an addition module, a normalization module, and a feedforward network.

[0055] In this exemplary embodiment, the image features are first encoded using a Transformer encoder. This involves using the image features as input feature A, adding positional encoding to feature A, and then feeding it into a Transformer encoder composed of multiple stacked identical feature encoding layers for independent inter-frame feature transformation to generate new image features. It should be noted that the specific number of feature encoding layers in the Transformer encoder can be set according to actual needs and is not limited here; for example, the number of feature encoding layers can be set to 4.

[0056] The following is a brief explanation of the workflow and principles of the Transformer encoder.

[0057] After adding a fixed positional code to the input feature A, it is fed into a multi-layer feature encoding layer for feature encoding. (See also...) Figure 3 As shown, since the encoding principle of each feature encoding layer is the same, for the sake of simplicity, this embodiment will take one of the feature encoding layers as an example for explanation: Input feature A will first form two identical branch features A1 and A2; branch feature A1 passes through a multi-scale deformable self-attention module (i.e., Figure 3 The MS-DSA algorithm performs global pixel-level correlation modeling to obtain output feature B, thereby improving the representational power of the features. The output feature B and branch feature A2 of MS-DSA are then subjected to bitwise addition by the addition module and layer normalization by the normalization module to obtain output feature C. Similarly, output feature C will form two identical branch features C1 and C2. Output feature C1 is then subjected to two linear transformation layers in the feedforward network to obtain output feature D. Then, output feature D and branch feature C2 are subjected to bitwise addition and layer normalization to obtain target feature E. The target feature will be used as the input of the next feature encoding layer or as the output of the encoder (i.e., new image features) and enter the decoder.

[0058] Step S20: Extract features from image features and word-level text features to obtain image frame features, and extract frame-level spatial information from image features and sentence-level text features to generate frame-level instance embeddings;

[0059] As an example, in the following embodiments, the target feature E will be used as the final output of the encoder. That is, the target feature is the new image feature obtained after processing by the Transformer encoder, and the first stage of frame-level spatial information extraction will be performed based on this new image feature. The target feature E will form two identical branch features E1 and E2, which will be processed by the feature pyramid network and the decoder, respectively.

[0060] It should be understood that in this embodiment, feature extraction is performed on branch features E1 and word-level text features through a cross-modal feature pyramid network to obtain image frame features. That is, in the cross-modal feature pyramid network, word-level text features and multi-level branch features E1 are independently fused across modally to generate text-enhanced multi-scale image features. The multi-scale image frame features are gradually upsampled from low-resolution features and fused with image features of the same size to finally generate single-scale high-resolution image frame features for convolution with subsequent conditional convolution kernels.

[0061] Specifically, in each level of the cross-modal fusion module of the feature pyramid network, branch feature E1 first undergoes downsampling to reduce the spatial dimension of the feature, thereby reducing the computational load of the subsequent multi-scale self-attention module. Then, after passing through the multi-head self-attention module, branch feature E1 is upsampled to restore the spatial dimension of the original input image, and after addition and layer normalization, feature G is generated. Feature G and word-level text features are input together into the multi-head mutual attention module for cross-modal interaction to generate text-enhanced image features. The text-enhanced image features are then added to the input feature G and layer normalized before being input into the feedforward neural network for feature transformation. Finally, after an addition and normalization operation, the text-enhanced multi-scale image features after cross-modal fusion are output.

[0062] Simultaneously, sentence-level text features are used as language query embeddings, and the branch features E2 output after encoding by the Transformer encoder are used in the Transformer decoder for frame-independent cross-modal feature interaction. That is, after multi-layer feature decoding operations, frame-level instance embeddings containing only target space representation information are generated. It should be noted that, see [link to relevant documentation]... Figure 3 As shown, the Transformer decoder in this embodiment is composed of multiple identical feature decoding layers stacked together. Each feature decoding layer includes a multi-head self-attention module, a multi-scale deformable mutual attention module (i.e., MS-DCA), an addition module, a normalization module, and a feedforward network, etc. The specific number of feature decoding layers in the Transformer decoder can be set according to actual needs and is not limited here. For example, the number of feature decoding layers can be set to 4.

[0063] The following will combine Figure 3 A brief explanation of the workflow and principles of the Transformer decoder.

[0064] Since the decoding principle of each feature decoding layer is the same, for the sake of simplicity, this embodiment will take one of the feature decoding layers as an example for explanation: Sentence-level text features as language query embeddings L will first form two identical branch features L1 and L2. Branch feature L1 is processed by a multi-head self-attention module to learn the global relevance of features and generate feature M. Feature M and branch feature L2 are processed by the addition module for bitwise addition and the normalization module for layer normalization to obtain output feature N. Output feature N also forms two identical branch features N1 and N2.

[0065] In this process, branch feature N1 and output feature E2 are input together into the MS-DCA module for cross-modal feature interaction and decoding to obtain output feature O. Output feature O and branch feature N2 are then subjected to bitwise addition and layer normalization to obtain output feature P, which in turn forms two identical branch features P1 and P2. Branch feature P1 is then passed through two linear transformation layers in the feedforward network to output feature Q. Output feature Q and branch feature P2 are then subjected to bitwise addition and layer normalization to obtain output feature R. Output feature R will be used as the input of the next feature decoding layer or as the output of the decoder (i.e., frame-level instance embedding).

[0066] Step S30: Perform spatiotemporal dimension merging on the frame-level instance embeddings, and perform spatiotemporal consistency modeling and spatiotemporal representation learning on the merged frame-level instance embeddings to generate target instance features;

[0067] As an example, in this embodiment, the frame-level instance embeddings are input into the spatiotemporal information aggregation unit for the second stage of spatiotemporal information aggregation, establishing the spatiotemporal consistency of the target and learning the spatiotemporal representation of the target with less computational cost. Specifically, in order to achieve spatiotemporal consistency modeling of the frame-level instance embeddings in the spatiotemporal dimension, the spatiotemporal dimensions of the frame-level instance embeddings generated in the first stage are merged before the frame-level instance embeddings are input into the encoder in the spatiotemporal information aggregation unit. Then, the frame-level instance embeddings are input into the encoder for spatiotemporal consistency modeling and learning the spatiotemporal representation of the target, that is, the frame-level instance embeddings that only have target spatial representation information are expanded in the spatiotemporal dimension, and then the target instance features are output.

[0068] Furthermore, the process of performing spatiotemporal consistency modeling and spatiotemporal representation learning on the merged frame-level instance embeddings to generate target instance features includes:

[0069] Add position encoding to the merged frame-level instance embedding to form a new frame-level instance embedding;

[0070] Based on the instance encoder, spatiotemporal consistency modeling and spatiotemporal representation learning are performed on the new frame-level instance embedding to generate target instance features. The instance encoder includes a multi-head self-attention module, an addition module, a normalization module, and a feedforward network.

[0071] As an example, in this embodiment, an instance encoder will be used to achieve spatiotemporal consistency modeling and spatiotemporal representation learning of frame-level instance embeddings, thereby generating target instance features. See also... Figure 4 As shown, the instance encoder consists of multiple identical instance coding layers stacked together. Each instance coding layer may specifically include a multi-head self-attention module, an addition module, a normalization module, and a feedforward network. It should be noted that the specific number of instance coding layers in the instance encoder can be set according to actual needs and is not limited here. For example, the number of instance coding layers can be set to 3.

[0072] The following will combine Figure 4 A brief explanation of the workflow and principles of the instance encoder.

[0073] After merging the spatial and temporal dimensions, and adding fixed positional encoding, a new frame-level instance embedding is formed. This new embedding undergoes feature encoding through multiple instance encoding layers to generate target instance features. Since the encoding principle of each instance encoding layer is the same, for the sake of simplicity, this embodiment will use one instance encoding layer as an example for explanation:

[0074] The new frame-level instance embedding H first forms two identical branch features H1 and H2. Branch feature H1 undergoes target spatiotemporal consistency modeling and spatiotemporal representation learning through a multi-head self-attention module to obtain output feature I, thereby improving the representational ability of the feature. Output feature I and branch feature H2 are then subjected to bitwise addition by an addition module and layer normalization by a normalization module to obtain output feature J. Output feature J also forms two identical branch features J1 and J2. Output feature J1 is then subjected to two linear transformation layers in a feedforward network to output feature K. Output feature K and branch feature J2 are then subjected to bitwise addition and layer normalization to obtain output feature U. Output feature U will be used as the input of the next instance coding layer or as the output of the instance encoder (i.e., the target instance feature) and fed into the decoder.

[0075] Step S40: Perform spatiotemporal information aggregation processing on the target instance features and the preset video-level query embedding to generate video-level instance embedding;

[0076] As an example, in this embodiment, a video-level query embedding and target instance features (i.e., the expanded target instance embedding) learned by the model are input into the decoder for multi-layer feature interaction, so that the video-level query embedding and target instance features interact and aggregate information in the spatiotemporal dimension, thereby generating a video-level instance embedding with global spatiotemporal representation information of the target, thereby realizing spatiotemporal consistency modeling and spatiotemporal representation learning of the target.

[0077] Furthermore, the process of aggregating spatiotemporal information on the target instance features and the preset video-level query embedding to generate a video-level instance embedding includes:

[0078] A video-level query embedding is generated through random initialization;

[0079] The instance decoder performs spatiotemporal information aggregation processing on target instance features and video-level query embeddings. The instance decoder includes a multi-head mutual attention module, an addition module, a normalization module, a multi-head self-attention module, and a feedforward network.

[0080] As an example, in this embodiment, an instance decoder will be used to perform spatiotemporal information aggregation processing on target instance features and video-level query embeddings, thereby generating video-level instance embeddings. See also... Figure 4 As shown, the instance decoder consists of multiple identical instance decoding layers stacked together. Each instance decoding layer may specifically include a multi-head mutual attention module, an addition module, a normalization module, and a feedforward network. It should be noted that the specific number of instance decoding layers in the instance decoder can be set according to actual needs and is not limited here. For example, the number of instance decoding layers can be set to 3.

[0081] The following will combine Figure 4 A brief explanation of the workflow and principles of the instance decoder.

[0082] In this embodiment, the output feature U is used as the final output of the instance encoder. That is, the output feature U is the target instance feature obtained after processing by the instance encoder. A video-level query embedding V is randomly initialized and generated, and after adding a fixed positional encoding, it is combined with the output feature U through multiple layers of feature decoding operations to generate a video-level instance embedding. Since the decoding principle of each instance decoding layer is the same, for the sake of simplicity, this embodiment will use one instance decoding layer as an example for explanation:

[0083] The video-level query embedding V, after adding positional encoding, will form two identical branch features V1 and V2. Branch feature V1 is first input together with the output feature U of the instance encoder into the multi-head mutual attention module for feature interaction and decoding, resulting in output feature W. Output feature W and branch feature V2 are then subjected to bitwise addition by the addition module and layer normalization by the normalization module to obtain output feature X. Output feature X will also form two identical branch features X1 and X2. Branch feature X1 is then subjected to global feature correlation learning by the multi-head self-attention module to generate feature Y. Feature Y and branch feature X2 are then subjected to bitwise addition and layer normalization to obtain output feature Z. Output feature Z will also form two identical branch features Z1 and Z2.

[0084] The output feature Z1 is passed through two linear transformation layers in the feedforward network to obtain the output feature S. The feature S and the branch feature Z2 are added bitwise and normalized by layer to obtain the output feature T. The output feature T will be used as the input of the next instance decoding layer or as the output of the instance decoder (i.e., video-level instance embedding).

[0085] Step S50: Perform a linear transformation on the video-level instance embedding to obtain the classification probability and conditional convolution kernel, and perform convolution operation on the conditional convolution kernel and image frame features to generate multiple mask prediction sequences;

[0086] As an example, in this embodiment, the video-level instance embeddings are subjected to linear transformations of classification and masking respectively to obtain the classification probability and conditional convolution kernel parameters for each frame. It is understood that the classification probability can be a binary classification probability, or the type of classification probability can be set according to actual needs. The binary classification probability is used to indicate whether the predicted instance is referred to by the text language and is visible in the current frame. The conditional convolution kernel is convolved with the image frame features generated in the cross-modal feature pyramid network to generate multiple mask prediction sequences corresponding to the target video sequence.

[0087] Furthermore, the linear transformation performed on the video-level instance embedding to obtain the classification probability and conditional convolution kernel includes:

[0088] The classification head network performs a linear transformation on the video-level instance embedding to obtain the classification probability. The classification head network includes a single-layer linear transformation layer.

[0089] The video-level instance embedding is linearly transformed by a masking terminal network to obtain a conditional convolution kernel. The masking terminal network includes multiple linear transformation layers.

[0090] As an example, in this embodiment, the video-level instance embeddings are subjected to linear transformations for classification and masking through a classification head network and a masking head network, respectively. Specifically, after the video-level instance embeddings are input into the classification head network for linear transformation, a binary classification probability for each frame is obtained. It should be noted that the classification head network preferably includes a single-layer linear transformation layer, which is used to transform the video-level instance embeddings with an input dimension of 256 into classification probabilities with an output dimension of 1.

[0091] Similarly, after the video-level instance embedding is input into the masking terminal network and linearly transformed, the conditional convolution kernel parameters of each frame are obtained. It should be noted that the masking terminal network preferably includes three linear transformation layers. The first two layers transform the video-level instance embedding with an input dimension of 256 into a feature embedding with an output dimension of 256, and the last layer transforms the video-level instance embedding with an input dimension of 256 into a feature embedding with an output dimension of 2153.

[0092] Step S60: Select the target mask prediction sequence from multiple mask prediction sequences based on classification probability as the video target segmentation result.

[0093] As an example, in this embodiment, for each mask prediction sequence result, the optimal target mask prediction sequence is determined according to its corresponding classification probability, and the target mask prediction sequence is used as the final video target segmentation result corresponding to the target language description content.

[0094] Furthermore, the step of selecting the target mask prediction sequence from multiple mask prediction sequences based on classification probability as the video target segmentation result includes:

[0095] For each mask prediction sequence, the average classification probability of the mask prediction sequence is calculated based on the classification probability corresponding to each frame;

[0096] The mask prediction sequence with the highest average classification probability is selected from multiple mask prediction sequences as the target mask prediction sequence;

[0097] The target mask prediction sequence is used as the video target segmentation result.

[0098] As an example, in this embodiment, for each mask prediction sequence, the average classification probability value of all its frames is calculated. For example, for mask prediction sequence 1, the average of all frames in mask prediction sequence 1 is calculated based on the classification probability value of each frame in mask prediction sequence 1 to obtain the average classification probability of mask prediction sequence 1. Then, the target mask prediction sequence with the highest average classification probability is selected from multiple mask prediction sequences as the final video target segmentation result, thereby completing the task of segmenting the target referred to by the text in the video sequence.

[0099] Therefore, this application proposes a two-level query mechanism for video target segmentation tasks, which constructs an algorithm architecture for target spatiotemporal consistency modeling and spatiotemporal representation learning in a more efficient manner, prioritizing spatial and then spatiotemporal approaches. This allows for the gradual implementation of target spatiotemporal consistency modeling and spatiotemporal representation learning according to the two-level strategy, enabling a more comprehensive interpretation of target information in video sequences. This improves the spatiotemporal consistency and accuracy of video target segmentation results, addressing the current challenges of insufficient target spatiotemporal consistency modeling and spatiotemporal representation learning in query-based and multimodal Transformer-based video target segmentation methods.

[0100] Meanwhile, by adopting a hierarchical mechanism (i.e., performing inter-frame feature extraction and spatiotemporal feature extraction) and the idea of ​​dimensionality reduction (i.e., transforming high-dimensional image features into low-dimensional instance embeddings), instances with smaller feature dimensions are selected and embedded in the spatiotemporal dimension for information interaction, thereby significantly alleviating the computational burden of the model.

[0101] It is understood that the video target segmentation method based on spatiotemporal hierarchical query provided in this embodiment is implemented based on a video target segmentation network model. The following will combine... Figure 2 This paper briefly explains the composition and construction process of the video target segmentation model. It should be noted that... Figure 2 The different shapes such as circles, triangles, and pentagrams in the image represent various dimensions and types of embeddings.

[0102] First, a video target segmentation network model based on spatiotemporal hierarchical query is constructed, which mainly consists of four units: feature extraction unit, frame-level spatial information extraction unit, spatiotemporal information aggregation unit, and instance sequence segmentation unit.

[0103] The feature extraction unit is used to extract multi-scale backbone features of image frames through a visual encoder, and to extract sentence-level and word-level text features of text descriptions using a text encoder.

[0104] The frame-level spatial information extraction unit mainly consists of three parts: a Transformer encoder, a Transformer decoder, and a feature pyramid network. The encoder and decoder are each composed of four identical encoding and decoding layers stacked together. In the encoder, the multi-scale backbone features of the image frame, after being augmented with positional encoding, are input to the Transformer encoder for frame-independent feature transformation. The output multi-scale image features, along with sentence-level text features, are used as input to the decoder. In the decoder, the sentence-level text features are used as a language query embedding (this language query embedding includes N...). lThe C-dimensional tensor and multi-scale image features are subjected to multiple cross-modal interactions to obtain frame-level instance embeddings. The frame-level instance embeddings are then input into the spatiotemporal information aggregation unit to model the spatiotemporal consistency of the target and learn the spatiotemporal representation of the target with less computational cost. The feature pyramid network is used to extract cross-modal features from the multi-scale image features and word-level text features output by the Transformer encoder to generate image frame features.

[0105] The spatiotemporal information aggregation unit mainly consists of two parts: an instance encoder and an instance decoder. Both the instance encoder and decoder are composed of three identical encoding and decoding layers stacked together. To achieve spatiotemporal consistency modeling of frame-level target instance embeddings in the spatiotemporal dimension, the spatiotemporal information aggregation unit merges the spatiotemporal dimensions of the frame-level instance embeddings before inputting them into the instance encoder. The frame-level instance embeddings are then input into the instance encoder for spatiotemporal consistency modeling and spatiotemporal representation learning of the target, outputting target instance features. In the instance decoder, the spatiotemporal information aggregation unit randomly initializes a video-level query embedding and performs multi-layer feature interactions with the target instance features in the decoder, finally outputting the video-level instance embedding.

[0106] The instance sequence segmentation unit is used to embed video-level instances through the transformation of the classification head network and the masking head network to obtain the binary classification probability and conditional convolution kernel parameters for each frame. Then, the conditional convolution kernel and the image frame features generated by the cross-modal feature pyramid network are convolved to generate multiple candidate mask prediction sequences for the video sequence.

[0107] Therefore, this embodiment treats the segmentation of referential video targets as a query-based sequence prediction problem and proposes a two-level query mechanism for spatiotemporal consistency modeling and spatiotemporal feature learning of the targets. Specifically, in the first stage, this embodiment uses a frame-level spatial information extraction unit to independently interact with each frame in the video sequence in the spatial dimension using linguistic features as query embeddings, generating instance embeddings containing target spatial information. In the second stage, this embodiment uses a spatiotemporal information aggregation unit to interact with the video-level learnable query embeddings and the instance embeddings generated in the first stage in the spatiotemporal dimension, generating video-level instance embeddings with spatiotemporal representation information. Finally, the video-level instance embeddings are linearly transformed into conditional convolution parameters and convolved with each frame in the video sequence to generate a mask prediction sequence for the targets.

[0108] After the model is built, it will be trained. Specifically, the Ref-COCO and Ref-YouTube-VOS datasets will be downloaded from the internet. The Ref-COCO dataset is an image-to-text segmentation dataset containing image-text description sample pairs, and the Ref-YouTube-VOS dataset is a video-to-video target segmentation dataset containing training samples consisting of video sequences and corresponding language description files. Training samples will be extracted from these datasets to train the model. Each video segment input to the model contains T = 5 image frames randomly sampled from the same video sequence. Data augmentation strategies, including random horizontal flipping, random cropping, and photometric distortion, will be applied to these image frames. All input images will be randomly resized before being input into the model, with the preferred image size range being [288, 640].

[0109] When training a model using a dataset, the AdamW optimizer can be used to calculate gradients and update network parameters, and supervised training of the model can be achieved by minimizing the matching loss; whereby the function for minimizing the matching loss is:

[0110] L match (y,y p )=λ cls L cls (y,y p )+λ mask L mask (y,y p )

[0111] In the formula, y represents the sequence segmentation mask annotation information (i.e., ground truth) of the target provided in the dataset. p L represents the predicted target segmentation sequence. match Let λ represent the total loss function of the model. cls L represents the weight parameters of the classification loss function. cls To represent the classification loss, the focus loss function, λ, can be preferably used. mask L represents the weight parameters of the segmentation loss function. mask Let represent the segmentation loss, which is the sum of the DICE loss function and the focus loss function.

[0112] The model training in this embodiment can be divided into two stages: (1) Pre-training stage: The model is first pre-trained on the referential image segmentation dataset, which can be the RefCOCO dataset. That is, the model can be trained on the RefCOCO dataset for 10 epochs, where the learning rate of the 6th and 8th epochs can be multiplied by a coefficient of 0.1 respectively; (2) Fine-tuning stage: The pre-trained model is fine-tuned on the RefYouTubeVOS training set for 6 epochs, where the learning rate of the 3rd and 5th epochs can be multiplied by a coefficient of 0.1 respectively. During the training process, the initial learning rate of the visual encoder can preferably be set to 5×10. -5 The initial learning rate of the text encoder can preferably be set to 1×10. -5 The initial learning rate for other parts of the model can preferably be set to 1×10. -4 Furthermore, during model training, an auxiliary loss function can be used to supervise the instance embeddings generated in the first stage of the spatiotemporal hierarchical query.

[0113] Since the model provided in this embodiment generates multiple candidate mask prediction sequences for the video sequence, and only one sequence in the entire video sequence is the target object referred to by the text, a binary matching method can be used to find the best prediction result from the multiple candidate mask prediction sequences by minimizing the matching loss. The final loss is then calculated, and an optimizer is used to perform gradient backpropagation and parameter updates on the model, thereby achieving optimized training. Finally, in the testing phase, for each mask prediction sequence result, the average classification probability value of all frames is calculated, and the target mask prediction sequence with the highest average classification probability value is selected as the final video target segmentation result.

[0114] It should be noted that the step numbers in the embodiments of this application do not limit the order of operations in the technical solution of this application.

[0115] It should be noted that those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the apparatus and each unit described above can be referred to the corresponding process in the aforementioned embodiments of the video target segmentation method, and will not be repeated here.

[0116] The apparatus provided in the above embodiments can be implemented as a computer program, which can be used in, for example... Figure 5 The image shown refers to the video target segmentation device that is running on it.

[0117] This application also provides a device for segmenting a video target, comprising: a memory, a processor, and a network interface connected via a system bus, wherein the memory stores at least one instruction, and the processor loads and executes the at least one instruction to implement all or part of the steps of the aforementioned method for segmenting a video target.

[0118] The network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0119] A processor can be a CPU, or other general-purpose processors, DSPs (Digital Signal Processors), ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor, or any conventional processor. The processor is the control center of a computer device, connecting all parts of the computer device through various interfaces and lines.

[0120] Memory can be used to store computer programs and / or modules. The processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for at least one function (such as video playback, image playback, etc.), etc.; the data storage area can store data created based on the use of the mobile phone (such as video data, image data, etc.). Furthermore, memory can include high-speed random access memory (RAM), and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, SMC (SmartMediaCard), SD (SecureDigital) cards, flash memory cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0121] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements all or part of the steps of the aforementioned method for segmenting referred video targets.

[0122] The embodiments of this application can implement all or part of the aforementioned processes, or they can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various methods described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, ROM (Read-Only memory), RAM (Random Access Memory), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0123] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, servers, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0124] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0125] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0126] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for segmenting a target in a video, characterized in that, Includes the following steps: Feature extraction is performed on the target video sequence and the target language description content to obtain image features, sentence-level text features, and word-level text features; Feature extraction is performed on image features and word-level text features to obtain image frame features, and frame-level spatial information is extracted from image features and sentence-level text features to generate frame-level instance embeddings; The spatiotemporal dimensions of the frame-level instance embeddings are merged, and the spatiotemporal consistency modeling and spatiotemporal representation learning are performed on the merged frame-level instance embeddings to generate target instance features; Spatiotemporal information aggregation processing is performed on the target instance features and the preset video-level query embedding to generate video-level instance embeddings; A linear transformation is performed on the video-level instance embedding to obtain the classification probability and conditional convolution kernel. Then, a convolution operation is performed on the conditional convolution kernel and the image frame features to generate multiple mask prediction sequences. The target mask prediction sequence is selected as the video target segmentation result from multiple mask prediction sequences based on classification probability. The step of performing spatiotemporal consistency modeling and spatiotemporal representation learning on the merged frame-level instance embeddings to generate target instance features includes: Add position encoding to the merged frame-level instance embedding to form a new frame-level instance embedding; Based on the instance encoder, spatiotemporal consistency modeling and spatiotemporal representation learning are performed on the new frame-level instance embedding to generate target instance features. The instance encoder includes a multi-head self-attention module, an addition module, a normalization module, and a feedforward network. The process of aggregating spatiotemporal information of the target instance features and the preset video-level query embedding to generate a video-level instance embedding includes: A video-level query embedding is generated through random initialization; The instance decoder performs spatiotemporal information aggregation processing on the target instance features and video-level query embeddings to generate video-level instance embeddings. The instance decoder includes a multi-head mutual attention module, an addition module, a normalization module, a multi-head self-attention module, and a feedforward network.

2. The method for segmenting a referred video target as described in claim 1, characterized in that, The linear transformation of the video-level instance embedding to obtain the classification probability and conditional convolution kernel includes: The classification head network performs a linear transformation on the video-level instance embedding to obtain the classification probability. The classification head network includes a single-layer linear transformation layer. The video-level instance embedding is linearly transformed by a masking terminal network to obtain a conditional convolution kernel. The masking terminal network includes multiple linear transformation layers.

3. The method for segmenting a referred video target as described in claim 1, characterized in that, The step of selecting the target mask prediction sequence from multiple mask prediction sequences based on classification probability as the video target segmentation result includes: For each mask prediction sequence, the average classification probability of the mask prediction sequence is calculated based on the classification probability corresponding to each frame; The mask prediction sequence with the highest average classification probability is selected from multiple mask prediction sequences as the target mask prediction sequence; The target mask prediction sequence is used as the video target segmentation result.

4. The method for segmenting a referred video target as described in claim 1, characterized in that, Before the step of feature extraction of image features and word-level text features, the method further includes: After extracting features from the image features using a Transformer encoder, new image features are generated. Based on the new image features, the step of extracting features from the image features and word-level text features is performed. The Transformer encoder includes a multi-scale deformable self-attention module, an addition module, a normalization module, and a feedforward network.

5. The method for segmenting a referred video target as described in claim 1, characterized in that, The feature extraction process for the target video sequence and the target language description content yields image features, sentence-level text features, and word-level text features, including: Image features are obtained by performing multi-scale feature extraction on the target video sequence using a visual encoder; Text features are extracted from the target language description by a text encoder to obtain sentence-level and word-level text features.

6. A device for segmenting video targets, characterized in that, include: The feature extraction unit is used to extract features from the target video sequence and the target language description content, respectively, to obtain image features, sentence-level text features and word-level text features; The frame-level spatial information extraction unit is used to extract features from image features and word-level text features to obtain image frame features, and to extract frame-level spatial information from image features and sentence-level text features to generate frame-level instance embeddings. The spatiotemporal information aggregation unit is used to merge the spatiotemporal dimensions of frame-level instance embeddings and to perform spatiotemporal consistency modeling and spatiotemporal representation learning on the merged frame-level instance embeddings to generate target instance features. Spatiotemporal information aggregation processing is performed on the target instance features and the preset video-level query embedding to generate video-level instance embeddings; The instance sequence segmentation unit performs a linear transformation on the video-level instance embedding to obtain classification probabilities and conditional convolution kernels. It then performs convolution operations on the conditional convolution kernels and image frame features to generate multiple mask prediction sequences. Based on the classification probabilities, it selects the target mask prediction sequence from the multiple mask prediction sequences as the video target segmentation result. Specifically, the spatiotemporal information aggregation unit is used for: The positional encoding is added to the merged frame-level instance embedding to form a new frame-level instance embedding; based on the instance encoder, spatiotemporal consistency modeling and spatiotemporal representation learning are performed on the new frame-level instance embedding to generate target instance features. The instance encoder includes a multi-head self-attention module, an addition module, a normalization module, and a feedforward network. A video-level query embedding is generated through random initialization; the target instance features and the video-level query embedding are aggregated based on the instance decoder to generate a video-level instance embedding. The instance decoder includes a multi-head mutual attention module, an addition module, a normalization module, a multi-head self-attention module, and a feedforward network.

7. A device for segmenting video targets, characterized in that, include: A memory and a processor, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the video target segmentation method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that, when executed by a processor, implements the specified video target segmentation method according to any one of claims 1 to 5.

Images