A motion representation video segmentation method and system based on task decoupling

Abstract

This invention discloses a motion representation video segmentation method and system based on task decoupling. The method involves inputting video and descriptive text into a decoupled motion representation video segmentation framework for processing. Specifically, a frozen video instance segmenter receives the video, tracks and segments all candidate objects within it, generates frame queries and video queries, and inputs them into a motion representation encoder. Motion word features and static word features are extracted from the descriptive text and input into the motion representation encoder. The motion representation encoder interacts with the frame queries, video queries, motion word features, and static word features to generate motion representation features. Simultaneously, the motion query is initialized using the video query. The initialized motion query and the motion representation features are then input into a motion query decoder for decoding, resulting in a decoded motion query. Based on the decoded motion query, the object specified in the descriptive text is identified. Compared to existing methods, this method achieves better performance with lower training costs.

Description

Technical Field

[0001] This invention relates to the field of video segmentation technology, and in particular to a method and system for motion representation video segmentation based on task decoupling. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] RVOS, or Video Object Segmentation, is a multimodal video task designed to segment a specified target object within a video based on a given linguistic representation. Current RVOS methods can be categorized into two types: multi-stage and single-stage methods. Multi-stage methods employ image-level segmentation models to process each frame of a video segment independently. A representative work is URVOS (Unified Referring Video Object Segmentation), which first performs initial mask prediction using an image-level segmentation model and then refines the mask through semi-supervised VOS propagation. Inspired by DETR (Detection Transformer), single-stage methods based on the Transformer architecture have been widely proposed. Existing techniques model the RVOS task as a sequence prediction problem, processing video and text simultaneously through a single multimodal Transformer model. The proposed MTTR (Multimodal Tracking Transformer) is trained end-to-end, without text-related inductive bias components or additional masking optimization post-processing steps. The proposed ReferFormer introduces a small set of target queries related to language input conditions, where all queries are forced to search only for the object they refer to, and object tracking is naturally achieved by connecting the corresponding queries across frames.

[0004] Existing RVOS datasets primarily focus on single salient objects and static attributes, which possess clear invariant features. Therefore, the aforementioned RVOS methods achieve good results on these datasets. To overcome this limitation of focusing only on single salient objects and static attributes, existing techniques propose Motion Representation Video Segmentation (MEVS), a task that emphasizes the importance of spatiotemporal motion features in videos. The dataset proposed with this paper, called MeViS, contains numerous motion expressions for representing target objects in complex environments, and a single expression may refer to multiple target objects. Current RVOS methods are designed for traditional RVOS datasets, and their performance significantly degrades when applied to MEVS tasks.

[0005] Because the dataset contains multi-object and inter-frame motion correlation features, researchers have attempted to inject textual information into Video Instance Segmentation (VIS) models to address this new task. The VIS task aims to simultaneously detect, segment, and track all object instances in a video. A major challenge in MEVS is the accurate localization and alignment of motion capture across time, thus offline VIS methods have been largely favored. LMPM (Language-guided Motion Perception and Matching) replaces random initial queries with linguistic conditional queries and selects matching object trajectories through thresholding, simply transforming the offline VIS model VITA into a MEVS model. The DsHmp framework introduces an expression decoupling method, using Mask2Former to segment potential targets based on static cues and leveraging motion cues to segment referential targets; it also proposes a hierarchical motion perception module to effectively capture temporal information at different time scales and uses a contrastive learning method to distinguish motion between visually similar objects.

[0006] DsHmp utilizes static cues to segment as many candidate objects as possible, but it lacks the true labels for all objects specified by the static cues. For example, given the text description "a bird flying away," DsHmp trains Mask2Former to segment all birds first, but in reality, only the birds flying away are supervised. This contradicts the goal of decoupling video segmentation into static perception and motion perception. Furthermore, existing methods train the entire framework from scratch, making optimization difficult and training costs high. Summary of the Invention

[0007] To overcome the shortcomings of the existing technologies, this invention provides a motion representation video segmentation method and system based on task decoupling. It constructs a decoupled motion representation video segmentation framework based on the existing VIS model, emphasizing the decoupling of the task into video instance segmentation and motion representation understanding. Compared with previous methods, it achieves better performance at a lower training cost.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] In a first aspect, the present invention provides a motion representation video segmentation method based on task decoupling, comprising:

[0010] Get the video and description text;

[0011] The video and the descriptive text are respectively input into the decoupled motion expression video segmentation framework for processing, and the objects specified by the descriptive text in the video are identified.

[0012] The decoupled motion expression video segmentation framework includes a frozen video instance segmenter and a decoupled motion expression video segmentation module. The decoupled motion expression video segmentation module includes a motion expression encoder and a motion query decoder. The frozen video instance segmenter receives the video and tracks and segments all candidate objects in the video, generates frame queries and video queries, and inputs them into the motion expression encoder. It extracts motion word features and static word features from the description text and inputs them into the motion expression encoder. The motion expression encoder interacts with the frame queries, video queries, motion word features, and static word features to generate motion expression features. At the same time, it initializes the motion query using the video query, inputs the initialized motion query and the motion expression features into the motion query decoder for decoding, and obtains the decoded motion query. Based on the decoded motion query, it identifies the object specified in the description text.

[0013] In a further technical solution, the decoupled motion representation video segmentation framework also includes a classification head and a mask generation head. The decoded motion query is input to the classification head and the mask generation head respectively to obtain the classification score and mask embedding.

[0014] In a further technical solution, the target decoder in the frozen video instance segmenter generates mask features, and the mask features are multiplied with the mask embedding to obtain a predicted mask.

[0015] A further technical solution involves inputting the descriptive text into a text encoder to extract word features, and then inputting the word features into a text decomposer to extract moving word features and static word features respectively.

[0016] A further technical solution involves the motion expression encoder interactively generating motion expression features by combining frame queries, video queries, and moving word features with static word features. Specifically, this process is as follows:

[0017] Enhanced frame queries are obtained by injecting static word features into frame queries using cross-attention, and enhanced video queries are obtained by injecting motion word features into video queries using cross-attention.

[0018] Word features are pooled to obtain sentence features. Serial cross-attention is used to interact the sentence features with the enhanced frame query to obtain sentence features that integrate frame-level static target information.

[0019] By leveraging serial cross-attention to interact sentence features that integrate frame-level static target information with enhanced video queries, motion expression features are obtained.

[0020] A further technical solution involves using video queries to initialize motion queries, specifically: the frozen video instance segmenter outputs the classification score corresponding to each video query, and a set number of queries with the highest classification scores are selected from the video queries to form a motion query.

[0021] In a further technical solution, the motion query decoder decodes the motion query, the word features output by the text encoder, and the motion expression features to obtain the decoded motion query.

[0022] Secondly, the present invention provides a motion representation video segmentation system based on task decoupling, comprising:

[0023] The data acquisition module is configured to acquire video and descriptive text.

[0024] The segmentation prediction module is configured to input the video and descriptive text into the decoupled motion representation video segmentation framework for processing, and identify the object specified by the descriptive text in the video.

[0025] The decoupled motion expression video segmentation framework includes a frozen video instance segmenter and a decoupled motion expression video segmentation module. The decoupled motion expression video segmentation module includes a motion expression encoder and a motion query decoder. The frozen video instance segmenter receives the video and tracks and segments all candidate objects in the video, generates frame queries and video queries, and inputs them into the motion expression encoder. It extracts motion word features and static word features from the description text and inputs them into the motion expression encoder. The motion expression encoder interacts with the frame queries, video queries, motion word features, and static word features to generate motion expression features. At the same time, it initializes the motion query using the video query, inputs the initialized motion query and the motion expression features into the motion query decoder for decoding, and obtains the decoded motion query. Based on the decoded motion query, it identifies the object specified in the description text.

[0026] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the motion representation video segmentation method based on task decoupling as described in the first aspect.

[0027] Fourthly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the motion representation video segmentation method based on task decoupling as described in the first aspect.

[0028] The above one or more technical solutions have the following beneficial effects:

[0029] This invention constructs a decoupled motion representation video segmentation framework based on an existing query-based VIS model, emphasizing the decoupling of the task into video instance segmentation and motion representation understanding. Specifically, a frozen video instance segmenter is first used to track and segment all candidate objects, and then the specified object is identified based on motion representation. This requires training the decoupled motion representation video segmentation module with fewer parameters. To further encode visually enhanced text information, a motion representation encoder is proposed. This encoder interacts with frame queries and static cues to focus on potential candidate objects of a specific category, and interacts with video queries and motion cues to focus on a specific motion target in a category-independent manner. To reduce optimization difficulty, a method for initializing motion queries based on video queries is proposed. Video queries are selected to initialize motion queries based on the Top K classification scores to eliminate interference from background queries. Finally, a mask and classification results are generated through a motion query decoder.

[0030] This invention proposes a simple yet effective decoupled motion representation video segmentation framework, DMVS, which is built on the existing query-based VIS model and emphasizes decoupling motion representation video segmentation into video instance segmentation and motion representation understanding.

[0031] This invention proposes a motion representation encoder that interacts with frame queries and video queries using static and motion cues, respectively, to further encode motion representations enhanced with visual information.

[0032] This invention proposes a novel query initialization strategy, which uses video queries guided by classification priors to initialize motion queries, greatly reducing the optimization difficulty and training cost. Attached Figure Description

[0033] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0034] Figure 1 This is an architecture diagram of the decoupled motion representation video segmentation framework according to an embodiment of the present invention;

[0035] Figure 2 This is a qualitative comparison diagram of the motion representation video segmentation method of this invention and DsHmp on the MeViS validation set. Detailed Implementation

[0036] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0037] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0038] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0039] Terminology Explanation: Motion representation video segmentation aims to segment a referred video object based on an input motion description. Task decoupling refers to decoupling the motion representation video segmentation task into video instance segmentation and motion representation understanding.

[0040] Motion expression video segmentation (MEVS) aims to segment objects in a video based on an input motion description. Compared to traditional referring video object segmentation (RVOS), which focuses on a single salient object and static representation, it places greater emphasis on motion representation and multi-object segmentation, making it more challenging. Classical RVOS models perform poorly on this task, and existing work has attempted to accomplish this by simply injecting textual information into video instance segmentation (VIS) models. However, this requires retraining the entire model, which is difficult to optimize and yields poor performance.

[0041] Example 1

[0042] like Figure 1 As shown in the figure, this embodiment discloses a motion representation video segmentation method based on task decoupling, which includes the following steps:

[0043] S1: Obtain the video and description text;

[0044] S2: The video and descriptive text are respectively input into the decoupled motion expression video segmentation framework for processing to identify the object specified in the descriptive text in the video. The decoupled motion expression video segmentation framework includes a frozen video instance segmenter and a decoupled motion expression video segmentation module. The decoupled motion expression video segmentation module includes a motion expression encoder and a motion query decoder. The frozen video instance segmenter receives the video and tracks and segments all candidate objects in the video, generates frame queries and video queries, and inputs them into the motion expression encoder. Motion word features and static word features are extracted from the descriptive text and input into the motion expression encoder. The motion expression encoder interacts with the frame query, video query, motion word features, and static word features to generate motion expression features. At the same time, the motion query is initialized using the video query. The initialized motion query and the motion expression features are input into the motion query decoder for decoding to obtain the decoded motion query. The object specified in the descriptive text is identified based on the decoded motion query.

[0045] The Decoupled Motion Representation Video Segmentation Framework (DMVS) employs a motion representation video segmentation method divided into three stages. The first stage uses a frozen video instance segmenter (VITA) as the video instance segmenter to identify, segment, and track all targets. The second stage is based on the frame query Q generated by VITA. f And video search Q v A motion expression encoder is used to query the frame Q. f Video search Q v With motion cue F m Static Clues F s Interact to generate motion representation features F that enhance visual information me The third stage uses video queries guided by classification priors to initialize motion query Q. m Then, a motion query decoder is used to decode the motion query layer by layer for classification and mask prediction. Finally, the motion query Q... m The mask embedding is generated by the mask generation head, and then the mask embedding is combined with the mask feature F. mask Multiply to obtain the prediction mask, and select the scores above the threshold as the output.

[0046] (I) Video Instance Splitter (VITA)

[0047] In this embodiment, VITA is used as the video instance segmenter, which is an offline VIS method built on an image instance segmentation model. VITA achieves video-level understanding by associating frame-level target queries without using spatiotemporal backbone features. Specifically, given a video input of H×W resolution consisting of T frames, VITA first uses Mask2Former to process each frame in a completely frame-independent manner, generating two parts: 1) a frame query containing target information for each frame. Where N f C is the number of target queries, and C is the number of channels. 2) Generate mask features (image features) from the target decoder. Where S is the stride of the feature map. Then, the object encoder establishes temporal communication between frame queries using a self-attention mechanism along the time axis. Finally, the object decoder extracts information from frame queries Q. f Aggregating information for video search Ultimately, it is used to predict the category and mask of an object in a video in one go.

[0048] The Video Instance Segmenter (VITA) consists of three parts: a frame-level detector, a target encoder, and a target decoder. First, VITA uses the Mask2Former frame-level detector to process each frame in a completely frame-independent manner, outputting a frame query. Then, the target encoder uses a window self-attention mechanism to perform inter-frame interactions between frame queries. Finally, the target decoder aggregates the frame query information into the video query to output the final video query.

[0049] (II) Decoupling the motion expression video segmentation module

[0050] (1) Motion expression encoder

[0051] The interaction between textual and visual features before query decoding can enhance information for each modality. This invention proposes that frame queries and video queries can provide sufficient object-specific visual information without interacting with dense backbone features. Frame queries independently represent all objects within each frame and require static information to determine object categories. Video queries, on the other hand, integrate global contextual information and are better suited for interacting with motion information in the temporal domain. Therefore, this invention decomposes a given expression into static and action words, which serve as cues for static-aware frame queries and motion-aware video queries.

[0052] like Figure 1 As shown, for the descriptive text "the cat turning around to play with its toys", the text encoder BERT is first used to extract word features. Where K wThis represents the longest word count in a sentence within the dataset. A text decomposer is used to identify nouns, adjectives, and prepositions in the sentence from word features, obtaining static cues (static word features), such as "cat" and "toy." Simultaneously, verbs and adverbs are extracted to obtain motion cues (motion word features), such as "turn around" and "play." Therefore, static word features are extracted. and motion word features Where K s / K m Indicates the length of static / dynamic words. Word feature F w For static word features F s Add motion word features F m Sentence features (sentence-level features) F are obtained by average pooling word features. se ∈R C .

[0053] First, static word features and dynamic word features are used to enhance the target query. Specifically, a cross-attention mechanism is used to inject static cues into the frame query:

[0054]

[0055] in, It is a frame query enhanced by static clues.

[0056] Similarly, motion cues are injected into video queries through cross-attention:

[0057]

[0058] in, It is a video query enhanced by motion cues.

[0059] DShMP only allows static and dynamic cues to interact with the target query at the start of initialization. Randomly initialized queries do not contain any object cues, thus failing to extract valid textual information. In fact, it relies on the combination of all textual features from the decoding phase to establish modal associations. This invention uses frame queries and video queries that already contain all object information, combining them with static and motion cues respectively, fully utilizing the crucial role of separating textual features.

[0060] Then, the present invention uses a target query containing specific cueing clues to enhance sentence features. Specifically, it uses serial cross-attention encoding to encode existing sentence features:

[0061]

[0062] Among them, F′ se ∈R C F represents sentence features that integrate frame-level static target information.me ∈R C It represents sentence features that integrate video-level moving target information, and it is also the final encoded motion representation used in the subsequent decoding process.

[0063] (2) Motion Query Decoder

[0064] The prerequisite for motion query decoding is the initialization of the motion query. Due to a lack of additional information, the target query in the VIS model is primarily initialized randomly. RVOS methods typically treat language as the query, for example, by replicating sentence embeddings to initialize the target query. However, regardless of the method used, optimization remains challenging due to the significant gap between the final decoded output query and the target. Ideally, RVOS would primarily select the VIS output based on textual information. Therefore, this invention proposes initializing the motion query using video queries. Since the default number of queries in VITA is relatively large, this invention selects the target query based on the Top K classification scores to eliminate interference from background queries.

[0065] Q m =TopK(Q v ,S v N m )

[0066] in, It's a sports query, S v The Top K represents the classification score corresponding to each video query output by the VIS model, where Q is the highest score among all video queries. v Select N with the highest classification score m This initialization method allows the motion query to encompass queries corresponding to almost all instances. The decoding process shifts from identifying and segmenting objects from scratch to matching and refining the most suitable object from all instances. This enables the optimization process to focus on the referential subtask, thereby reducing optimization difficulty and training cost.

[0067] Next, a motion query decoder is employed to extract information from action representation features, rather than frame or video queries. Specifically, the action query interacts with word features using a cross-attention mechanism to extract the most basic lexical-level information and determine the importance of each word. Then, it interacts with visually enhanced sentence embeddings through a cross-attention mechanism to extract the overall action representation.

[0068] Q′ m =Decoder(Q) m ,F w ,F me )

[0069] Among them, Q′ mThe motion query is decoded and used for final classification and mask prediction.

[0070] A decoder consists of two cross-attention layers, one self-attention layer, and one feedforward neural network layer. The motion query decoder effectively captures video context and aggregates motion representation information into motion queries. Therefore, the motion query decoder exhibits fast convergence speed while achieving high accuracy, and significantly reduces training memory compared to previous RVOS methods.

[0071] (3) Model training and inference

[0072] Finally, the motion query Q′ output from the motion query decoder m Passed to the classification header H c and mask generation header H m :

[0073] S = H c (Q′ m )

[0074] in, It is a binary classification score, H c It is a single-layer linear layer.

[0075] M = F mask ·H m (Q′ m )

[0076] in, It is a prediction mask, H m It consists of three MLP layers to generate mask embeddings, and · represents the dot product operation.

[0077] Finally, the masks with classification scores greater than the threshold from all predicted masks are output to identify the object mask specified in the description text. In this embodiment, the threshold is chosen to be 0.7.

[0078] This invention adds the proposed DMVS module to the video instance segmenter, and the entire model is trained end-to-end. It's important to note that the frame-level output of Mask2Former and the video-level output of VITA are not used for loss calculation; only the video-level output loss of the DMVS module is considered. The total learning loss of the model is as follows:

[0079] L total =L mask +λ cls L cls

[0080] Among them, L mask It is the mask loss, composed of binary cross-entropy loss and Dice loss; L cls It is classification loss.

[0081] This invention uses the entire video as input to the inference process, and DMVS learns to represent the motion query of the referential instance throughout the video. In traditional object video segmentation, when processing datasets with single-objective expressions, the mask with the highest prediction score is selected as the final prediction result. For the MeViS dataset with multi-objective expressions, the mask with a prediction confidence score greater than a confidence threshold σ is selected as the final prediction result.

[0082] This invention presents a Decoupled Motion Expression Video Segmentation (DMVS) method, built upon the existing VIS model, emphasizing the decoupling of the task into video instance segmentation and motion expression understanding. It first uses a frozen video instance segmenter to track and segment all candidate objects, then identifies the specified object based on motion expression, requiring only training a DMVS module with fewer parameters. The DMVS module proposes a motion expression encoder that interacts with frame and video queries using static and motion cues respectively to further encode visually enhanced motion expressions. A novel query initialization strategy is proposed, utilizing classification prior-guided video queries to initialize motion queries, significantly reducing optimization difficulty. Compared to the best existing techniques, this invention achieves better performance at a lower training cost, as shown in the table below:

[0083] Table 1

[0084] method Number of learnable parameters Training uses GPU memory Training time J&F Indicator DsHmp 102.3M 22G 17 hours 46.4 DMVS 12.7M 6G 7 hours 48.6

[0085] Extensive experiments on three benchmark datasets demonstrate that the DMVS of this invention outperforms the state-of-the-art method DsHmp in both qualitative and quantitative metrics. Figure 2 The present invention method is qualitatively compared with the main comparison method DsHmp on the MeViS validation set.

[0086] Example 2

[0087] This embodiment discloses a motion representation video segmentation system based on task decoupling, including:

[0088] The data acquisition module is configured to acquire video and descriptive text.

[0089] The segmentation prediction module is configured to input the video and descriptive text into the decoupled motion representation video segmentation framework for processing, and identify the object specified by the descriptive text in the video.

[0090] The decoupled motion expression video segmentation framework includes a frozen video instance segmenter and a decoupled motion expression video segmentation module. The decoupled motion expression video segmentation module includes a motion expression encoder and a motion query decoder. The frozen video instance segmenter receives the video and tracks and segments all candidate objects in the video, generates frame queries and video queries, and inputs them into the motion expression encoder. It extracts motion word features and static word features from the description text and inputs them into the motion expression encoder. The motion expression encoder interacts with the frame queries, video queries, motion word features, and static word features to generate motion expression features. At the same time, it initializes the motion query using the video query, inputs the initialized motion query and the motion expression features into the motion query decoder for decoding, and obtains the decoded motion query. Based on the decoded motion query, it identifies the object specified in the description text.

[0091] Example 3

[0092] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method of Embodiment 1.

[0093] Example 4

[0094] The purpose of this embodiment is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the method of Embodiment 1.

[0095] The steps and methods involved in the apparatuses of Embodiments 3 and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0096] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0097] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0098] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A motion representation video segmentation method based on task decoupling, characterized in that, include: Get the video and description text; The video and the descriptive text are respectively input into the decoupled motion expression video segmentation framework for processing, and the objects specified by the descriptive text in the video are identified. The decoupled motion expression video segmentation framework includes a frozen video instance segmenter and a decoupled motion expression video segmentation module. The decoupled motion expression video segmentation module includes a motion expression encoder and a motion query decoder. The frozen video instance segmenter receives the video and tracks and segments all candidate objects in the video, generates frame queries and video queries, and inputs them into the motion expression encoder. It extracts motion word features and static word features from the description text and inputs them into the motion expression encoder. The motion expression encoder interacts with the frame queries, video queries, motion word features, and static word features to generate motion expression features. At the same time, it initializes the motion query using the video query, inputs the initialized motion query and the motion expression features into the motion query decoder for decoding, and obtains the decoded motion query. Based on the decoded motion query, it identifies the object specified in the description text.

2. The motion representation video segmentation method based on task decoupling as described in claim 1, characterized in that, The decoupled motion representation video segmentation framework also includes a classification head and a mask generation head. The decoded motion query is input into the classification head and the mask generation head respectively to obtain the classification score and mask embedding.

3. The motion representation video segmentation method based on task decoupling as described in claim 2, characterized in that, In the frozen video instance segmenter, the target decoder generates mask features, and the mask features are multiplied with the mask embedding to obtain a predicted mask.

4. The motion representation video segmentation method based on task decoupling as described in claim 1, characterized in that, The descriptive text is input into a text encoder to extract word features, and the word features are then input into a text decomposer to extract moving word features and static word features respectively.

5. The motion representation video segmentation method based on task decoupling as described in claim 1, characterized in that, The motion representation encoder interacts with frame queries, video queries, and moving word features and static word features to generate motion representation features. Specifically: Enhanced frame queries are obtained by injecting static word features into frame queries using cross-attention, and enhanced video queries are obtained by injecting motion word features into video queries using cross-attention. Word features are pooled to obtain sentence features. Serial cross-attention is used to interact the sentence features with the enhanced frame query to obtain sentence features that integrate frame-level static target information. By leveraging serial cross-attention to interact sentence features that integrate frame-level static target information with enhanced video queries, motion expression features are obtained.

6. The motion representation video segmentation method based on task decoupling as described in claim 1, characterized in that, The process of initializing motion queries using video queries is as follows: the frozen video instance segmenter outputs the classification score corresponding to each video query, and a set number of queries with the highest classification scores are selected from the video queries to form a motion query.

7. The motion representation video segmentation method based on task decoupling as described in claim 6, characterized in that, The motion query decoder decodes the motion query, the word features output by the text encoder, and the motion expression features to obtain the decoded motion query.

8. A motion representation video segmentation system based on task decoupling, characterized in that, include: The data acquisition module is configured to acquire video and descriptive text. The segmentation prediction module is configured to input the video and descriptive text into the decoupled motion representation video segmentation framework for processing, and identify the object specified by the descriptive text in the video. The decoupled motion expression video segmentation framework includes a frozen video instance segmenter and a decoupled motion expression video segmentation module. The decoupled motion expression video segmentation module includes a motion expression encoder and a motion query decoder. The frozen video instance segmenter receives the video and tracks and segments all candidate objects in the video, generates frame queries and video queries, and inputs them into the motion expression encoder. It extracts motion word features and static word features from the description text and inputs them into the motion expression encoder. The motion expression encoder interacts with the frame queries, video queries, motion word features, and static word features to generate motion expression features. At the same time, it initializes the motion query using the video query, inputs the initialized motion query and the motion expression features into the motion query decoder for decoding, and obtains the decoded motion query. Based on the decoded motion query, it identifies the object specified in the description text.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the motion representation video segmentation method based on task decoupling as described in any one of claims 1-7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the motion representation video segmentation method based on task decoupling as described in any one of claims 1-7.

Images