Reference video segmentation method for complex motion description of body-robot

Abstract

The application discloses a reference video segmentation method under complex action description of a body robot, which is based on a DINO-SAM model and constructs a reference video segmentation framework under complex text description. The framework contains three modules, namely, a DINO-SAM segmentation module, an action perception aggregation module and a text-label matching module. The DINO-SAM module utilizes the powerful segmentation capability of SAM, performs preliminary segmentation on the video through text prompts, and generates object labels. The action perception aggregation module can effectively capture object action information at different time scales and enhance the global understanding of the video. The text-label matching module enhances the consistency between the segmentation result and the text description. The method effectively improves the segmentation performance, enhances the perception and aggregation capability of the object action, realizes accurate matching between the language description and the object, improves the generalization capability and the few-shot detection capability.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of robot vision processing, and in particular, relates to a reference video segmentation method for complex action description of embodied robots. BACKGROUND

[0002] An embodied robot is generally defined as an intelligent agent with a physical body and the ability to interact with the environment. The core lies in "intelligence" rather than "form", that is, the robot can actively perceive the environment, understand the task requirements, plan the action path, and complete the task through interaction with the environment. For example, an embodied robot can obtain information through various sensors such as vision, touch, and force, and make autonomous decisions using reinforcement learning and other technologies. In the robot vision system, video segmentation can be used to identify and track target objects in the environment, helping the robot to better navigate and perform tasks.

[0003] Reference video segmentation is based on natural language description to accurately segment and track specific targets in a video. This technology can effectively utilize complex language descriptions to provide key information related to target motion, such as "fly away" and "circle", to understand video content. Recent datasets in this field, such as MeViS, have highlighted the key role of motion description in understanding multi-modal information.

[0004] This task was first proposed by Gavrilyuk et al. in the paper Actor and Action Video Segmentation from a Sentence, and the introduction of Ref-DAVIS17, Ref-Youtube-VOS and MeViS datasets has promoted the continuous development of this field. Many early segmentation methods in this field mainly rely on frame-by-frame object segmentation of videos, focusing on the separate processing of image features and text features, while ignoring the correlation in the time dimension. Seonguk Seo et al. proposed a large-scale reference video object segmentation benchmark and a unified segmentation framework URVOS in the paper URVOS: unified referring video object segmentation network with a large-scale benchmark. Early methods mainly rely on complex pipelines, in order to simplify the workflow, Jiannan Wu et al. adopted an end-to-end framework based on queries to decode objects in multi-modal features in the paper Language as queries for referring video object segmentation, but only focused on object information within each frame during decoding, failing to fully utilize the temporal context information across frames. The recently proposed MeViS dataset aims to highlight the importance of motion expression and points out the shortcomings of existing methods in understanding language and motion information in videos.

[0005] With the development and rise of large models pre-trained on network-scale datasets, the Segment Anything Model (SAM) proposed by Alexander Kirillov et al. in the paper Segment Anything has received widespread attention as an important large-scale model in the field of computer vision. SAM can generate high-quality object masks based on various prompt methods such as points, boxes, and text, and has strong zero-shot ability in multiple segmentation tasks. However, the current version of SAM does not support complex text prompts for reference object segmentation and other advanced tasks that require semantic understanding. SUMMARY

[0006] The purpose of the present application is to provide a method for reference video segmentation under complex action description of embodied robots, which can segment objects that meet the text description in the video through natural language prompts, helping embodied robots better navigate and perform tasks.

[0007] To achieve the above-mentioned purpose, the technical solutions adopted by the present application are as follows:

[0008] A method for reference video segmentation under complex action description of a humanoid robot, comprising the following steps:

[0009] S1, input the text prompt and the video frame by frame into the DINO-SAM segmentation module;

[0010] S2, based on one frame image in T frame video Get the video features of this segment of video with the corresponding video text description And its corresponding mask feature , wherein H, W is the length and width of the single frame image feature tensor, C is the number of channels;

[0011] S3, based on video features And language features Get the output action perception aggregation label , wherein is the length of the text;

[0012] S4, based on object action label And language features And video mask features Get the target segmentation mask.

[0013] Further, the specific process of step S2 is as follows:

[0014] S21, obtain the object detection box corresponding to the text description through the Grounding DINO model;

[0015] S22, input the object detection box and the frame image into SAM as a box prompt to obtain the image feature and mask feature of ;

[0016] S23, obtain the image feature and mask feature of the overall features of this segment of video and its corresponding mask feature .

[0017] Further, the specific process of step S3 is as follows: video features And language features get video object labels through the Transformer decoder, then get object label tracking results through Hungarian matching, and finally based on object label tracking results with language features obtaining the output of the action perception aggregation module through a self-attention layer and a multi-layer cross-attention layer .

[0018] Further, in the step S4, the language features obtaining the filtered objects through a cross-attention mechanism a plurality of learnable queries , and then outputting the filtered objects based on the action perception aggregation module labeling the learnable queries obtaining the filtered objects through a Transformer decoder, and finally multiplying the video mask features obtained in the step S2 with the filtered objects to obtain a segmentation mask of the described object.

[0019] Further, the method is based on a DINO-SAM model to construct a reference video segmentation framework under complex text description; the framework contains three modules: a DINO-SAM segmentation module, an action perception aggregation module and a text-label matching module; the DINO-SAM segmentation module realizes preliminary segmentation of video objects and generates video features and mask features ; the action perception aggregation module combines the video features and language features to process the object action trajectory in the time dimension of the video; and the text-label matching module realizes effective matching between object labels and text descriptions.

[0020] Compared with the prior art, the present application has the following beneficial effects:

[0021] The present application obtains video object labels based on video features and language features, and obtains object motion trajectories in candidate object labels that match text descriptions in a decoder, effectively improving segmentation performance, enhancing the perception and aggregation ability of object actions, and realizing precise matching between language descriptions and objects, improving generalization ability and few-shot detection ability, and effectively improving the navigation performance of embodied robots. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The present application is a method flowchart. DETAILED DESCRIPTION

[0023] The present application will be further described below in conjunction with the drawings and examples, and the modes of the present application include but are not limited to the following examples. EMBODIMENTS

[0024] ​This paper discloses a method for segmenting reference videos under complex action descriptions of embodied robots. This method is based on the DINO-SAM model and constructs a reference video segmentation framework under complex text descriptions. The framework consists of three modules: the DINO-SAM segmentation module, the action perception aggregation module, and the text-token matching module. The DINO-SAM segmentation module performs preliminary segmentation of video objects and generates video features. With mask features ; Action perception aggregation module combined with video features and language features , to process the object motion trajectory in the time dimension of the video; the text-token matching module realizes the effective matching between object tags and text descriptions.

[0025] like Figure 1 The following is the structure of the DINO-SAM segmentation module. The initial segmentation and feature generation process of the video object is as follows:

[0026] (1) First, T A frame of image in a frame video Together with the corresponding video text description, it is input into the Grounding DINO model. Grounding DINO adopts a dual encoder-single decoder architecture, and performs feature extraction through the image backbone (Swin Transformer) and the text backbone (BERT) to generate image features and text features. Subsequently, these features are fed into the feature enhancer for cross-modal fusion. The enhancer includes a self-attention layer, a deformable self-attention layer, an image-to-text cross-attention layer, and a text-to-image cross-attention layer, which can optimize the alignment of features of different modalities. The enhanced features are then input into the language-guided query selection module to select fused features that are more relevant to the input text as queries for the decoder. Finally, the image features, text features, and the retrieved cross-modal query are input into the cross-modal decoder to generate the same Object detection boxes corresponding to text descriptions;

[0027] (2) Align the object detection frame with the frame image It is input into SAM as a box hint. After being processed by the image encoder and hint encoder, the mask decoder generates Image features and mask features .

[0028] (3) After inputting the video frames into DINO-SAM, all Image features and mask features Add in the time dimension to get the video features And video mask features .

[0029] In this embodiment, the action-aware aggregation module captures the action information flow at different time scales as follows:

[0030] (1) Through the cross-attention mechanism, inject the text description clues into a learnable static query to obtain the query that extracts static and dynamic information in the text description:

[0031] ;

[0032] (2) Based on and video features , obtain video object labels through the Transformer decoder , and then match the object labels between adjacent frames through Hungarian matching to obtain object label tracking results :

[0033]

[0034] After matching, get candidate objects and their motion trajectories ;

[0035] (3) The video object labels and the language features are combined through the action aggregation block to produce the final action-aware aggregation label . The action aggregation block is composed of hierarchical layers, and each layer contains three core components: self-attention, multi-layer cross-attention, and feed-forward network (FFN) layer. Self-attention is used to calculate the long-distance dependencies in each object motion trajectory at the previous hierarchical stage:

[0036]

[0037] where denotes the number of stages of hierarchical operation of the motion aggregation block, , denotes the object label combined through self-attention with global and local information;

[0038] (4) The similarity between the language features and the motion trajectories of each object after self-attention processing can be used to highlight the objects described in the text description:

[0039]

[0040] wherein, denotes the number of stages of the multi-layer cross-attention hierarchical operation, , , denotes the object motion trajectory of the T frame and language feature. Combining with the language feature can obtain more rich object motion features :

[0041]

[0042]

[0043] wherein, is the sum over the dimension of , i.e., the row-wise sum of , which represents the sum of the influence of the language feature on the motion trajectory in each frame of ; can be regarded as the frame weight indicating the importance of the single-frame object label in the motion trajectory across frames relative to the long-term motion. Then the object motion trajectory is summed with its corresponding attention map to perform the label merging operation, which merges two adjacent cues into one cue:

[0044]

[0045] wherein, the merged object cue . After the multi-layer cross-attention goes through rounds of iterations, the output of the multi-layer cross-attention is combined with the input of the action aggregation:

[0046]

[0047] wherein, . is input to the feed-forward network (FFN) to obtain the output of the action aggregation block :

[0048]

[0049] wherein, is the input of the next stage action aggregation block, i.e., the formula: After ​After the layer is polymerized, the output of the action perception aggregation module is obtained .

[0050] In the embodiment, the text description and the object segmentation process are as follows:

[0051] (1) The same strategy as the action perception aggregation module is used to inject the language features into the learnable query through the cross attention:

[0052]

[0053] wherein, is an initialized learnable query;

[0054] (2) The query injected through the cross attention mechanism and the object action label are input into the Transformer decoder to obtain the object motion trajectory that matches the text description in the candidate object label, so as to realize accurate alignment and segmentation of the object;

[0055] (3) The object label obtained through screening is multiplied with the video mask feature output by the DINO-SAM segmentation module frame by frame to obtain the target segmentation mask.

[0056] Through the above design, the segmentation performance is effectively improved, the perception and aggregation ability of the object action are enhanced, the accurate matching between the language description and the object is realized, and the generalization ability and the few-shot detection ability are improved.

[0057] The above embodiment is only one of the preferred embodiments of the present application and should not be used to limit the protection scope of the present application. Any modification or polishing made within the main design idea and spirit of the present application without substantial meaning, which still solves the technical problems consistent with the present application, should be included in the protection scope of the present application.​

Claims

1. A reference video segmentation method for describing complex actions of an embodied robot, characterized by: The following steps are involved: S1, inputs the text prompt and video frame by frame into the DINO-SAM segmentation module; S2, based on inclusion Frame Image The video clip and the corresponding video text description are used to obtain the video features of the video and its mask features ,in 、 is the length and width of the single-frame image feature tensor, is the number of channels; S3, based on video features The text features extracted from the text prompt by the text backbone in the DINO-SAM segmentation module , get the output action-aware aggregation tag , and output object action tags ,in is the text length; the specific process is as follows: video features and text features Obtaining video object tags through the Transformer decoder , and then obtain the object tag tracking results through Hungarian matching , and finally based on the object tag tracking results With text features Obtain the object action labels output by the action-aware aggregation module through the self-attention layer and the multi-layer cross-attention layer ; S4, based on object action labeling and text features And the mask features obtained by aggregation in step S2 Get the target segmentation mask; among them, the text feature Acquired through the cross attention mechanism Learnable Queries , then based on the output action-aware aggregation tag and learnable queries , get the filtered objects through the Transformer decoder, and finally get the mask features obtained in step S2 Multiply frame by frame to obtain the segmentation mask of the described object.

2. A reference video segmentation method for complex action description of an embodied robot according to claim 1, characterized in that: The specific process of step S2 is as follows: S21, obtained through the Grounding DINO model Object detection boxes corresponding to text descriptions; S22, the object detection frame is aligned with the frame image Enter into SAM as a box prompt to get Image features and mask features ; S23, will Image features and mask features Add in the time dimension to obtain the overall video features of the video segment and its corresponding mask features .

3. The method for segmenting reference videos for complex action description of an embodied robot according to claim 2, characterized in that: This method is based on the DINO-SAM model and constructs a reference video segmentation framework under complex text descriptions. The framework consists of three modules: the DINO-SAM segmentation module, the action-aware aggregation module, and the text-token matching module. The DINO-SAM segmentation module performs preliminary segmentation of video objects and generates video features. With mask features ; Action-aware aggregation module combined with video features With text features , to process the object motion trajectory in the time dimension of the video; the text-token matching module realizes the effective matching between object tags and text descriptions.

Images