Video object segmentation method, device, equipment and storage medium

Abstract

The application discloses a video target segmentation method and device, equipment and a storage medium, and relates to the technical field of video image processing. The video target segmentation method comprises the following steps: acquiring video features of a video to be segmented and initial text features of a description text; decoupling the initial text features based on a subject decoupler to obtain subject text features and context text features; performing information interaction between the subject text features, the context text features and the video features respectively to obtain visual decoupling information and language decoupling information; and performing target segmentation on the video to be segmented according to the visual decoupling information and the language decoupling information to obtain a video segmentation result. Through the above scheme, the anti-interference capability of the same target can be improved, stable anti-interference language reference video target segmentation can be realized, and the stability and accuracy of video target segmentation can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video image processing, and particularly relates to a video target segmentation method and device, equipment and a storage medium. BACKGROUND

[0002] Language referring video target segmentation aims to segment and track the target described by text from the video. The existing language referring video target segmentation technology usually uses a structure based on a multi-modal transformer to realize the interaction of visual features and text features. However, since the multi-modal transformer easily ignores the syntax structure of the text, in fact, all the text components are mixed together without structure in the interaction process, thus causing the ambiguity of visual-language information alignment, and reducing the accuracy of video target segmentation.

[0003] In addition, due to the limitation of the diversity of training data, in most training samples, the target category referred to by the text is unique in the video, and there is no interference of the same target. However, in real scenarios, the interference of the same target is common, so the existing technology is sensitive to the interference of the same target, and there is a problem of low stability and accuracy of video segmentation.

[0004] Based on the above, how to improve the stability and accuracy of video target segmentation is a problem to be solved.

[0005] The above content is only used to assist in understanding the technical solutions of the present application, and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0006] The main purpose of the present application is to provide a video target segmentation method, device, equipment and storage medium, which aims to solve the technical problem of how to improve the stability and accuracy of video target segmentation.

[0007] To achieve the above purpose, the present application provides a video target segmentation method, which comprises:

[0008] obtaining video features of a video to be segmented and initial text features of a description text;

[0009] decoupling the initial text features based on a subject decoupler to obtain subject text features and context text features;

[0010] respectively performing information interaction of the subject text features, the context text features and the video features to obtain visual decoupling information and language decoupling information;

[0011] performing target segmentation on the video to be segmented according to the visual decoupling information and the language decoupling information to obtain a video segmentation result.

[0012] In an embodiment, the step of obtaining video features of the video to be segmented and initial text features of the description text comprises:

[0013] obtaining a video to be segmented and a description text;

[0014] encoding features of the description text based on a preset text encoder to obtain initial text features of the description text;

[0015] encoding features of the video to be segmented based on a preset video encoder to obtain video features of the video to be segmented.

[0016] In an embodiment, the description text comprises a plurality of word pieces, the initial text features comprise text features corresponding to the plurality of word pieces, and the step of decoupling the initial text features based on a subject decoupler to obtain subject text features and context text features comprises:

[0017] performing subject prediction on the word pieces in the description text based on the subject decoupler to obtain subject probabilities of the word pieces;

[0018] performing vocabulary division on the description text to obtain a plurality of word piece groups;

[0019] calculating average subject probabilities of the word piece groups and selecting a word piece group with the largest average subject probability as a subject word piece group;

[0020] taking text features corresponding to the word pieces in the subject word piece group as the subject text features and taking the initial text features as the context text features.

[0021] In an embodiment, the step of respectively interacting the subject text features and the context text features with the video features to obtain visual decoupling information and language decoupling information comprises:

[0022] interacting the subject text features with the video features between modalities based on a multi-head cross-attention mechanism and reinforcing the features after the interaction within a modality based on a multi-head self-attention mechanism to obtain first fused video features;

[0023] interacting the context text features with the video features between modalities based on a multi-head cross-attention mechanism and reinforcing the features after the interaction within a modality based on a multi-head self-attention mechanism to obtain second fused video features;

[0024] reinforcing the subject text features within a modality based on a multi-head self-attention mechanism to obtain first text features;

[0025] reinforcing the context text features within a modality based on a multi-head self-attention mechanism to obtain second text features;

[0026] splicing the first fusion video feature and the second fusion video feature to obtain visual decoupling information;

[0027] splicing the first text feature and the second text feature to obtain language decoupling information.

[0028] In an embodiment, the step of performing target segmentation on the video to be segmented according to the visual decoupling information and the language decoupling information to obtain a video segmentation result comprises:

[0029] determining a segmentation target from the video to be segmented based on a target decoder and a pixel decoder according to the visual decoupling information, the language decoupling information, and the video feature;

[0030] annotating the segmentation target in the video to be segmented to obtain a video segmentation result.

[0031] In an embodiment, after the step of performing feature encoding on the video to be segmented based on a preset video encoder to obtain a video feature of the video to be segmented, the method further comprises:

[0032] performing pre-alignment processing on the initial text feature of the description text and the video feature of the video to be segmented based on a preset video encoder to obtain a video feature after pre-alignment processing.

[0033] In an embodiment, the step of performing subject prediction on the word units in the description text based on a subject decoupler to obtain a subject probability of the word units comprises:

[0034] performing information enhancement on the initial text feature based on a multi-head self-attention mechanism, and performing layer normalization on the enhanced text feature to obtain normalized initial text feature;

[0035] performing classification mapping on the text feature corresponding to each word unit in the normalized initial text feature by using a multi-layer perception and a linear mapping layer to obtain a subject probability of the word unit.

[0036] In addition, to achieve the above object, the application further provides a video target segmentation device, which comprises:

[0037] an acquisition module configured to acquire a video feature of a video to be segmented and an initial text feature of a description text;

[0038] a subject decoupling module configured to perform decoupling on the initial text feature based on a subject decoupler to obtain a subject text feature and a context text feature;

[0039] a modal fusion module configured to interact information between the subject text feature and the context text feature respectively and the video feature, to obtain visual decoupling information and language decoupling information;

[0040] a target segmentation module configured to perform target segmentation on the video to be segmented according to the visual decoupling information and the language decoupling information, to obtain a video segmentation result.

[0041] In addition, to achieve the above-mentioned purpose, the present application also provides a video target segmentation device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the video target segmentation method as described above.

[0042] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer readable storage medium, and a computer program is stored on the storage medium, and the computer program is executed by a processor to implement the steps of the video target segmentation method as described above.

[0043] The one or more technical solutions provided by the present application have at least the following technical effects:

[0044] The video target segmentation method, device, equipment and storage medium provided by the present application specifically obtain the video feature of the video to be segmented and the initial text feature of the description text; decouple the initial text feature based on a subject decoupler to obtain the subject text feature and the context text feature; interact information between the subject text feature and the context text feature respectively and the video feature to obtain visual decoupling information and language decoupling information; and perform target segmentation on the video to be segmented according to the visual decoupling information and the language decoupling information to obtain a video segmentation result.

[0045] The present application obtains the video feature of the video to be segmented and the initial text feature of the description text, then decouples the initial text feature by using a subject decoupler to obtain the subject text feature and the context text feature, so as to realize the subsequent decoupled visual-language interaction process; then interacts information between the subject text feature and the context text feature respectively and the video feature to obtain visual decoupling information and language decoupling information, and finally performs target segmentation on the video to be segmented according to the visual decoupling information and the language decoupling information, so as to obtain a video segmentation result, thereby realizing the exchange and fusion of multi-modal information under the premise of clearly describing the subject information and the description information, realizing stable and anti-interference language reference video target segmentation, and effectively improving the stability and accuracy of video target segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the application and, together with the description, further serve to explain the principles of the application.

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the accompanying drawings required to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those of ordinary skill in the art, the other drawings can also be obtained based on these drawings without any creative work.

[0048] Figure 1 The flowchart provided for the video target segmentation method embodiment one of the present application;

[0049] Figure 2 The flowchart provided for the video target segmentation method embodiment two of the present application;

[0050] Figure 3 The flowchart provided for the video target segmentation method embodiment three of the present application;

[0051] Figure 4 The flowchart provided for the video target segmentation method embodiment four of the present application;

[0052] Figure 5 The process diagram of the video target segmentation related to the embodiments of the present application;

[0053] Figure 6 The module structure diagram of the video target segmentation device of the embodiments of the present application;

[0054] Figure 7 The device structure diagram of the hardware running environment related to the video target segmentation method in the embodiments of the present application.

[0055] The purpose realization, functional features and advantages of the present application will be further explained with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0056] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0057] In order to better understand the technical solutions of the present application, the following will be described in detail with reference to the drawings and specific embodiments of the specification.

[0058] The main solution of the embodiment of the present application is: obtaining video features of a to-be-segmented video and initial text features of a description text; decoupling the initial text features based on a subject decoupler to obtain subject text features and context text features; respectively performing information interaction between the subject text features, the context text features and the video features to obtain visual decoupling information and language decoupling information; and performing target segmentation on the to-be-segmented video according to the visual decoupling information and the language decoupling information to obtain a video segmentation result.

[0059] The technical terms involved in the embodiments of the present application include:

[0060] Transformer: Transformer is a neural network architecture based entirely on attention mechanisms, aiming to solve the parallelism and long-distance dependency problems of traditional recurrent neural networks (RNN) and convolutional neural networks (CNN) when processing sequential data.

[0061] Transformer mainly consists of an encoder and a decoder, both of which contain multiple identical layers, each of which integrates a multi-head self-attention mechanism and a feed-forward neural network. In addition, the decoder also contains a multi-head cross-attention mechanism. The Transformer model has been widely applied to various natural language processing tasks, such as text generation, text classification, named entity recognition, question-answering systems, machine translation, etc. In addition, Transformer has also been extended to the field of computer vision.

[0062] Multi-head self-attention mechanism: Multi-head self-attention mechanism is one of the key components in Transformer, which is used to establish connections between different positions within a sequence. Specifically, the multi-head self-attention mechanism allows elements at each position to "focus" on other elements in the sequence, thereby obtaining more rich contextual information.

[0063] Multi-head cross-attention mechanism: Multi-head cross-attention mechanism is used to interact across two different sequences. Multi-head cross-attention mechanism is particularly suitable for tasks involving two different modalities, such as machine translation, text generation, visual question answering, etc.

[0064] Existing language-to-video object segmentation techniques usually use a multi-modal transformer-based structure to realize the interaction between visual features and text features. However, due to the fact that multi-modal transformers tend to ignore the syntactic structure of text, and are limited by the diversity of training data, when applying existing techniques in real-world scenarios, they are sensitive to similar target interference, and there are problems of low stability and accuracy of video segmentation.

[0065] From the above analysis, it can be seen that the existing video object segmentation technology has the problem of low stability and accuracy of video object segmentation.

[0066] The application provides a solution by obtaining video features of a video to be segmented and initial text features of a description text, then decoupling the initial text features by using a subject decoupler to obtain subject text features and context text features, so as to realize a decoupled visual-linguistic interaction process subsequently; then performing information interaction between the subject text features, the context text features and the video features to obtain visual decoupling information and linguistic decoupling information, and finally performing target segmentation on the video to be segmented according to the visual decoupling information and the linguistic decoupling information, so as to obtain a video segmentation result, thereby realizing communication and fusion of multi-modal information under the premise of subject information and description information in the explicit description text, realizing stable and anti-interference linguistic reference video target segmentation, and effectively improving the stability and accuracy of video target segmentation.

[0067] It should be noted that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a video target segmentation device capable of realizing the above functions. The embodiments will be described below taking the video target segmentation device as an example.

[0068] Based on this, the embodiment of the application provides a video target segmentation method, which is described with reference to Figure 1 , Figure 1 The flowchart of the first embodiment of the video target segmentation method of the application is shown in FIG. 1.

[0069] In the embodiment, the video target segmentation method includes steps S110-S140:

[0070] Step S110, obtaining video features of a video to be segmented and initial text features of a description text;

[0071] First, for the video target segmentation task of the text reference, the video target segmentation device needs to obtain the video features of the video to be segmented and the initial text features of the description text, so as to interact and fuse the video features and the text features subsequently, so that the visual features can perceive the text description information, thereby improving the accuracy and stability of the linguistic reference video target segmentation.

[0072] In a feasible implementation, the step S110 can include steps A01-A03:

[0073] Step A01, obtaining a video to be segmented and a description text;

[0074] Step A02, performing feature encoding on the description text based on a preset text encoder to obtain initial text features of the description text;

[0075] Step A03, encoding features of the to-be-segmented video based on a preset video encoder to obtain video features of the to-be-segmented video.

[0076] In this embodiment, the preset text encoder is a model for extracting text features in the descriptive text, which can be a self-encoder, a BERT (Bidirectional Encoder Representations from Transformers) model, a RoBERTa (Robustly Optimized BERT Pretraining Approach) model, etc.

[0077] The preset video encoder is a model for extracting spatial features of each frame of image in the video and time dynamic features across frames, which can be constructed based on a convolutional neural network, a spatio-temporal convolutional network (STConv), a Transformer, etc.

[0078] Specifically, the video target segmentation device first acquires a to-be-segmented video and a descriptive text, wherein the descriptive text refers to a text describing a segmentation target, for example, "a white car located on the left side of another car".

[0079] Then, the descriptive text is input into the preset text encoder, and the preset text encoder is used to convert the descriptive text into a text feature vector. Specifically, text features of each word piece are extracted from the descriptive text, and the text features of each word piece are encoded to obtain a text feature representation corresponding to the descriptive text, i.e., initial text features The word piece refers to a smaller, processable unit or segment in text data, which can be a word, a character, a sub-word, or even a symbol.

[0080] Then, the to-be-segmented video is input into the preset video encoder, and the preset text encoder is used to extract spatial features of each frame of image in the to-be-segmented video and time dynamic features across frames, and the extracted spatial and time features are encoded to obtain a feature representation containing spatio-temporal information of the entire to-be-segmented video, i.e., video features

[0081] wherein, v i represents the visual features of the i-th scale, I represents the number of scales, T, H i , W i and C represent the dimensions in terms of time, width, height and information amount, respectively, and R refers to the real number field. con,l ∈R C represents the text features of the l-th word piece, and L represents the number of all word pieces.

[0082] In step S120, the initial text features are decoupled based on a subject decoupler to obtain subject text features and context text features.

[0083] In step S130, the subject text features and the context text features are respectively interacted with the video features to obtain visual decoupling information and language decoupling information.

[0084] It should be noted that the subject decoupler refers to a model or tool capable of separating the subject (i.e., the main object or entity described in the text) from the context (i.e., other information or description surrounding the subject) in the text features.

[0085] Specifically, since the initial text features contain comprehensive description information about a certain scene, event or object in the video frame, in order to enable the video features to better perceive deeper semantic information in the description text, the video target segmentation device separates the initial text features, in which the subject information is mixed with the description information, into two parts by applying the subject decoupler: one part is the subject text features focusing on the subject, including information of core attributes of the target object; the other part is the context text features describing the environment, actions or states around the subject, including all auxiliary descriptive information other than the subject, such as time, place, other related objects or event description information.

[0086] Then, the video target segmentation device realizes information interaction between the text features and the video features through feature fusion, attention mechanism, etc. The video target segmentation device first performs feature matching between the subject text features and the video features to identify relevant video features of the segmentation target corresponding to the subject text features, thereby obtaining first visual information, and performs feature matching between the context text features and the video features to identify relevant video features corresponding to the context text features, thereby obtaining second visual information, and the first visual information and the second visual information constitute the visual decoupling information.

[0087] In addition, the subject text features and the context text features are information-enhanced through a pre-trained language model and an attention mechanism, thereby obtaining the language decoupling information.

[0088] In step S140, the target segmentation is performed on the video to be segmented according to the visual decoupling information and the language decoupling information, thereby obtaining a video segmentation result.

[0089] In this embodiment, the video target segmentation device uses the segmentation target information contained in the above visual decoupling information, and simultaneously uses the language decoupling information to enhance the understanding of the segmentation target, ensures that even in multiple instances of the same category in the video to be segmented, the instance that best matches the description can be accurately distinguished, thereby determining the segmentation target from the video to be segmented, and generating a binary segmentation mask for identifying the segmentation target, and further obtaining the video segmentation result.

[0090] To further improve the perception of video features to text features, after the above step A03, step B01 is further included:

[0091] Step B01, based on a preset video encoder, pre-aligning the initial text features of the description text and the video features of the video to be segmented to obtain pre-aligned video features.

[0092] The step S130 includes:

[0093] The main text features and the context text features are respectively interacted with the pre-aligned video features to obtain visual decoupling information and language decoupling information.

[0094] Specifically, the video target segmentation device matches or aligns the initial text features and the video features based on similarity measures (such as cosine similarity, Euclidean distance, etc.) in time, space or other dimensions through a preset video encoder, realizes pre-alignment of the initial text features and the video features, thereby establishing the association between the textual description of the description text and the visual information in the video to be segmented, obtaining the pre-aligned video features, so that the subsequent video target segmentation task can be more effectively performed.

[0095] Then, the pre-aligned video features can be respectively interacted with the main text features and the context text features for multi-modal information interaction, which helps to more accurately understand the relationship between the video content and the text description, so that the fused visual features can realize deep perception of the segmentation target, thereby effectively improving the accuracy of video target segmentation.

[0096] The embodiment provides a video target segmentation method, which comprises the following steps: obtaining video features of a video to be segmented and initial text features of a description text; decoupling the initial text features based on a main decoupler to obtain main text features and context text features; respectively interacting the main text features and the context text features with the video features for information interaction to obtain visual decoupling information and language decoupling information; and performing target segmentation on the video to be segmented according to the visual decoupling information and the language decoupling information to obtain a video segmentation result.

[0097] The application obtains video features of a to-be-segmented video and initial text features of a description text, then decouples the initial text features by using a subject decoupler to obtain subject text features and context text features, so as to realize a decoupled visual-linguistic interaction process subsequently; then the subject text features and the context text features are respectively interacted with the video features to obtain visual decoupling information and linguistic decoupling information, and finally the to-be-segmented video is segmented according to the visual decoupling information and the linguistic decoupling information, so as to obtain a video segmentation result, thereby realizing the exchange and fusion of multi-modal information under the premise of clearly describing subject information and description information, realizing stable and anti-interference linguistic reference video target segmentation, and effectively improving the stability and accuracy of video target segmentation.

[0098] Based on the first embodiment of the application, in the second embodiment of the application, the same or similar contents as the above-mentioned first embodiment can be referred to the above introduction, and will not be described in detail hereinafter. On this basis, the description text includes a plurality of word units, and the initial text features include text features corresponding to the plurality of word units. Please refer to Figure 2 , the step S120 includes steps S210-S240:

[0099] Step S210, according to the subject decoupler, the subject prediction is carried out on the word units in the description text, and the subject probability of the word units is obtained.

[0100] Specifically, the video target segmentation device uses the subject decoupler to first reduce the dimension of the text features corresponding to all the word units in the description text , and map them to a low-dimensional binary classification space (such as subject and non-subject). By predicting whether each word unit is part of the subject, the subject probability of each word unit is obtained, where 0≤p l ≤1.

[0101] In a feasible implementation manner, the step S210 can include steps C01-C02:

[0102] Step C01, based on the multi-head self-attention mechanism, the information of the initial text features is enhanced, and the enhanced text features are normalized by layer normalization to obtain normalized initial text features.

[0103] Step C02, using a multi-layer perception and a linear mapping layer, the text features corresponding to each word unit in the normalized initial text features are classified and mapped to obtain the subject probability of the word unit.

[0104] In this embodiment, the subject decoupler uses multi-head self-attention, layer normalization, multi-layer perception and linear mapping layer to realize subject prediction by inputting the text features of all the word units: Output the subject probability corresponding to all word pieces wherein 0≤p l ≤1.

[0105] Specifically, the video target segmentation device first uses a multi-head self-attention mechanism to perform global information enhancement on the initial text features, uses the global information of the initial text features itself to strengthen the text feature information corresponding to each word piece, and further increases the accuracy of information expression. At the same time, the layer normalization technology is used to further normalize the data distribution of the information-enhanced text features, so as to ensure the stability of the training and inference of the entire video target segmentation architecture, and finally obtain the normalized initial text features. The normalized initial text features can be expressed as follows:

[0106]

[0107] wherein LN represents layer normalization processing, and MHSA represents a multi-head self-attention module, which is used to use the global information itself to strengthen the feature expression.

[0108] Then, the normalized initial text features are sent into a classification module composed of a multi-layer perceptron and a linear mapping layer, and the subject probability wherein the subject probability can be expressed as follows:

[0109]

[0110] wherein p l represents the probability that the lth word piece belongs to the subject. MLP represents a multi-layer perceptron, which is used to obtain the mutual relationship between word pieces and adjust the text feature distribution, so as to facilitate accurate classification. Linear represents a linear mapping layer, which is used to reduce the dimension of the input text features and convert them into classification probabilities.

[0111] In step S220, the description text is divided into word pieces to obtain a plurality of word piece groups.

[0112] Then, the description text is divided into word pieces by using a pre-defined dictionary matching method, or by using the point mutual information between word pieces to measure the correlation of the word pieces, or by using a pre-trained language model BERT or RoBERTa to divide the description text into word pieces to obtain a plurality of word piece groups. For example, when the description text is "a white car located on the left side of another car", the word piece division can obtain two word groups or word piece groups "white car" and "car". The word piece group refers to text containing a plurality of word pieces, that is, a word group.

[0113] In step S230, the average subject probability of the word piece groups is calculated, and the word piece group with the maximum average subject probability is selected as the subject word piece group.

[0114] Step S240, taking the text features corresponding to the word units in the subject word unit group as subject text features, and taking the initial text features as context text features.

[0115] Specifically, since each word unit group is composed of a series of word units, in order to determine the subject word unit group in the description text, the video target segmentation device can calculate the average subject probability of each word unit group obtained by the above vocabulary division, and select the word unit group with the maximum average subject probability as the subject word unit group.

[0116] The selection of the subject word unit group can be expressed as the following formula:

[0117]

[0118] Wherein, argmax is used to find the function of the parameter (or index) that makes the probability value maximum in the average subject probability formula; is the average subject probability calculation formula of the word unit group; W represents the number of word unit groups, N w represents the number of word units included in the wth word unit group. The average subject probability of the word unit group is obtained by calculating the average of the subject probabilities of all word units included in the word unit group.

[0119] Through the above formula, the subject word unit group can be determined, and the subject word unit group is written as L s represents the number of word units in the subject word unit group.

[0120] Finally, the text features corresponding to the word units in the subject word unit group are taken as the subject text features, and the initial text features corresponding to the description text are taken as the context text features.

[0121] The embodiment provides a video target segmentation method, which comprises the following steps: obtaining video features of a video to be segmented and initial text features of a description text; performing subject prediction on word units in the description text by using a subject decoupler to obtain subject probabilities of the word units; performing vocabulary division on the description text to obtain a plurality of word unit groups; calculating average subject probabilities of the word unit groups, and selecting a word unit group with the maximum average subject probability as a subject word unit group; taking text features corresponding to word units in the subject word unit group as subject text features, and taking the initial text features as context text features; performing information interaction between the subject text features, the context text features and the video features respectively to obtain visual decoupling information and language decoupling information; and performing target segmentation on the video to be segmented according to the visual decoupling information and the language decoupling information to obtain a video segmentation result.

[0122] The subject decoupler is used to predict the subject of the word in the description text, obtain the subject probability of the word, select the word group with the maximum average subject probability as the subject word group, determine the corresponding subject text feature, and then implement the decoupled visual-language interaction process, so as to realize the exchange and fusion of multi-modal information under the premise of subject information and description information in the explicit description text, realize stable and anti-interference language reference video target segmentation, and effectively improve the stability and accuracy of video target segmentation.

[0123] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as the above-mentioned embodiment one and / or embodiment two can be referred to the above introduction, and the subsequent will not be described in detail. On this basis, please refer to Figure 3 , the step S130 includes steps S310-S360:

[0124] Step S310, based on the multi-head cross attention mechanism, the subject text feature and the video feature are interacted between modalities, and based on the multi-head self-attention mechanism, the features after interaction are reinforced within modalities, to obtain the first fusion video feature;

[0125] Step S320, based on the multi-head cross attention mechanism, the context text feature and the video feature are interacted between modalities, and based on the multi-head self-attention mechanism, the features after interaction are reinforced within modalities, to obtain the second fusion video feature;

[0126] Specifically, the video target segmentation device first uses the multi-head cross attention mechanism to fuse the subject text feature and the video feature, realizes the interaction of inter-modal information, so that the video feature can better perceive the part related to the subject text feature, realizes the fusion of the subject text feature into the visual feature, and obtains the feature after interaction.

[0127] Then, the multi-head self-attention mechanism is applied to further strengthen the internal relationship of the features after interaction, to ensure that each feature can fully utilize the global information and improve its expression ability, so as to output the visual feature containing the subject text feature and the strengthened global information, i.e. the first fusion video feature

[0128]

[0129] It can be understood that, similar to the above steps, the multi-head cross attention mechanism is used to fuse the context text feature and the video feature, to realize the interaction of inter-modal information. Then, the multi-head self-attention mechanism is applied to further strengthen the internal relationship of the features after interaction, to output the visual feature containing the context text feature and the strengthened global information, i.e. the second fusion video feature

[0130]

[0131] Wherein, MHCA represents a multi-head cross attention module, used for injecting text feature information into visual information; MHSA represents a multi-head self-attention module, used for using self-global information to strengthen feature expression.

[0132] In step S330, the subject text feature is strengthened within the mode based on the multi-head self-attention mechanism, to obtain a first text feature;

[0133] In step S340, the context text feature is strengthened within the mode based on the multi-head self-attention mechanism, to obtain a second text feature;

[0134] Specifically, the multi-head self-attention mechanism is used to strengthen the information within the mode for the subject text feature and the context text feature respectively, and the first text feature and the second text feature after information strengthening are obtained respectively, wherein the first text feature and the second text feature The calculation formula can be expressed as follows:

[0135]

[0136] Wherein, represents the subject text feature, represents the context text feature.

[0137] In step S350, the first fusion video feature and the second fusion video feature are spliced to obtain visual decoupling information;

[0138] In step S360, the first text feature and the second text feature are spliced to obtain language decoupling information.

[0139] Specifically, in order to ensure that the feature vector after splicing not only has a reasonable size, but also can retain the key information of the original feature, and at the same time improve the generalization ability and calculation efficiency of the model, the video target segmentation device needs to compress the dimension of the first fusion video feature and the second fusion video feature after interaction and strengthening in the process of feature splicing, to appropriately reduce the feature dimension and improve the data calculation efficiency, and finally obtain the decoupled visual information expression, i.e. visual decoupling information

[0140] Similarly, in the process of feature splicing of the first text feature and the second text feature after information strengthening, dimension compression and average pooling processing are needed, so as to obtain language decoupling information e dec . Wherein, the visual decoupling information and the language decoupling information e decThe calculation formula can be expressed as follows:

[0141]

[0142] wherein Resize represents compression of the information dimension. AvgPool represents averaging the information of all word units in the text information to realize higher-dimensional information aggregation.

[0143] The embodiment provides a video target segmentation method, which realizes visual-linguistic interaction process by inter-modal interaction and intra-modal reinforcement through a multi-head cross attention mechanism and a multi-head self-attention mechanism, so as to obtain visual decoupling information and linguistic decoupling information, and further realizes communication and fusion of multi-modal information on the premise of clearly describing subject information and description information in the text, realizes stable and anti-interference language reference video target segmentation, and effectively improves the stability and accuracy of video target segmentation.

[0144] Based on any one of the above embodiments of the application, in the fourth embodiment of the application, the same or similar contents as any one of the above embodiments can be referred to the above introduction, and will not be repeated hereinafter. On this basis, please refer to Figure 4 , the step S140 includes steps S410-S420:

[0145] Step S410, based on the target decoder and the pixel decoder, determining a segmentation target from the video to be segmented according to the visual decoupling information, the linguistic decoupling information and the video features;

[0146] Step S420, labeling the segmentation target in the video to be segmented to obtain a video segmentation result.

[0147] It should be noted that the target decoder is used for segmentation target query and generates a binary mask or segmentation result corresponding to the segmentation target. In video target segmentation, the target decoder can predict the class label of each pixel according to the feature map related to the segmentation target. The implementation of the target decoder is based on a deep learning framework, which can be a decoder of a Transformer model.

[0148] The pixel decoder is used to restore the original video according to the video features, and can realize a pixel-level segmentation task, that is, according to the class label corresponding to each pixel in the video frame, the segmentation target and the background are distinguished.

[0149] Specifically, the application target decoder predicts the bounding box or mask of the segmentation target in the to-be-segmented video according to the visual decoupling information and the language decoupling information. Then, the predicted bounding box or mask information is input into the pixel decoder, and the video features of the to-be-segmented video are combined to identify the target or object that needs to be segmented from each video frame of the to-be-segmented video and label it, such as color labeling, bounding box labeling, etc., so as to realize the video target segmentation of the language reference and obtain the video segmentation result.

[0150] In this embodiment, Figure 5 Fig. 1 shows a schematic diagram of a video target segmentation process.

[0151] In Figure 5 , the text encoder is used to encode the input text (i.e., the description text) ("a white car, located on the left side of another car") to obtain the corresponding initial text features Then the video encoder is used to encode the input video (i.e., the to-be-segmented video) while combining the initial text features to perform feature pre-alignment, and then obtain the pre-aligned video features

[0152] Then the text features corresponding to all the word units are sent to the main decoupler to obtain the main probability of the word unit At the same time, the input text is divided into two word groups (i.e., word units), i.e., "white car" and "car". Since each word group is composed of a series of word units, the division of the main information (e.g., "white car") can be realized by selecting the word group with the maximum average main probability. After confirming the main word group, the text features corresponding to the main word group are written as , which is the main text feature.

[0153] Through the "main-aware visual-language interaction" and "context-aware visual-language interaction" in Figure 5 , the main text feature and the context text feature are respectively interacted with the visual features . Both the "main-aware visual-language interaction" and the "context-aware visual-language interaction" modules are composed of two basic modules, and the basic module structure is shown in the dashed box in Figure 5 . First, the text features are integrated into the visual features through inter-modal interaction, so that the visual features can perceive the text description information (main or context). Then, through intra-modal reinforcement, the expression ability of visual and language information is further improved.

[0154] Then the multi-round interaction and the reinforced visual information and language information are spliced and fused to obtain visual decoupling information and language decoupling information. A target decoder and a pixel decoder are applied to obtain a video target segmentation result according to the visual decoupling information, the language decoupling information and the video features.

[0155] The embodiment provides a video target segmentation method. Video features of a video to be segmented and initial text features of a description text are obtained. The initial text features are decoupled based on a subject decoupler to obtain subject text features and context text features. The subject text features and the context text features are respectively interacted with the video features to obtain visual decoupling information and language decoupling information. A target decoder and a pixel decoder are applied to determine a segmentation target from the video to be segmented according to the visual decoupling information, the language decoupling information and the video features. The segmentation target is labeled in the video to be segmented to obtain a video segmentation result.

[0156] The application realizes the communication and fusion of multi-modal information under the premise that the subject information and the description information in the explicit description text are realized, realizes the stable and anti-interference language reference video target segmentation, and effectively improves the stability and accuracy of the video target segmentation.

[0157] The application also provides a video target segmentation device, please refer to Figure 6 , the video target segmentation device comprises:

[0158] The acquisition module 10 is configured to acquire video features of a video to be segmented and initial text features of a description text.

[0159] The subject decoupling module 20 is configured to decouple the initial text features based on a subject decoupler to obtain subject text features and context text features.

[0160] The modal fusion module 30 is configured to respectively interact the subject text features and the context text features with the video features to obtain visual decoupling information and language decoupling information.

[0161] The target segmentation module 40 is configured to perform target segmentation on the video to be segmented according to the visual decoupling information and the language decoupling information to obtain a video segmentation result.

[0162] The video target segmentation device provided in the present application adopts the video target segmentation method in the above embodiments, and can solve the technical problem of how to improve the stability and accuracy of video target segmentation. Compared with the prior art, the video target segmentation device provided in the present application has the same beneficial effects as the video target segmentation method provided in the above embodiments, and other technical features in the video target segmentation device are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0163] The present application provides a video target segmentation device, which comprises at least one processor and a memory connected with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the video target segmentation method in the above embodiments.

[0164] Reference will be made to the following description of the embodiments of the present application, taken in conjunction with the accompanying drawings, in which Figure 7 which shows a structural schematic diagram of a video target segmentation device suitable for implementing the embodiments of the present application. The video target segmentation device in the embodiments of the present application can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 7 The video target segmentation device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0165] As Figure 7As shown, the video object segmentation device can include a processing apparatus 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 1002 or loaded from a storage apparatus 1003 into a random access memory (RAM) 1004. Various programs and data required for the operation of the video object segmentation device are also stored in the RAM 1004. The processing apparatus 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input apparatus 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output apparatus 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage apparatus 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication apparatus 1009. The communication apparatus 1009 can allow the video object segmentation device to communicate wirelessly or wired with other devices to exchange data. Although the video object segmentation device with various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0166] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication apparatus, or installed from the storage apparatus 1003, or installed from the ROM 1002. When the computer program is executed by the processing apparatus 1001, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0167] The video object segmentation device provided by the present disclosure adopts the video object segmentation method in the above-mentioned embodiments, and can solve the technical problem of how to improve the stability and accuracy of video object segmentation. Compared with the prior art, the video object segmentation device provided by the present disclosure has the same beneficial effects as the video object segmentation method provided by the above-mentioned embodiments, and other technical features in the video object segmentation device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0168] It should be understood that various aspects of the disclosure can be implemented in hardware, software, firmware, or combinations thereof, to achieve the various aspects of the disclosure. In the description above, specific features, structures, materials or characteristics can be combined in any suitable manner without necessarily being limited to one or more embodiments or examples.

[0169] The above description is merely illustrative of the application and is not intended to limit the scope of the application. Any modifications or equivalents of the application should be construed as falling within the scope of the application. The scope of the application should be determined by the appended claims.

[0170] The application provides a computer readable storage medium having computer readable program instructions (i.e., computer programs) stored thereon, the computer readable program instructions being used to perform the video object segmentation method in the above-described embodiments.

[0171] The computer readable storage medium provided by the application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection having one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), etc., or any suitable combination thereof.

[0172] The above-described computer readable storage medium can be contained in the video object segmentation device; or can exist separately without being assembled into the video object segmentation device.

[0173] The computer readable storage medium described above carries one or more programs, when the one or more programs are executed by the video object segmentation device, cause the video object segmentation device to: acquire video features of a video to be segmented and initial text features of description text; based on a subject decoupler, decouple the initial text features to obtain subject text features and context text features; respectively perform information interaction of the subject text features, the context text features and the video features to obtain visual decoupling information and language decoupling information; and according to the visual decoupling information and the language decoupling information, perform object segmentation on the video to be segmented to obtain a video segmentation result.

[0174] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0175] The flow diagrams and the block diagrams in the drawings are illustrations of possible architectures, functions, and operations for systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0176] The modules described in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0177] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the video target segmentation method described above, and can solve the technical problem of how to improve the stability and accuracy of video target segmentation. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the video target segmentation method provided by the above-mentioned embodiments, and will not be described here.

[0178] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent processing scope of the present application.

Claims

1. A video target segmentation method, characterized in that, The method includes: Obtain the video features of the video to be segmented and the initial text features of the descriptive text; Based on the main decoupler, the initial text features are decoupled to obtain main text features and context text features. The descriptive text includes several word units, and the initial text features include text features corresponding to several word units. Based on a multi-head cross-attention mechanism, the main text features and video features are subjected to intermodal interaction, and the interacting features are enhanced intramodally based on a multi-head self-attention mechanism to obtain a first fused video feature; based on a multi-head cross-attention mechanism, the context text features and video features are subjected to intermodal interaction, and the interacting features are enhanced intramodally based on a multi-head self-attention mechanism to obtain a second fused video feature; based on a multi-head self-attention mechanism, the main text features are enhanced intramodally to obtain a first text feature; based on a multi-head self-attention mechanism, the context text features are enhanced intramodally to obtain a second text feature; the first fused video feature and the second fused video feature are concatenated to obtain visual decoupling information; the first text feature and the second text feature are concatenated to obtain language decoupling information; Based on the visual decoupling information and the language decoupling information, the video to be segmented is segmented to obtain the video segmentation result. The step of decoupling the initial text features based on the subject decoupler to obtain subject text features and context text features includes: Based on the subject decoupling device, subject prediction is performed on the words in the descriptive text to obtain the subject probability of the words; The descriptive text is segmented into words to obtain several word tuples; The average subject probability of the word tuples is calculated, and the word tuple with the highest average subject probability is selected as the subject word tuple. The text features corresponding to the lexical units in the main lexical group are used as the main text features, and the initial text features are used as the context text features.

2. The method as described in claim 1, characterized in that, The steps of obtaining the video features of the video to be segmented and the initial text features of the descriptive text include: Obtain the video to be segmented and the description text; Based on a preset text encoder, feature encoding is performed on the descriptive text to obtain the initial text features of the descriptive text; Based on a preset video encoder, feature encoding is performed on the video to be segmented to obtain the video features of the video to be segmented.

3. The method as described in claim 1, characterized in that, The step of performing target segmentation on the video to be segmented based on the visual decoupling information and the language decoupling information to obtain the video segmentation result includes: Based on the target decoder and pixel decoder, the segmentation target is determined from the video to be segmented according to the visual decoupling information, language decoupling information and the video features; The segmentation target is labeled in the video to be segmented to obtain the video segmentation result.

4. The method as described in claim 2, characterized in that, After the step of performing feature encoding on the video to be segmented based on a preset video encoder to obtain the video features of the video to be segmented, the method further includes: Based on a preset video encoder, the initial text features of the description text and the video features of the video to be segmented are pre-aligned to obtain pre-aligned video features.

5. The method as described in claim 1, characterized in that, The step of performing subject prediction on the lexical units in the descriptive text based on the subject decoupling device to obtain the subject probability of the lexical units includes: The initial text features are augmented using a multi-head self-attention mechanism, and the augmented text features are then normalized to obtain the normalized initial text features. By using a multilayer perceptron and a linear mapping layer, the text features corresponding to each word in the normalized initial text features are classified and mapped to obtain the subject probability of the word.

6. A video target segmentation device, characterized in that, The video target segmentation device includes: The acquisition module is used to acquire the video features of the video to be segmented and the initial text features of the descriptive text; The main decoupling module is used to decouple the initial text features based on the main decoupling device to obtain the main text features and the context text features. The descriptive text includes several word units, and the initial text features include text features corresponding to several word units. The modality fusion module is used to perform intermodal interaction between the main text features and the video features based on a multi-head cross-attention mechanism, and to perform intramodal enhancement on the interacted features based on a multi-head self-attention mechanism to obtain a first fused video feature; to perform intermodal interaction between the context text features and the video features based on a multi-head cross-attention mechanism, and to perform intramodal enhancement on the interacted features based on a multi-head self-attention mechanism to obtain a second fused video feature; to perform intramodal enhancement on the main text features based on a multi-head self-attention mechanism to obtain a first text feature; to perform intramodal enhancement on the context text features based on a multi-head self-attention mechanism to obtain a second text feature; to concatenate the first fused video feature and the second fused video feature to obtain visual decoupling information; and to concatenate the first text feature and the second text feature to obtain linguistic decoupling information. The target segmentation module is used to perform target segmentation on the video to be segmented based on the visual decoupling information and the language decoupling information, so as to obtain the video segmentation result; The subject decoupling module is further configured to perform subject prediction on the lexical units in the descriptive text based on the subject decoupling device, and obtain the subject probability of the lexical units; The descriptive text is segmented into words to obtain several word tuples; The average subject probability of the word tuples is calculated, and the word tuple with the highest average subject probability is selected as the subject word tuple. The text features corresponding to the lexical units in the main lexical group are used as the main text features, and the initial text features are used as the context text features.

7. A video target segmentation device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the video target segmentation method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the video target segmentation method as described in any one of claims 1 to 5.

Images