A small sample reference video target segmentation method based on cross-modal affinity

Abstract

The application discloses a small sample reference video target segmentation method based on cross-modal affinity, and mainly solves the problems that the existing method needs to depend on a large amount of labeled data and cannot be generalized to a new scene. The application mainly comprises the following steps: preparing a small sample reference video target segmentation data set; constructing a cross-modal affinity network for model training; training the network model based on the constructed cross-modal affinity network and the prepared data set; and inputting a video frame sequence and a natural language description according to learned model parameters to obtain a segmentation result. The method is scientific and reasonable in design, effectively improves the generalization and accuracy of small sample reference video target segmentation by establishing a cross-modal affinity relationship, and has a wide application prospect in video editing and human-computer interaction.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and further relates to video target segmentation technology. Specifically, it relates to a few-sample reference video target segmentation method based on cross-modal affinity that can be used in video editing, human-computer interaction and other fields. Background Technology

[0002] Computer vision is a research field that aims to help computers use complex algorithms (both traditional and deep learning-based) to understand digital images and videos and extract useful information. The main goal of computer vision is to first understand the content of videos and still images, and then extract useful information from them to solve an increasing number of problems. As a subfield of artificial intelligence (AI) and deep learning, computer vision trains convolutional neural networks (CNNs) to develop human-like visual capabilities for a variety of applications. Computer vision involves specifically training CNNs to perform data segmentation, classification, and detection using images and videos.

[0003] Video object segmentation is a crucial task in video processing, aiming to separate objects of interest from the background in a series of video sequences. In recent years, due to the excellent performance of deep learning technology in computer vision tasks (such as image recognition, object tracking, and action recognition), deep learning-based video object segmentation algorithms have become the mainstream method for solving this task. The performance of deep learning-based video object segmentation algorithms depends on the size of the neural network used. The performance of the neural network relies on a large amount of training data; the larger the training dataset, the better the generalization and robustness of the trained neural network. In the supervised learning paradigm, creating a training dataset for video object segmentation is costly and time-consuming, requiring not only spatial annotation of every pixel in the image but also temporal annotation of every frame in the video sequence. The performance of the video object segmentation model is also closely related to its structure. By reasonably optimizing the inference process of the video object segmentation model, errors in the video object segmentation process can be effectively reduced.

[0004] Reference video object segmentation aims to segment target objects in videos that have natural language descriptions. It has wide applications in real-world scenarios, such as video editing and human-computer interaction, thus this novel task has attracted significant attention from the research community. Unlike traditional semi-supervised video object segmentation, reference video object segmentation is more challenging because it not only lacks the correct mask from the first frame of the video but also requires interaction of multimodal information, including visual and linguistic data.

[0005] The immense success of deep learning in various tasks is attributed to the vast amounts of labeled data. However, real-world data exhibits a long-tail distribution, and obtaining high-quality labeled data is often extremely costly. For example, in video object segmentation tasks, a large amount of video data and manually labeled natural language descriptions are required, and the high cost of annotation significantly limits the applicability of the model. Furthermore, video object segmentation tasks are often designed for specific, single scenarios and lack good generalization ability when faced with new scenarios. This is because real-world datasets often contain overly simplistic scenarios, leading to poor performance when transferring models to new datasets.

[0006] Given input video and text, the goal of reference video object segmentation is to segment and associate textual reference objects across all video frames. To effectively link textual cues with visual objects, current reference video object segmentation methods primarily employ three techniques for interaction between multimodal elements: dynamic convolution, cross-modal attention, and Transformer.

[0007] Dynamic convolution was first applied to object segmentation in reference videos, where text features were encoded into kernels to convolve visual features. Subsequent work improved this idea by modulating the text kernels with the visual context required for convolution, thus providing greater robustness to visually similar distractors.

[0008] Cross-modal attention is a widely used technique in reference video object segmentation because it can establish fine-grained and semantic correspondences between visual and linguistic elements. Early work utilized these properties to improve visual context and reduce linguistic variation. To make fuller use of text, some schemes incorporate specific linguistic components into attention-based interactions.

[0009] The success of Transformers in Natural Language Processing (NLP) and Computer Vision (CV) has encouraged related applications in multimodal analysis. Unlike the aforementioned approaches, transformer-based schemes rely entirely on attention mechanisms to achieve visual-language interaction. Early methods segmented each video frame individually and utilized only the transformer for feature fusion. Inspired by the application of DETR in video instance segmentation, recent work employs a DETR-like architecture, where Transformers are used in a parallel and global manner for feature fusion and object localization.

[0010] However, the above methods are limited by datasets specific to certain scenarios. They fail to achieve satisfactory results when measuring effectiveness in unknown scenarios.

[0011] Based on the above analysis, it is necessary to provide a method that enables models to learn new semantic information from a small number of samples, thereby quickly adapting to complex and diverse scenarios in the real world. Summary of the Invention

[0012] To overcome the limitations of existing technologies, which often suffer from overly simplistic real-world datasets and require large amounts of labeled data, this invention proposes a few-sample reference video target segmentation method based on cross-modal affinity. The cross-modal affinity module calculates the multimodal information affinity between the support set and the query set, enabling the learning of new semantic information from a small number of samples, thus allowing for rapid adaptation to complex and diverse real-world scenarios.

[0013] To achieve the above-mentioned technical objectives and effects, the present invention is implemented through the following technical solution:

[0014] A few-sample reference video target segmentation method based on cross-modal affinity includes the following steps:

[0015] S1. Construct a dataset suitable for small-sample reference video object segmentation tasks using existing reference video object segmentation datasets;

[0016] S2. Construct a cross-modal affinity network for model training;

[0017] S3. Train the network constructed in step S2 based on the dataset created in step S1, and save the training parameters.

[0018] S4. Using the video frame sequence and natural language description as input to the network, the segmentation result is obtained as output using the parameters learned in step S3.

[0019] Furthermore, the process of creating a dataset suitable for the few-sample reference video object segmentation task in step S1 is as follows: For each video and its corresponding natural language description in the existing reference video object segmentation dataset, firstly select videos containing two different categories of segmentation targets and discard them; then sort the remaining videos according to the number of segmentation target categories, and discard videos of categories with fewer than 10 or more than 50 categories; by performing the above processing on all videos in the dataset, a dataset suitable for the few-sample reference video object segmentation task can be obtained. The dataset contains 1668 videos and is divided into 48 categories.

[0020] Furthermore, the cross-modal affinity network constructed in step S2 takes data from two sets, the support set and the query set, as input. First, it extracts visual and textual features separately through a weight-shared feature extraction network. The feature extraction network employs a residual network consisting of multiple sequentially connected convolutional layers, multiple pooling layers, multiple residual unit modules, a single fully connected layer, and a large-scale pre-trained text model based on Transformer. Then, a multi-head cross-modal attention mechanism is used to fuse the visual and textual features from the support set and the query set respectively. The fused multimodal features are used to calculate the self-affinity within the query set. The obtained self-affinity features are then compared with the multimodal features of the support set to calculate the affinity relationship, resulting in more robust features. Finally, in order to locate the most relevant target and progressively decode the features, the obtained cross-modal affinity features are fed into the mask generation module to obtain the final segmentation mask.

[0021] Furthermore, the cross-modal affinity network constructed in step S2 is divided into four modules, namely:

[0022] The multimodal fusion module fuses visual and textual features;

[0023] The self-affinity module is used to calculate the self-affinity features in the query set;

[0024] The cross-affinity module is used to calculate cross-affinity features between the query set and the support set;

[0025] The mask generation module is used to generate the final segmentation result.

[0026] Furthermore, in the multimodal fusion module, before feature fusion, it is necessary to perform dimensional mapping on the visual features to uniformly change the number of channels of the multi-scale features to 256. The specific steps are as follows: First, the first three layers of 3×3 convolutional layers and regularization layers are passed through to map the number of channels of the first three layers of multi-scale features to 256; then, the last layer of multi-scale features is passed through a 1×1 convolutional layer and regularization layer to obtain the final feature with a uniform number of channels.

[0027] The multimodal fusion module includes visual features and text features. Specifically, for the interaction between visual features and text features, multi-head cross-attention (MCA) is used to fuse multimodal information to obtain a new multi-scale feature map. The specific process is represented by formula (1):

[0028] f′ vs =MCA(f s ,f i)

[0029] f′ vq =MCA(f q ,f j (1)

[0030] Among them, f s The visual features representing support, f i The linguistic feature that indicates support, f q The visual characteristics of a query, f j The language features that represent a query.

[0031] Furthermore, the self-affinity module includes a 3×3 convolutional layer, a matrix calculation module, a normalization layer, and residual connections. The multimodal features fused in the query set are processed by a 3×3 convolutional layer to obtain three different vectors q, k, and v. These three vectors are fed into the matrix calculation module to calculate the self-affinity, and then normalized and connected by residuals. The specific process is represented by formula (2):

[0032]

[0033] q s =q+LN(Softmax(A) Q (2)

[0034] Where, d head It is the number of dimensions of the hidden layers, A Q It is the calculated affinity matrix, and LN is the normalization operation.

[0035] Furthermore, the cross-affinity module includes a 3×3 convolutional layer, a matrix calculation module, a normalization layer, and residual connections; the fused multimodal features in the support set are processed by a 3×3 convolutional layer to obtain two different vectors as k. s v s Two different vectors and the output q of the self-affinity module are fed into the matrix calculation module to calculate the cross-affinity. Then, after normalization and residual connection, the specific process is represented by formula (3):

[0036]

[0037] Where, d head It represents the number of dimensions of the hidden layer, and LN is the normalization operation.

[0038] Furthermore, the mask generation module includes a Transformer architecture and a feature pyramid network; the Transformer architecture contains a six-layer encoder-decoder structure; the decoder contains two lightweight auxiliary heads for obtaining the final target mask; the feature pyramid network contains four layers of hierarchical features at different scales, and uses cross-attention to interact with visual and linguistic features. The specific process is represented by formula (4):

[0039]

[0040] in, It refers to the visual features at each level, f tq This represents the text features corresponding to the query set; finally, the features from the last layer are passed through a 3×3 convolutional layer to obtain the final feature map.

[0041] Furthermore, the Adam optimization algorithm is used to optimize the network model. The network loss function consists of two parts, and the specific process is expressed by formula (5):

[0042]

[0043] Where, λ cls , λ kernel This is a hyperparameter used to balance the loss; y represents the true mask of the target. This indicates the predicted segmentation result.

[0044] Furthermore, step S4 updates the network parameters using gradient descent, as expressed by formula (6) as follows:

[0045]

[0046] Among them, V i+1 This represents the weight update value for this time, while V... i This represents the previous weight update value, where μ is the weight of the previous gradient value, and α is the learning rate. It is the gradient.

[0047] The beneficial effects of this invention are:

[0048] 1. The present invention provides a small sample reference video target segmentation method based on cross-modal affinity. Considering the need for a large amount of labeled data in real-world scenarios, only a small amount of data is needed for training to complete the segmentation of unknown category data. This method meets the requirement of a large amount of data in reality.

[0049] 2. This invention provides a few-sample reference video target segmentation method based on cross-modal affinity. It proposes a cross-modal affinity module to calculate the affinity of multimodal information from the support set to the query set. First, it fuses the multimodal features within the support set and the query set respectively. Then, it aggregates the information between the support set and the query set, thereby effectively avoiding attention being biased by irrelevant features.

[0050] 3. This invention provides a few-sample reference video target segmentation method based on cross-modal affinity. By learning semantic information of a new category using a small amount of data, it can handle complex and diverse real-world scenarios. It not only improves segmentation performance but also utilizes far fewer resources than existing methods and outperforms current advanced methods in objective quantitative measurement, showing broad application prospects in video editing and human-computer interaction.

[0051] Of course, any product implementing this invention does not necessarily need to achieve all of the above advantages at the same time. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart of the small sample reference video target segmentation method based on cross-modal affinity of the present invention;

[0054] Figure 2 This is a flowchart illustrating the creation process of the dataset for the small sample reference video target segmentation task in this invention.

[0055] Figure 3 This is a schematic diagram of the cross-modal affinity network constructed in this invention;

[0056] Figure 4 This is a schematic diagram of the mask generation module of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Specific embodiments of the present invention are as follows:

[0059] Combination Figure 1 This embodiment provides a small sample reference video target segmentation method based on cross-modal affinity, specifically including the following steps:

[0060] S1. Utilize existing reference video object segmentation datasets, such as the Ref-YouTube-VOS dataset, to create a dataset suitable for small-sample reference video object segmentation tasks. Specific steps are as follows: Figure 2 As shown, that is:

[0061] The existing dataset contains 3471 videos, 12913 natural language descriptions, and annotated instances covering over 60 categories. However, some videos in this dataset contain instances of multiple categories. Therefore, during data preparation, these videos were removed, and only those containing instances of a single category were retained, resulting in a total of 2387 videos. The video data in the dataset should be class-balanced, with the number of samples in each category not differing too much to avoid overfitting to any one class. Therefore, for a particular category, if the number of videos is too small, they are discarded directly. If the number of videos is too large, a portion of them are randomly selected and the rest are discarded. After these processing steps, a dataset suitable for small-sample reference video object segmentation tasks is obtained, containing 1668 videos and 48 categories.

[0062] S2. Construct a cross-modal affinity network for model training;

[0063] 2-1. The constructed cross-modal affinity network has the following specific structure: Figure 3 As shown, it consists of the following parts: a multimodal fusion module, which fuses visual and textual features; a self-affinity module, which calculates the self-affinity features in the query set; a cross-affinity module, which calculates the cross-affinity features between the query set and the support set; and a mask generation module, which generates the final segmentation result.

[0064] 2-2. In the multimodal fusion module, before feature fusion, it is necessary to perform dimensional mapping on the visual features and uniformly change the number of channels of multi-scale features to 256. The specific steps are as follows:

[0065] (1) First, after passing through 3 layers of 3×3 convolutional layers and regularization layers, the number of feature channels in the first 3 layers of multi-scale features is mapped to 256.

[0066] (2) Then the last layer of multi-scale features is passed through a 1×1 convolutional layer and a regularization layer to obtain the final unified channel number feature.

[0067] The multimodal fusion module contains both visual and textual information. Specifically, for the interaction between visual and textual features, multi-head cross-attention (MCA) is used to fuse the multimodal information, resulting in a new multi-scale feature map. The specific process can be represented by formula (1):

[0068] f′ vs =MCA(f s ,f i )

[0069] f′ vq =MCA(f q ,f j (1)

[0070] Among them, f s f q This represents the visual characteristics of support and query, f i f j This corresponds to its language features. The number of heads in MCA is 8.

[0071] 2-3. The self-affinity module includes a 3×3 convolutional layer, a matrix calculation module, a normalization layer, and residual connections. The multimodal features fused in the query set are passed through a 3×3 convolutional layer to obtain three different vectors, q, k, and v, which are then fed into the matrix calculation module to calculate the self-affinity. Then, normalization and residual connections are performed. The specific process can be represented by formula (2):

[0072]

[0073] q s =q+LN(Softmax(A) Q (2)

[0074] Where, d head It is the number of dimensions of the hidden layers, A Q It is the calculated affinity matrix, and LN is the normalization operation.

[0075] 2-4. The cross-affinity module includes a 3×3 convolutional layer, a matrix calculation module, a normalization layer, and residual connections. The fused multimodal features in the support set are passed through a 3×3 convolutional layer to obtain two distinct vectors as k. s v s The output of the self-affinity module is fed as q into the matrix calculation module to calculate cross-affinity. Then, it undergoes normalization and residual connection. The specific process can be represented by formula (3):

[0076]

[0077] Where, d head It represents the number of dimensions of the hidden layer, and LN is the normalization operation.

[0078] 2-5. The mask generation module includes a Transformer architecture and a feature pyramid network, with the specific structure as follows: Figure 4 As shown. The Transformer architecture contains a 6-layer encoder-decoder structure. The decoder contains two lightweight auxiliary heads to obtain the final target mask. The feature pyramid network contains 4 layers of hierarchical features at different scales, and uses cross-attention to interact with visual and linguistic features. The specific process can be represented by formula (4):

[0079]

[0080] in, It refers to the visual features at each level, f tq This represents the text features corresponding to the query set. Finally, the features from the last layer are passed through a 3×3 convolutional layer to obtain the final feature map.

[0081] S3. Based on the training set obtained in step S1 and the network constructed in step S2, perform network training;

[0082] 3-1. Train the network using the PyTorch deep learning platform. For the cross-modal affinity network constructed in step S2, first initialize the super-resolution network using the Xavier method, setting all biases to 0. The specific process is as follows:

[0083] 1) After initializing the weights W using the Xavier method in the cross-modal affinity network, W satisfies the following Gaussian distribution:

[0084]

[0085] Where n represents the number of input units in this layer of the network, that is, the number of input feature maps of the convolutional layer.

[0086] 2) In the entire network, all biases are initialized to 0, i.e., bi = 0.

[0087] 3-2. The Adam optimization algorithm is used to optimize the network model. The network loss function consists of two parts, and the specific process can be expressed by formula (5):

[0088]

[0089] Where, λ cls , λ kernelThis is a hyperparameter used to balance the loss; y represents the true mask of the target. This indicates the predicted segmentation result.

[0090] S4. After training, the video frame sequence and natural language description are used as input to the network. The parameters learned in step S3 are used to gradually locate and decode the video frames, and finally the segmentation result is obtained.

[0091] This invention presents a few-sample reference video target segmentation method based on cross-modal affinity. By learning semantic information of a new category using a small amount of data, it can handle complex and diverse real-world scenarios. It not only improves segmentation performance but also utilizes far fewer resources than existing methods and outperforms current advanced methods in objective quantitative measurement, showing broad application prospects in video editing and human-computer interaction.

[0092] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to specific implementations. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A small sample reference video object segmentation method based on cross-modal affinity, characterized in that, Comprising the following steps: S1, using the existing reference video target segmentation dataset, a dataset suitable for small sample reference video target segmentation task is constructed; S2, a cross-modal affinity network is constructed for model training; The constructed cross-modal affinity network takes the data of the support set and the query set as input, first extracts visual features and text features through a weight-shared feature extraction network, the feature extraction network adopts a residual network including sequentially connected multiple convolution layers, multiple pooling layers, multiple residual unit modules, a single fully connected layer, and a large-scale pre-trained text model based on Transformer; then the multi-head cross-modal attention mechanism is used to fuse the visual features and text features in the support set and the query set, the fused multi-modal features are used to calculate the self-affinity within the query set, and the obtained self-affinity features and the multi-modal features of the support set are used to calculate the affinity relationship to obtain more robust features; finally, in order to locate the most relevant target and gradually decode the features, the obtained cross-modal affinity features are sent to a mask generation module to obtain the final segmentation mask. The constructed cross-modal affinity network is divided into four modules, which are: A multi-modal fusion module for fusing visual features and text features; A self-affinity module for calculating self-affinity features in the query set; A cross-affinity module for calculating cross-affinity features between the query set and the support set; A mask generation module for generating the final segmentation result; S3, train the network constructed in step S2 according to the dataset prepared in step S1, and save the training parameters; S4, taking the video frame sequence and the natural language description as the input of the network, and taking the segmentation result obtained by using the parameters learned in step S3 as the output.

2. The method of claim 1, wherein the method is based on cross-modal affinity. The process of preparing a dataset suitable for small sample reference video target segmentation task in step S1 is as follows: for each video and its corresponding natural language description in the existing reference video target segmentation dataset, first select the video containing two different categories of segmentation targets and discard it; then sort the remaining videos by the number of segmentation target categories, and discard the categories with less than 10 videos and more than 50 videos; after the above processing of all videos in the dataset, a dataset suitable for small sample reference video target segmentation task can be obtained, which contains 1668 videos and is divided into 48 categories.

3. The method of claim 2, wherein the method is based on cross-modal affinity. In the multi-modal fusion module, before feature fusion, the visual features need to be dimensionally mapped, and the channel number of the multi-scale features is unified to 256, the specific steps are as follows: first pass through three 3x3 convolution layers and a regularization layer to map the channel number of the first three layers of the multi-scale features to 256; then pass the last layer of the multi-scale features through a 1x1 convolution layer and a regularization layer to obtain the final feature with a unified channel number; The multimodal fusion module contains visual features and text features. For the interaction between visual features and text features, specifically, multi-modal information is fused using multi-head cross-attention to obtain a new multi-scale feature map. The specific process is represented by formula (1): （1） wherein, represent visual features of the support, represent linguistic features of the support, represent visual features of the query, represent linguistic features of the query.

4. The small sample reference video object segmentation method based on cross-modal affinity according to claim 3, characterized in that: In the self-affinity module, including 3x3 convolution layer, matrix calculation module, normalization layer, residual connection; the fused multi-modal features in the query set pass through the 3x3 convolution layer to obtain three different vectors , , , the three vectors are sent to the matrix calculation module to calculate the self-affinity, and then pass through the normalization operation and the residual connection, and the specific process is represented by formula (2): （2） wherein, is the number of dimensions of the hidden layer, is the computed affinity matrix, LN is a normalization operation.

5. The small sample reference video object segmentation method based on cross-modal affinity according to claim 4, characterized in that: In the cross-affinity module, including 3x3 convolution layer, matrix calculation module, normalization layer, residual connection; the fused multi-modal features in the support set pass through the 3x3 convolution layer to obtain two different vectors as , , two different vectors, the output of the self-affinity module are jointly input into the matrix calculation module to calculate the cross-affinity, and then pass through the normalization operation and the residual connection, and the specific process is represented by formula (3): （3） wherein, is the number of dimensions of the hidden layer, LN is a normalization operation.

6. The small sample reference video object segmentation method based on cross-modal affinity according to claim 5, characterized in that: In the mask generation module, a Transformer architecture and a feature pyramid network are included. The Transformer architecture includes a six-layer encoder-decoder structure. In the decoder, two lightweight auxiliary heads are included to obtain the final target mask. In the feature pyramid network, four layers of hierarchical features of different scales are included. Cross-attention is used to interact visual features and language features. The specific process is represented by formula (4): （4） wherein, is the visual feature of each level, represents the text feature corresponding to the query set; finally, the features of the last layer are passed through a 3x3 convolution layer to obtain the final feature map.

7. The method of claim 6, wherein the method is based on cross-modal affinity. The Adam optimization algorithm is used to optimize the network model. The loss function of the network contains two parts. The specific process is represented by formula (5): （5） wherein, , is a hyperparameter to balance the loss, denotes the ground truth mask of the target, denotes the predicted segmentation result.

8. The small sample reference video object segmentation method based on cross-modal affinity according to claim 7, characterized in that: Step S4 updates the network parameters using the gradient descent method, which is represented by formula (6): （6） wherein, denotes the weight update value of this time, while denotes the weight update value of the last time, while is the weight of the last gradient value, is the learning rate, is the gradient.

Images