An unsupervised video object segmentation algorithm based on heterogeneous transformer

Abstract

An unsupervised video object segmentation algorithm based on heterogeneous Transformer, two different fusion strategies based on Transformer are designed for the shallow and deep stages of the appearance-motion feature fusion of the unsupervised video object segmentation network, which are global context sharing Transformer and semantic aggregation-embedding Transformer. The global context sharing Transformer can learn the global shared context information between video frames with lower computational complexity, and the semantic aggregation-embedding Transformer models the semantic correlation of the foreground and background respectively and further reduces the computational complexity through the soft aggregation of the feature vector. Based on the two fusion modules, a hierarchical heterogeneous Transformer architecture is designed for the unsupervised video object segmentation task, which can achieve the most advanced performance with lower computational complexity.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of machine learning, semantic segmentation, unsupervised video object segmentation, and relates to feature extraction network Swin-Transformer, semantic segmentation decoder SegformerMLPHead, global context network GCNet and visual Transformer algorithm, and specifically relates to an unsupervised video object segmentation algorithm based on heterogeneous Transformer. BACKGROUND

[0002] As one of the basic tasks in the field of computer vision, semantic segmentation is a core technology for understanding complex scenes. Semantic segmentation is usually defined as a task of predicting the class of each pixel, i.e., the class of the object to which the pixel belongs. The first work FCN based on deep learning method in the field of semantic segmentation first uses a fully convolutional neural network and pooling operation to extract low-resolution features with deep semantic information. Subsequent works represented by the DeepLab series and PSPNet enhance the global spatial information by expanding the receptive field of the neural network to obtain more accurate segmentation results. Most of the subsequent works are inspired by Non-local Network to explore capturing global semantic context information through attention mechanism to improve the performance of segmentation. Recent works introduce visual Transformer into the field of semantic segmentation and have achieved great success.

[0003] As a branch of the semantic segmentation task, unsupervised video object segmentation aims to discover the most eye-catching objects in a video sequence, and therefore can be defined as a video semantic segmentation problem with two classes. Unlike static image segmentation which mainly relies on appearance features, unsupervised video object segmentation further explores temporal motion information to obtain reliable and temporally consistent segmentation results. Mainstream methods such as FSNet(Full-duplex Strategy Network) proposed by Ji Geng et al. and AMCNet(Attentive Multi-Modality Collaboration Network) proposed by Yang Shu et al. mainly use hand-designed feature fusion modules to aggregate appearance and motion information and apply the designed fusion modules indiscriminately to the multi-stage feature fusion process. Although these works have promoted the progress and development of unsupervised video object segmentation, how to design a more suitable method for multi-stage appearance-motion feature fusion for the task of unsupervised video object segmentation is still an open problem.

[0004] Recently, visual Transformer has made great breakthroughs in many computer vision tasks, thanks to its powerful global attention modeling capability and flexibility of multi-modal fusion. However, this advantage has not been fully explored in the field of unsupervised video object segmentation. The baseline method of the present application is to use a standard visual Transformer module as a fusion module for appearance and motion features. Preliminary experiments show that for each feature fusion stage, directly feeding the appearance and motion features into a standard visual Transformer module after concatenation can achieve state-of-the-art performance, but at the cost of excessive computational complexity and long inference time. Therefore, how to effectively reduce the computational cost while maintaining high accuracy is a key problem for the successful application of visual Transformer in the field of unsupervised video object segmentation. SUMMARY

[0005] To solve the above problems, the present application designs two Transformer-based modules, namely context-shared Transformer module and semantic aggregation-embedding Transformer module. Both modules can greatly reduce the computational cost while maintaining the accuracy of the standard visual Transformer, thereby enabling the efficient application of visual Transformer in the task of unsupervised video object segmentation. Based on these two modules, the present application proposes a high-performance, lightweight heterogeneous Transformer network architecture to solve the task of unsupervised video object segmentation.

[0006] Technical scheme of the present application:

[0007] An unsupervised video object segmentation algorithm based on heterogeneous Transformer, comprising a context-shared Transformer module and a semantic aggregation-embedding Transformer module, and a heterogeneous Transformer network architecture designed based on the two modules:

[0008] The heterogeneous Transformer network architecture comprises an appearance feature extraction network, a motion feature extraction network, two context-shared Transformer fusion modules, two semantic aggregation-embedding Transformer fusion modules, and a decoder. Both feature extraction networks use Swin-Tiny, and the decoder uses the segmentation head based on the full connection layer designed in Segformer. The two feature extraction networks extract appearance and motion features to four stages, respectively, and at each stage l(l∈{1,2,3,4}) the preliminary fusion features are obtained by concatenating the channel dimensions of the appearance and motion features where c l represents the fusion feature dimension of the lth stage, wl and h l respectively represent the width and height of the fused feature resolution at stage l. For convenience, the subscript l is omitted in the following equations after specifying the fused feature stage l.

[0009] A standard visual Transformer module consists of a multi-head attention computation module with a residual structure and a feed-forward neural network module with a residual structure. The context-shared Transformer module simplifies the multi-head attention computation in the standard visual Transformer module by global context modeling, which computes a shared and query-independent weight map for all query feature vectors. The global context modeling contains a query-independent spatial attention computation and a channel attention computation. Specifically, after obtaining the fused feature Xl (simultaneously as the query feature vector for global context modeling), a single-channel attention weight map Wl is first generated by a 1 x 1 convolution and a SoftMax function To further model the correlation between channels, two groups of channel attention modules composed of a 1 x 1 convolution, a batch normalization, and a ReLU function are used to adjust the weighted representation Wl g The residual structure is used to aggregate the global context information and the fused feature Xl after global context modeling. The output of the global context modeling is sent to the feed-forward neural network module with a residual structure in the standard visual Transformer to obtain the final fused feature Xl

[0010] Although the algorithm is relatively simple, the context-shared Transformer module significantly accelerates the inference speed of the standard Transformer (from 3 frames per second to 36 frames per second) without affecting high performance.

[0011] The core idea of the semantic aggregation-embedding Transformer is to model the semantic correlation of foreground and background respectively and reduce the computational cost at the same time. This module contains two parallel and symmetric branches to process the features of foreground and background respectively, and each branch mainly contains the selection of query feature vectors (query), key-value feature vector (key-value) soft aggregation, correlation modeling computation, and query feature vector embedding. Among them, the query feature vector (query) and the key-value feature vector (key-value) are the inputs of the standard visual Transformer module.

[0012] For the fused representation Xl First, a 1x1 convolution and a sigmoid function are used to generate a single-channel feature vector selection heat map Based on this heat map, the query feature vector X F = X[H i ≥ F th ] belonging to the foreground and the query feature vector X B = X[H i < B th ] belonging to the background are selected respectively, wherein F th and B th are two thresholds for determining foreground and background selection, and H i is the corresponding value of the heat map at position i.

[0013] Taking the foreground branch as an example, in order to obtain the key-value feature vector pair highlighted by the foreground, first, the dot product of the fused feature X and the heat map H is performed to obtain the mask-enhanced foreground feature vector sequence Then, a feature vector soft aggregation mechanism is used to obtain a more compact compressed representation This mechanism compresses the foreground feature vector sequence by learning a set of representation conversion matrices . Then the query feature vector X F corresponding to the foreground and the foreground-enhanced compressed key-value feature vector X ce are sent to the standard visual Transformer for attention calculation to model and enhance semantic correlation and update the corresponding semantic representation. The background branch process is consistent with the foreground. This way can greatly reduce the computational cost. The output of the visual Transformer module is then embedded back into the initial fused feature X according to the corresponding index of the query feature vector selection stage and obtains the final fused feature S.

[0014] In actual implementation, in the key-value feature vector aggregation stage, k is set to 1 / 9n, which can reduce the computational amount of multi-head attention calculation to 10 / 81 without affecting the performance.

[0015] Compared with the baseline method of using a standard visual Transformer to indiscriminately fuse multi-scale features, the heterogeneous Transformer network architecture can improve the segmentation performance and increase the inference speed from 3 frames per second to 39 frames per second, meeting the task requirements of high precision and real-time of unsupervised video object segmentation.

[0016] The beneficial effects of the present application are:

[0017] (1) The appearance motion information is fused in multiple levels using the architecture based on the Transformer, including a feature extraction network and a feature fusion network, so that higher accuracy can be achieved compared to the unsupervised video target segmentation network based on the convolutional neural network.

[0018] (2) The heterogeneous feature fusion mode is used in the shallow and deep fusion stages, so that the needs and characteristics of feature fusion at different levels can be more targeted. This heterogeneous fusion strategy not only can fully exert the advantages of high precision of the Transformer structure, but also can make the network more lightweight, so that the task requirements of high precision and real-time of the unsupervised video target segmentation task are met. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The algorithm flowchart of the heterogeneous Transformer for the unsupervised video target segmentation network.

[0020] Figure 2 The algorithm flowchart of the context-shared Transformer and the semantic aggregation-embedding Transformer. DETAILED DESCRIPTION

[0021] The specific embodiments of the present application will be further described below in combination with the drawings and technical solutions.

[0022] Figure 1 The algorithm flowchart of the heterogeneous Transformer for the unsupervised video target segmentation network. The heterogeneous Transformer includes an appearance feature extraction network, a motion feature extraction network, four multi-scale fusion modules, and a decoder. An input video frame and an optical flow map calculated from the current frame and adjacent frames are input, and the appearance feature extraction network and the motion feature extraction network can extract multi-stage appearance and motion features, respectively. The multi-stage fusion module receives the multi-stage appearance and motion features and outputs the final to the multi-scale feature fusion decoder. The present application adopts an advanced Swin-Tiny backbone network as the feature extraction network and a lightweight SegformerMLPHead as the multi-scale fusion feature decoder.

[0023] Figure 2 The algorithm flowchart of the context-shared Transformer and the semantic aggregation-embedding Transformer. The four-stage appearance and motion features extracted by the appearance and motion feature extraction networks are first fused in the channel dimension using a 1x1 convolution to obtain preliminary fusion representations The first two stages of the fusion representation are sent to the context-shared Transformer and the output is obtained The fusion representation of the last two stages is fed into a semantic aggregation-de-embedding Transformer and the output is obtained

[0024] The present application uses a pre-trained RAFT to generate an optical flow map for video data. All input pictures are scaled to a spatial resolution of 512x512. In the model training phase, the present application uses data augmentation including random horizontal flipping and random photometric distortion transformation to enhance the generalization of the model, and uses an AdamW optimizer with a fixed learning rate of 6e-5 and a binary cross-entropy loss function to train the model end-to-end. The present application pre-trains the model for 300 rounds using the Youtube-VOS dataset and fine-tunes the model for 100 rounds using the DAVIS-2016 and FBMS datasets. The binary threshold of the segmentation result is set to 0.5. The model is trained throughout using 4 NVIDIA 3090 graphics cards with a batch size of 8.

[0025] The feature extraction network structure is as follows:

[0026] Operation Downsampling ratio Number of attention heads Dimension Input size Output size Swin Stage1 4× 3 96 384×384 96×96 Swin Stage2 8× 6 192 96×96 48×48 Swin Stage3 16× 12 384 48×48 24×24 Swin Stage4 32× 24 768 24×24 12×12 .

Claims

1. A method for unsupervised video object segmentation based on heterogeneous Transformer, characterized in that, The unsupervised video object segmentation algorithm based on the heterogeneous Transformer comprises a context-shared Transformer module, a semantic aggregation-embedding Transformer module and a heterogeneous Transformer network architecture designed based on the two modules: The heterogeneous Transformer network architecture comprises an appearance feature extraction network, a motion feature extraction network, two context-shared Transformer fusion modules, two semantic aggregation-embedding Transformer fusion modules, and a decoder, wherein the appearance feature extraction network and the motion feature extraction network both use Swin-Tiny, and the decoder uses a segmentation head based on a full connection layer designed in Segformer; the appearance feature extraction network and the motion feature extraction network extract appearance features and motion features of four stages respectively, and in each stage The appearance features and the motion features are spliced in a channel dimension to obtain preliminary fusion features , wherein represents the fusion feature dimension of the first stage , wherein and respectively represent the width and the height of the fusion feature resolution of the first stage ; and . The standard visual Transformer module mainly consists of a multi-head attention computation module with a residual structure and a feed-forward neural network module with a residual structure; the context-shared Transformer fusion module simplifies the multi-head attention computation module with a residual structure in the standard visual Transformer module by global context modeling to compute a shared and independent weight map for all query feature vectors; the global context modeling includes a query feature vector independent spatial attention computation and a channel attention computation; specifically, the fusion feature of the shallow stage is used as the query feature vector for the global context modeling, and first generates a single-channel attention weight map through a convolution layer and a function; the single-channel attention weight map obtains a query feature vector shared weighted representation through the fusion feature of the shallow stage ; to further model the correlation between channels, two groups of channel attention modules composed of a convolution layer, batch normalization and a ReLU function are used to optimize the weighted representation .​​​​​​ Residual structure is used to aggregate global context information and shallow stage after global context modeling Fused features The output of global context modeling is fed into the feedforward neural network module of the residual structure of the standard visual Transformer module to obtain the final fused features ; The semantic aggregation-embedded Transformer fusion module models the feature correlation of foreground and background respectively, and contains two parallel branches to process the features of foreground and background respectively, wherein each branch contains query feature vector selection, key-value feature vector soft aggregation, correlation modeling calculation and query feature vector embedding; wherein the query feature vector and the key-value feature vector are the inputs of the standard visual Transformer module; for the fusion representation from the deep stage , first, a convolutional layer and function are used to generate a single-channel feature vector selection heat map ; based on the feature vector selection heat map, the query feature vector belonging to the foreground and the query feature vector belonging to the background are selected respectively, wherein and are two thresholds for determining the foreground and background selection, is the corresponding value of the heat map at position ;​​ The process of calculating the relevance of the foreground branch: In order to obtain the key-value feature vector pairs that are prominent in the foreground, the deep stage is first... Fusion characteristics and heat map Perform a dot product to obtain a sequence of foreground feature vectors for mask enhancement. Subsequently, a feature vector soft aggregation mechanism is employed to obtain a more compact compressed representation. This mechanism learns a set of representation transformation matrices. To compress the foreground feature vector sequence; then query the feature vector corresponding to the foreground. Key-value eigenvectors for foreground enhancement compression The input is fed into a standard visual Transformer for attention computation to model and enhance semantic relevance and update the corresponding semantic representation; due to the symmetry of the foreground and background branch processes, the background branch process is consistent with the foreground branch; the output of the visual Transformer module is then embedded back into the initial fused features according to the corresponding index of the query feature vector selection stage. The final fusion characteristics are obtained in the process. .

Images