Unsupervised video target segmentation method based on motion-appearance multi-level clustering

Abstract

The invention provides an unsupervised video target segmentation method based on motion-appearance multi-level clustering. The method comprises the following steps: inputting a video frame into a backbone network to extract motion features; designing a position-channel attention module to enhance spatio-temporal context modeling; motion and appearance features are fused through a multi-level clustering refinement strategy; a reconstructed optical flow is used as a motion intensity indicator, and the complex scene processing capability is improved by combining subspace clustering multi-view analysis; and finally outputting the target segmentation mask. According to the method, the discriminative motion features can be extracted from the FlowFormer + + optical flow network and the position-channel attention module, and the motion features and the appearance features in the Vision Transform backbone network can be effectively fused by the proposed multi-level clustering refinement strategy; according to subspace clustering multi-view analysis designed by the invention, larger weights in features can be concentrated in a foreground target area, useless background information is filtered out, and compared with an existing unsupervised video target segmentation method, the method has higher segmentation precision.

Description

Technical Field

[0001] The invention relates to the field of video processing, and in particular to an unsupervised video target segmentation method based on motion-appearance multi-level clustering. Background Art

[0002] Video object segmentation (VOS) is a long-standing and important task in the field of computer vision. Its goal is to assign pixels in a video frame to different clusters so as to identify and segment the main objects in the video. According to the processing method in the test phase, VOS can be divided into three methods: interactive, semi-supervised, and unsupervised. Among them, unsupervised video object segmentation (UVOS) has received special attention because it does not rely on ground truth masks. It aims to automatically identify and segment the most visually salient objects in video sequences.

[0003] Early UVOS methods mainly segmented video pixels into continuous spatiotemporal groups based on motion and appearance information. Representative methods include motion analysis, trajectory analysis, and hierarchical segmentation based on superpixels or supervoxels. Although these methods can effectively perform preliminary over-segmentation of videos, they often fail to achieve object-level pattern modeling. In recent years, thanks to the availability of large-scale datasets, deep learning methods have shown great potential in the field of UVOS. Deep convolutional neural networks (CNNs) have shown powerful feature extraction and learning capabilities in various visual tasks, and have therefore been widely introduced into UVOS tasks.

[0004] At present, the most effective UVOS methods have made significant progress by mainly utilizing various visual cues (such as motion and appearance) and attention mechanisms. In 2021, Yang et al. (C. Yang, H. Lamdouar, E. Lu, A. Zisserman, W. Xie, Self-supervised video object segmentation by motion grouping, in: Proceedings of the IEEE / CVF International Conference on Computer Vision, 2021, pp. 7177-7188.) developed a self-supervised task of reconstructing the optical flow of unlabeled video data using transformers to segment foreground objects in videos. In 2020, Lu et al. (X. Lu, W. Wang, J. Shen, D. Crandall, J. Luo, Zero-shot video object segmentation with co-attention siamese networks, IEEE Trans. Pattern Anal. Mach. Intell. 44 (4) (2020) 2228-2242.) used the intrinsic correlation between video frames and combined with the global co-attention mechanism to achieve video object segmentation without relying on optical flow. In 2024, Wang et al. (J. Wang, H. Cao, C. Sun, Z. Huang, Y Zhang, Motion perception-driven multimodal self-supervised video object segmentation, The Visual Computer (2024) 1-18.) proposed a two-stream network to extract features from RGB frames and optical flow, and adopted a dual-path encoder-decoder structure to model spatiotemporal features through self-supervised training to segment the main objects in the video.

[0005] Although the deep learning-based UVOS method has made great progress compared to traditional methods, it is still insufficient in obtaining segmentation results with uniform salient areas and precise boundaries. The main problems are: the existing methods are not effective in fusing motion and appearance features, and it is difficult to effectively balance these two types of information in different scenes, resulting in unstable and inaccurate segmentation results in complex video sequences; the spatiotemporal consistency of foreground objects in appearance and motion features is often ignored, resulting in possible jumps in segmentation results between adjacent frames, especially in scenes where objects move quickly or lighting conditions change dramatically. Summary of the invention

[0006] In order to solve the problems existing in the background technology, the present invention proposes an unsupervised video object segmentation method based on motion-appearance multi-level clustering.

[0007] The present invention provides the following technical solutions:

[0008] An unsupervised video object segmentation method based on motion-appearance multi-level clustering includes the following steps:

[0009] S1, input the video frame into the FlowFormer++ optical flow network, extract motion features, and generate forward optical flow F t→t+n and reverse optical flow F t+n→t , and input the video frame into the Vision Transformer network to extract static salient features D t ;

[0010] S2, the optical flow feature F t→t+n and F t+n→t Input into the autoencoder with position-channel attention module, initialized by minimizing the reconstruction loss, and obtain enhanced optical flow features and reconstructed optical flow;

[0011] S3, combined with D t 、F t→t+n and F t+n→t The dynamic similarity of the features is calculated to form a similarity matrix A; a multi-level clustering refinement strategy is adopted to perform power iteration clustering, subspace clustering and K-means|| clustering in sequence to gradually refine the segmentation results;

[0012] S4 uses the reconstructed optical flow as a motion intensity indicator and combines it with multi-view analysis of subspace clustering to finally output the target segmentation mask.

[0013] Furthermore, the position-channel attention module includes position attention and channel attention, wherein the position attention is used to encode spatial context information to enhance local feature expression, and the channel attention is used to model the interdependence between different channels to improve semantic feature representation.

[0014] Furthermore, the optical flow feature Ft →t+n Through the encoder Φ enc Downsample to obtain a feature map Z of size 10×60×107 t→t+n , the downsampled feature Z t→t+nThen it undergoes a 1×1 convolution operation to adjust the number of feature channels while keeping the spatial dimension unchanged; then the convolved features are simultaneously subjected to maximum pooling and average pooling operations to capture the most significant features and retain the overall feature distribution information, respectively; the pooled results are concatenated together to generate two feature maps B and C of size 1×60×107; B and C are reshaped into one-dimensional vectors of 1×6420; in the position attention mechanism, B is first transposed, then matrix multiplied with C, and the softmax function is applied to the result to obtain the position attention matrix; at the same time, the input is processed through another 1×1 convolution layer to obtain a feature map D of size 10×6420; finally, D is matrix multiplied with the position attention matrix to obtain the feature Z containing global context information p , the formula is as follows

[0015]

[0016] In the formula, α is the learnable coefficient, and its initial value is 0.1. t+n→t , and execute the same processing flow.

[0017] Furthermore, the optical flow feature F t→t+n Through the encoder Φ enc The feature Z obtained by downsampling t→t+n , the input feature Z t→t+n Reshape into feature maps E and F, where the size of E is 6420×10 and the size of F is 10×6420; perform matrix multiplication on E and F to calculate the channel attention matrix; process the channel attention matrix through the softmax function to weight the feature F; reshape to obtain the feature Z strengthened by the channel attention c , the formula is as follows

[0018]

[0019] In the formula, β is a learnable parameter with an initial value of 0.1. t+n→t , and execute the same processing flow.

[0020] Furthermore, the implementation process of the power iteration clustering includes the following steps:

[0021] T1, calculate the degree matrix Λ based on the similarity matrix A, and then obtain the random walk probability matrix W;

[0022] T2, define the initial distribution vector v0, use the random walk probability matrix W to calculate the initial eigenvector Update, where the size of W is H×W;

[0023] T3, when ||∈ m+1 -∈m || ∞ When ≤τ, the iteration stops and the final feature vector is obtained. This vector groups the contents with high similarity together. The operation is as follows

[0024] Λ=diag({∑ j＝1 A ij} i )

[0025] W=Λ -1 A

[0026]

[0027] In the formula, diag means calculating a diagonal matrix, In the initial feature vector represents the proportion of the i-th row in the similarity matrix A in the sum of the elements of the entire matrix A.

[0028] Furthermore, the implementation process of the subspace clustering includes the following steps:

[0029] F1, define the subspace prototype Each of these prototypes is a 10-dimensional subspace S j The unit vector of

[0030] F2, fusion of features X optimized by position-channel attention module t→t+n , X t+n→t , get the weighted features

[0031] F3, calculate the affinity matrix M between the pixel and the prototype;

[0032] F4, by optimizing the problem Determine the optimal allocation matrix P;

[0033] F5, update subspace prototype The specific operations of each step are as follows:

[0034]

[0035] Where, [;] represents the concatenation operation, ο represents the Hadamard product, for The nth 10-dimensional vector, v (n) for The nth 30-dimensional vector, |S j | represents the subspace S j The total number of elements in .

[0036] Furthermore, the process of K-means ‖‖ clustering and final segmentation mask generation includes the following steps:

[0037] H1, use K-means||clustering to calculate the cluster label I of each subspace j (j=1,...,30), labels are based on subspace prototypes The affinity between the pixels M _，j get;

[0038] H2, introduces motion intensity information to reconstruct optical flow To adjust the clustering labels and improve the accuracy of foreground area positioning;

[0039] H3, combining subspace multi-angle analysis to generate foreground areas and background area The formula is as follows

[0040]

[0041] In the formula, and Represent the foreground and background areas respectively, represents the normalized motion intensity of the reconstructed optical flow at position i, is the indicator function.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. The present invention designs an autoencoder with a position-channel attention module to learn distinctive motion features and enhance spatiotemporal context modeling. In addition, the present invention proposes a multi-level clustering optimization strategy, which uses power iteration clustering, subspace clustering and K-means|| clustering to gradually optimize the segmentation results, thereby achieving a synergistic fusion of appearance and motion features, thereby bringing more accurate and coherent target segmentation.

[0044] 2. The present invention utilizes reconstructed optical flow as an indicator of motion intensity and combines it with multi-view analysis of subspace clustering, which greatly improves the accuracy and robustness in processing complex scenes with diverse motion patterns. Compared with existing unsupervised video target segmentation methods, the present invention shows significant advantages in terms of regional similarity and contour accuracy indicators. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 It is the overall network block diagram of the present invention.

[0047] Figure 2 This is the position-channel attention module diagram of the present invention.

[0048] Figure 3 This is a flow chart of feature fusion optimization based on subspace clustering of the present invention.

[0049] Figure 4 This is a comparison chart of the subjective visual effects of the present invention.

[0050] Figure 5 The present invention is a flow chart of the method. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] An unsupervised video object segmentation method based on motion-appearance multi-level clustering, see Figure 1-5 As shown, including:

[0054] Step 1: First, input the video frame into FlowFormer++ and Vision Transformer to extract optical flow and appearance features; input the optical flow into the autoencoder with a position-channel attention module, initialize it by minimizing the reconstruction loss, and obtain enhanced optical flow features and reconstructed optical flow.

[0055] Step 2: Calculate the random transfer matrix based on the optical flow and appearance features, and use the dynamic similarity calculation to initialize the distribution vector. Perform power iteration clustering on the random transfer matrix and the initial distribution vector, iteratively update until convergence, and obtain a stable feature distribution; then initialize the prototype of subspace clustering, use Gaussian distribution and perform L2 normalization; then fuse the enhanced optical flow features generated by the position-channel attention module with the stable feature distribution generated by power iteration clustering, and perform subspace clustering, including determining cluster assignments, updating prototypes, and optimizing network parameters. This process continues until convergence conditions are met;

[0056] Step 3: Perform K-means|| clustering on the results of subspace clustering, combine the reconstructed optical flow generated by the autoencoder as a motion clue, and subspace clustering multi-view analysis to refine the results of K-means|| clustering, and finally assign foreground and background labels to each pixel to generate the segmentation result.

[0057] Based on the above, the position-channel attention module is divided into two parts: position attention and channel attention. Position attention is to weight the input features in the spatial dimension, and channel attention is to weight the features in the channel dimension. Channel attention processing is performed after position attention processing.

[0058] Based on the above, the position attention process first uses a 1×1 convolution layer to process the optical flow features to obtain three new feature maps. Subsequently, two of the feature maps are subjected to maximum pooling and average pooling operations to capture the most significant features and overall feature distribution information. The pooled results are concatenated together to form two new feature maps. The two feature maps are reshaped and matrix multiplied, and then the position attention matrix is ​​obtained through a softmax operation. This matrix is ​​then matrix multiplied with the third feature map (i.e., the original 1×1 convolution output without pooling) to obtain features containing global context information. The formula is as follows

[0059]

[0060] In the formula, α is a learnable coefficient, B, C and D represent the three processed feature maps respectively, and HW represents the number of pixels in the feature map.

[0061] Based on the above, the channel attention process directly reshapes the optical flow features to obtain two new feature maps. The two feature maps are matrix multiplied, and then the channel attention matrix is ​​obtained through softmax operation. Then this matrix is ​​matrix multiplied with one of the feature maps to obtain the channel attention enhanced features. The formula is as follows

[0062]

[0063] Where β is a learnable parameter, E and F represent two reshaped feature maps, and p = 10 represents the number of channels of the feature map.

[0064] Based on the above, the features obtained by position attention and channel attention are weighted fused with the original input features in multiple stages to obtain the final dual attention processed feature X t-n→t . Specifically, the spatial information of the features is first refined through two consecutive 3×3 convolutional layers, and each convolutional layer is followed by batch normalization and ReLU activation function to enhance the expressiveness of the features. Subsequently, a 1×1 convolutional layer is used to reduce the dimension of the features and reorganize the information between channels, and a dropout mechanism is introduced to prevent overfitting. This multi-stage processing method can effectively fuse feature information of different scales and types, and improve the discriminability and robustness of the features. When the input is the reverse optical flow feature, the corresponding weighted feature is obtained through the same processing steps, ensuring the consistency of forward and reverse feature processing.

[0065] Based on the above, the multi-level clustering refinement strategy includes three parts: power iteration clustering, subspace clustering and K-means|| clustering. These methods are applied in sequence to gradually refine the clustering results and improve the segmentation accuracy. The strategy first calculates the dynamic similarity matrix, then starts from a macro perspective and gradually introduces more detailed perspectives to ensure that the segmentation process is comprehensive and in-depth.

[0066] Based on the above, the power iteration clustering process first uses the similarity matrix A to calculate the random walk probability matrix W, and then uses the power iteration method to calculate the main eigenvector of the matrix. It is calculated by the rows and proportions of the similarity matrix, and then the random walk probability matrix W is used to iteratively update the vector until convergence. The final feature vector effectively groups similar content, and the formula is as follows

[0067] Λ=diag({∑ j＝1 A ij} i )

[0068] W=Λ -1 A

[0069]

[0070] In the formula, m represents the number of iterations.

[0071] Based on the above, the subspace clustering process first defines the subspace prototype These prototypes summarize the feature directions of the corresponding subspace. Before comparing the feature vector with the prototype, the features optimized by the channel position attention module are fused. Specifically, the cosine similarity of these features is calculated and used as weights for fusion. Then, the optimized features are used Initialize the prototype to enhance the ability to capture dynamic content. Calculate the affinity between pixels and all prototypes through softmax normalization to form a global affinity matrix M. In order to enhance the global clustering balance and reduce the computational complexity, define an optimization problem for local prototype allocation and use the Sinkhom algorithm to solve it. Finally, update the prototype according to the optimized allocation results. The operation is as follows

[0072]

[0073] In the formula, [;] represents the splicing operation, represents the Hadamard product, After remodeling The nth 10-dimensional vector, v (n) After remodeling The nth 30-dimensional vector, m (n) Contains the nth pixel in the video frame and all prototypes Affinity, 1 HW and 1 k are unit vectors of length HW and length k, respectively, r = 1 HW , h(P) represents the entropy of the allocation matrix P, and k>0 is a parameter that controls smoothing.

[0074] Based on the above, the K-means|| clustering process uses the results of the previous operation to calculate cluster labels for each subspace. In order to improve the distinction between foreground and background, the reconstructed optical flow motion information is taken into account and the cluster labels are adjusted. If a pixel is classified as foreground in at least 10 subspaces and its weighted optical flow intensity exceeds the threshold, the pixel is considered to belong to the foreground, otherwise it belongs to the background. This method improves the robustness of clustering and the accuracy of foreground positioning. The formula is as follows

[0075]

[0076] In the formula, and Represent the foreground and background areas respectively, represents the normalized motion intensity of the reconstructed optical flow at position i, is the indicator function, 10 is the prototype The dimension of determines the maximum number of orthogonal pixels, representing the different angles from which pixels in a video frame can be viewed.

[0077] In step 1, the video frames are input into the FlowFormer++ optical flow network and the Vision Transformer network. The FlowFormer++ network extracts motion features and generates forward and backward optical flows. The network decides whether to process in blocks based on the image size and calculates motion information in the horizontal and vertical directions. The Vision Transformer network extracts static salient features, processes them through multiple Transformer blocks, and uses the self-attention mechanism to extract keys as appearance features.

[0078] The optical flow information is input to the autoencoder integrated with the position-channel attention module. The encoder consists of multiple convolutional layers to extract and compress feature information; the decoder uses multiple transposed convolutional layers to reconstruct the optical flow. The encoder maps the optical flow to the latent space and obtains the embedded features.

[0079] The position-channel attention module contains position attention and channel attention. Position attention encodes spatial context information and enhances local feature expression. Channel attention models inter-channel dependencies and improves semantic feature representation. The two mechanisms process features through matrix multiplication and softmax operations to enhance remote context representation. The network parameters are initialized by minimizing the reconstruction loss to obtain enhanced optical flow features and reconstructed optical flow. The loss function is:

[0080]

[0081] Where j is the pixel index and Ω is the entire spatial grid. t→t+n is the original optical flow, To reconstruct the optical flow.

[0082] In step 2, a random transfer matrix is ​​calculated based on the optical flow and appearance features, and the distribution vector is initialized using dynamic similarity calculation. Power iteration clustering is performed on the random transfer matrix and the initial distribution vector, and iterative updates are performed until convergence to obtain a stable feature distribution. Then the prototype of the subspace clustering is initialized Gaussian distribution is used and L2 normalization is performed. In order to make different prototypes learn features from different angles, it is necessary to ensure the orthogonality between prototypes. The formula is as follows

[0083]

[0084] In the formula, τ is a small constant. Based on the orthogonalization constraint, we introduce the prototype contrast loss function To ensure that the distinction between cluster prototypes is maximized:

[0085]

[0086] Where Ω is the entire spatial grid.

[0087] Then, the enhanced optical flow feature X generated by the position-channel attention module is t→t+n It is fused with the stable feature distribution generated by power iteration clustering, and subspace clustering is performed, including determining cluster assignment, updating prototypes, and optimizing network parameters. This process continues until the convergence condition is met. In order to enhance the similarity between the prototype and the features within the cluster to which it belongs, while reducing the similarity with the features outside the cluster, the intra-cluster and extra-cluster contrast loss function is used, which is expressed as:

[0088]

[0089] In step 3, the K-means|| algorithm is first used to calculate cluster labels for each subspace, which are based on the affinity between the subspace prototype and the pixel. Then, the motion information of the reconstructed optical flow is introduced to adjust the cluster labels to improve the accuracy of foreground-background distinction. When determining the foreground and background areas, the classification results of multiple subspaces and the motion intensity of the reconstructed optical flow are considered. If a pixel is classified as foreground in enough subspaces and its weighted optical flow intensity exceeds a preset threshold, the pixel is considered to belong to the foreground. In order to maximize the representation distance between the foreground and the background in the predicted mask, the contrast loss function is used The loss function is expressed as:

[0090]

[0091]

[0092] In the formula, Indicates prospects, Indicates background.

[0093] The entire loss process uses a multi-level contrast loss function to supervise the multi-layer network, including 1 loss function for feature reconstruction and 3 loss functions for clustering optimization. The full loss function is defined as follows:

[0094]

[0095] In the formula, To rebuild the losses; Foreground-background contrast loss, maximize the distance between foreground and background representations; is the prototype contrast loss, which ensures the orthogonality between cluster prototypes; It is the intra-class and inter-class contrast loss, which enhances the similarity between the prototype and the features of the cluster to which it belongs. r ,λ f ,λ p ,λ i is the weight coefficient of each loss function.

[0096] Specifically, Figure 1As shown, the present invention provides an unsupervised video target segmentation method based on motion-appearance multi-level clustering, and the operation of the method is as follows:

[0097] a. The method first perceives the dynamic changes of video content through an optical flow enhancement model. Specifically, the present invention uses an autoencoder framework to process optical flow information. The encoder consists of a multi-layer convolutional network and a maximum pooling layer with a structure of [64, MP, 128, MP, 256], where "MP" represents a maximum pooling with a step size of 2 and a convolution kernel size of 3×3. The decoder consists of a transposed convolutional layer and BatchNorm2d, which is used to restore the encoded features to the original resolution to reconstruct the optical flow. For the input forward optical flow F with a resolution of 480×856 t→t+n , we use the encoder Φ enc Map it to the r-dimensional latent space U and get the 10-dimensional embedding feature Z t→t+n , where Z t→t+n The size of is 60×107. The formula is as follows:

[0098]

[0099] In the formula, is a feature multi-layer perceptron (MLP) of size 10×10. For the reverse optical flow feature F t+n→t , performing the same processing flow. In appearance feature extraction, we use the vision transformer. Based on the assumption that the features of the same object at different positions should show consistent performance, we remove the last transformer block in the backbone network and use self-attention to extract its key as the appearance feature.

[0100] b. Figure 2 The position-channel attention module in is divided into two parts: position attention and channel attention. The position attention part firstly takes the feature Z obtained from the encoder as t-t+n Through a convolution layer with a stride of 1 and a kernel size of 1×1, and then a maximum pooling and average pooling operation with a stride of 2 and a kernel size of 3×3, two new feature maps B and C are generated. Then, the transpose of B and C are matrix multiplied, and the softmax function is applied to calculate the position attention matrix to reveal the spatial correlation between pixels. At the same time, Z t→t+n After another convolutional layer processing and reshaping, the feature map D is obtained. D is matrix multiplied with the position attention matrix and reshaped to obtain the feature Z containing global context information. p .

[0101] The channel attention part directly converts Z t→t+nReshape into two new feature maps E and F. The channel attention matrix is ​​calculated by transposing E and matrix multiplication of F. After softmax processing, the matrix is ​​weighted with feature F and then reshaped to obtain the channel attention enhanced feature Z c .

[0102] Finally, Z p and Z c With the original embedding feature Z t→t+n Fusion, get the feature X after dual attention processing t→t+n The same process is applied to Z t+n→t , and get the corresponding weighted feature X t+n→t .

[0103] c. Figure 1 In the power iteration clustering, the dynamic similarity is first calculated by combining the appearance features and optical flow features to form a similarity matrix A. Specifically, the appearance features D obtained from the vision transformer t , calculate the cosine similarity and weightedly fuse it with the similarity of the forward and backward optical flow features. The fused similarity matrix A is normalized and thresholded. Next, calculate the similarity matrix D, whose diagonal elements are the sums of the corresponding rows in the similarity matrix A. By multiplying A by the inverse matrix of D on the right, we get the random walk probability matrix W. The initial distribution vector It is obtained by dividing the row sum of A by the sum of A, which reflects the overall similarity of each position with all other positions. The power iteration process is implemented by iteratively updating the distribution vector. At each iteration, W is added to the current distribution vector Multiply and then perform L1 norm normalization to get The iteration continues until the difference between two consecutive iterations is less than the preset threshold τ. The formula is as follows:

[0104]

[0105] Λ=diag({∑ j＝1 A ij} i )

[0106] W=Λ -1 A

[0107]

[0108] ||∈ m+1 -∈ m || ∞ ≤τ

[0109] In the formula, α D =1,α F = 1. || ||1 represents the first norm, || || ∞Represents the infinite norm.

[0110] d. in Figure 3 In the proposed subspace clustering, feature fusion is performed. First, the optimized feature X from the position-channel attention module is t→t+n and X t+n→t The specific operation is as follows: calculate the cosine similarity between the two features, and use this similarity as a weight to fuse the features to obtain the weighted feature This similarity-based weighted fusion enhances spatiotemporal consistency and leverages the contextual information from the attention module to improve the representation of dynamic scenes. Next, we define the subspace prototypes Each of these prototypes is the corresponding subspace S j The unit vector of . When using weighted features To enhance dynamic content capture. During the optimization process, the feature vector With prototype Compare to update the clustering prototypes to enhance the representation and clustering efficiency. and The Hadamard product of is then normalized using softmax to form a global affinity matrix M, where the rows of M correspond to pixels and the columns correspond to prototypes. For each pixel, the subspace to which it is most likely to belong is determined by selecting the prototype with the largest affinity. Subsequently, based on these matching results, each prototype is updated by aggregating the pixel features corresponding to each prototype to more accurately represent the center of the subspace. The formula is as follows:

[0111]

[0112] Among them, argmax is used to determine the parameter position when the function reaches its maximum value.

[0113] e. Figure 1 In the above, based on the previous operation, the present invention successfully groups pixels according to the motion and saliency features of the input frame through dynamic iteration and subspace clustering methods. In order to make full use of the information obtained from different subspaces, the present invention applies the K-means|| clustering algorithm to calculate cluster labels for each subspace. These labels are obtained by evaluating the subspace prototypes. The affinity between the pixels M ，j To determine. Further, in order to improve the distinction between foreground and background, the present invention integrates the motion information extracted from the reconstructed optical flow to adjust these clustering labels. Finally, the clustering process is supervised and optimized by a multi-level contrast loss function, and a segmentation mask is output.

[0114] In order to verify the effectiveness of this method in the unsupervised video object segmentation task, the present invention conducted comparative experiments with multiple state-of-the-art methods on three public datasets (DAVIS2016, FBMS-59, and SegTrackv2). The experiment built a network based on the deep learning framework Pytorch. The experimental environment was set up on a Linux system, and an NVIDIA GeForce RTX3080 16GB graphics card was used to train and test the neural network model. In order to adapt to network training, the present invention adjusted the video frame to a resolution of 480×856 by random cropping. The experimental results are shown in Table 1 below:

[0115]

[0116] As shown in Table 1, the comparison between the experimental results of the present method on these three data sets and the latest methods shows that the PR curve of the present method performs well. Figure 4 As shown in the figure, from the perspective of subjective visual effect comparison, this method has strong capabilities in edge segmentation, position detection and detail representation of salient objects.

[0117] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. An unsupervised video object segmentation method based on motion-appearance multi-level clustering, characterized in that: The following steps are involved: S1, input the video frame into the FlowFormer++ optical flow network, extract motion features, and generate forward optical flow F t→t+n and reverse optical flow F t+n→t , and input the video frame into the Vision Transformer network to extract static salient features D t ; S2, the optical flow feature F t→t+n and F t+n→t Input into the autoencoder with position-channel attention module, initialized by minimizing the reconstruction loss, and obtain enhanced optical flow features and reconstructed optical flow; S3, combined with D t 、F t→t+n and F t+n→t The dynamic similarity of the features is calculated to form a similarity matrix A; a multi-level clustering refinement strategy is adopted to perform power iteration clustering, subspace clustering and K-means|| clustering in sequence to gradually refine the segmentation results; S4, uses the reconstructed optical flow as a motion intensity indicator, combined with multi-view analysis of subspace clustering, and finally outputs the target segmentation mask.

2. The unsupervised video object segmentation method based on action-appearance multi-level clustering according to claim 1, characterized in that: The position-channel attention module includes position attention and channel attention. The position attention is used to encode spatial context information to enhance local feature expression, and the channel attention is used to model the interdependence between different channels to improve semantic feature representation.

3. The unsupervised video object segmentation method based on action-appearance multi-level clustering according to claim 2, characterized in that: The optical flow feature F t→t+n Through the encoder Φ enc Downsample to obtain a feature map Z of size 10×60×107 t→t+n , the downsampled feature Z t→t+n Then, a 1×1 convolution operation is performed to adjust the number of feature channels while keeping the spatial dimension unchanged. The convolved features are then subjected to maximum pooling and average pooling operations to capture the most significant features and retain the overall feature distribution information, respectively. The pooled results are concatenated to generate two feature maps B and C of size 1×60×107; B and C are reshaped into one-dimensional vectors of 1×6420; in the position attention mechanism, B is first transposed, then matrix multiplied with C, and the softmax function is applied to the result to obtain the position attention matrix; at the same time, the input is processed through another 1×1 convolution layer to obtain feature map D, whose size is 10×6420; finally, D is matrix multiplied with the position attention matrix to obtain feature Z containing global context information p , the formula is as follows In the formula, α is the learnable coefficient, and its initial value is 0.

1. t+n→t , and execute the same processing flow.

4. The unsupervised video object segmentation method based on action-appearance multi-level clustering according to claim 2, characterized in that: The optical flow feature F t→t+n Through the encoder Φ enc The feature Z obtained by downsampling t→t+n , the input feature Z t→t+n Reshape into feature maps E and F, where the size of E is 6420×10 and the size of F is 10×6420; perform matrix multiplication on E and F to calculate the channel attention matrix; process the channel attention matrix through the softmax function to weight the feature F; reshape to obtain the feature Z strengthened by the channel attention c , the formula is as follows In the formula, β is a learnable parameter with an initial value of 0.

1. t+n→t , and execute the same processing flow.

5. The unsupervised video object segmentation method based on action-appearance multi-level clustering according to claim 1, characterized in that: The implementation process of the power iteration clustering includes the following steps: T1, calculate the degree matrix Λ based on the similarity matrix A, and then obtain the random walk probability matrix W; T2, defines the initial distribution vector Use the random walk probability matrix W to calculate the initial eigenvector Update, where the size of W is H×W; T3, when ||∈ m+1 -∈ m || ∞ When ≤τ, the iteration stops and the final feature vector is obtained. This vector groups the contents with high similarity together. The operation is as follows Λ=diag({Σ j＝1 A ij } i ) In the formula, diag means calculating a diagonal matrix, In the initial feature vector represents the proportion of the i-th row in the similarity matrix A in the sum of the elements of the entire matrix A.

6. The unsupervised video object segmentation method based on action-appearance multi-level clustering according to claim 1, characterized in that: The implementation process of the subspace clustering includes the following steps: F1, define the subspace prototype Each of these prototypes is a 10-dimensional subspace S j The unit vector of F2, fusion of features X optimized by position-channel attention module t→t+n , X t+n→t , get the weighted features F3, calculate the affinity matrix M between the pixel and the prototype; F4, by optimizing the problem Determine the optimal allocation matrix P; F5, update subspace prototype The specific operations of each step are as follows: In the formula, [;] represents the concatenation operation, ○ represents the Hadamard product, for The nth 10-dimensional vector, v (n) for The nth 30-dimensional vector, |S j | represents the subspace S j The total number of elements in .

7. The unsupervised video object segmentation method based on action-appearance multi-level clustering according to claim 1, characterized in that: The process of K-means|| clustering and final segmentation mask generation The following steps are involved: H1, use K-means||clustering to calculate the cluster label I of each subspace j (j=1,...,30), labels are based on subspace prototypes The affinity between the pixels M _，j get; H2, introduces motion intensity information to reconstruct optical flow To adjust the clustering labels and improve the accuracy of foreground area positioning; H3, combining subspace multi-angle analysis to generate foreground areas and background area The formula is as follows In the formula, is the indicator function, is the normalized motion intensity of the reconstructed optical flow at position i, and Represent the foreground and background areas respectively.

Images