An Unsupervised Video Target Segmentation Method Based on Frequency Domain Global Filtering

Abstract

This invention discloses an unsupervised video target segmentation method based on frequency domain global filtering. Based on video frames and their corresponding optical flow estimation images, appearance and motion features are extracted using a dual-stream CNN backbone network. A frequency domain global filtering module fuses each set of appearance and motion features and transforms them from the spatial domain to the frequency domain using a Fast Fourier Transform (FFT). In the frequency domain, a learnable weight matrix captures global dependencies, and then an inverse FFT transforms the results back to the spatial domain, achieving the capture of global feature dependencies. Multi-scale feature extraction is then performed. Low-resolution features are continuously upsampled to restore high resolution and concatenated with high-resolution features to finally output a segmented video target mask. This unsupervised video target segmentation method significantly improves the accuracy and robustness of segmentation results even in scenes with diverse interfering targets and cluttered backgrounds, while maintaining low computational cost.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to an unsupervised video target segmentation method based on frequency domain global filtering. Background Technology

[0002] Unsupervised video object segmentation (UVOS) is a task that, given a video segment, automatically segments the main objects in the video without providing additional segmentation mask information. This task has wide applications in fields such as autonomous driving, video understanding, and online conferencing.

[0003] In recent years, many deep learning-based methods have been proposed to better guide models in segmenting target regions with the same semantic information by capturing the global dependencies of image features. These methods generally employ self-attention mechanisms or globally sized convolutional kernels in the spatial domain to capture these global dependencies. However, this approach leads to a significant increase in computational cost when dealing with large input images, placing higher demands on the operating equipment and impacting the practical application of segmentation models. Summary of the Invention

[0004] The technical problem to be solved by this invention is the excessive computational cost caused by the use of self-attention mechanisms to capture global dependencies of features in unsupervised video object segmentation (UVOS) tasks.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] An unsupervised video target segmentation method based on frequency domain global filtering performs the following steps on a video containing target objects to obtain a video target mask with the target objects segmented out:

[0007] Step 1: For a video containing the target object, obtain each video image frame included in the video; and based on the segmentation and annotation of the target object in the video, obtain the optical flow estimation image corresponding to each video image frame;

[0008] Step 2: For each video image frame and the optical flow estimation image corresponding to each video image frame, use a pre-trained unsupervised video target segmentation model that takes the video image frame and the optical flow estimation image corresponding to the video image frame as input and the image target mask of the segmented target object corresponding to the video image frame as output to obtain the image target mask of the segmented target object corresponding to each video image frame, and then obtain the video target mask of the segmented target object.

[0009] As a preferred technical solution of the present invention, in step 2, the pre-trained unsupervised video target segmentation model, which takes video image frames and optical flow estimation images corresponding to the video image frames as inputs and the image target mask of the segmented target objects corresponding to the video image frames as outputs, includes a dual-stream CNN backbone network module, a frequency domain global filtering module, a multi-scale feature extraction module, a residual optimization module, and an upsampling decoding module.

[0010] The dual-stream CNN backbone network module includes feature extraction network modules corresponding to video image frames and optical flow estimation images, respectively. The video image frame is input into its corresponding feature extraction network module and undergoes feature extraction stages at preset resolutions to obtain the appearance features corresponding to each stage. The optical flow estimation image is input into its corresponding feature extraction network module and undergoes feature extraction stages at preset resolutions to obtain the motion features corresponding to each stage. The feature extraction network modules corresponding to the video image frame and the optical flow estimation image have the same number of feature extraction stages. The higher the number of stages, the lower the output feature resolution. The same stage outputs the same feature resolution.

[0011] The frequency domain global filtering module performs feature fusion on the appearance and action features output by the dual-stream CNN backbone network module at each stage to obtain the fused features corresponding to each stage. For the fused features corresponding to each stage, the fused features are converted into frequency domain features. The frequency domain features are then multiplied with the learnable weight matrix to achieve global filtering of the frequency domain features. Then, the frequency domain features are converted into spatial features and multiplied with the fused features to obtain the first feature corresponding to that stage. This process is repeated to obtain the first features corresponding to each stage.

[0012] The multi-scale feature extraction module performs convolution operations on the first feature corresponding to each stage, combined with convolution kernels of preset sizes, and then concatenates the outputs of the convolution kernels of preset sizes. After 1×1 convolution dimensionality reduction and activation function, the multi-scale features corresponding to that stage are obtained, and thus the multi-scale features corresponding to each stage are obtained.

[0013] The residual optimization module performs a dot product between the multi-scale features corresponding to each stage and the multi-scale features of that stage after 3×3 convolution, and then adds the multi-scale features of that stage to obtain the optimized features corresponding to that stage, thereby obtaining the optimized features corresponding to each stage.

[0014] The upsampling decoding module iteratively upsamples the optimized features corresponding to the last stage based on the optimized features corresponding to each stage until the upsampling reaches the resolution of the video image frame, thus obtaining the upsampled features. Then, the upsampled features are subjected to 1×1 convolution dimensionality reduction and activation function to obtain the target features, that is, the image target mask of the segmented target object corresponding to the video image frame.

[0015] As a preferred embodiment of the present invention, the frequency domain global filtering module specifically performs the following process for the appearance features and action features output by the dual-stream CNN backbone network module at each stage, to obtain the first feature corresponding to each stage:

[0016] Step A1: For the appearance features A of the k-th stage output by the dual-stream CNN backbone network module k With motion characteristics M k Feature extraction is performed, and the appearance feature A is processed through a 1×1 convolution and activation function module. k With motion characteristics M k Feature A is obtained k ′ and M k The specific formula is:

[0017] A k ′=σ(conv(A k ),

[0018] M k ′=σ(conv(M k ),

[0019] in, Let A represent the activation function, where conv represents a 1×1 convolution. k ′ indicates appearance feature A k The corresponding feature after feature extraction, M k ′ represents motion feature M k The corresponding features after feature extraction;

[0020] Step A2: Based on the appearance feature A of stage k k With motion characteristics M k Feature A obtained by feature extraction k ′、M k ′, feature A is expressed by the following formula k ′ and M k 'Merge, and obtain the fusion feature F of the k-th stage. k :

[0021]

[0022] in, For dot product, max is the max pooling operation, and concat is the concatenation operation.

[0023] Step A3: Based on the Fast Fourier Transform, fuse the k-th stage features F k Transform into frequency domain features, and then match the frequency domain features with their corresponding learnable weight matrix X. kThe frequency domain global enhancement features are obtained by dot product, and then the frequency domain global enhancement features are transformed into spatial features by inverse fast Fourier transform. The spatial features are then fused with the k-th stage feature F. k Perform a dot product to obtain the first feature F corresponding to this stage. k The specific formula is:

[0024]

[0025] in, For dot product operation, FFT is Fast Fourier Transform, iFFT is Inverse Fast Fourier Transform, X k This is a learnable weight matrix.

[0026] As a preferred embodiment of the present invention, the multi-scale feature extraction module obtains the multi-scale features corresponding to each stage for each stage using the following formula:

[0027] F k "=σ(conv(concat(conv1(F k ′),conv2(F k ′),conv3(F k ′)...convn(F k ′)))),

[0028] Among them, F k "" represents the multi-scale feature corresponding to the k-th stage, F k ′ represents the first feature F corresponding to the k-th stage. k ′, This represents the activation function, convn represents the convolution operation corresponding to the nth pre-defined kernel size, and concat is the concatenation operation.

[0029] As a preferred embodiment of the present invention, the residual optimization module obtains the optimization features corresponding to each stage for each stage using the following formula:

[0030]

[0031] Where conv3*3 represents the convolution operation corresponding to a 3×3 size convolution kernel. For dot product operation, F k "′" represents the optimization feature corresponding to the k-th stage, F k "" represents the multi-scale feature corresponding to the k-th stage.

[0032] As a preferred embodiment of the present invention, the upsampling decoder module performs the following steps to obtain the image target mask of the segmented target object corresponding to the video image frame:

[0033] Step B1: Based on the optimization features corresponding to each stage, iteratively upsample the optimization features corresponding to the last stage until all stages have been traversed, obtaining the first upsampled feature; in each iteration, upsample to the resolution corresponding to the optimization features of the previous stage, and then concatenate with the optimization features of the previous stage, as shown in the following formula:

[0034] U k-1 =concat(Upsample(F k "′),F k-1 "′),

[0035] Where Upsample represents the upsampling operation, concat represents the concatenation operation, and F k "′" represents the optimization feature corresponding to the k-th stage, U k-1 This represents the feature obtained by upsampling the optimized feature of the k-th stage to the resolution corresponding to the optimized feature of the previous stage, and then concatenating it with the optimized feature of the previous stage.

[0036] Step B2: Further upsample the first upsampled feature to the resolution of the video image frame to obtain the second upsampled feature, i.e., the upsampled feature;

[0037] Step B3: The second upsampled features are subjected to 1×1 convolution dimensionality reduction and activation function to obtain the target features, that is, the image target mask of the segmented target object corresponding to the video image frame.

[0038] As a preferred embodiment of the present invention, the loss function L of the pre-trained unsupervised video object segmentation model takes a video image frame and the optical flow estimation image corresponding to the video image frame as input, and the image target mask of the segmented target object corresponding to the video image frame as output. total (M,G) are shown below:

[0039] L total (M,G)=L IOU (M,G)+L BEC (M,G),

[0040] in,

[0041] M represents the image target mask result predicted and segmented by the model, G represents the image target result actually segmented by the model, H represents the height of the video image frame, and W represents the width of the video image frame. i,j G represents the value in the i-th row and j-th column of the image target mask result after predicting and segmenting the target object. i,j This represents the value in the i-th row and j-th column of the image target mask result that truly segments the target object.

[0042] The beneficial effects of this invention are as follows: This invention provides an unsupervised video target segmentation method based on frequency domain global filtering, comprising: acquiring video frames and their corresponding optical flow estimation images; extracting four sets of appearance features and motion features respectively through a dual-stream CNN backbone network in the decoding stage; constructing a frequency domain global filtering module to fuse each set of appearance features and motion features and converting them from the spatial domain to the frequency domain through Fast Fourier Transform (FFT); capturing global dependencies in the frequency domain using a learnable weight matrix and then converting them back to the spatial domain through Inverse FFT, thereby capturing the global dependencies of features; constructing a feature pyramid module to extract features at multiple scales; in the decoding stage, continuously upsampling low-resolution features to restore high resolution and concatenating them with high-resolution features to finally output a segmented video target mask. This invention utilizes Fast Fourier Transform to convert features from the spatial domain to the frequency domain, and in the frequency domain, it uses the dot product operation of frequency domain features and weight matrix to achieve global feature learning, replacing the global size convolution operation in the spatial domain, thereby reducing the computational load while enabling the model to accurately segment targets in the video from the background. The unsupervised video target segmentation method of the present invention can still significantly improve the accuracy and robustness of the segmentation results in scenes with diverse interfering targets and cluttered backgrounds, while maintaining a low computational cost. Attached Figure Description

[0043] Figure 1 This is a flowchart of an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of the frequency domain global filtering module according to an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram of the multi-scale feature extraction module according to an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram of the residual optimization module according to an embodiment of the present invention;

[0047] Figure 5 This is an example of the segmentation result in an embodiment of the present invention. Detailed Implementation

[0048] The present invention will be further described below with reference to the accompanying drawings. The following embodiments will enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way.

[0049] An unsupervised video target segmentation method based on frequency domain global filtering is illustrated in the flowchart below. Figure 1 As shown, for a video containing a target object, the following steps are performed to obtain a video target mask with the target object segmented out:

[0050] Step 1: For a video containing the target object, obtain each video image frame included in the video; and based on the segmentation annotation of the target object in the video, obtain the optical flow estimation image corresponding to each video image frame.

[0051] In this embodiment, the video containing the target object is first converted into consecutive video image frames arranged in time. The target object segmentation and annotation results corresponding to each video image frame are obtained. Then, using existing optical flow estimation methods, optical flow estimation images calculated for each video image frame and its adjacent frames are generated. In this embodiment, the optical flow estimation image corresponding to each video image frame is the optical flow estimation image calculated for that video image frame and the video image frame at the previous time. Optical flow refers to the displacement of the target object between two adjacent frames when the camera moves or an object moves in space. The optical flow estimation method can generate optical flow estimation images of two adjacent frames of the video.

[0052] Step 2: For each video image frame and its corresponding optical flow estimation image, a pre-trained unsupervised video object segmentation model is used. This model takes the video image frame and its corresponding optical flow estimation image as input and outputs the segmented target object image mask for that video image frame. This process yields the segmented target object image mask for each video image frame, and thus the video target mask. Each frame is processed, and the target mask results for each frame are continuously output to obtain the video target mask.

[0053] In step 2, the pre-trained unsupervised video target segmentation model, which takes a video image frame and the optical flow estimation image corresponding to the video image frame as input and the image target mask of the segmented target object corresponding to the video image frame as output, includes a dual-stream CNN backbone network module, a frequency domain global filtering module, a multi-scale feature extraction module, a residual optimization module, and an upsampling decoding module.

[0054] The dual-stream CNN backbone network module includes feature extraction network modules corresponding to video image frames and optical flow estimation images, respectively. The video image frame is input into its corresponding feature extraction network module and undergoes feature extraction stages at preset resolutions to obtain the appearance features corresponding to each stage. The optical flow estimation image is input into its corresponding feature extraction network module and undergoes feature extraction stages at preset resolutions to obtain the motion features corresponding to each stage. The feature extraction network modules corresponding to the video image frame and the optical flow estimation image have the same number of feature extraction stages. The higher the number of stages, the lower the output feature resolution. The output feature resolution is the same for the same stage.

[0055] In this embodiment, the video image frame and the optical flow estimation image have the same resolution: 672 pixels wide, 384 pixels high, and 3 channels. After four stages of feature extraction (downsampling), four sets of appearance features at different resolutions are obtained. With motion characteristics Where k∈[1,4] represents the k-th feature group, C represents the number of feature channels, H represents the feature height, and W represents the feature width. Specifically, A1, A2, A3 A4

[0056] The frequency domain global filtering module performs feature fusion on the appearance and action features output by the dual-stream CNN backbone network module at each stage to obtain the fused features corresponding to each stage. For the fused features corresponding to each stage, the fused features are converted into frequency domain features. The frequency domain features are then multiplied with the learnable weight matrix to achieve global filtering of the frequency domain features. Then, they are converted into spatial features and multiplied with the fused features to obtain the first feature corresponding to that stage. This process is repeated to obtain the first features corresponding to each stage.

[0057] Figure 2 The frequency domain global filtering module is demonstrated. Specifically, this module performs the following process to obtain the first feature corresponding to each stage, targeting the appearance and action features output by the dual-stream CNN backbone network module at each stage:

[0058] Step A1: For the appearance features A of the k-th stage output by the dual-stream CNN backbone network module k With motion characteristics M k Feature extraction is performed, and the appearance feature A is processed through a 1×1 convolution and activation function module. k With motion characteristics M k Feature A is obtained k ′ and M k The specific formula is:

[0059] A k ′=σ(conv(A k ),

[0060] M k ′=σ(conv(M k ),

[0061] in, Let A represent the activation function, where conv represents a 1×1 convolution. k ′ indicates appearance feature A k The corresponding feature after feature extraction, M k ′ represents motion feature Mk The corresponding features after feature extraction;

[0062] Step A2: Based on the appearance feature A of stage k k With motion characteristics M k Feature A obtained by feature extraction k ′、M k ′, feature A is expressed by the following formula k ′ and M k 'Merge, and obtain the fusion feature F of the k-th stage. k :

[0063]

[0064] in, For dot product, max is the max pooling operation, and concat is the concatenation operation.

[0065] Step A3: Based on the Fast Fourier Transform, fuse the k-th stage features F k Transform into frequency domain features, and then match the frequency domain features with their corresponding learnable weight matrix X. k The frequency domain global enhancement features are obtained by dot product, and then the frequency domain global enhancement features are transformed into spatial features by inverse fast Fourier transform. The spatial features are then fused with the k-th stage feature F. k Perform a dot product to obtain the first feature F corresponding to this stage. k That is, to obtain the feature F that captures global dependencies in the spatial domain. k The specific formula is:

[0066]

[0067] in, For dot product operation, FFT is Fast Fourier Transform, iFFT is Inverse Fast Fourier Transform, X k This is a learnable weight matrix.

[0068] In this embodiment, the frequency domain global filtering module's function is to fuse the four sets of appearance features A1, A2, A3, A4 with motion features M1, M2, M3, M4 obtained from the dual-stream CNN backbone network module at different resolutions, perform a dot product with the learnable weight matrix in the frequency domain, and then convert it back to the spatial domain to obtain the feature F that captures global dependencies. k ′。 Wherein, the learnable weight matrix Initial values ​​are all 1, C is 1 to represent the number of channels, H and W are the height and width of the weight matrix, and F is the value of the weight matrix. k The height and width are the same. As the training learns the importance of different positions of the features, the corresponding weight coefficients are obtained.

[0069] The multi-scale feature extraction module performs convolution operations on the first feature corresponding to each stage, combined with convolution kernels of preset sizes, to extract multi-scale information of the features. Then, the outputs of the convolution kernels of preset sizes are concatenated, and after 1×1 convolution dimensionality reduction and activation function, the multi-scale features corresponding to that stage are obtained, and thus the multi-scale features corresponding to each stage are obtained.

[0070] Figure 3 A multi-scale feature extraction module is demonstrated. This module obtains the multi-scale features corresponding to each stage based on the first feature of each stage using the following formula:

[0071] F k "=σ(conv(concat(conv1(F k ′),conv2(F k ′),conv3(F k ′)...convn(F k ′)))),

[0072] Among them, F k "" represents the multi-scale feature corresponding to the k-th stage, F k ′ represents the first feature F corresponding to the k-th stage. k ′, This represents the activation function, convn represents the convolution operation corresponding to the nth pre-defined kernel size, and concat is the concatenation operation.

[0073] In this embodiment, the convolution kernels of each preset size are 1×1, 7×7, and 13×13, respectively. The multi-scale feature extraction module then obtains the multi-scale features corresponding to each stage for the first feature at each stage using the following formula:

[0074] F k ″=σ(conv1(concat(conv1*1(F k ′),conv7*7(F k ′),conv13*13(F k ′)))),

[0075] The residual optimization module performs a dot product between the multi-scale features corresponding to each stage and the multi-scale features of that stage after a 3×3 convolution. Then, it adds the multi-scale features of that stage to the original module and optimizes the result using the residual structure, yielding the optimized feature F. k "′ means obtaining the optimization features corresponding to this stage, and then obtaining the optimization features corresponding to each stage respectively;

[0076] Figure 4The residual optimization module is demonstrated. This module, for each stage's multi-scale features, obtains the optimized features for each stage using the following formula:

[0077]

[0078] Where conv3*3 represents the convolution operation corresponding to a 3×3 size convolution kernel. For dot product operation, F k "′" represents the optimization feature corresponding to the k-th stage, F k "" represents the multi-scale feature corresponding to the k-th stage.

[0079] The upsampling decoding module iteratively upsamples the optimized features corresponding to the last stage based on the optimized features corresponding to each stage until the upsampling reaches the resolution of the video image frame, thus obtaining the upsampled features. Then, the upsampled features are subjected to 1×1 convolution dimensionality reduction and activation function to obtain the target features, that is, the image target mask of the segmented target object corresponding to the video image frame.

[0080] In this embodiment, for the constructed upsampling decoding module, the optimized features corresponding to the last stage are continuously upsampled and concatenated with the features of the previous stage to recover the feature F up to the original image width and height. Pred Then, the feature F is activated by an activation function. Pred The numerical range is scaled to the 0-1 interval, and the channel dimension is reduced to 1 through a 1×1 convolution, where feature F Pred The value of each pixel represents the probability that the pixel is a foreground object. The final output is a segmented binary video target mask image, that is, the segmented target object image mask corresponding to the video image frame.

[0081] The upsampling decoder module performs the following steps to obtain the image target mask of the segmented target object corresponding to the video image frame:

[0082] Step B1: Based on the optimization features corresponding to each stage, iteratively upsample the optimization features corresponding to the last stage until all stages have been traversed, obtaining the first upsampled feature; in each iteration, upsample to the resolution corresponding to the optimization features of the previous stage, and then concatenate with the optimization features of the previous stage, as shown in the following formula:

[0083] U k-1 =concat(Upsample(F k "′),F k-1 "′),

[0084] Where Upsample represents the upsampling operation, concat represents the concatenation operation, and F k"′" represents the optimization feature corresponding to the k-th stage, U k-1 This represents the feature obtained by upsampling the optimized feature of the k-th stage to the resolution corresponding to the optimized feature of the previous stage, and then concatenating it with the optimized feature of the previous stage;

[0085] Specifically, in this embodiment, the optimized feature corresponding to the last stage, namely the lowest resolution feature F4″′, is upsampled to the resolution of F3″′ and spliced ​​with F3″′ along the channel direction to obtain U3. U3 is upsampled to the resolution of F2″′ and spliced ​​with F2″′ along the channel direction to obtain U2. U2 is upsampled to the resolution of F1″′ and spliced ​​with F1″′ along the channel direction to obtain U1.

[0086] Step B2: Further upsample the first upsampled feature to the resolution of the video image frame to obtain the second upsampled feature, i.e., the upsampled feature;

[0087] Specifically, in this embodiment, U1 performs a 4x upsampling to restore the resolution to the original image size, with a height and width of 384,672 pixels. Pred ;

[0088] Step B3: The second upsampled features are subjected to 1×1 convolution dimensionality reduction and activation function to obtain the target features, that is, the image target mask of the segmented target object corresponding to the video image frame.

[0089] Furthermore, the loss function of the pre-trained unsupervised video object segmentation model, which takes a video image frame and the corresponding optical flow estimation image as input, and outputs the image target mask of the segmented target object corresponding to the video image frame, is the total loss function L. total By IOU loss function L IOU and cross-entropy loss function L BCE Confirmed, L total (M,G) are shown below:

[0090] L total (M,G)=L IOU (M,G)+L BEC (M,G),

[0091] in,

[0092] M represents the image target mask result predicted and segmented by the model, G represents the image target result actually segmented by the model, H represents the height of the video image frame, and W represents the width of the video image frame. i,j G represents the value in the i-th row and j-th column of the image target mask result after predicting and segmenting the target object. i,j This represents the value in the i-th row and j-th column of the image target mask result that truly segments the target object.

[0093] Figure 5 The following is a partial example of the results of processing using the method of the present invention. It can be clearly seen from the figure that foreground targets such as swans, camels, dogs, cars, horses, and cows have been segmented from the video image frames.

[0094] This invention designs an unsupervised video target segmentation method based on frequency domain global filtering, comprising: acquiring video frames and their corresponding optical flow estimation images; extracting four sets of appearance features and motion features respectively through a dual-stream CNN backbone network in the decoding stage; constructing a frequency domain global filtering module to fuse each set of appearance features and motion features and converting them from the spatial domain to the frequency domain through Fast Fourier Transform (FFT); capturing global dependencies in the frequency domain using a learnable weight matrix and then converting them back to the spatial domain through Inverse FFT to achieve the capture of global dependencies of features; constructing a feature pyramid module to extract features at multiple scales; in the decoding stage, continuously upsampling low-resolution features to restore high resolution and concatenating them with high-resolution features to finally output a segmented video target mask. This invention utilizes FFT to convert features from the spatial domain to the frequency domain, and in the frequency domain, it uses the dot product operation of frequency domain features and weight matrix to achieve global feature learning, replacing the global size convolution operation in the spatial domain, thereby reducing the amount of computation while enabling the model to accurately segment targets in the video from the background. The unsupervised video target segmentation method of the present invention can still significantly improve the accuracy and robustness of the segmentation results in scenes with diverse interfering targets and cluttered backgrounds, while maintaining a low computational cost.

[0095] The above are merely preferred embodiments of the present invention, but do not limit the patent scope of the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of the present invention specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the patent protection scope of the present invention.

Claims

1. An unsupervised video target segmentation method based on frequency domain global filtering, characterized in that: For a video containing the target object, perform the following steps to obtain a video target mask from which the target object is segmented: Step 1: For a video containing the target object, obtain each video image frame included in the video; and based on the segmentation and annotation of the target object in the video, obtain the optical flow estimation image corresponding to each video image frame; Step 2: For each video image frame and the optical flow estimation image corresponding to each video image frame, use a pre-trained unsupervised video target segmentation model that takes the video image frame and the optical flow estimation image corresponding to the video image frame as input and the image target mask of the segmented target object corresponding to the video image frame as output to obtain the image target mask of the segmented target object corresponding to each video image frame, and then obtain the video target mask of the segmented target object. The unsupervised video target segmentation model includes a dual-stream CNN backbone network module, a frequency domain global filtering module, a multi-scale feature extraction module, a residual optimization module, and an upsampling decoding module. The frequency domain global filtering module performs the following process for the appearance features and action features output by the dual-stream CNN backbone network module at each stage, respectively, to obtain the first feature corresponding to each stage: Step A1: For the appearance features A of the k-th stage output by the dual-stream CNN backbone network module k With motion characteristics M k Feature extraction is performed, and the appearance feature A is processed through a 1×1 convolution and activation function module. k With motion characteristics M k Feature A is obtained k ′ and M k The specific formula is: A k ′=σ(conv(A k ), M k ′=σ(conv(M k ), in, Let A represent the activation function, where conv represents a 1×1 convolution, and A... k ′ indicates appearance feature A k The corresponding feature after feature extraction, M k ′ represents motion feature M k The corresponding features after feature extraction; Step A2: Based on the appearance feature A of stage k k With motion characteristics M k Feature A obtained by feature extraction k ′、M k ′, feature A is expressed by the following formula k ′ and M k 'Merge, and obtain the fusion feature F of the k-th stage. k : in, For dot product, `max` is the max pooling operation, and `concat` is the concatenation operation. Step A3: Based on the Fast Fourier Transform, fuse the k-th stage features F k Transform into frequency domain features, and then match the frequency domain features with their corresponding learnable weight matrix X. k The frequency domain global enhancement features are obtained by dot product, and then the frequency domain global enhancement features are transformed into spatial features by inverse fast Fourier transform. The spatial features are then fused with the k-th stage feature F. k Perform a dot product to obtain the first feature F corresponding to this stage. k The specific formula is: in, For dot product operation, FFT is Fast Fourier Transform, iFFT is Inverse Fast Fourier Transform, X k This is a learnable weight matrix.

2. The unsupervised video target segmentation method based on frequency domain global filtering according to claim 1, characterized in that: In step 2, the dual-stream CNN backbone network module includes feature extraction network modules corresponding to video image frames and optical flow estimation images, respectively. The video image frame is input into its corresponding feature extraction network module and undergoes feature extraction stages at preset resolutions to obtain appearance features corresponding to each stage. The optical flow estimation image is input into its corresponding feature extraction network module and undergoes feature extraction stages at preset resolutions to obtain motion features corresponding to each stage. The feature extraction network modules corresponding to the video image frame and the optical flow estimation image have the same number of feature extraction stages. The higher the number of stages, the lower the output feature resolution. The same stage outputs the same feature resolution. The frequency domain global filtering module performs feature fusion on the appearance and action features output by the dual-stream CNN backbone network module at each stage to obtain the fused features corresponding to each stage. For each stage, the fusion features are converted into frequency domain features. The frequency domain features are then multiplied with the learnable weight matrix to achieve global filtering of the frequency domain features. These features are then converted into spatial features and multiplied with the fusion features to obtain the first feature corresponding to that stage. This process is repeated to obtain the first features corresponding to each stage. The multi-scale feature extraction module performs convolution operations on the first feature corresponding to each stage, combined with convolution kernels of preset sizes, and then concatenates the outputs of the convolution kernels of preset sizes. After 1×1 convolution dimensionality reduction and activation function, the multi-scale features corresponding to that stage are obtained, and thus the multi-scale features corresponding to each stage are obtained. The residual optimization module performs a dot product between the multi-scale features corresponding to each stage and the multi-scale features of that stage after 3×3 convolution, and then adds the multi-scale features of that stage to obtain the optimized features corresponding to that stage, thereby obtaining the optimized features corresponding to each stage. The upsampling decoding module, based on the optimization features corresponding to each stage, iteratively upsamples the optimization features corresponding to the last stage until it is upsampled to the video image frame resolution to obtain the upsampled features; Then, the upsampled features are subjected to 1×1 convolution dimensionality reduction and activation function to obtain the target features, that is, the image target mask of the segmented target object corresponding to the video image frame.

3. The unsupervised video target segmentation method based on frequency domain global filtering according to claim 2, characterized in that: The multi-scale feature extraction module obtains the multi-scale features corresponding to each stage for the first feature corresponding to each stage using the following formula: F k ″=σ(conv(concat(conv1(F k ′),conv2(F k ′),conv3(F k ′)...convn(F k ′))))), Among them, F k "" represents the multi-scale feature corresponding to the k-th stage, F k ′ represents the first feature F corresponding to the k-th stage. k ′, This represents the activation function, convn represents the convolution operation corresponding to the nth pre-defined kernel size, and concat is the concatenation operation.

4. The unsupervised video target segmentation method based on frequency domain global filtering according to claim 2, characterized in that: The residual optimization module obtains the optimized features for each stage based on the multi-scale features corresponding to each stage using the following formula: Where conv3*3 represents the convolution operation corresponding to a 3×3 size convolution kernel. For dot product operation, F k "′" represents the optimization feature corresponding to the k-th stage, F k "" represents the multi-scale feature corresponding to the k-th stage.

5. The unsupervised video target segmentation method based on frequency domain global filtering according to claim 2, characterized in that: The upsampling decoding module performs the following steps to obtain the image target mask of the segmented target object corresponding to the video image frame: Step B1: Based on the optimization features corresponding to each stage, iteratively upsample the optimization features corresponding to the last stage until all stages have been traversed, and obtain the first upsampled feature; In each iteration, the sample is upsampled to the resolution corresponding to the optimized features of the previous stage, and then concatenated with the optimized features of the previous stage. The specific formula is as follows: U k-1 =concat(Upsample(F k ″′),F k-1 ″′), Where Upsample represents the upsampling operation, concat represents the concatenation operation, and F k "′" represents the optimization feature corresponding to the k-th stage, U k-1 This represents the feature obtained by upsampling the optimized feature of the k-th stage to the resolution corresponding to the optimized feature of the previous stage, and then concatenating it with the optimized feature of the previous stage; Step B2: Further upsample the first upsampled feature to the resolution of the video image frame to obtain the second upsampled feature, i.e., the upsampled feature; Step B3: The second upsampled features are subjected to 1×1 convolution dimensionality reduction and activation function to obtain the target features, that is, the image target mask of the segmented target object corresponding to the video image frame.

6. The unsupervised video target segmentation method based on frequency domain global filtering according to claim 1, characterized in that: The pre-trained unsupervised video object segmentation model takes video image frames and corresponding optical flow estimation images as input, and outputs the segmented target mask of the target object corresponding to the video image frame. The loss function L is... total (M,G) are shown below: L total (M,G)=L IOU (M,G)+L BEC (M,G), in, M represents the image target mask result predicted and segmented by the model, G represents the image target result actually segmented by the model, H represents the height of the video image frame, and W represents the width of the video image frame. i,j G represents the value in the i-th row and j-th column of the image target mask result after predicting and segmenting the target object. i,j This represents the value in the i-th row and j-th column of the image target mask result that truly segments the target object.

Images