An alternating dynamic fusion spatio-temporal attention video salient object detection method

Abstract

The application discloses an alternating dynamic fusion spatio-temporal attention video salient object detection method, which comprises the following steps: firstly, a video frame sequence and a corresponding visualized optical flow map are divided into a plurality of video slices; for each group of slices, the video frame sequence and the visualized optical flow map are respectively input into an encoder to obtain four levels of static appearance and dynamic optical flow feature maps; then, the four levels of feature maps are input into a fusion module in correspondence with the same level, cross-modal multi-scale interaction is performed through an iterative spatial attention mode, and four levels of fusion feature maps are output; then, the last two levels of fusion feature maps in the deep layer are respectively input into a spatio-temporal attention module for independent time sequence modeling, and two layers of deep spatio-temporal feature maps with reinforced time sequence relationship are obtained; finally, the deep spatio-temporal feature maps and the first two levels of fusion feature maps without spatio-temporal attention are jointly input into a cascaded decoder, progressive upsampling and feature fusion are performed from the deep layer to the shallow layer, and finally, the prediction mask of the current video frame is output.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, specifically to a video salient target detection method based on alternating dynamic fusion and spatiotemporal attention. Background Technology

[0002] In the digital age, the widespread adoption of smart cameras, mobile devices, and other equipment has generated massive amounts of video sequence data. This data not only grows exponentially in scale but also contains rich dynamic spatiotemporal information such as target movement, scene changes, and attention shifts, necessitating the development of targeted algorithms to extract its value. Video salient object detection (VSOD) aims to detect and segment the most visually salient objects or regions from dynamic videos. Compared to static image salient object detection (ISOD), it places greater emphasis on the fusion of temporal and spatial information, requiring the acquisition of motion relationships between consecutive frames, making it more challenging. It is often used as an upstream task in many real-time applications, such as video compression, video object tracking, and medical segmentation.

[0003] In recent years, with the rapid development of deep learning technologies such as Convolutional Neural Networks (CNNs) and Transformers, deep learning-based methods for video salient object detection have achieved remarkable success. However, existing video salient object detection technologies perform poorly in complex real-world scenarios, with limited computing resources, or in mobile terminal scenarios with high real-time requirements. In particular, when optical flow features are introduced to perceive motion temporal information, the inherent noise and coarse edges of optical flow features, coupled with the susceptibility of static image features to cluttered background interference, make it difficult for existing cross-modal fusion methods to achieve deep bidirectional complementarity and mutual correction, resulting in blurred fine-grained boundaries of salient objects. Furthermore, existing long-range spatiotemporal context modeling mechanisms are computationally too complex, often requiring spatial downsampling at the expense of resolution for computational efficiency, making it difficult to balance fine-grained spatial edge structures with global computational overhead.

[0004] Therefore, designing a lightweight and efficient video salient object detection model that, under limited resources, can achieve deep alternation and dynamic correction of RGB and optical flow features to strengthen fine-grained boundaries, while reducing the computational complexity of spatiotemporal attention and preserving the full-resolution spatial structure, is a key technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] This invention aims to overcome the shortcomings of existing video salient object detection methods, such as insufficient cross-modal fusion leading to blurred edges and excessive complexity in long-distance spatiotemporal modeling resulting in loss of spatial structure. Specifically, it addresses two technical problems: First, existing cross-modal fusion methods are mostly unidirectional and single-order cascading, unable to achieve deep bidirectional correction between modalities, leading to blurred fine-grained edges of salient objects in complex scenes. Second, it addresses the problems of high computational redundancy in existing spatiotemporal self-attention mechanisms, which rely on spatial downsampling, resulting in loss of salient object edge structure information and difficulty in balancing computational efficiency with fine-grained spatial resolution.

[0006] To address the aforementioned technical problems, this invention provides a method for detecting salient targets in alternating dynamic fusion spatiotemporal attention videos, specifically comprising the following steps:

[0007] Step 1. Obtain the continuous video frame sequence to be detected and the corresponding visualized optical flow sequence. Extract and output the static appearance features and dynamic optical flow motion features of the image at four different resolutions through the encoder. The spatial resolution of the feature map gradually decreases and the number of feature channels gradually increases.

[0008] Step 2. Input the static appearance features and dynamic optical flow motion features of the four levels of images into the alternating bidirectional fusion module for iterative fusion at the same level, and output the fused feature maps of the four levels. The specific steps are as follows:

[0009] Step 21. Extracting the first The static appearance features of the hierarchical image are denoted as , No. The hierarchical dynamic optical flow motion characteristics are denoted as ,in Set the current iteration round number. Let the static features of the initial input be denoted as Dynamic features are denoted as .

[0010] Step 22. When the number of iterations m is odd (inclusive) When ), the dynamic characteristics of the current round will be... The input space collaborative self-attention module generates the spatial attention weight matrix. The specific steps of the spatial collaborative self-attention module are as follows:

[0011] Step 221. Denote the input four-dimensional tensor feature map as... Where B, C, H, and W represent batch size, number of channels, height, and width, respectively. Global average pooling is performed along the height and width dimensions of the feature map F, compressing the two-dimensional spatial features into one-dimensional vectors in the height and width directions, forming a one-dimensional representation sequence. and .

[0012] Step 222. To capture diverse implicit multi-semantic patterns, the decoupled one-dimensional feature vectors are... and Each feature is divided into four subgroups along the channel dimension, and then a set of multi-scale one-dimensional convolutional kernels of different sizes are used. In parallel, depth-wise convolution operations are performed on each feature subspace to adaptively learn spatial geometric contexts at different scales, generating spatial intermediate representation sequences with multiple semantic scales.

[0013] Step 223. Concatenate the four feature subgroups processed by one-dimensional depthwise convolution at different scales along the channel dimension to restore them, thus reconstructing them into multi-scale horizontal aggregated features. and multi-scale vertical aggregation features .

[0014] Step 224. Perform layer normalization on the aggregated features. Input the normalized horizontal axis features and vertical axis features into the sigmoid activation function layer for nonlinear probability mapping, and explicitly constrain the feature response values ​​to a bounded weight range.

[0015] Step 225. Through a dimension expansion operation, the activated horizontal and vertical features are expanded and reshaped into a one-dimensional horizontal attention weight matrix. and , and Element-wise broadcast multiplication is performed, and while maintaining channel dimension C independence, the horizontal and vertical probability weights are fused into a full-resolution 2D network spatial grid, ultimately outputting the required spatial attention weight matrix. .

[0016] Step 23. Introduce a gating coefficient (scaling factor α) to the generated spatial attention weight matrix. The intensity of spatial attention is adjusted by multiplying the gating coefficient α; subsequently, the static features of the input are modulated using a gating residual modulation formula. Element-level dynamic weighted modulation is performed to obtain the modulated appearance features. The formula is as follows:

[0017]

[0018] in This represents element-wise matrix multiplication. Spatial features are amplified to ensure that the inherent detail and texture information of the original static appearance is not lost while introducing motion prior guidance. Dynamic features remain unchanged in this round.

[0019] Step 24. When the number of iterations m is even (inclusive) When using the high-purity appearance characteristics output after the previous modulation, Reverse generation of static appearance attention weight matrix The steps for generating the attention weight matrix are the same as in step 221.

[0020] Step 25. Combining the gating coefficient α, utilize the gating residual modulation formula to analyze the dynamic characteristics of the input. Element-level inverse weighting is performed to obtain the corrected motion characteristics. The formula is as follows:

[0021]

[0022] This reverse feedback utilizes high-purity static appearance boundaries to forcibly constrain and correct naturally occurring motion noise and rough edges in the optical flow characteristics. During this round, the appearance characteristics remain unchanged.

[0023] Step 26. Let Repeat the iterative steps until the number of iterations m reaches the preset total number of iterations M. Then, alternately modulate the output appearance features in the last round. and the corrected motion characteristics Perform channel-wise concatenation (Concat) and linear channel mapping fusion to generate the fused feature map of the i-th level, and output it to the subsequent attention or decoding stage.

[0024] Step 3. Input the last two high-level fusion feature maps from the four levels of fusion feature maps in Step 2 into the spatiotemporal decoupling attention module. Independent spatiotemporal long-distance dependency modeling is performed within each level, correspondingly outputting the spatiotemporal context features of the two levels. The specific steps are as follows:

[0025] Step 31. Extract the fused feature maps of the 3rd and 4th layers output in Step 26. Each layer independently performs spatiotemporal decoupling attention modeling. Taking any layer as an example, the input is transformed into spatial pixel sequence features. Where B is the batch size, T is the number of consecutive video frames, and N is the total number of spatial tokens at full resolution (N=H×W).

[0026] Step 32. Extract spatial pixel sequence features After layer normalization, the corresponding spatial query matrix is ​​generated through three independent linear mapping transformations. Space bond matrix Spatial value matrix .Will , , The frame is divided into multiple independent subspaces along the channel dimension. Intra-frame spatial self-attention computation is performed independently within each subspace. Full-resolution spatial aggregation features are obtained through channel concatenation and linear output transformation. The formula is as follows:

[0027]

[0028] Where d is the scaling factor.

[0029] Step 33. Utilize residual connections and a feedforward network (FFN) to aggregate spatial features. The original input features X are fused and nonlinearly mapped to produce spatially enhanced features. The formula is as follows:

[0030]

[0031] Where LN represents layer normalization and FFN represents feedforward network.

[0032] Step 34. Enhance spatial features through tensor dimension permutation. Reconstructed into time series features At this point, pixel tokens with the same spatial coordinates in multiple consecutive frames are recombined along the time axis into an independent temporal evolution sequence of length T.

[0033] Step 35. Transfer the time series features After layer normalization, the input to the temporal self-attention layer generates a time-domain query matrix through independent linear mappings. Key matrix Value matrix For each spatial pixel, a cross-frame temporal global multi-head self-attention calculation of up to T frames is performed independently, outputting temporal aggregated features. .Will The 3D tensor, consistent with the spatially enhanced feature structure, is restored through inverse dimensional reshaping and permutation operations. .

[0034] Step 36. Using residual connections and feedforward networks, Spatial enhancement features output from step 33 The spatiotemporal characteristics of the final output are obtained by performing fusion and nonlinear transformation. The formula is as follows:

[0035]

[0036] Step 4. Input the two deep-level spatiotemporal context features from Step 3 and the two shallow-level fused feature maps from Step 2 into the decoder, perform multi-scale progressive decoding and reconstruction from deep to shallow, and output the final video salient object detection mask. The specific steps are as follows:

[0037] Step 41. To capture local spatial boundaries and detailed textures while reducing computational overhead, the shallow-level fusion features from Step 2 are first... The input is a depthwise separable convolutional layer (DWConv), which undergoes batch normalization and ReLU activation for nonlinear mapping and feature reconstruction, outputting a spatial feature resonant map rich in local spatial geometric details. Then, a channel attention (CA) module dynamically generates a one-dimensional channel attention weight vector, which is multiplied back into the original features to suppress background noise interference and explicitly enhance the boundary details of the target foreground, ultimately generating multi-dimensional refined shallow features. The formula is as follows:

[0038]

[0039] Step 42. Obtain deep-level features Using bilinear interpolation and other upsampling operators to... Spatial dimensionality is expanded to improve its spatial resolution and refine the shallow features obtained in step 41. Strict alignment; then the aligned deep semantic features Compared with refined shallow features Element-wise addition is performed and then fed into a second depthwise separable convolutional layer to further eliminate semantic fragmentation and scale heterogeneous noise caused by direct addition across layers. Simultaneously, to prevent gradient vanishing, the above output is combined with the initial shallow-level features. A second element-wise residual summation is performed to construct the fused feature matrix F. The equation is as follows:

[0040]

[0041] Up represents upsampling.

[0042] Step 43. Input the fused feature matrix F sequentially into the Layer Normalization (LN) layer and the Multilayer Perceptron (MLP), and then connect the output to the residual F via a jump connection; the equation is as follows:

[0043]

[0044] Finally, 1×1 convolution and bilinear interpolation are used as a mask to output the final predicted mask.

[0045] The method of this invention utilizes alternating iterative self-attention to dynamically refine the fusion of cross-modal features, and relies on the spatiotemporally decoupled self-attention topology and residual Transformer decoding link to significantly reduce the computational redundancy of long-distance modeling of spatiotemporal context while maintaining the full-resolution spatial structure, and at the same time improve the accuracy of saliency mask prediction. Attached Figure Description

[0046] Figure 1 This is a flowchart of the cross-modal fusion module algorithm framework of the present invention;

[0047] Figure 2 This is a flowchart of the spatiotemporal decoupling self-attention module algorithm framework of the present invention;

[0048] Figure 3 This is a flowchart of a video salient target detection method according to an embodiment of the present invention. Detailed Implementation

[0049] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that all other embodiments obtained by those skilled in the art without creative effort, as described herein, fall within the scope of protection of the present invention.

[0050] This invention provides a video salient target detection method based on alternating dynamic fusion and spatiotemporal attention, the steps of which include:

[0051] Step 1: Divide the video frame sequence and its corresponding visual optical flow map into several video slices. For each slice, input the video frame sequence and the visual optical flow map into the encoder to obtain four levels of static appearance feature maps and dynamic optical flow feature maps.

[0052] Step 2: Input the static appearance feature maps and dynamic optical flow feature maps of the four levels into the fusion module according to the same level, and perform cross-modal multi-scale interaction through iterative spatial attention to output the fused feature maps of the four levels.

[0053] Step 3: Input the fused feature maps of the last two deep layers into the spatiotemporal attention module for independent temporal modeling to obtain two deep spatiotemporal feature maps that strengthen the temporal relationship.

[0054] Step 4: Input the deep spatiotemporal feature map and the shallow first two layer fused feature maps that have not undergone spatiotemporal attention into the cascaded decoder, perform progressive upsampling and feature fusion from deep to shallow layers, and finally output the prediction mask of the current video frame.

[0055] In step 2, cross-modal interaction is performed through iterative spatial attention, which differs from the linear fusion method used in existing cross-modal feature fusion technologies.

[0056] This invention inputs the static appearance features and dynamic optical flow features processed by the encoder into the fusion module for a total of m cross-iterative modulations:

[0057] In odd-numbered iterations, the current optical flow features are input into the spatial attention unit to generate the first weight matrix, and then matrix multiplication is performed with the current image features to output the modulated appearance features;

[0058] In even-numbered iterations, the modulated appearance features are input into the spatial attention unit to generate a second weight matrix, which is then multiplied with the original dynamic optical flow features to output the modulated optical flow features.

[0059] After m iterations, the modulated appearance features and modulated optical flow features are concatenated by channels, and the fused feature map of the current level is output through a linear mapping layer.

[0060] The two spatiotemporal attention mechanisms in step 3:

[0061] The first stage processes spatial dimension features. For any single frame feature map within the current time window, it is divided into multiple spatial tokens. Global spatial self-attention calculation is performed within a single frame to capture the spatial topology and contextual target contour information within the frame. The output features are then processed by the FFN feedforward network and residual processing before being input into the second stage.

[0062] The second stage is responsible for processing temporal dimension features. For the feature map processed by the first stage, the corresponding tokens at the same spatial position in consecutive T frames are locked, and cross-frame temporal self-attention calculation is performed independently along the time axis to capture long-term dynamic motion cues. Then, the final output features are obtained by passing through an FFN feedforward network and residual processing.

[0063] refer to Figure 3 This embodiment includes the following steps:

[0064] Step 101. Use the public dataset DAVIS-16 as the training dataset, and use the RAFT (Recurrent All-Pairs Field Transforms) technique to generate the corresponding visual optical flow map as another modality of the training dataset.

[0065] Step 102. Process the continuous video frame sequence and the corresponding visualized optical flow sequence into a 512×512 size, batch size of 4, and three frames per group of data. Input the data into the backbone network ConvNext for feature extraction and output four layers of feature maps with different resolutions, with sizes of 128 (H / 4), 64 (H / 8), 32 (H / 16), and 16 (H / 32), respectively, where H is the size of the input sequence.

[0066] Step 103. Combine static image features of the same level and dynamic optical flow characteristics The input is used to perform iterative fusion in an alternating bidirectional fusion module, and the output is a fused feature map at four levels.

[0067] Step 104. Input the last two high-level fusion feature maps from the four-level fusion feature maps in Step 103 into the spatiotemporal decoupling attention module, and perform spatiotemporal long-distance dependency modeling independently within each level, and output the spatiotemporal context features of the two levels accordingly.

[0068] Step 105. Input the two deep-level spatiotemporal context features from Step 104 and the two shallow-level fused feature maps from Step 103 into the decoder, perform cross-scale progressive decoding and reconstruction from deep to shallow, and output the final video salient object detection mask.

[0069] In this embodiment, step 103 specifically includes the following steps:

[0070] Step 1031. Preset the total number of iterations M (M is better if it is 3 or 5). When the number of iterations m is odd, the optical flow characteristics... As input to the spatial collaborative self-attention module, the input features are subjected to global average pooling along both the height and width dimensions, compressing the two-dimensional spatial features into one-dimensional vectors in the height and width directions, forming a one-dimensional representation sequence. and The one-dimensional feature vectors in both height and width dimensions are divided into four feature subgroups along the channel dimension. A set of one-dimensional convolutional kernels with kernel sizes {3, 5, 7, 9} are used to perform depth-wise convolution operations on each feature subspace in parallel. The four feature subgroups are then concatenated along the channel dimension to reconstruct multi-scale horizontal aggregated features. Layer normalization and sigmoid activation are then performed sequentially. Finally, the enhanced features in the height and width dimensions are multiplied element-wise to output the required spatial attention weight matrix. .

[0071] Step 1032. Introduce a gating coefficient α, multiply the generated spatial attention weight matrix by the gating coefficient α, and then use the gating residual modulation formula to perform element-wise dynamic weighted modulation on the input image features to obtain the modulated image features. The formula is as follows:

[0072] ,

[0073] Step 1033. When the number of iterations m is even, utilize the high-purity image features output after the previous modulation round. The attention weight matrix for the static image is generated in reverse, following the same steps as in step 1031. Similarly, using the gating coefficient α, the input optical flow features are element-wise inversely weighted using the gating residual modulation formula to obtain the corrected optical flow features. The formula is as follows:

[0074] ,

[0075] In one specific embodiment, setting α to 1 yields better prediction results.

[0076] Step 1034. After the iteration is completed, the image features output by the last round of alternating modulation and the corrected optical flow features are fused by channel-by-channel concatenation (Concat) and linear channel mapping to generate the corresponding level of fused feature map, and output to the subsequent attention or decoding stage.

[0077] In this embodiment, step 104 specifically includes the following steps:

[0078] Step 1041. Convert the fused feature maps of the 3rd and 4th layers output in Step 103 into spatial pixel sequence features. T is the number of consecutive video frames, and N is the total number of spatial tokens at full resolution (N=H×W). After layer normalization, the corresponding spatial query matrix is ​​generated through three independent linear mapping transformations. Space bond matrix Spatial value matrix .Will , , The system is divided into four independent subspaces along the channel dimension, and intra-frame spatial self-attention computation is performed independently within each subspace. Then, residual connections and a feedforward network (FFN) are used to fuse and nonlinearly map the spatial aggregated features with the original input features X, outputting spatially enhanced features. The formula is as follows:

[0079] ,

[0080] Step 1042. Enhance spatial features through tensor dimension permutation. Reconstructed into time series features , After layer normalization, the input to the temporal self-attention layer generates a time-domain query matrix through independent linear mappings. Key matrix Value matrix For each spatial pixel, a temporal global multi-head self-attention computation spanning three frames is performed independently, outputting temporal aggregated features. These features are then reconstructed into a three-dimensional tensor consistent with the spatial enhanced feature structure through inverse dimensionality reshaping and permutation operations. Then, residual connections and a feedforward network (FFN) are used to integrate the temporal aggregation features with... The final spatiotemporal enhanced features are output through fusion and nonlinear mapping. The formula is as follows:

[0081] .

[0082] In this embodiment, step 105 specifically includes the following steps:

[0083] Step 1051. Calculate the shallow-level fusion features output in Step 103. The process involves sequentially performing depthwise separable convolution, batch normalization layers, and ReLU activation operations. Then, the input channel attention (CA) module dynamically generates a one-dimensional channel attention weight vector, which is multiplied back into the original features to ultimately generate multi-dimensional refined shallow-level features. .

[0084] Step 1052. Process the deep-level features output in step 104. Upsampling is performed using bilinear interpolation to improve the deep semantic features. Compared with refined shallow features Element-wise addition is performed and then fed into a second depthwise separable convolution layer. Simultaneously, the output of this process is combined with the initial shallow-level features. A second element-wise residual summation is performed to construct the fused feature matrix F. The equation is as follows:

[0085] .

[0086] Step 1053. Input the fused feature matrix F into the Layer Normalization (LN) layer and the Multilayer Perceptron (MLP) in sequence, and connect the output with the F residual jump connection. Finally, use 1×1 convolution and bilinear interpolation as a mask to output the prediction mask.

[0087] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Other variations and modifications are possible without departing from the technical solutions described in the claims.

Claims

1. A method for detecting salient targets in alternating dynamic fusion spatiotemporal attention videos, characterized in that, The steps include: (i) Divide the video frame sequence and its corresponding visual optical flow graph into several groups of video slices; (ii) For each group of slices, perform the following processing: Step 1: Input the video frame sequence and the visualized optical flow map into the encoder to obtain four levels of static appearance feature maps and dynamic optical flow feature maps; each level is distinguished from light to dark according to resolution. Step 2: For the feature maps of the four levels, the static appearance feature maps and their corresponding dynamic optical flow feature maps of the same level are input into the fusion module according to the level. Cross-modal multi-scale interaction is performed through iterative spatial attention, and the fused feature maps of the four levels are output. Step 3: Input the fused feature maps of the last two deep layers into the spatiotemporal attention module for independent temporal modeling to obtain two deep spatiotemporal feature maps that strengthen the temporal relationship. Step 4: Input the deep spatiotemporal feature map obtained in Step 3 and the fused feature map of the first two shallow layers without spatiotemporal attention into the cascaded decoder to perform progressive upsampling and feature fusion from deep to shallow layers, and finally output the prediction mask of the current video frame. In step 2, cross-modal interaction is performed through an iterative spatial attention approach. This involves inputting the encoder-processed static appearance features and dynamic optical flow features into the fusion module for m iterations of cross-modal modulation, where: In the odd-numbered iterations, the current dynamic optical flow features are input into the spatial attention unit to generate the first weight matrix, and matrix multiplication is performed with the current static appearance features to output the modulated appearance features. In even-numbered iterations, the modulated appearance features are input into the spatial attention unit to generate a second weight matrix, which is then multiplied with the original dynamic optical flow features to output the modulated optical flow features. After m iterations, the modulated appearance features and modulated optical flow features are concatenated by channels, and the fused feature map of the current level is output through a linear mapping layer. Step 3 employs a two-stage spatiotemporal attention mechanism, namely: The first stage processes spatial dimension features. For any single frame feature map within the current time window, it is first divided into multiple spatial tokens. Then, global spatial self-attention calculation is performed within the single frame to capture the spatial topology and contextual target contour information within the frame. Finally, the output features are processed by the FFN feedforward network and residual processing before being input into the second stage. The second stage processes temporal dimension features. For the feature map processed in the first stage, the corresponding tokens at the same spatial position in consecutive T frames are locked first, and cross-frame temporal self-attention calculation is performed independently along the time axis to capture long-term dynamic motion cues. Then, the final output features are obtained by passing through an FFN feedforward network and residual processing.

2. The video salient target detection method according to claim 1, characterized in that, The visual optical flow graph corresponding to the video frame sequence is generated using the Recursive All-Pair Transform (RAFT) method.

3. The video salient target detection method according to claim 1, characterized in that, In step 1, the continuous video frame sequence to be detected and the corresponding visualized optical flow sequence are obtained. The encoder extracts and outputs the static appearance features and dynamic optical flow motion features of the image at four different resolutions. The spatial resolution of the feature map gradually decreases, while the number of feature channels gradually increases.

4. The video salient target detection method according to claim 1, characterized in that, In step 2, the static appearance features and dynamic optical flow motion features of the four levels of images are input into the alternating bidirectional fusion module for alternating iterative fusion within the same level, resulting in the output of fused feature maps of the four levels. The specific steps are as follows: Step 21. Extracting the first The static appearance features of the hierarchical image are denoted as , No. The hierarchical dynamic optical flow motion characteristics are denoted as ,in Set the current iteration round number. Let the static features of the initial input be denoted as Dynamic features are denoted as ; Step 22. When the iteration number m is odd, the dynamic features of the current iteration are... The input space collaborative self-attention module generates the spatial attention weight matrix. ; Step 23. Introduce a gating coefficient α to the generated spatial attention weight matrix. Multiply by the gating coefficient α to adjust the intensity of spatial attention; Then, the gated residual modulation formula is used to analyze the input static features. Element-level dynamic weighted modulation is performed to obtain the modulated appearance features. The formula is as follows: （1） in This represents element-wise matrix multiplication. Enlarged spatial features; In this round, the dynamic characteristics remain unchanged; Step 24. When the iteration number m is even, use the output after modulation from the previous round. Reverse generation of static appearance attention weight matrix Generate the attention weight matrix; Step 25. Combining the gating coefficient α, utilize the gating residual modulation formula to analyze the dynamic characteristics of the input. Element-level inverse weighting is performed to obtain the corrected motion characteristics. The formula is as follows: (2) In this round, the appearance remains unchanged; Step 26. Let Repeat the iteration steps until the number of iterations m reaches the preset total number of iterations M; The appearance characteristics of the last round of alternating modulation output and the corrected motion characteristics Channel-by-channel cascading and linear channel mapping fusion are performed to generate the fused feature map of the i-th level, which is then output to the subsequent attention or decoding stages.

5. The video salient target detection method according to claim 4, characterized in that, In step 3, the last two high-level fusion feature maps from the four levels of fusion feature maps in step 2 are input into the spatiotemporal decoupling attention module. Spatiotemporal long-distance dependency modeling is performed independently within each level, and the spatiotemporal context features of the two levels are output accordingly. The specific steps are as follows: Step 31. Extract the fused feature maps of the 3rd and 4th layers output in Step 26, and perform spatiotemporal decoupling attention modeling independently for each of the two layers; For any level: transform the input into spatial pixel sequence features. Where B is the batch size, T is the number of consecutive video frames, and N is the total number of spatial tokens at full resolution, N=H×W; Step 32. Extract spatial pixel sequence features After layer normalization, the corresponding spatial query matrix is ​​generated through three independent linear mapping transformations. Space bond matrix and spatial value matrix ; Will , and The frame is divided into multiple independent subspaces along the channel dimension. Intra-frame spatial self-attention computation is performed independently within each subspace. Full-resolution spatial aggregation features are obtained through channel concatenation and linear output transformation. The formula is as follows: （3） Where d is the scaling factor; Step 33. Utilize residual connections and a feedforward network (FFN) to aggregate spatial features. The original input features X are fused and nonlinearly mapped to produce spatially enhanced features. The formula is as follows: （4） Where LN represents layer normalization and FFN represents feedforward network; Step 34. Enhance spatial features through tensor dimension permutation. Reconstructed into time series features At this point, pixel tokens with the same spatial coordinates in multiple consecutive frames are recombined along the time axis into an independent temporal evolution sequence of length T. Step 35. Extract time series features After layer normalization, the input to the temporal self-attention layer generates a time-domain query matrix through independent linear mappings. Key matrix Sum matrix ; Perform independent multi-head self-attention calculations for each spatial pixel for up to T frames across the time frame, and output temporal aggregated features. ; Will The 3D tensor, consistent with the spatially enhanced feature structure, is restored through inverse dimensional reshaping and permutation operations. ; Step 36. Using residual connections and feedforward networks, The output of step 33 The spatiotemporal characteristics of the final output are obtained by performing fusion and nonlinear transformation. The formula is as follows: （5）。 6. The video salient target detection method according to claim 5, characterized in that, In step 4, the two deep-level spatiotemporal context features from step 3 and the two shallow-level fused feature maps from step 2 are input into the decoder to perform cross-scale progressive decoding and reconstruction from deep to shallow layers, outputting the final video salient object detection mask. The specific steps are as follows: Step 41. First, fuse the shallow-level features from Step 2. The input is a depthwise separable convolutional layer DWConv, which is then processed by a batch normalization layer and a ReLU activation function for nonlinear mapping and feature reconstruction, and outputs a spatial feature response map containing rich local spatial geometric details. Next, the input channel attention CA module dynamically generates a one-dimensional channel attention weight vector and multiplies it back into the original feature. Finally, shallow features refined in multiple dimensions are generated. The formula is as follows: （6）； Step 42. First, obtain deep-level features. Using the upsampling operator Expand the spatial dimensions to align the spatial resolution. ; Next, the aligned deep semantic features and Add elements one by one, input them together into a second depthwise separable convolution, and then combine the output with the initial shallow-level features. A second element-wise residual summation is performed to construct the fused feature matrix F, as shown in the following formula: （7）， Where Up represents upsampling; Step 43. Input F sequentially into the normalized LN layer and the multilayer perceptron (MLP), and then connect the output to the residual F via a jump concatenation. Finally, use 1×1 convolution and bilinear interpolation as a mask to output the prediction mask, as shown in the following formula: （8）。 7. The video salient target detection method according to claim 4, characterized in that, In step 22, the processing steps of the spatial collaborative self-attention module are as follows: Step 221. Denote the input four-dimensional tensor feature map as... Where B, C, H and W represent batch size, number of channels, height and width, respectively; Global average pooling is performed on the feature map F along both the height and width dimensions, compressing the two-dimensional spatial features into one-dimensional vectors in the height and width directions, forming a one-dimensional representation sequence. and ; Step 222. Decouple the one-dimensional feature vector and The feature groups are divided into four subgroups along the channel dimension; a set of multi-scale one-dimensional convolutional kernels of different sizes are used. In parallel, depthwise convolution operations are performed on each feature subspace to adaptively learn spatial geometric contexts at different scales, generating spatial intermediate representation sequences with multiple semantic scales. Step 223. Concatenate and reconstruct the four feature subgroups after one-dimensional depthwise convolution at different scales along the channel dimension to reconstruct the corresponding multi-scale horizontal aggregated features. and multi-scale vertical aggregation features ; Step 224. First, perform layer normalization on the two aggregated features. Then, input the horizontal and vertical features obtained by layer normalization into the sigmoid activation function layer for nonlinear probability mapping, and explicitly constrain the feature response values ​​to the bounded weight range. Step 225. Through a dimension expansion operation, the activated horizontal and vertical features are expanded and reshaped into a one-dimensional horizontal attention weight matrix. and ; and Element-level broadcast multiplication is performed, and while maintaining the independence of the channel dimension C, the horizontal and vertical probability weights are fused into a full-resolution two-dimensional network spatial grid, ultimately outputting the required spatial attention weight matrix.

Images