Video saliency prediction method based on multi-scale bidirectional gate and perception guide fusion

Abstract

The application discloses a video saliency prediction method based on multi-scale bidirectional gating and perception guidance fusion, aiming at solving the problems of insufficient multi-scale feature fusion and poor time consistency in the prior art. The method adopts a Video Swin Transformer to extract multi-scale spatiotemporal features; then, a multi-scale bidirectional gating fusion module is used to enhance feature interaction in the top-down and bottom-up paths; a time attention module is used to strengthen the time sequence information of the highest layer features; and finally, a perception guidance fusion module is used to adaptively aggregate features of each scale to generate a final saliency map. Through the combined mechanism of bidirectional gating, time enhancement and perception guidance, the comprehensiveness of feature expression is strengthened, the temporal coherence is ensured, and the accuracy and stability of video saliency prediction are improved.

Description

Technical Field

[0001] This invention relates to the fields of video processing and computer vision technology, and in particular to a video saliency prediction method based on multi-scale bidirectional gating and perception-guided fusion. Background Technology

[0002] Video saliency prediction aims to simulate the human visual attention mechanism, predicting the salient regions in a video that the human eye focuses on, and representing them in the form of a saliency map. The results have wide applications in tasks such as video compression, object detection, visual coding, and action recognition.

[0003] Existing video saliency prediction methods are mostly based on 3D convolutional neural networks (3D CNNs) or VideoSwin Transformers, capturing salient regions by stacking spatiotemporal feature extraction modules. However, these methods generally suffer from the following problems: First, in the decoding stage, they often employ simple top-down or bottom-up feature fusion paths, making it difficult to effectively combine deep semantic information with shallow detail information, resulting in blurred boundaries of salient regions or incomplete internal responses. Second, they lack sufficient modeling of long-term dependencies in the temporal dimension, which can easily cause jitter and instability in the prediction results of consecutive frames when dealing with complex dynamic scenes, lacking temporal consistency. Third, in multi-scale feature aggregation, they usually use simple concatenation or addition operations, lacking adaptive feature selection and enhancement mechanisms, which may introduce redundant information or noise, affecting the final prediction accuracy.

[0004] To address the aforementioned issues, this invention proposes a video saliency prediction method based on multi-scale bidirectional gating and perception-guided fusion. By constructing a bidirectional gating feature fusion framework, introducing temporal attention, and designing a perception-guided fusion module, the method significantly improves the accuracy, spatial detail integrity, and temporal stability of video saliency prediction. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies, such as low saliency prediction accuracy and poor temporal consistency due to insufficient multi-scale feature fusion and inadequate time-dependent modeling. This invention proposes a video saliency prediction method based on multi-scale bidirectional gating and perception-guided fusion, which achieves high-precision and stable prediction of salient regions in videos.

[0006] To achieve the above objectives, this invention provides a video saliency prediction method based on the fusion of multi-scale bidirectional gating and perception guidance, the method comprising the following steps: S1: Obtain the video sequence to be predicted, select consecutive frames and input them into the video saliency prediction network; S2: The video sequence is processed by an encoder using a Video Swin Transformer as the backbone network to extract multi-scale spatiotemporal features and obtain feature maps at different scales. ; S3: Input the feature maps of different scales into the multi-scale bidirectional gating fusion module to perform bidirectional gating fusion of the multi-scale features from top to bottom and from bottom to top to obtain the fused features of each scale; S4: Perform temporal attention enhancement on the highest-level features in the fusion features of each scale, that is, extract feature change information in the time dimension through the temporal attention module to generate high-level reference features with consistent time sequence. S5: Input the high-level reference features and the fusion features at each scale into the perception-guided fusion module to achieve adaptive aggregation of multi-scale features under perception guidance, and generate a saliency prediction map consistent with the input resolution through the prediction module.

[0007] A further technical solution involves the Video Swin Transformer, which serves as the encoder of the backbone network, comprising a four-layer structure. As the layers deepen, the temporal and spatial dimensions of the output feature map decrease, while the channel dimension increases. This pyramid structure effectively captures multi-layered spatiotemporal information, from low-level texture details to high-level semantic concepts.

[0008] A further technical solution is that the multi-scale bidirectional gating fusion module in step S3 includes two fusion paths: a top-down fusion path and a bottom-up fusion path. The top-down gating fusion path starts from the highest layer features. Initially, deep semantic features are propagated step-by-step to shallow features through trilinear upsampling, guided by gated fusion units; the bottom-up gated fusion path starts from the lowest layer features. Initially, shallow detail features are propagated to deeper features through max pooling, and gated fusion units are used to supplement them.

[0009] A further technical solution is that the implementation mechanism of the gated fusion unit is as follows: First, the two input features X and Y are concatenated along the channel dimension, and a spatially adaptive gated weight (GA) is generated through a 1×3×3 convolution and a sigmoid function. Then, the second input feature is weighted using this weight. Finally, the weighted feature is adaptively fused with the first input feature through two learnable residual fusion parameters. The specific formula is as follows: , , , Where G represents the concatenated feature. This represents the Sigmoid function. This represents a 3D convolution with a kernel size of 1×3×3. The output characteristics after passing through the gated fusion unit, , These are learnable scalar parameters that are automatically optimized through network training and are used to control the weighting ratio of the two input features in the fused output.

[0010] A further technical solution is that, in step S4, the highest-level features among the fused features at each scale undergo temporal attention enhancement. Specifically, the highest-level features are passed through a temporal attention module, and the processing includes: processing the input features... Adaptive average pooling, adaptive max pooling, and standard deviation pooling are performed along the time dimension to preserve the time dimension. The multi-scale pooling results are concatenated and channel attention weights are learned through a multilayer perceptron. The attention weights TA are multiplied by the original features and the features are enhanced through residual connections. The specific formula is as follows: , , in, Here, S is the Sigmoid function, MLP represents a multilayer perceptron, and AvgPool, MaxPool, and StdPool represent adaptive average pooling, max pooling, and standard deviation pooling operations, respectively. These are features enhanced by the temporal attention module, and also one of the input features of the perception-guided fusion module.

[0011] A further technical solution is that the fusion feature obtained by the perception-guided fusion module in step S5 is obtained by stitching together multi-scale features. High-level reference features output by the time attention module As input, its processing includes: firstly, generating attention weights (PEUs) through a perceptual enhancement unit to apply to the fused features. Feature enhancement is performed; subsequently, the enhanced perceptual features are combined with high-level reference features. By learnable weights The specific formula for weighted fusion is as follows: , , , in This represents a 1×1×1 3D convolution, with ReLU as the activation function.

[0012] A further technical solution is that the prediction module in step S5 gradually restores the feature space resolution through cascaded three-dimensional convolution and upsampling operations, and finally generates a normalized saliency prediction map. The cascaded structure consists of four sets of alternating three-dimensional convolutional layers and upsampling layers, wherein the convolutional kernel size includes 4×3×3, 2×3×3, 2×1×1 and 1×1×1, the upsampling operation adopts the trilinear interpolation method, and finally outputs a normalized saliency prediction map through the Sigmoid function.

[0013] A further technical solution involves the following training steps for the video saliency prediction network: building a video saliency prediction network; constructing a training dataset and a validation dataset, wherein each dataset contains a video frame sequence and its corresponding ground truth saliency label; inputting the training dataset into the prediction network, calculating the loss function based on the saliency prediction map output by the network and the ground truth saliency label, and performing backpropagation optimization; repeating multiple rounds of training, evaluating the model performance on the validation dataset, and saving the network weights with the minimum loss on the validation set as the final model.

[0014] Compared with the prior art, the present invention has at least the following beneficial effects: 1. The multi-scale bidirectional gating fusion module proposed in this invention achieves effective guidance of shallow features by deep semantic information and accurate supplementation of deep features by shallow detail information through the synergistic effect of two paths: top-down and bottom-up. This significantly improves the spatial positioning accuracy and boundary integrity of salient targets.

[0015] 2. The invention introduces a temporal attention module in the decoding stage. By performing temporal dependency modeling on high-level features, it effectively enhances the model's ability to perceive dynamic saliency changes, ensuring that the generated saliency map is smoother and more stable over time, and reducing flickering.

[0016] 3. The perception-guided fusion module designed in this invention adaptively enhances the saliency response in the aggregated features using the perception enhancement unit, and uses the temporally enhanced high-level features as guidance to achieve synergistic optimization of spatial details and global semantics, thereby further improving the accuracy of the final prediction results.

[0017] 4. The overall structure of this invention balances spatial accuracy and temporal consistency. Comparative experiments on multiple public video saliency datasets, including DHF1K, Hollywood-2, and UCF-Sports, demonstrate that this method exhibits optimal performance on key evaluation metrics such as AUC-J, SIM, CC, and NSS. Attached Figure Description

[0018] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating the overall process of the video saliency prediction method based on the fusion of multi-scale bidirectional gating and perception guidance in this embodiment of the invention.

[0019] Figure 2 This is a detailed architecture diagram of the video saliency prediction network in an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the gated fusion unit in an embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram of the time attention module in an embodiment of the present invention.

[0022] Figure 5 This is a schematic diagram of the perception enhancement unit structure in an embodiment of the present invention.

[0023] Figure 6 This is a visualization of the saliency prediction results of embodiments of the present invention on different datasets. Detailed Implementation

[0024] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0025] Example 1 This invention provides a video saliency prediction method based on multi-scale bidirectional gating and perception-guided fusion, the overall process of which is as follows: Figure 1 As shown, the detailed network architecture is as follows: Figure 2 As shown. This method mainly includes the following steps: S1: Obtain the video sequence to be predicted, select consecutive frames and input them into the video saliency prediction network; S2: The video sequence is processed by an encoder using a Video Swin Transformer as the backbone network to extract multi-scale spatiotemporal features and obtain feature maps at different scales. ; S3: Input the feature maps of different scales into the multi-scale bidirectional gating fusion module to perform bidirectional gating fusion of the multi-scale features from top to bottom and from bottom to top to obtain the fused features of each scale; S4: Perform temporal attention enhancement on the highest-level features in the fusion features of each scale, that is, extract feature change information in the time dimension through the temporal attention module to generate high-level reference features with consistent time sequence. S5: Input the high-level reference features and the fusion features at each scale into the perception-guided fusion module to achieve adaptive aggregation of multi-scale features under perception guidance, and generate a saliency prediction map consistent with the input resolution through the prediction module.

[0026] The present invention provides a preferred embodiment for executing S1. This method first acquires the video sequence to be predicted and processes it into a video segment consisting of T consecutive frames (T=32 in this embodiment). The video clip is then input into the trained video saliency prediction network.

[0027] This invention provides a preferred embodiment for executing S2. A Video Swin Transformer (VST) is used as the encoder of the backbone network to extract multi-scale spatiotemporal features from the input video segment. The encoder contains four hierarchically progressive VST coding blocks, forming a pyramid structure. Each coding block outputs a spatiotemporal feature map at one scale, thus obtaining four feature maps at different scales. .in As the most basic feature, This represents the highest level of features. As the encoding layers deepen, the spatial and temporal dimensions of the feature maps decrease accordingly, while the channel dimension increases. This structural design aims to capture multi-level information, from low-level texture and motion to high-level semantics.

[0028] The present invention provides a preferred embodiment for executing S3, which involves processing the four-layer features output by the encoder. The input is fed into a multi-scale bidirectional gated fusion module. This module contains two parallel paths: Top-down path: from highest-level features Initially, the feature map resolution is progressively increased through upsampling operations, and then fused with the features from the next layer using gated fusion units. This approach aims to pass high-level semantic information to the shallower layers to guide the localization of salient targets.

[0029] Bottom-up approach: Starting from the lowest-level features Initially, the feature map resolution is progressively reduced through max pooling operations, and then fused with the features from the previous layer using gated fusion units. This approach aims to supplement the deeper layers with details and boundary information from the lower layers to optimize the contours of salient regions.

[0030] The structure of the gate fusion unit is as follows: Figure 3As shown in the diagram, this module receives two input features: feature X from the previous fusion stage and feature Y from the same level of the encoder. The specific calculation process is as follows: X and Y are concatenated along the channel dimension to obtain the combined feature G=[X,Y]. G is then fed into a 3D convolutional layer with a kernel size of 1×3×3, and an adaptive gated weight GA is generated using a sigmoid activation function. This is achieved through two learnable scalar parameters. and The gated weighted features are residually fused with the original input features X to obtain the output. This mechanism can adaptively control the contribution ratio of the two input features in the fused output.

[0031] The two paths ultimately output multi-scale features after bidirectional gating fusion. .

[0032] The present invention provides a preferred embodiment for executing S4, which, in order to enhance the model's ability to model temporal dynamics, incorporates the highest-level features obtained through bidirectional fusion. The input is fed into a time-attention module, whose structure is as follows: Figure 4 As shown.

[0033] This module first processes the input features. Adaptive average pooling, adaptive max pooling, and standard deviation pooling operations are performed along the time dimension to obtain three statistical descriptors that can represent temporal information. These three descriptors are then concatenated and passed through a multilayer perceptron (MLP) and a sigmoid function to learn the channel-level temporal attention weights TA. .

[0034] Finally, the learned attention weights TA are compared with the original features. Element-wise multiplication is performed, and the enhanced high-level reference features are obtained through residual concatenation. : .

[0035] The present invention provides a preferred embodiment for performing step S5, which aims to aggregate all scale information and generate the final saliency map. Its specific implementation and the process obtained by the network are described below: First, the multi-scale features after bidirectional gating fusion are... The spatial resolution is aligned by upsampling, and then the data is concatenated along the channel dimension to obtain an aggregated feature containing information from all scales. .

[0036] Subsequently, the aggregation feature The input is fed into the perception-guided fusion module. The core of this module is a perception enhancement unit, the structure of which is as follows: Figure 5 As shown, it aggregates features through two cascaded 1×1×1 3D convolutional layers and a ReLU activation function. Generate an attention weight. This attention weight is used to... Weighting is applied to enhance the response of salient regions and suppress background noise, thereby obtaining perceptual enhancement features. .

[0037] Then, the perception enhancement features Compared with high-level reference features enhanced by the time attention module Weighted fusion is performed to obtain the final fusion features. : , in, It is a learnable scalar fusion weight.

[0038] Finally, the final fusion features The data is fed into the prediction module. This module typically consists of several cascaded convolutional layers and upsampling layers. Its function is to progressively restore the spatial resolution of the feature map to match that of the input video frame, and finally normalize it using a sigmoid function to generate the final saliency map. .

[0039] To obtain a trained video saliency prediction network that performs the prediction function in the above steps, the training steps in this embodiment include: Training and validation sets were constructed using three publicly available video saliency datasets: DHF1K, Hollywood-2, and UCF-Sports. DHF1K is a large-scale video gaze database covering various scenes and motion types; its 1000 videos were divided into training, validation, and test sets of 600, 100, and 300 videos, respectively. Hollywood-2 consists of 1707 video clips from Hollywood movies, with 823 used for training and 884 for testing. UCF-Sports contains videos of sports scenes, with 103 videos used for training and 47 for testing.

[0040] Loss Function Definition and Backpropagation: The training set is input into the video saliency prediction network of this invention to obtain the saliency prediction map S output by the network, and loss function calculation and backpropagation optimization are performed between the network and the corresponding ground truth saliency map G. This embodiment uses multiple saliency loss terms as the total loss function, including Kullback-Leibler Divergence loss and Linear Correlation Coefficient (CC) loss. Assuming G and S represent the ground truth map and the prediction map, respectively, and F represents the fixation point annotation data, the calculation method for each loss term is as follows: , Where i represents the pixel index. , Let represent the covariance and mean, respectively, and N be the number of fixations. The final total loss function is expressed as follows: .

[0041] Iterative Training and Model Saving: The Adam optimizer is used to train the network through multiple rounds of iterative training, and the model performance is evaluated on the validation set after each round of training. When the loss on the validation set reaches its minimum, the current network parameters are saved, thus obtaining the final trained video saliency prediction model that can be used for saliency prediction. This model can stably output saliency maps in different scenarios and maintain cross-frame consistency and spatial structure integrity.

[0042] To verify the effectiveness of the method of this invention, comparative experiments were conducted with several mainstream video saliency prediction methods. The experiments were performed on the DHF1K, Hollywood-2, and UCF-Sports datasets, and four metrics were used as performance evaluation standards: AUC-Judd (AUC-J), similarity measure (SIM), linear correlation coefficient (CC), and normalized scan path saliency (NSS). Higher values ​​for these metrics indicate that the predicted saliency region is closer to the actual human eye gaze distribution.

[0043] Table 1 shows a performance comparison of this embodiment with other state-of-the-art methods on three datasets. As can be seen from the results in Table 1, this embodiment outperforms existing methods in multiple metrics across various datasets. Figure 6 The saliency prediction results of this embodiment on different datasets are visualized. As can be seen, the prediction results generated by this embodiment are highly consistent with the ground truth map in spatial distribution, and can completely and accurately highlight the key salient targets in the video.

[0044] Table 1. Comparative experimental results of this embodiment with other advanced methods on three datasets.

[0045] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A video saliency prediction method based on multi-scale bidirectional gating and perception-guided fusion, characterized in that, Includes the following steps: S1: Obtain the video sequence to be predicted, select consecutive frames and input them into the video saliency prediction network; S2: The video sequence is processed by an encoder using a Video Swin Transformer as the backbone network to extract multi-scale spatiotemporal features and obtain feature maps at different scales. ; S3: Input the feature maps of different scales into the multi-scale bidirectional gating fusion module to perform bidirectional gating fusion of the multi-scale features from top to bottom and from bottom to top to obtain the fused features of each scale; The implementation mechanism of the gating unit in the multi-scale bidirectional gating fusion module is as follows: First, the feature X from the previous fusion stage and the same-level feature Y from the encoder are concatenated along the channel dimension. Then, a spatially adaptive gating weight GA is generated through convolution and the sigmoid function. Next, the weight is used to weight the second input feature. Finally, the weighted feature is adaptively fused with the first input feature through two learnable residual fusion parameters. The specific formula is as follows: ， ， ， Where G represents the concatenated feature. This represents the Sigmoid function. This represents a 3D convolution with a kernel size of 1×3×3. The output characteristics after fusion by the gating unit, , These are learnable scalar parameters that are automatically optimized through network training and are used to control the weighting ratio of the two input features in the fused output. S4: Perform temporal attention enhancement on the highest-level features in the fusion features of each scale, that is, extract feature change information in the time dimension through the temporal attention module to generate high-level reference features with consistent time sequence. The highest-level features in the fused features at each scale are enhanced with temporal attention, specifically by making the highest-level features... The processing steps using the temporal attention module include: processing the input features... Adaptive average pooling, adaptive max pooling, and standard deviation pooling are performed along the time dimension to preserve the time dimension. The multi-scale pooling results are concatenated and channel attention weights are learned through a multilayer perceptron. The attention weights TA are multiplied by the original features and the features are enhanced through residual connections. The calculation formula is as follows: ， ， in, Here, S is the Sigmoid function, MLP represents a multilayer perceptron, and AvgPool, MaxPool, and StdPool represent adaptive average pooling, max pooling, and standard deviation pooling operations, respectively. These are features enhanced by the temporal attention module; S5: Input the high-level reference features and the fusion features at each scale into the perception-guided fusion module to achieve adaptive aggregation of multi-scale features under perception guidance, and generate a saliency prediction map consistent with the input resolution through the prediction module. The perception-guided fusion module uses the fused features obtained by stitching together multi-scale features. High-level reference features output by the time attention module As input, its processing includes: firstly, the perceptual enhancement unit generates attention weights (PEUs) to adjust the fused features. Feature enhancement is performed; subsequently, perceptual enhancement features are applied. High-level reference features By learnable weights Weighted fusion is performed to obtain the final output features. The calculation formula is as follows: ， ， ， in This represents a 1×1×1 3D convolution, with ReLU as the activation function.

2. The video saliency prediction method based on multi-scale bidirectional gating and perception-guided fusion as described in claim 1, characterized in that, The encoder, which uses Video Swin Transformer as its backbone network, comprises a four-layer structure. As the layers deepen, the temporal and spatial dimensions of the output feature map decrease, while the channel dimension increases.

3. The video saliency prediction method based on multi-scale bidirectional gating and perception-guided fusion as described in claim 1, characterized in that, The multi-scale bidirectional gating fusion module described in step S3 includes two fusion paths: top-down and bottom-up. The top-down gating fusion path starts from the highest-level features. Initially, the spatial resolution of each layer of features is aligned by a trilinear upsampling operation, and deep semantic features are propagated to shallow features step by step. Adaptive fusion is achieved using gating units. The bottom-up gating fusion path starts from the most basic features. Initially, the spatial resolution of features in each layer is aligned through max pooling, and shallow detailed features are propagated to deep features step by step. Adaptive fusion is achieved using gating units.

4. The video saliency prediction method based on multi-scale bidirectional gating and perception-guided fusion as described in claim 1, characterized in that, The prediction module in step S5 gradually restores the feature space resolution through cascaded three-dimensional convolution and upsampling operations, and finally generates a normalized saliency prediction map. The cascaded structure consists of four sets of alternating three-dimensional convolutional layers and upsampling layers, wherein the convolutional kernel size includes 4×3×3, 2×3×3, 2×1×1 and 1×1×1, the upsampling operation adopts the trilinear interpolation method, and finally outputs the normalized saliency prediction map through the Sigmoid function.

5. The video saliency prediction method based on multi-scale bidirectional gating and perception-guided fusion as described in claim 1, characterized in that, The training steps for the video saliency prediction network include: A video saliency prediction network is constructed, wherein the network adopts the network structure defined in the method as described in any one of claims 1 to 4; Construct training and validation datasets, where each dataset contains a sequence of video frames and their corresponding real salient icons; The training dataset is input into the prediction network, and the loss function is calculated based on the saliency prediction map output by the network and the true saliency map, and then backpropagation optimization is performed. After repeated training rounds, the model performance is evaluated on the validation dataset, and the network weights with the smallest loss on the validation set are saved as the final model.

Images