A Video Salience Prediction Method Based on a Two-Stream Architecture

Abstract

The present invention discloses a video saliency prediction method based on a two-stream architecture. First, the temporal and spatial features contained in video frames and optical flow maps are extracted; the temporal features are enhanced through an attention module and the temporal and spatial features are fused; then the spatio-temporal features are filtered using the mask map generated by the temporal decoder; a GRU module is used to obtain the long-term spatio-temporal information between video frames; and finally a spatial decoder is used to generate a prediction map. The present invention more effectively extracts the temporal and spatial features contained in the optical flow map and video frames by using a novel two-stream architecture; more effectively fuses the temporal and spatial features by using an attention module to enhance the temporal features at different scales; more accurately locates the positions of salient objects in the spatio-temporal features by using a temporal masking method; and can capture longer spatio-temporal information in the video by using a GRU module.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and particularly relates to a video saliency prediction method based on a two-stream architecture. Background Art

[0002] Humans can quickly focus their attention on important regions in the field of vision and filter out irrelevant information. This selection process is called the visual attention mechanism, which helps humans process a large amount of visual information in real time. In the 1990s, visual attention began to be studied in the field of computer vision and has shown extensive applications, such as object segmentation, video compression, video captioning, etc. Applying the visual attention mechanism to the above tasks can allocate limited computing resources to the objects most relevant to the task.

[0003] In recent years, many deep learning-based models have been proposed to simulate the visual attention mechanism (image saliency prediction) of humans in static scenes, but there are few models for simulating the visual attention mechanism (video saliency prediction) of humans in dynamic scenes. Different from image saliency prediction, in videos, both temporal and spatial information are important factors affecting the direction of human attention. How to effectively integrate saliency features from different domains and different scales is a fundamental but long-standing unsolved problem in video saliency prediction.

[0004] Currently, there are mainly three types of deep learning-based video saliency prediction models. The first is the model based on the two-stream architecture, where the optical flow backbone is used to extract temporal information from the optical flow map, and the appearance backbone is used to extract spatial information from video frames. However, most current two-stream models simply fuse the spatio-temporal information at the end of the backbone, making it impossible to fully utilize the temporal information. The second is the model based on the long short-term memory network (LSTM). First, a convolutional network is used to extract spatial information, and then LSTM is used to obtain temporal information. However, the separate processing of spatio-temporal information limits the accuracy of saliency prediction. The third is the model based on 3D convolution, where 3D convolution can process both temporal and spatial information simultaneously. However, 3D convolution can only encode visual information in a fixed local spatio-temporal according to its kernel size and cannot effectively obtain long-term spatio-temporal information in the video. Summary of the Invention

[0005] The technical problem to be solved by the present invention: The existing video saliency prediction methods cannot effectively extract and fuse temporal and spatial features, and cannot effectively capture long-term spatio-temporal information in the video, resulting in inaccurate predicted saliency regions.

[0006] The technical solution adopted by the present invention to solve this technical problem: Use a video saliency prediction model based on a two-stream architecture to improve the accuracy of video saliency prediction. In order to effectively extract temporal and spatial features, a brand-new two-stream architecture is proposed, in which the temporal encoder is used to extract temporal features from the optical flow map, and the spatial encoder is used to extract spatial features from video frames. In order to effectively fuse temporal and spatial features, an attention module is proposed and deployed at each level of the two-stream architecture. This module first performs channel attention and spatial attention on the input temporal features in sequence, and then transmits the enhanced temporal features to the spatial encoder for fusion. In order to remove the interference information in the spatial features, a temporal masking method is proposed, that is, first generate a mask map through the temporal decoder, and then use the mask map to weight and filter the spatial features at the top layer of the encoder, highlighting the salient features and removing redundant features. In order to effectively capture the long-term spatio-temporal information in the video, a gated recurrent unit (GRU) is used in the model, which can capture the effective spatio-temporal information contained in consecutive video frames by controlling the opening degrees of the internal reset gate and update gate.

[0007] A video saliency prediction method based on a two-stream architecture, the steps are as follows:

[0008] Step 1: Extract the temporal and spatial features contained in the video frames and optical flow maps.

[0009] First, preprocess the video frames to be predicted and the corresponding optical flow maps, including size adjustment, data type conversion, and numerical normalization. The preprocessed pictures can improve the running efficiency of the model. Then, input the preprocessed video frames and optical flow maps into the temporal encoder and spatial encoder of the model respectively to extract temporal features and spatial features.

[0010] Step 2: Enhance the temporal features through the attention module and fuse the temporal and spatial features.

[0011] First, input the temporal features extracted by the temporal encoder into the attention module for enhancement processing. The processing of the attention module includes two steps. The first step is to use the channel attention mechanism to weight the feature map, and the second step is to use the spatial attention mechanism to weight different regions in the feature map. The use of the attention module can not only maintain the semantic consistency between temporal and spatial features, but also reduce the interference caused by poor-quality optical flow maps to the model performance. Then, fuse the enhanced temporal features with the spatial features extracted by the spatial encoder, and the fused spatio-temporal features will be input into a higher-level spatial encoder for further processing. It should be noted that the attention module is deployed at each level of the encoder, so for temporal features of different scales, the enhancement and fusion of features will be performed sequentially from low to high.

[0012] Step 3: Filter the spatio-temporal features using the mask map generated by the temporal decoder.

[0013] First, input the temporal features at the top layer of the temporal encoder into the temporal decoder for processing. The temporal decoder includes two convolutional layers and a Sigmoid activation function. After processing, the temporal decoder outputs a mask map, which contains the approximate positions and contours of the moving objects in the video frames. Then, use the mask map to perform weighted filtering on the spatio-temporal features at the top layer of the spatial decoder, so as to effectively remove the interference information contained in the spatio-temporal features and highlight the salient features.

[0014] Step 4: Use the GRU module to obtain the long-term spatio-temporal information between video frames.

[0015] First, input the spatio-temporal features filtered by the mask map into the gated recurrent unit GRU for processing. The GRU module will capture the effective spatio-temporal information contained in consecutive video frames by controlling the opening degrees of the internal reset gate and update gate. Then, input the processed spatio-temporal features into the spatial decoder for further processing.

[0016] Step 5: Use the spatial decoder to generate the prediction map.

[0017] Input the spatio-temporal features processed by the GRU module into the spatial decoder for processing, that is, first process them through two convolutional layers to reduce their channel dimension to 1, then process them through a Sigmoid activation function, and finally use the bilinear upsampling function to enlarge their size to the size of the original video frame to obtain the final prediction map.

[0018] Furthermore, the specific method of step 1 is as follows:

[0019] First, preprocess the video frame to be predicted and the corresponding optical flow map. The content includes: reading the picture according to the picture address and converting it to the RGB format; adjusting the size to 224×224; converting the data type to Tensor type; mapping the numerical range from [0, 255] to [-1, 1].

[0020] Then, input the preprocessed video frame X S and the optical flow map X T into the temporal encoder and spatial encoder of the model respectively for extracting temporal features and spatial features The temporal encoder and spatial encoder have the same structure, both of which are constructed based on the ResNet-50 backbone. Among them, the convolutional blocks T1-T5 of the temporal encoder and the convolutional blocks S1-S5 of the spatial encoder correspond to the convolutional blocks Conv1-Conv5 of the ResNet-50 backbone.

[0021] Furthermore, the specific method of step 2 is as follows:

[0022] First, the temporal features extracted by the temporal encoder are input into the attention module for enhancement processing. The processing of the attention module includes two parts. The first part is to use the channel attention mechanism to weight the feature map. That is, first use global average pooling to change the feature dimension to C×1×1, and then use two consecutive convolutional layers to first compress the feature dimension to C / 16×1×1 and then expand it back to the original dimension C×1×1. Then use the Sigmoid activation function to map the numerical range of the features to [0,1] to generate a weight vector C i , and finally multiply the weight vector C i and the original temporal features and use residual connection to obtain the channel-weighted features The above process can be represented by Formula 1.

[0023]

[0024] Among them, i∈{1,2,3,4,5} represents the 5 levels of the encoder; GAP represents the global average pooling layer; Conv represents the 1×1 convolutional layer; Relu represents the Relu activation function; Sigmoid represents the Sigmoid activation function; ⊙ represents element-wise multiplication; represents element-wise addition.

[0025] The second part is to use the spatial attention mechanism to weight different regions in the feature map. That is, first use a 1×1 convolutional layer to change the dimension of the channel-weighted features to 1×H×W, and then use the Sigmoid activation function to generate a weight matrix T i , and finally multiply the weight matrix T i and the original channel-weighted features and use residual connection to obtain the final temporally enhanced features The above process can be represented by Formula 2.

[0026]

[0027] Then, the enhanced temporal features are fused with the spatial features extracted by the spatial encoder , and the fused spatio-temporal features will be input into a higher-level spatial encoder for further processing. The above process can be represented by Formula 3.

[0028]

[0029] It should be noted that the attention module is deployed at each layer of the encoder. Therefore, for time features of different scales, feature enhancement and fusion are performed in sequence from low to high.

[0030] Furthermore, the specific method of step 3 is as follows:

[0031] First, the time features at the top layer of the time encoder are input to the time decoder for processing. The time decoder will first use two consecutive 1×1 convolutional layers to reduce the feature channel dimension to C / 2 and 1 in sequence, and then use a Sigmoid activation function for processing to obtain a mask map P T . Then, the mask map P T is used to weightedly filter the spatio-temporal features at the top layer of the spatial decoder to obtain the weighted spatio-temporal features The above process can be represented by Equation 4.

[0032]

[0033] Furthermore, the specific method of step 4 is as follows:

[0034] First, the spatio-temporal features filtered by the mask map are input to the gated recurrent unit GRU for processing. The GRU module captures the effective spatio-temporal information contained in consecutive video frames by controlling the opening degrees of the internal reset gate and update gate. Its specific execution process can be represented by Equation 5.

[0035]

[0036] Among them, R t represents the reset gate; Z t represents the update gate; represents the candidate hidden state; H t represents the new hidden state; X t represents the input feature; H t-1 represents the old hidden state; W xr , W hr , W xz , W hz , W xh , W hh represent the weight parameters of the convolutional layer; b r , b z , b h represent the bias parameters of the convolutional layer; σ represents the Sigmoid activation function; tanh represents the tanh activation function.

[0037] Then, the hidden state H containing rich spatio-temporal information output by the GRU module tIt is input into the spatial decoder for further processing. It should be noted that in order to use the GRU module, the batch size is set to 6, that is, the model needs to input 6 consecutive video frames of the same video into the spatial encoder each time for processing, and input the corresponding 6 optical flow maps into the temporal encoder. After a series of processes, the model will predict the saliency map corresponding to the last video frame.

[0038] Furthermore, the specific method of step 5 is as follows:

[0039] The hidden state H output by the GRU module t is input into the spatial decoder to obtain the final prediction map. Specifically, first use two consecutive 1×1 convolutional layers to sequentially reduce the feature channel dimension to C / 2 and 1, then use a Sigmoid activation function for processing, and finally use a bilinear upsampling function to enlarge its size to the size of the original video frame to obtain the final saliency prediction map P S . The above process can be represented by formula 6.

[0040] P S = Up(Sigmoid(Conv2(Relu(Conv1(H t ))))) (6)

[0041] where P S represents the predicted saliency map; Up represents the bilinear upsampling function.

[0042] The beneficial effects of the present invention are as follows:

[0043] By using a novel two-stream architecture, the temporal and spatial features contained in the optical flow map and video frames are extracted more effectively; by using the attention module to enhance the temporal features at different scales, the temporal and spatial features are fused more effectively; by using a temporal masking method, the positions of salient objects in the spatio-temporal features are located more accurately; by using the GRU module, longer spatio-temporal information in the video can be captured. Brief Description of the Drawings

[0044] Figure 1 is the structural diagram of the video saliency prediction model based on the two-stream architecture;

[0045] Figure 2 is the structural diagram of the attention module;

[0046] Figure 3 is the structural diagram of the gated recurrent unit (GRU). Specific Embodiments

[0047] The implementation steps of the present invention will be discussed in detail in combination with the content of the invention and the drawings.

[0048] A video saliency prediction method based on a two-stream architecture, the steps are as follows:

[0049] Step 1: Extract the temporal and spatial features contained in the video frames and optical flow maps.

[0050] First, preprocess the video frames to be predicted and the corresponding optical flow maps, including: reading the pictures according to the picture addresses and converting them to RGB format; adjusting the size to 224×224; converting the data type to Tensor type; mapping the numerical range from [0,255] to [-1,1]. It should be noted that the optical flow maps used in the present invention are generated by the existing optical flow model RAFT.

[0051] Then, input the preprocessed video frame X S and the optical flow map X T into the temporal encoder and spatial encoder of the model respectively to extract temporal features and spatial features The temporal encoder and spatial encoder have the same structure, both are constructed based on the ResNet-50 backbone, where the convolutional blocks T1-T5 of the temporal encoder and the convolutional blocks S1-S5 of the spatial encoder correspond to the convolutional blocks Conv1-Conv5 of the ResNet-50 backbone.

[0052] Step 2: Enhance the temporal features through the attention module and fuse the temporal and spatial features.

[0053] First, input the temporal features extracted by the temporal encoder into the attention module for enhancement processing. Specifically, as Figure 2 shown, the processing of the attention module includes two parts. The first part is to use the channel attention mechanism to weight the feature map, that is, first use global average pooling to change the feature dimension to C×1×1, and then use two consecutive convolutional layers to first compress the feature dimension to C / 16×1×1 and then expand it back to the original dimension C×1×1. Then use the Sigmoid activation function to map the numerical range of the features to [0,1] to generate a weight vector C i , and finally multiply the weight vector C i and the original temporal features and use residual connection to obtain the channel-weighted features The above process can be expressed by formula 1.

[0054]

[0055] Among them, \(i\in\{1,2,3,4,5\}\) represents the 5 levels of the encoder; GAP represents the global average pooling layer; Conv represents the \(1\times1\) convolutional layer; Relu represents the Relu activation function; Sigmoid represents the Sigmoid activation function; \(\odot\) represents element-wise multiplication; represents element-wise addition.

[0056] The second part is to use the spatial attention mechanism to weight different regions in the feature map, that is, first use a \(1\times1\) convolutional layer to change the dimension of the channel-weighted feature to \(1\times H\times W\), then use the Sigmoid activation function to generate a weight matrix \(T\) i , and finally multiply the weight matrix \(T\) i and the original channel-weighted feature and use the residual connection to obtain the final temporal enhanced feature The above process can be represented by Equation 2.

[0057]

[0058] Then the enhanced temporal feature is fused with the spatial feature extracted by the spatial encoder , and the fused spatio-temporal feature will be input into a higher-level spatial encoder for further processing. The above process can be represented by Equation 3.

[0059]

[0060] It should be noted that the attention module is deployed at each level of the encoder. Therefore, for temporal features of different scales, the enhancement and fusion of features will be performed successively from low to high.

[0061] Step 3: Filter the spatial feature using the mask map generated by the temporal decoder.

[0062] First, input the temporal feature at the top layer of the temporal encoder into the temporal decoder for processing. The temporal decoder will first use two consecutive \(1\times1\) convolutional layers to successively reduce the feature channel dimension to \(C / 2\) and 1, and then use a Sigmoid activation function to process it to obtain a mask map \(P\) T . Then use the mask map \(P\) T to weight and filter the spatio-temporal feature at the top layer of the spatial decoder to obtain the weighted spatio-temporal feature The above process can be represented by Equation 4.

[0063]

[0064] Step 4: Use the GRU module to obtain the long-term spatio-temporal information between video frames.

[0065] First, input the spatio-temporal features filtered by the mask image into the gated recurrent unit (GRU) for processing. The GRU module captures the effective spatio-temporal information contained in consecutive video frames by controlling the opening degrees of the internal reset gate and update gate. Its specific execution process can be represented by Equation 5.

[0066]

[0067] Among them, R t represents the reset gate; Z t represents the update gate; represents the candidate hidden state; H t represents the new hidden state; X t represents the input feature; H t-1 represents the old hidden state; W xr , W hr , W xz , W hz , W xh , W hh represent the weight parameters of the convolutional layer; b r , b z , b h represent the bias parameters of the convolutional layer; σ represents the Sigmoid activation function; tanh represents the tanh activation function.

[0068] Then, input the hidden state H containing rich spatio-temporal information output by the GRU module t into the spatial decoder for further processing. It should be noted that to use the GRU module, the batch size should be set to 6, that is, the model needs to input 6 consecutive video frames of the same video in the spatial encoder each time it processes, and input the corresponding 6 optical flow maps in the temporal encoder. After a series of processes, the model will predict the saliency map corresponding to the last video frame.

[0069] Figure 3 is the structural diagram of the gated recurrent unit (GRU).

[0070] Step 5: Use the spatial decoder to generate the prediction map.

[0071] Input the hidden state H output by the GRU module t into the spatial decoder to obtain the final prediction map. Specifically, first use two consecutive 1×1 convolutional layers to reduce the feature channel dimension to C / 2 and 1 in sequence, then use a Sigmoid activation function for processing, and finally use a bilinear upsampling function to enlarge its size to the size of the original video frame to obtain the final saliency prediction map P SThe above process can be represented by Equation 6.

[0072] P S = Up(Sigmoid(Conv2(Relu(Conv1(H t )))) (6)

[0073] where P S represents the predicted saliency map; Up represents the bilinear upsampling function.

[0074] Model training details:

[0075] Figure 1 is the structural diagram of the video saliency prediction model based on the two-stream architecture;

[0076] The proposed model is implemented using Pytorch on an NVIDIA TITAN X Pascal GPU. The parameters of the temporal and spatial encoders are both initialized using ResNet-50, and the parameters of the remaining parts are initialized with the default settings of Pytorch. The entire model is trained using the Adam optimizer, and the initial learning rate is set to 10 -5 .

[0077] The model is trained using the training sets of the DHF1K, Hollywood-2, and UCF Sports datasets, the validation set of DHF1K is used to monitor the entire training process, and the test sets of these three datasets are used to evaluate the model performance. The input video frames and optical flow maps are both resized to 224×224, the batch size is set to 6, that is, 6 consecutive video frames in the same video will be processed in each training iteration. In order to predict the first 5 frames of the video, the first frame needs to be copied 5 times. It should be noted that the model only uses the prediction map P S during the test process, but during the training process, it is necessary to simultaneously supervise the mask map P T output by the temporal decoder and the prediction map P S output by the spatial decoder. The total training loss can be represented by Equation 7.

[0078] L(P, G) = L loss (P S [[ID=4C]] loss (P T , G) (7)

[0079] where P represents the predicted saliency map; G represents the ground-truth saliency map; L loss represents the hybrid loss function, and its calculation process can be represented by Equation 8.

[0080] L loss (P, G) = KL(P, G) + CC(P, G) (8) ​

[0081] Among them, KL is a commonly used metric for measuring the difference between two probability distributions, and its calculation process can be expressed by Equation 9.

[0082]

[0083] Among them, ε represents the regularization constant.

[0084] The metric CC is used to measure the dependence relationship between two distribution maps, and is expressed by Equation 10.

[0085]

[0086] Among them, sd represents the standard deviation, and cov represents the covariance.

Claims

1. A video saliency prediction method based on a two-stream architecture, characterized in that The steps are as follows: Step 1: Extract the temporal and spatial features contained in the video frames and optical flow maps; First, preprocess the video frames to be predicted and the corresponding optical flow maps, including resizing, data type conversion, and numerical normalization; then input the preprocessed optical flow maps and video frames into the temporal encoder and spatial encoder of the model respectively to extract temporal features and spatial features; Step 2: Enhance the temporal features through the attention module and fuse the temporal and spatial features; First, input the temporal features extracted by the temporal encoder into the attention module for enhancement processing. The processing of the attention module includes two steps. The first step is to use the channel attention mechanism to weight the feature map, and the second step is to use the spatial attention mechanism to weight different regions in the feature map; then fuse the enhanced temporal features with the spatial features extracted by the spatial encoder. The fused spatio-temporal features will be input into a higher-level spatial encoder for further processing; it should be noted that the attention module is deployed at all levels of the encoder, so for temporal features of different scales, feature enhancement and fusion will be performed sequentially from low to high; Step 3: Filter the spatio-temporal features using the mask map generated by the temporal decoder; First, input the temporal features at the top layer of the temporal encoder into the temporal decoder for processing. The temporal decoder includes two convolutional layers and a Sigmoid activation function. After processing, the temporal decoder outputs a mask map, which contains the positions and contours of moving objects in the video frames; then use the mask map to weight and filter the spatio-temporal features at the top layer of the spatial decoder, so as to effectively remove the interference information contained in the spatio-temporal features and highlight the significant features; Step 4: Use the GRU module to obtain the long-term spatio-temporal information between video frames; First, input the spatio-temporal features filtered by the mask map into the gated recurrent unit GRU for processing. The GRU module will capture the effective spatio-temporal information contained in consecutive video frames by controlling the opening degrees of the internal reset gate and update gate; then input the processed spatio-temporal features into the spatial decoder for further processing; Step 5: Use the spatial decoder to generate the prediction map; Input the spatio-temporal features processed by the GRU module into the spatial decoder for processing, that is, first reduce its channel dimension to 1 through two convolutional layers, then process it through a Sigmoid activation function, and finally use the bilinear upsampling function to enlarge its size to the size of the original video frame to obtain the final prediction map.

2. The video saliency prediction method based on a two-stream architecture according to claim 1, wherein, The specific method of Step 1 is as follows: First, preprocess the video frames to be predicted and the corresponding optical flow maps, including: reading the pictures according to the picture addresses and converting them to RGB format; resizing them to a size of 224×224; converting the data type to Tensor type; mapping the numerical range from [0,255] to [-1,1]; Then the preprocessed optical flow map X T and the video frame X S are respectively input into the temporal encoder and the spatial encoder of the model for extracting temporal features and spatial features The temporal encoder and the spatial encoder have the same structure and are both built based on the ResNet-50 backbone. Among them, the convolutional blocks T1-T5 of the temporal encoder and the convolutional blocks S1-S5 of the spatial encoder correspond to the convolutional blocks Conv1-Conv5 of the ResNet-50 backbone.

3. The video saliency prediction method based on a two-stream architecture according to claim 2, characterized in that, The specific method of Step 2 is as follows: First, the time feature F extracted by the time encoder is input into the attention module for enhancement processing. The processing of the attention module includes two parts. The first part is to use the channel attention mechanism to weight the feature map. That is, first use global average pooling to change the feature dimension to C×1×1, and then use two consecutive convolutional layers to first compress the feature dimension to C / 16×1×1 and then expand it back to the original dimension C×1×1. Then use the Sigmoid activation function to map the numerical range of the feature to [0,1] to generate a weight vector C. i T Input to the attention module for enhancement processing; the processing of the attention module includes two parts. The first part is to use the channel attention mechanism to weight the feature map, that is, first use global average pooling to change the feature dimension to C×1×1, then use two consecutive convolutional layers to first compress the feature dimension to C / 16×1×1 and then expand it back to the original dimension C×1×1, and then use the Sigmoid activation function to map the numerical range of the feature to [0,1] to generate a weight vector C. i , and finally the weight vector C i and the original time feature are multiplied and a residual connection is used to obtain the channel-weighted feature The above process can be represented by Formula 1; Among them, \(i\in\{1,2,3,4,5\}\) represents the 5 levels of the encoder; GAP represents the global average pooling layer; Conv represents the \(1\times1\) convolutional layer; Relu represents the Relu activation function; Sigmoid represents the Sigmoid activation function; \(\odot\) represents element-wise multiplication; represents element-wise addition; The second part is to use the spatial attention mechanism to weight different regions in the feature map. First, a 1×1 convolutional layer is used to change the dimension of the channel-weighted feature to 1×H×W, and then the Sigmoid activation function is used to generate a weight matrix T i . Finally, the weight matrix T i is multiplied by the original channel-weighted feature , and the residual connection is used to obtain the final temporal enhanced feature The above process can be represented by Equation 2; Then the enhanced temporal features are fused with the spatial features extracted by the spatial encoder to obtain the fused spatio-temporal features which will be input into a higher-level spatial encoder for further processing; the above process can be represented by Equation 3; It should be noted that the attention module is deployed at all levels of the encoder, so for temporal features of different scales, feature enhancement and fusion will be performed sequentially from low to high.

4. A video saliency prediction method based on a two-stream architecture according to claim 3, characterized in that The specific method of Step 3 is as follows: First, input the time features of the top layer of the time encoder into the time decoder for processing. The time decoder will first use two consecutive 1×1 convolutional layers to reduce the feature channel dimension to C / 2 and 1 in sequence, and then use a Sigmoid activation function to process it to obtain a mask map P T ; then use the mask map P T to perform weighted filtering on the spatio-temporal features of the top layer of the spatial decoder to obtain the weighted spatio-temporal features The above process can be represented by Equation 4; 5. A video saliency prediction method based on a two-stream architecture according to claim 4, characterized in that, The specific method of Step 4 is as follows: First, input the spatio-temporal features filtered by the mask image into the gated recurrent unit (GRU) for processing. The GRU module captures the effective spatio-temporal information contained in consecutive video frames by controlling the opening degrees of the internal reset gate and update gate. Its specific execution process can be represented by Equation 5; Among them, R t represents a reset gate; Z t represents an update gate; represents a candidate hidden state; H t represents a new hidden state; X t represents an input feature; H t-1 represents an old hidden state; W xr ,W hr ,W xz ,W hz ,W xh ,W hh represents the weight parameter of the convolutional layer; b r ,b z ,b h represents the bias parameter of the convolutional layer; σ represents the Sigmoid activation function; tanh represents the tanh activation function; Then, the hidden state H containing rich spatio-temporal information output by the GRU module t is input into the spatial decoder for further processing. It should be noted that, in order to use the GRU module, the batch size is set to 6, that is, the model needs to input 6 consecutive video frames of the same video into the spatial encoder each time for processing, and input the corresponding 6 optical flow maps into the temporal encoder. After a series of processes, the model will predict the saliency map corresponding to the last video frame.

6. A video saliency prediction method based on a two-stream architecture according to claim 5, characterized in that The specific method of Step 5 is as follows: The hidden state H output by the GRU module t is input into the spatial decoder for processing to obtain the final prediction map. Specifically, two consecutive 1×1 convolutional layers are first used to sequentially reduce the feature channel dimension to C / 2 and 1, then processed using a Sigmoid activation function, and finally, the bilinear upsampling function is used to enlarge its size to the original video frame size to obtain the final saliency prediction map P S ; the above process can be represented by Equation 6; P S = Up(Sigmoid(Conv2(Relu(Conv1(H t ))))) (6) Among them, P S represents the predicted saliency map; Up represents the bilinear upsampling function.

Images