A video saliency prediction method mimicking human attention mechanism

Abstract

The present invention discloses a video saliency prediction method that mimics the human attention mechanism. The method first randomly extracts a video sequence of n consecutive frames from each video as an input training set. Next, a video saliency prediction model based on the human attention mechanism is constructed. The model comprises an encoder module, a multi-level feature integration module, a hierarchical feature enhancement module, and an output module. Finally, the obtained training set video sequence is input into the video saliency prediction model for training, resulting in prediction results. The present invention achieves improved prediction performance and can better predict the most salient regions in a video.

Description

Technical Field

[0001] The present invention relates to the technical field of video image processing, and in particular to a video saliency prediction method that imitates the human attention mechanism. Background Art

[0002] The human visual attention mechanism enables people to quickly focus on the most interesting areas in their field of view and also helps them process a large amount of visual information. In the field of computer vision, saliency prediction is an important research topic in modeling the visual attention mechanism. Its main research content is to build a saliency prediction model that mimics the visual attention mechanism and determines the saliency of each region in the image.

[0003] In recent years, with the development of high-quality video saliency datasets (DHF1K) and benchmarks, numerous video saliency prediction models have been proposed. Compared to image saliency prediction, video saliency prediction is more challenging because human attention not only focuses on the current field of view but is also influenced by memory cues. Therefore, video saliency prediction models must not only consider spatial information but also effectively utilize temporal cues. Existing VSP models can be roughly divided into three categories: two-stream methods, LSTM-based methods, and 3D convolution-based methods. Two-stream and LSTM are two methods based on 2D convolution-based VSP models. Two-stream methods take an image and an optical flow map as input, integrating the motion information of the optical flow as temporal information with the spatial information extracted from the image to achieve spatiotemporal feature fusion. However, optical flow information only represents the flow of information between two frames and cannot represent longer-term cues. LSTM methods, on the other hand, typically use 2D convolution to obtain spatial information and then feed it into the LSTM to integrate temporal information. However, their limitations lie in their inability to simultaneously utilize both temporal and spatial information for modeling, failing to effectively represent spatiotemporal information.

[0004] Compared to 2D convolution, 3D convolution can simultaneously combine temporal and spatial information. The separate spatiotemporal convolution proposed in S3D can effectively extract the spatiotemporal information of features and reduce the model's computational complexity. Many existing 3D convolution-based VSP models use S3D as an encoder to extract effective spatiotemporal feature information. However, due to the limitations of convolutional neural networks, 3D convolution-based VSP models can only represent local spatiotemporal connections and cannot effectively represent long-term spatiotemporal dependencies. Although some models have attempted to introduce attention mechanisms to compensate for these shortcomings and have achieved good results, the limitations of encoder feature extraction remain unaddressed.

[0005] In recent years, the Transformer has been proposed and has achieved great success in the field of computer vision. While the Transformer uses an attention mechanism to capture long-term dependencies between features, this approach requires significant computational resources. Compared to image and natural language processing tasks, video-related tasks require even more computational resources. VST, a variant of the Transformer, uses a 3D sliding window approach to calculate multi-head attention within a window, reducing computational effort while also extracting spatiotemporal features with long-term dependencies. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the present invention proposes a video saliency prediction method that mimics the human attention mechanism. The method mainly consists of an encoder module, a multi-level feature integration part, and a hierarchical feature enhancement part. By training the neural network, the optimal parameters are obtained to achieve the prediction of video saliency targets.

[0007] In order to solve the above technical problems, the technical solution of the present invention is:

[0008] A video saliency prediction method that mimics the human attention mechanism includes the following steps:

[0009] S1, randomly extract a video sequence of n consecutive frames from each video as the input training set;

[0010] S2. Build a model for video saliency prediction based on mimicking the human attention mechanism, the model comprising an encoder module, a multi-level feature integration part, a hierarchical feature enhancement part, and an output module;

[0011] S3. Input the video sequence of the training set into the model for training, which is specifically divided into:

[0012] The encoder module outputs 4-level encoder features with long-term spatiotemporal dependencies, and then passes the 4-level encoder features through the 3D convolution layer to obtain 4-level spatiotemporal information features;

[0013] The 4-level spatiotemporal information features are integrated into a new 4-level integrated feature through a multi-level feature integration part. The multi-level feature integration unit in the multi-level feature integration part can make the features have richer context information through feature transmission and integration.

[0014] The 4-level integrated features are enhanced by the hierarchical feature enhancement part to obtain the 4-level enhanced features. The hierarchical feature enhancement part is divided into a multi-scale spatiotemporal fusion unit and a channel feature enhancement unit, which act on the first 2-level integrated features and the last 2-level integrated features respectively to highlight the salient areas of the features.

[0015] Fusing 4 levels of enhanced features, through the output module, generates an accurate prediction map;

[0016] Specifically: First, a continuous n-frame video sequence is randomly extracted from each video in the training dataset and input into the encoder module of the model (i.e., VST network). The encoder module includes 4 stages. Stage 1 consists of a 3D convolution, a linear layer, and 2 consecutive VST blocks in series; Stage 2 and Stage 4 consist of a patch merging layer and 2 consecutive VST blocks in series; Stage 3 consists of a patch merging layer and 18 consecutive VST blocks in series. The patch merging layer consists of four resolution downsampling and fully connected layers in series, and the VST block consists of 2 consecutive sliding window attention parts. The encoder extracts the input video sequence into 4 levels of encoder features F with different channel dimensions and resolution sizes. i (i=1,2,3,4). Then, each level of the obtained encoder features is input into a 3D convolution layer consisting of a 3D convolution, a GN layer, and a ReLU activation function to compress the number of channels and the time dimension of the encoder features into the same size to obtain a 4-level spatiotemporal information feature.

[0017] Then, the 4-level spatiotemporal information features are gradually integrated into the spatiotemporal information features of each level in the multi-level feature integration part by encoding and decoding. The multi-level feature integration unit cascades the features in the time dimension, and contains three parallel branches consisting of 3D convolutions of convolution kernels of different time scales and 3D convolutions of 2×3×3 convolution kernels. The branch features are aggregated by element-wise addition and integrated by a 3D convolution to obtain the integrated features. The input of the multi-level feature integration unit is the corresponding hierarchical spatiotemporal features and the output of the multi-level feature integration unit of the previous level, where

[0018] Secondly, the 4-level integrated features are input into the hierarchical feature enhancement part. (in After a single upsampling step, the feature vector is input to the multi-scale spatiotemporal fusion feature unit. The multi-scale spatiotemporal fusion feature unit consists of three branches composed of 3D convolutions with different convolution kernel sizes, plus a fourth branch composed of a maximum pooling layer and 3D convolution. All branch features are integrated by element-wise addition, and the output is enhanced by integrating the spatiotemporal scales through two 3D convolutions. Post-2 level integration features (in After one upsampling, the feature is input to the channel feature enhancement unit. The channel feature enhancement unit first integrates the features through two 3D convolutions and then splits the features into four proportional features along the channel. The proportional features are enhanced step by step through element addition and 3D convolution, and the enhanced proportional features are aggregated again through channel cascade and 3D convolution. The enhanced features are then output through dense connection, 3D convolution and upsampling.

[0019] Finally, the four enhanced features are aggregated in a channel-wise cascade and fed into the output module, which consists of four 3D convolutions, followed by a sigmoid activation function, to output the predicted image P.

[0020] The present invention has the following characteristics and beneficial effects:

[0021] By adopting the above technical solution and combining it with the encoder to extract more comprehensive encoder features, the input video sequence can be comprehensively modeled with richer spatiotemporal information. The context information is transferred to the spatiotemporal information features through the multi-level feature integration unit combined in the encoding-decoding manner in the multi-level feature integration part. At the same time, taking into account the information differences between different integrated features, the hierarchical feature enhancement part adopts a multi-scale spatiotemporal fusion unit and a channel feature enhancement unit to enhance the integrated features. It acts on the first 2 levels of integrated features and the last 2 levels of integrated features respectively to highlight the significant areas of the hierarchical features while further fusing the hierarchical spatiotemporal features to generate 4-level enhanced features. Finally, these 4-level enhanced features are fused and the final prediction map is generated through the output module. At the same time, compared with the existing video saliency prediction model, the present invention has a better prediction effect and can better predict the most significant areas in the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention.

[0023] Figure 1 The overall network diagram of the proposed model;

[0024] Figure 2 It is a multi-level feature integration unit;

[0025] Figure 3(a) shows the multi-scale spatiotemporal fusion unit;

[0026] Figure 3(b) shows the channel feature enhancement unit. DETAILED DESCRIPTION

[0027] The present invention provides a video saliency prediction method that imitates the human attention mechanism. Figure 1 As shown, the following steps are included:

[0028] The training input video sequence is obtained, and 32 consecutive frames of images are randomly selected from each video to form a video sequence. The present invention mainly uses the DHF1K dataset (the most extensive and high-quality dataset in this field) as the main dataset for model training, of which the first 600 videos are used as the training set for training, and the last 100 videos are used as the validation set to test the effect of the trained model and find the optimal model parameter weights.

[0029] Existing common spatiotemporal feature encoders include the 3D convolution-based S3D encoder (a backbone network based on separable spatiotemporal 3D convolutions) and the Transformer-based VST encoder. Limited by the size of the convolution kernel, 3D convolutions only model local spatiotemporal features and fail to comprehensively extract spatiotemporal information. Furthermore, for video saliency prediction tasks, they prioritize global spatiotemporal correlations over spatial details. In contrast, the VST encoder, pre-trained on the Kinetics-400 action recognition dataset, employs a sliding window attention mechanism to comprehensively model spatiotemporal information, fully capturing the inherent spatiotemporal relationships within the video. The VST encoder consists of four stages. The 3D convolution and linear layers in stage 1 compress the spatial and temporal scales of the input video, reducing the computational cost of subsequent stages. The patch merging layers in stages 2-4 halve the spatial resolution of the features and double the number of channels. The VST block utilizes a windowed attention mechanism to calculate spatiotemporal correlations within the window size, capturing spatiotemporal relationships between windows through a sliding window approach.

[0030] A video segment T×H×W (T=32, H=224, W=384) with T consecutive frames is input into the encoder module (i.e., the Video Swim Transformer backbone network pre-trained on the Kinetics-400 action recognition dataset) to obtain a 4-level encoder feature Then the 4-level encoder features are put into the 3D convolution layer, the channel dimension of the features are compressed to 192, and the time (T) dimension is compressed by half to obtain the spatiotemporal information features. In order to be more consistent with the human attention mechanism and pay more attention to global information, we will High-level semantic features are encoded and decoded in combination with multi-scale spatiotemporal fusion units to gradually transform spatiotemporal information features. Pass more context information to get integrated features in The multi-scale spatiotemporal fusion unit (such as Figure 2As shown in the figure, the high-level features are first upsampled and then cascaded with the next-level spatiotemporal information features through the time dimension. Then, 3D convolutions with convolution kernels of different time scales and 3D convolutions with a convolution kernel of (2×3×3) are combined into three different branches. The spatiotemporal information of different time steps is obtained to fully consider the influence of time clues on spatiotemporal information. Then, the branch features of different scales are obtained. The most significant area is highlighted by adding. It can be specifically expressed as:

[0031]

[0032] in{;} T Indicates cascade features in the time dimension; Conv t×1×1 Represents 3D convolution of convolution kernels of different time scales; Conv 2×3×3 Represents a 3D convolution with a (2×3×3) convolution kernel and a (2×1×1) step size, which is used to restore the time dimension of the original spatiotemporal integration feature; Represents the element addition method.

[0033] For the hierarchical feature enhancement part, different feature enhancement modules are deployed for different hierarchical features taking into account the differences between the integrated features at different levels, so that it can effectively fuse spatiotemporal information and highlight the salient areas. As shown in Figure 3, a multi-scale spatiotemporal fusion unit (shown in Figure 3(a)) and a channel feature enhancement unit (shown in Figure 3(b)) are used to enhance the hierarchical features. and In the low-level integrated features, the channel feature enhancement unit is used to and Among the advanced integration features, and All are first upsampled once and then input into their respective units. as input to their respective enhancement units.

[0034] Regarding the multi-scale spatiotemporal fusion unit, it adopts a method similar to ASPP, using 3D convolution with multiple convolution kernels of different scales and a maximum pooling layer to form a 4-branch structure, extracting relevant spatiotemporal information from different time steps and achieving efficient fusion of spatiotemporal information. Specifically, the feature The input is fed into three branches, including 3D convolutions with kernels of different scales, and a fourth branch, including a maximum pooling layer and a 3D convolution layer, to extract multi-scale information and fuse the spatiotemporal dimensions. We then fuse the information element-wise and further integrate the temporal features through 3D convolution. The multi-scale spatiotemporal fusion unit process can be defined as follows:

[0035]

[0036] Among them, Conv 2×3×3 3D convolution with (2×3×3) convolution kernel and (2×1×1) step size is used to compress the time channel to achieve effective interaction of spatiotemporal information; MaxPool 2×3×3 The maximum pooling layer with (2×3×3) convolution kernel and (2×1×1) step size is used to compress the time dimension and highlight the salient areas. Conv 1×3×3 、 Conv 5×1×1 、 Conv 3×1×1 and Conv 2×3×3 The channel dimension has been changed to 128.

[0037] Regarding the channel feature enhancement unit, it highlights the significant area by splitting and integrating the channel dimension, and finds the significant location information of the entire spatiotemporal feature in a similar intersection method. Specifically, first, a 3D convolution with a (2×3×3) convolution kernel and a (2×1×1) step size is used to compress the time dimension. This not only reduces computing resources but also effectively prevents information redundancy caused by long time series. The features are then first integrated through a 3D convolution with a (1×1×1) convolution kernel to integrate the channels, and then split into four proportional features in equal proportions according to the channel dimension. The split features are operated step by step by combining element addition and 3D convolution to highlight the common significant areas in the features and to make the semantic position information of the features clearer. The obtained features are then re-integrated by channel cascade and the original features are residually linked to ensure that the most significant areas of the features are highlighted while also preserving the semantic information related to the original features. Finally, upsampling and 3D convolution are used to ensure that the output feature size remains consistent. The channel feature enhancement unit process can be defined as follows:

[0038]

[0039] in{;} C Indicates cascading features in the channel dimension; split indicates the geometric split operation according to the channel; Conv 2×3×3 The 3D convolution with (2×3×3) convolution kernel and (2×1×1) step size not only compresses the time dimension to 4 but also compresses the channel to 128; Conv 1×1×1 and Conv 1×3×3 They only integrate the features but do not change the feature size. Get 4 levels of enhanced features

[0040] Finally, the four enhanced features are aggregated in a channel cascade manner and then input into the output module to output the final prediction map. The output module consists of a 2×3×3 convolution kernel 3D convolution, a 1×1×1 convolution kernel 3D convolution, two 1×3×3 convolution kernel 3D convolution, two upsampling components and a sigmoid activation function. The specific formula is:

[0041]

[0042] The 3D convolution operation Conv in P 2×3×3 It is used to compress the time dimension to 1 while reducing the number of channels. The remaining convolution operations gradually reduce the number of channel dimensions to 1.

[0043] During the training process, a combination of KL loss and SSIM loss is used: L(P,G)=L KL (P,G)+L CC (P, G) The predicted image P and the true image G are added to the loss calculation for supervision. The last frame of the input video is used as the prediction target and compared with its corresponding GT image. In general, the final predicted image P result is supervised by adjusting the size of the GT image to the size of the predicted image for gradient calculation and backpropagation, and observing the training loss value and whether it converges to judge the network training process. On the validation set, the result is passed through the resize function to make it have the same size as the GT image, and several quantitative indicators are calculated to find the optimal model parameters. In step S6, the size of the input data is uniformly adjusted to 3×32×224×384, where 32 represents the number of input frames T, 224 represents the image height H, and 384 represents the image width W. The batch size is set to 1, and the Adam optimizer is used to update the model parameters during training, and the initial learning rate is set to 1e-5.

[0044] Table 1 Experimental results

[0045] Model CC NSS SIM AUC-J ViNet 0.521 2.957 0.388 TSFP-Net 0.529 3.009 0.398 0.919 STSANet 0.539 3.082 0.411 0.920 The present invention 0.5507 3.1594 0.4306 0.9228

[0046] Compared to existing video saliency prediction models, as shown in Table 1, our proposed model achieves excellent experimental results by comparing relevant metrics (CC, NSS, SIM, AUC-J) with state-of-the-art VSP models (ViNet, TSFP-Net, and STSANet) on the DHF1K100 validation set. By efficiently utilizing the spatiotemporal features extracted by the VST encoder, it significantly improves prediction accuracy and can be widely applied in many fields, such as video compression, video subtitling, video segmentation, and video coding.

Claims

1. A video saliency prediction method that mimics the human attention mechanism, characterized by: The steps include: S1, randomly extract a video sequence of n consecutive frames from each video as the input training set; S2. Constructing a video saliency prediction model based on mimicking the human attention mechanism, the model includes an encoder module, a multi-level feature integration part, a hierarchical feature enhancement part, and an output module; S3. Input the obtained training set video sequence into the video saliency prediction model for training to obtain the prediction results. The specific process is as follows: S3.

1. Output the four-level encoder features with spatiotemporal dependencies through the encoder module, and then pass the four-level encoder features through the 3D convolution layer to obtain the four-level spatiotemporal information features; S3.

2. The 4-level spatiotemporal information features are integrated through the multi-level feature integration part to obtain the 4-level integrated features. Specifically, the multi-level feature integration unit cascades the features in the time dimension, including 3 parallel branches consisting of a 3D convolution with a convolution kernel of different time scales and a 3D convolution with a 2×3×3 convolution kernel. The branch features are aggregated in an element-wise addition manner and integrated with a 3D convolution to obtain the integrated features. The input of the multi-level feature integration unit is the corresponding hierarchical spatiotemporal features and the output of the multi-level feature integration unit of the previous level, where S3.

3. The four-level integrated features are processed through the hierarchical feature enhancement part to obtain four-level enhanced features. The hierarchical feature enhancement part is divided into a multi-scale spatiotemporal fusion unit and a channel feature enhancement unit, which act on the first two levels of integrated features and the last two levels of integrated features respectively. The specific process is as follows: The 4-level integrated features are input into the hierarchical feature enhancement part, and the first 2-level integrated features in After an upsampling, it is input into the multi-scale spatiotemporal fusion feature unit; the multi-scale spatiotemporal fusion feature unit contains three branches composed of 3D convolutions with different convolution kernel sizes, plus a fourth branch composed of a maximum pooling layer and 3D convolution. All branch features are integrated in an element-wise manner, and the integrated output is enhanced by two 3D convolutions on the spatiotemporal scale. Post-2 level integration features in After one upsampling, the image is input to the channel feature enhancement unit. The channel feature enhancement unit first integrates the features through two 3D convolutions, then splits the features into four geometric features along the channel. The geometric features are enhanced step by step through element addition and 3D convolution, and the enhanced geometric features are aggregated again by channel cascade and 3D convolution. Then enhance the features through dense connection, 3D convolution and upsampling output S3.4, fuse the 4-level enhanced features and output the prediction map through the output module.

2. The video saliency prediction method imitating the human attention mechanism according to claim 1, characterized in that: Step S3.1 specifically includes: randomly extracting a continuous n-frame video sequence from each video and inputting it into the encoder module, i.e., the VST network; The encoder module consists of four stages: stage 1 consists of a 3D convolution, a linear layer, and two consecutive VST blocks in series; stages 2 and 4 consist of a patch merging layer and two consecutive VST blocks in series; stage 3 consists of a patch merging layer and 18 consecutive VST blocks in series; The patch merging layer consists of four resolution downsampling and fully connected layers in series, and the VST block consists of two consecutive sliding window attention parts; The encoder extracts 4 levels of encoder features F with different channel dimensions and resolution sizes from the input video sequence i , i = 1, 2, 3, 4, and then each level of the obtained encoder features is input into a 3D convolution layer consisting of a 3D convolution, a GN layer, and a ReLU activation function, and the number of channels and the time dimension of the encoder features are compressed to the same size to obtain a 4-level spatiotemporal information feature 3. The video saliency prediction method imitating the human attention mechanism according to claim 2, characterized in that: Step S3.4 is as follows: the output module includes 4 3D convolutions, which pass the sigmoid activation function and output the prediction map P.

Images