A lightweight video saliency prediction method based on spatiotemporal octave convolution modules

Abstract

This invention discloses a lightweight video saliency prediction method based on a spatiotemporal octave band convolution module, primarily addressing the problems of excessively large 3D deep convolutional networks, which easily lead to resource constraints and overfitting in existing technologies. The implementation scheme involves: acquiring and preprocessing training and testing datasets separately; constructing a lightweight spatiotemporal convolutional unit and using it to build a multi-scale feature extraction chain; constructing a feature fusion module and connecting it to the multi-scale feature chain to form a lightweight multi-scale video saliency prediction model; iteratively training the video saliency prediction model using the preprocessed training data through the Adam optimization method; and inputting the preprocessed testing data into the trained lightweight multi-scale video saliency prediction model to obtain the video saliency prediction result. This invention reduces the size of the video saliency prediction model and improves the accuracy of the video saliency prediction results, which can be used to help machines understand human visual focus in videos.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a lightweight video saliency prediction method, which can be used to help machines understand human visual focus on videos, thereby improving the efficiency and accuracy of information transmission. Background Technology

[0002] Video saliency prediction is a computer vision technique that analyzes video content to identify and predict which areas or objects viewers are most likely to focus on. This technology has important applications in various fields, such as automatically highlighting key content in video editing, determining optimal display locations in advertising, and quickly locating abnormal behavior in security monitoring. In short, video saliency prediction helps machines understand human visual focus, thereby improving the efficiency and accuracy of information delivery. Early video saliency prediction techniques have evolved from traditional methods relying on handcrafted features and simple algorithms. For example, Gao et al. added additional motion information to their image saliency detection model; Guo et al. used the phase spectrum of Fourier transform to calculate dynamic saliency; and Rahtu et al. utilized statistical models and the contrast of local features to calculate video saliency.

[0003] In recent years, this technology has evolved from model-based methods to breakthroughs in deep learning, especially convolutional neural networks, and further to the introduction of multimodal fusion and attention mechanisms, continuously optimizing models to improve real-time performance and interpretability. For video saliency prediction models, researchers have proposed benchmarks for video saliency prediction, such as DHF1K and LEDOV. By treating video saliency prediction as a regression problem, researchers can efficiently train deep learning models in an end-to-end manner.

[0004] With video processing technology becoming increasingly popular, researchers have leveraged the powerful temporal modeling capabilities of Long Short-Term Memory (LSTM) networks to apply them to video saliency prediction, aiming to capture dynamic attention patterns in videos. Jiang et al. combined a two-layer LSTM neural network with a static appearance feature extraction network; Wang et al. proposed a dynamic human eye attention detection model based on ConvLSTM. However, LSTM networks typically require a large number of parameters to store long-term memory and learn complex temporal patterns, and require forward and backward propagation step by step, which becomes very time-consuming when processing long sequences.

[0005] Therefore, the approach of gradually aggregating spatiotemporal cues using 3D fully convolutional models has attracted researchers' attention and exploration. These methods primarily utilize local details and global features, attention information, and edge information to improve state-of-the-art saliency detection performance. For example, Min et al. proposed the TASED-Net model, which applies 3D fully convolution to a standard encoder-decoder architecture for video saliency prediction. Giovanni et al. proposed the HD2S model, which uses a 3D fully convolutional architecture and multi-feature learning to generate saliency prediction results.

[0006] Patent document with application number CN201910916306.4 discloses a video saliency region detection method and system based on deep C3D features. It uses a deep 3DResNext-101 network pre-trained on ImageNet to extract features from video sequences, obtain spatiotemporal feature convolutional feature blocks, then processes them with a softmax function to obtain spatiotemporal three-dimensional context features, and then uses the ConvLSTM algorithm to update and obtain saliency map sequences.

[0007] The aforementioned models and methods all utilize 3D deep convolutional networks. To achieve better saliency prediction performance, the network size needs to be increased, requiring the introduction of additional parameters. Therefore, they are not suitable for resource-constrained environments or situations requiring real-time processing. Furthermore, the large size of 3D deep convolutional networks increases the risk of overfitting, especially when the amount of data is limited, potentially leading to overfitting.

[0008] To achieve lightweight networks, researchers have made a series of efforts. Howard et al. proposed MobileNets, which uses inverted blocks and SE attention modules to build lightweight deep neural networks. Ma et al. proposed ShuffleNetV2, which uses channel shuffling to change the channel order in convolutional neural networks, reducing the limitations of parameter sharing. Tan et al. proposed EfficientNet, which uses simple and efficient composite coefficients to uniformly scale all dimensions.

[0009] However, the above methods for achieving network lightweighting are all designed for classification tasks, focusing on the global features of images. Therefore, they only target the lightweighting of shallow networks and use a large number of downsampling strategies to reduce multiply-accumulate operations, resulting in low accuracy and incomplete prediction results. Summary of the Invention

[0010] The purpose of this invention is to address the shortcomings of the prior art by proposing a lightweight video saliency prediction method based on a spatiotemporal octave convolution module. This method aims to solve the resource constraints and overfitting problems caused by the excessive size of 3D deep convolutional networks and improve the accuracy of saliency prediction results.

[0011] To achieve the above objectives, the technical solution of the present invention includes the following steps:

[0012] (1) Obtain the training and testing datasets:

[0013] 1a) Obtain the DHF1K dataset and UCF-Sports dataset from public websites, and take M videos from the DHF1K dataset as the training dataset and N videos from the UCF-Sports dataset as the test dataset, where M is greater than or equal to 600 and N is greater than or equal to 45.

[0014] 1b) Read the training and testing datasets according to the data path, perform frame separation on all videos to obtain video data frames, and further create a list of frame names and annotation names for each video. Extract training segments from the list of frame names and annotation names according to the target segment length parameter. Training annotation fragments With test fragments ;

[0015] (2) Dataset preprocessing:

[0016] 2a) Adjust, flip, and normalize the video image frames and corresponding annotation image frames of the training data according to the target size, then convert them into tensor data to obtain the corresponding training video image frame tensor data. and the corresponding training video annotation frame tensor data ;

[0017] 2b) Adjust and normalize the video image frames of the test data sequentially according to the target size, then convert them into tensor data to finally obtain the corresponding test video image frame tensor data. ;

[0018] (3) Construct a lightweight multi-scale video saliency prediction model:

[0019] 3a) Constructing primarily 3D octave-length convolutional layers Spatiotemporal multiscale octave convolutional layers Lightweight spatiotemporal convolutional units composed of cascades ;

[0020] 3b) A multi-scale feature extraction chain consisting of multiple feature extraction modules with different structures and multiple cascaded feature separation layers is used to extract features at different scales. Each module consists of a different number of lightweight spatiotemporal convolutional units. Connecting components;

[0021] 3c) Construct multiple parallel dilated convolutional layers and spatiotemporal multi-scale octave band convolutional layers. A feature fusion module is composed of a pointwise convolutional module and an upsampling layer connected in sequence;

[0022] 3d) Concatenate the multi-scale feature extraction chain constructed in step 3b) and the feature fusion module constructed in step 3c) to form a lightweight multi-scale video saliency prediction model, and use KL divergence as the loss function of the model. ;

[0023] (4) Input the training video image frame tensor dataset into the lightweight multi-scale video saliency prediction model, and train it using the Adam optimization method to obtain the updated lightweight multi-scale video saliency prediction model.

[0024] (5) Transfer the tensor data of the test video image frames The input is fed into the trained lightweight multi-scale video saliency prediction model to obtain the saliency prediction results.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] First, this invention incorporates a spatiotemporal multi-scale octave band convolution module in its lightweight multi-scale video saliency prediction model. It uses parameters with the same size as traditional 3D convolution modules, and additionally realizes the decomposition and fusion of features of any number of different dimensions, thus improving accuracy, reducing model size, saving computational costs, and reducing spatial redundancy.

[0027] Second, this invention incorporates lightweight spatiotemporal convolutional units in its lightweight multi-scale video saliency prediction model. It is used to construct a multi-scale feature extraction chain. By extracting abstract image features step by step, the model can not only establish a gradual understanding of the semantic information of the image, but also capture the micro and macro features of the video at the same time, thereby improving the accuracy of prediction. Attached Figure Description

[0028] Figure 1 It is a flowchart of the invention's implementation process;

[0029] Figure 2 This is a diagram of the 3D octave band convolutional layer structure in this invention;

[0030] Figure 3 This is a diagram of the spatiotemporal multi-scale octave band convolutional layer structure in this invention;

[0031] Figure 4 This is a structural diagram of the lightweight spatiotemporal convolutional unit in this invention;

[0032] Figure 5 This is a structural diagram of the lightweight multi-scale video saliency prediction model in this invention. Detailed Implementation

[0033] The embodiments and effects of the present invention will be further described below with reference to the accompanying drawings.

[0034] Reference Figure 1 The implementation steps of this embodiment are as follows:

[0035] Step 1: Obtain the training and testing datasets.

[0036] 1.1) Obtain the DHF1K dataset and UCF-Sports dataset from public websites, and take M videos from the DHF1K dataset as the training dataset and N videos from the UCF-Sports dataset as the test dataset. In this example, M=600 and N=45.

[0037] 1.2) Store the training dataset and the test dataset in the training data folder and the test data folder respectively within the main dataset folder;

[0038] Step 2: Preprocess the data.

[0039] 2.1) Read all data according to the total dataset folder path, perform frame separation on all videos, and further create a list of frame names and annotation names for each video:

[0040] 2.1.1) Perform frame extraction on all videos in the dataset, separating individual image frames from the video sequence frame by frame, and save them as independent image files in a folder named after the video.

[0041] 2.1.2) Creation , , and Four empty training set lists, use the first training set list. Store the training video names using the second training set list. Store the frame name of each training video using the third training set list. Store the frame names of each training video annotation using the fourth training set list. Store the number of frames for each training video;

[0042] 2.1.3) Creation , and Three empty test set lists, use the first test set list. Store the test video names using the second test set list. Store the frame name of each test video using the third test set list. Store the number of frames for each test video;

[0043] 2.2) According to the length of the target segment Extract training segments from the list of frame names and annotation names. Training annotation fragments and test fragments :

[0044] 2.2.1) Randomly select a training start frame index. Starting from this index, from the second training set list and the third training set list Extract training segments of length T from each. and the corresponding training annotation fragments ;

[0045] 2.2.2) Set the test start frame index Starting from this initial frame index, the second test set list... Extract a test segment of length T. In this example ;

[0046] 2.3) Based on the target size The video image frames and their corresponding annotation image frames in the training data are adjusted, flipped, and normalized sequentially:

[0047] 2.3.1) Training segments The video image frames are converted from BGR format to RGB format, and the video image frame size is adjusted to match the target size. Then, the video frames are randomly flipped with a probability of 0.5, and then normalized to reduce the pixel value range of each frame from the original image pixel value. Convert to normalized image pixel values The normalization formula is as follows:

[0048] ;

[0049] 2.3.2) Create an empty list and use The list stores the normalized training video image frames;

[0050] 2.3.3) Regarding The list is concatenated along the last axis to obtain a three-dimensional array, which is then converted into a three-dimensional PyTorch tensor to obtain the training video image frame tensor data. ;

[0051] 2.3.4) Training Annotated Fragments Perform the same adjustments, flips, and normalization as in step 2.3.1) to create an empty list. ,use The list stores the normalized training annotated video and image frames.

[0052] 2.3.5) will Each video annotation frame in the list is converted into a two-dimensional PyTorch tensor to obtain the training video annotation frame tensor data. In this example ;

[0053] 2.4) Based on the target size The video image frames of the test data were adjusted and normalized sequentially:

[0054] 2.4.1) For the test segment The video image frames are converted from BGR format to RGB format, and the video image frame size is adjusted to match the target size. Then, normalize it to reduce the range of pixel values ​​for each frame from the original image pixel values. Convert to normalized image pixel values :

[0055] ;

[0056] 2.4.2) Creating an empty list and use The list stores the normalized test video image frames;

[0057] 2.4.3) Regarding The list is concatenated along the last axis to obtain a three-dimensional array, which is then converted into a three-dimensional PyTorch tensor to obtain the tensor data of the test video image frames. In this example, the settings are as follows: .

[0058] Step 3: Construct 3D octave-length convolutional layers Spatiotemporal multiscale octave convolutional layers Lightweight spatiotemporal convolutional units ,like Figure 4 As shown.

[0059] The 3D octave convolutional layer like Figure 2 As shown, it includes four 3D convolutional layers, one 3D pooling layer, and one trilinear interpolation upsampling layer. The 3D pooling layer and the 3D convolutional layer are connected in series to form a first series module. The trilinear interpolation upsampling layer and the 3D convolutional layer are connected in series to form a second series module. These two series modules are then connected in parallel with the remaining two 3D convolutional layers to form a 3D octave band convolutional layer. It is used to process two input features and The processing yields two output features. and input features After parallel processing by the first concatenated module and a 3D convolutional layer, the output features are obtained by weighted summation. Input features The output features are obtained by weighted summation after parallel processing by a second concatenated module and a three-dimensional convolutional layer. ;

[0060] The spatiotemporal octave convolutional layer ,like Figure 3 As shown, it includes n 3D convolutional layers, m-1 3D pooling layers, and nm trilinear interpolation upsampling layers. The m-1 3D pooling layers are concatenated with each of the m-1 3D convolutional layers, and the nm trilinear interpolation upsampling layers are concatenated with each of the nm 3D convolutional layers. These n-1 concatenated modules are then connected in parallel with the remaining 3D convolutional layer, where n ≥ 1, m ≥ 1, and n ≥ m. This spatiotemporal octave convolutional layer... Used to process n input features to obtain an output feature. The output features are obtained by weighted summation after all input features are processed in parallel by n-1 concatenated modules and a three-dimensional convolutional layer. .

[0061] Step 4: Construct a lightweight multi-scale video saliency prediction model.

[0062] Reference Figure 5 The implementation of this step is as follows:

[0063] 4.1) Construct a multi-scale feature extraction chain consisting of multiple feature extraction modules with different structures and multiple cascaded feature separation layers:

[0064] The feature separation layer consists of a parallel three-dimensional convolutional layer with a 1×1×1 kernel and a stride of 1, and a three-dimensional pooling layer with a scaling factor of 2. It is used to separate the input... According to the scale decomposition ratio Decomposed into two intermediate features of different scales and In this example ;

[0065] The multiple feature extraction modules with different structures are composed of different numbers of lightweight spatiotemporal convolutional units constructed in step 3. The module consists of lightweight spatiotemporal convolutional units connected in series in each feature extraction module. The number is determined by the module order i, that is, each module consists of 2×i lightweight spatiotemporal convolutional units. In this example, four feature extraction modules are constructed in a series. The first module, Stage 1, the second module, Stage 2, the third module, and the fourth module, Stage 4, are composed of 2, 4, 6, and 8 lightweight spatiotemporal convolutional units connected in series, respectively. All modules are connected by a spatiotemporal multi-scale octave convolutional layer with a kernel size of 1×1×1. The feature extraction modules are connected in series with a feature separation layer. The structures of multiple feature extraction modules with different structures are shown in Table 1. The interior contains lightweight spatiotemporal convolutional units. The structure, This represents a 3D octave convolutional layer. This represents a spatiotemporal multi-scale octave band convolutional layer;

[0066] Table 1. Structure of multiple feature extraction modules with different structures

[0067]

[0068] 4.2) Construct multiple parallel dilated convolutional layers and spatiotemporal multi-scale octave band convolutional layers. A feature fusion module consisting of a pointwise convolutional module and an upsampling layer cascaded in sequence:

[0069] The dilated convolutional layer and the spatiotemporal multi-scale octave band convolutional layer It consists of M parallel dilated convolutional layers with kernel size 3 and dilation coefficients of [1, 2, 4, 8, 16] and a spatiotemporal multi-scale octave convolutional layer with kernel size 1×1×1. The components are connected in series; in this example, M=3.

[0070] The pointwise convolution module consists of N pointwise convolutional layers connected in series. The kernel size of each convolutional layer is 3×1×1 and the stride is 2×1×1. In this example, N=2.

[0071] The upsampling layer includes a bilinear interpolation upsampling layer with an amplification factor of s, where s=2 in this example;

[0072] 4.3) Concatenate the multi-scale feature chain constructed in step 4.1) with the feature fusion module constructed in step 4.2) to form a lightweight multi-scale video saliency prediction model.

[0073] Step 5: Train the lightweight multi-scale video saliency prediction model using training data.

[0074] 5.1) Process the training video frame tensors with a scale of 3×16×128×192 obtained in step 2. The input is fed into a multi-scale feature extraction chain, which passes through four stages of feature extraction modules to obtain four spatiotemporal features, namely the first spatiotemporal feature with a scale of 40×16×128×192 extracted in the first stage. The second stage extracts a second spatiotemporal feature with a scale of 80×8×64×96. The third stage extracts the third spatiotemporal features from an image with a scale of 160×4×32×48. The fourth stage extracts fourth spatiotemporal features with a scale of 240×2×16×24. ;

[0075] 5.2) The second spatiotemporal features obtained in step 5.1) Third spatiotemporal characteristics and fourth spacetime characteristics The three spatiotemporal features are input into the feature fusion module, where they undergo parallel dilated convolutions with invariant scale. Then, the three spatiotemporal features are input into a spatiotemporal octave convolutional layer. Feature fusion was performed to obtain spatiotemporal features with dimensions of 480×8×64×96. Then, it is input into the pointwise convolution module to remove its time dimension, resulting in a 480×64×96 two-dimensional feature tensor; then, through pointwise convolution and upsampling, a 1×128×192 saliency prediction tensor is obtained. ;

[0076] 5.3) Iteratively update the lightweight multi-scale video saliency prediction model using the Adam optimization method:

[0077] 5.3.1) Initialize the first-order moment estimate Second-order moment estimation Initial learning rate The number of iterations is T=2500;

[0078] 5.3.2) Using KL divergence as the loss function for model training, as follows:

[0079]

[0080] Where S represents the significance prediction tensor obtained in step 5.2). A represents the video annotation image frame tensor obtained in step 2. ;

[0081] 5.3.3) Using the loss function Calculate the current network parameters gradient ;

[0082] 5.3.4) Utilize the current gradient and the current first moment estimate Update the unbiased first moment estimate: The first-order hyperparameters ;

[0083] 5.3.5) Utilize the current gradient and the current second moment estimate Update the standard unbiased second moment estimate: The second-order hyperparameters ;

[0084] 5.3.6) Using the updated unbiased first moment estimate and standard unbiased second moment estimation Calculate the updated network parameters :

[0085]

[0086] in = ;

[0087] 5.3.7) Repeat steps 5.3.2) to 5.3.6) until the number of iterations reaches 2500, and obtain the iteratively updated lightweight multi-scale video saliency prediction model.

[0088] Step 6: Transfer the tensor data of the test video image frames. The results are input into the lightweight multi-scale video saliency prediction model trained in step 5 to obtain the saliency prediction results.

[0089] The invention will be further illustrated below with simulation experiments:

[0090] 1. Experimental conditions

[0091] The computer used in the simulation experiment had a 13th Gen Intel(R) Core(TM) i9-13900KF×32 processor, 96GB of RAM, and an NVIDIA GeForce RTX 3090 24 GB graphics card.

[0092] The software platform for the simulation experiment was Ubuntu 22.04.3 (LTS) operating system, Python 3.6, and PyTorch 1.10.3.

[0093] 2. Experiment Content

[0094] Experiment 1: Using the present invention and existing methods ACLNet, SalEMA, TASED-Net and HD2S, video saliency prediction was performed. Five different evaluation indicators were calculated to objectively evaluate the saliency prediction results. The results are shown in Table 2.

[0095] Table 2 Objective Evaluation of Prediction Results of the Invention and Other Existing Methods

[0096]

[0097] In Table 2, NSS represents the significance index of the normalized scan path, AUC-J represents the area under the curve - exact fit, s-AUC represents the area under the curve - exact fit after reducing center sensitivity, SIM represents similarity, and CC represents the correlation coefficient.

[0098] As shown in Table 2, the results of this invention outperform all other methods in terms of AUC-J, s-AUC, SIM, and CC metrics, and rank second in terms of NSS metric, indicating that this invention can effectively improve the accuracy of video saliency prediction results. The model size of this invention is only 11.4 MB, which is one-fifth to one-thirtieth of other methods, making it the smallest among all models. Moreover, this model outperforms other methods in inference speed, processing one frame of image in just 0.01 seconds, and can achieve fast inference while maintaining high performance.

[0099] Experiment 2: Using the method of this invention, along with existing SalEMA, TASED-Net, and HD2S methods, to predict the saliency of a video in the UCF-Sports dataset.

[0100] The first row contains three consecutive video frames selected from the image.

[0101] The second line is the corresponding video annotation image frame.

[0102] The third row shows the prediction results of the SalEMA method for video frame images.

[0103] The fourth row shows the prediction results of the TASED-Net method for video frame images.

[0104] The fifth line shows the prediction results of the HD2S method for video frame images.

[0105] The sixth line shows the prediction results of the video frame image by the method of the present invention.

[0106] The saliency prediction region of this invention is closer to the saliency region shown in the video annotation image frame than the saliency prediction regions of other methods, indicating that this invention can effectively improve the accuracy of video saliency prediction results.

[0107] The existing methods used in the experiment are from the following sources:

[0108] ACLNet is a classic video saliency prediction method proposed by Wang et al., which uses ConvLSTM networks to obtain temporal information.

[0109] SalEMA is a video saliency prediction method proposed by LINARDOS P et al., which modifies the convolution based on ConvLSTM.

[0110] TASED-Net is a method for video saliency prediction using a 3D fully convolutional network architecture proposed by MIN K et al.

[0111] HD2S is a video saliency prediction method proposed by G. Bellitto et al., which uses a multi-feature learning strategy based on a 3D fully convolutional network.

[0112] The above description is merely a specific example of the present invention and does not constitute any limitation on the present invention. Obviously, those skilled in the art, after understanding the content and principles of the present invention, may make various modifications and changes in form and detail without departing from the principles and structure of the present invention. For example, in the training of the lightweight multi-scale video saliency prediction model in the present invention, in addition to the Adam optimization method used in this example, gradient descent can also be used; in the increase of feature resolution in the present invention, in addition to the bilinear interpolation upsampling layer used in this example, nearest neighbor interpolation upsampling layer can also be used; in the training dataset in the present invention, in addition to the DHF1K dataset used in this example, the Holwood2 dataset can also be used. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

[0113] It should be noted that the step numbers in the specification and claims of this invention are only for the purpose of clearly describing the embodiments of this invention and facilitating understanding, and their order is not limited.

Claims

1. A lightweight video saliency prediction method based on a spatiotemporal octave band convolution module, characterized in that, Includes the following steps: (1) Obtain the training and testing datasets: 1a) Obtain the DHF1K dataset and UCF-Sports dataset from public websites, and take M videos from the DHF1K dataset as the training dataset and N videos from the UCF-Sports dataset as the test dataset, where M is greater than or equal to 600 and N is greater than or equal to 45. 1b) Read the training dataset and test dataset according to the data path, perform frame separation on all videos to obtain video data frames, and further create a list of frame names and annotation names for each video. Extract the training clip Clip1, training annotation clip2 and test clip3 from the list of frame names and annotation names according to the target clip length parameter. (2) Dataset preprocessing: 2a) Adjust, flip, and normalize the video image frames and corresponding annotation image frames of the training data according to the target size, then convert them into tensor data to finally obtain the corresponding training video image frame tensor data Train. T And the corresponding training video annotation frame tensor data Annt T ; 2b) Adjust and normalize the video image frames of the test data sequentially according to the target size, then convert them into tensor data to finally obtain the corresponding test video image frame tensor data Test. T ; (3) Construct a lightweight multi-scale video saliency prediction model: 3a) Construct a lightweight spatiotemporal convolutional unit M, mainly composed of cascaded 3D octave convolutional layers (3DOctave) and spatiotemporal multi-scale octave convolutional layers (SPM). LW ; The 3D octave convolutional layer 3DOctave includes four three-dimensional convolutional layers, one three-dimensional pooling layer and one trilinear interpolation upsampling layer. One of the three-dimensional pooling layers is connected in series with one of the three-dimensional convolutional layers, and one of the trilinear interpolation upsampling layers is connected in series with one of the three-dimensional convolutional layers. These two series modules are then connected in parallel with the remaining two three-dimensional convolutional layers. The Spatiotemporal Multiscale Octave Band Convolutional Layer (SPM) comprises x three-dimensional convolutional layers, y three-dimensional pooling layers, and z trilinear interpolation upsampling layers. The y three-dimensional pooling layers are concatenated with the y three-dimensional convolutional layers, and the z trilinear interpolation upsampling layers are concatenated with the z three-dimensional convolutional layers. These y+z concatenated modules are then connected in parallel with the remaining xyz three-dimensional convolutional layers. Here, x, y, and z are all integers, x ≥ 1 and x > y, x > z. 3b) A multi-scale feature extraction chain consisting of multiple feature extraction modules with different structures and multiple cascaded feature separation layers is used to extract features at different scales. Each module consists of a different number of lightweight spatiotemporal convolutional units M. LW Connecting components; The multiple feature separation layers have the same structure, and each feature separation layer consists of a three-dimensional convolutional layer and a three-dimensional pooling layer connected in parallel. The multiple feature extraction modules with different structures are composed of different numbers of lightweight spatiotemporal convolutional units M LW The module is composed of a series of lightweight spatiotemporal convolutional units M, each of which is connected in series in the feature extraction module. LW The number is determined by the module order i, that is, each module consists of 2×i lightweight spatiotemporal convolutional units M. LW The system is composed of a series of modules, all of which are connected in series through a spatiotemporal multi-scale octave band convolutional layer (SPM) and a feature separation layer. 3c) Construct a feature fusion module consisting of multiple parallel dilated convolutional layers, a spatiotemporal multi-scale octave band convolutional layer (SPM), a pointwise convolutional module, and an upsampling layer connected in sequence; The dilated convolutional layer and the spatiotemporal multi-scale octave band convolutional layer (SPM) are composed of M parallel dilated convolutional layers and one spatiotemporal multi-scale octave band convolutional layer (SPM) connected in series, where M≥2. The pointwise convolution module consists of N pointwise convolutional layers connected in series. The kernel size of each convolutional layer is 3×1×1 and the stride is 2×1×1, and N≥1. The upsampling layer includes a bilinear interpolation upsampling layer with an amplification factor of s, where s>1; 3d) Concatenate the multi-scale feature extraction chain constructed in step 3b) and the feature fusion module constructed in step 3c) to form a lightweight multi-scale video saliency prediction model, and use the KL divergence as the loss function L of the model. s (S,A); (4) Input the training video image frame tensor dataset into the lightweight multi-scale video saliency prediction model, and train it using the Adam optimization method to obtain the updated lightweight multi-scale video saliency prediction model. (5) Transfer the tensor data T of the test video image frames test The input is fed into the trained lightweight multi-scale video saliency prediction model to obtain the saliency prediction results.

2. The method according to claim 1, characterized in that, In step 1b), a list of frame names and annotation names is created for each video. Based on the target segment length parameter, the training segment Clip1, training annotation segment Clip2, and test segment Clip3 are extracted from the list of frame names and annotation names, as follows: 3.1) Create four empty lists L1, L2, L3 and L4. Use L1 to store the names of the training videos, L2 to store the frame names of each training video, L3 to store the frame names of the annotations in each training video, and L4 to store the number of frames in each training video. 3.2) Randomly select a training start frame index Idx, and starting from this index, extract the training fragment Clip1 with target fragment length T and the corresponding training annotation fragment Clip2 from the L2 list and L3 list respectively; 3.3) Create three empty lists L1', L2', and L3'. Use list L1' to store the test video names; use list L2' to store the frame names of each test video; and use list L3' to store the number of frames in each test video. 3.4) Set the test start frame index Idx' = 0, and extract the test segment Clip3 with target segment length T from the L2' list starting from this start frame index.

3. The method according to claim 1, characterized in that, In step 2a), the video image frames of the training data and their corresponding annotated video image frames are adjusted, flipped, and normalized sequentially according to the target size, and then converted into tensor data, as follows: 2a1) Convert the video image frames in the training clip Clip1 from BGR format to RGB format, adjust the size of the video image frames to match the target size W×H, then randomly flip the video frames with a probability of 0.5, and then normalize them, reducing the pixel value range of each frame from the original image pixel value Ori. PixValue Transform ∈ [0,255] to normalized image pixel values ​​(Norm) PixValue ∈[-1,1], the normalization formula is as follows: 2a2) Create an empty list Train and use the Train list to store the normalized training video image frames; 2a3) Concatenate the Train list along the last axis to obtain a three-dimensional array, then convert this three-dimensional array into a three-dimensional PyTorch tensor to obtain the training video image frame tensor data Train. T ∈R [3×T×W×H] ; 2a4) Perform the same adjustment, flipping and normalization on the training annotation clip Clip2 as in step 2a1), create an empty list Annt, and use the Annt list to store the normalized training annotation video image frames. 2a5) Convert each video annotation frame in the Annt list into a two-dimensional PyTorch tensor to obtain the training video annotation frame tensor data Annt. T ∈R [3×W×H] .

4. The method according to claim 1, characterized in that, In step 2b), the video image frames of the test data are adjusted and normalized sequentially according to the target size, and then converted into tensor data, as follows: 2b1) Convert the video image frames in the test clip Clip3 from BGR format to RGB format, adjust the size of the video image frames to match the target size of W×H, and then normalize them, reducing the pixel value range of each frame from the original image pixel value Ori. PixValue Transform ∈ [0,255] to normalized image pixel values ​​(Norm) PixValue ∈[-1,1]: 2b2) Create an empty list Test, and use the Test list to store the normalized test video image frames; 2b3) Concatenate the Test list along the last axis to obtain a three-dimensional array, then convert this three-dimensional array into a three-dimensional PyTorch tensor to obtain the test video image frame tensor data Test. T ∈R [3×T×W×H] Where W≥128, N≥196.

5. The method according to claim 1, characterized in that, Step 3d) Define the loss function L of the model. s (S,A) represents the following: Where S represents the saliency prediction result obtained by the model, and A represents the video annotation image frame tensor.

6. The method according to claim 1, characterized in that, In step (4), the lightweight multi-scale video saliency prediction model is trained using the Adam optimization algorithm, as follows: 9a) Initialize the first-order moment estimate m0 = 0, the second-order moment estimate v0 = 0, the initial learning rate η > 0.0001, and the maximum number of iterations T ≥ 2500; 9b) Using the loss function L s (S,A), calculate the current network parameters θ t-1 gradient 9c) Utilizing the current gradient g t And the current first moment estimate m t-1 Update the unbiased first moment estimate: The first-order hyperparameter β1 = 0.99; 9d) Using the current gradient g t And the current second moment estimate v t-1 Update the standard unbiased second moment estimate: The second-order hyperparameter β2 = 0.999; 9e) Using the updated unbiased first moment estimate and standard unbiased second moment estimation Calculate the updated network parameters θ t : Where ε is less than 10 -8 A constant; 9c) Repeat steps 9b) to 9e) until the set number of iterations is reached, to obtain a lightweight multi-scale video saliency prediction model after T iterations.

Images