A video saliency prediction method and system based on audio-video features

Abstract

The application aims at the field of video saliency prediction, and discloses a video saliency prediction method and system based on audio-video features. First, the video to be predicted is processed into video frames and log mel spectrum maps through data preprocessing, and spatial features of the video frames, time features, and semantic features of audio synchronized with the video frames in time are extracted through three parallel encoders. Then, through an audio-video feature fusion module based on a channel attention mechanism, the inter-channel weights of the visual features are adaptively learned by using the semantic information of the visual features and the audio features, so that the fusion of the audio-video features is realized. Finally, the audio-video features are obtained through a decoder to obtain a saliency prediction map of the video frames.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video saliency prediction, and particularly relates to a video saliency prediction method and system based on audio-video features. BACKGROUND

[0002] When observing a scene, the human visual system quickly and selectively identifies the most attractive part from the large amount of information received, rather than processing the information of all regions. Even in a complex environment, humans can quickly focus their attention on the important part of the scene. This is called the visual attention mechanism, and the region of the scene that can attract the attention of the human visual system is called the saliency region. Understanding and modeling this mechanism to predict which regions of the video are more attractive to human attention is the research content of video saliency. With the continuous development and widespread use of shooting devices, video data can be very conveniently obtained and used. Compared with images, videos have stronger expression ability and richer information. The research on video saliency prediction is of great significance for video analysis and processing. By applying the human visual attention mechanism to the field of video processing, computers can assist in selectively processing information in videos. It can make computers allocate limited computing resources to more important regions first, which will greatly improve the work efficiency of video processing methods. The research on video saliency prediction can be applied to the field of video processing, such as video quality evaluation, video compression, video monitoring, target recognition, target segmentation, etc.

[0003] In the past few decades, video saliency prediction has made significant progress in prediction accuracy from traditional saliency prediction algorithms using hand-crafted features to models using deep learning methods. Video saliency prediction can be divided into methods based on signal compression and methods based on image pixel processing. Methods based on signal compression consider saliency as a measure of signal compressibility, and use video bit streams to extract features for saliency prediction. They first encode and compress the video, then measure the reconstruction error of the decoded data, and the region with larger reconstruction error will be considered as salient. Methods based on image pixels start from the feature integration theory, and use spatio-temporal information extracted from video frames to predict saliency regions. However, existing mainstream video saliency prediction methods only use visual information in the scene, ignoring the synchronous audio information, which is different from the daily situation of people watching videos. Although vision is the main way for humans to perceive the external environment, hearing also provides a lot of important information. Humans receive related information from the same object or event through hearing and vision, for example, hearing engine sound and seeing a moving vehicle, hearing conversation and seeing a person's moving lips. These synchronized information is combined in the human brain and jointly affects the human visual attention distribution in the scene. Cognitive neuroscience research shows that the combination of audiovisual stimuli can enhance the response of the human perception system, and integrating audiovisual information can more quickly locate the region in the scene that attracts human visual attention. Eye movement experiment results under different audio conditions of the same video more intuitively prove that audio information will affect the region of interest of the human visual system. Therefore, the prediction results of video saliency prediction methods based on audiovisual features will be more consistent with the viewing state of humans in real scenes. Compared with mature video saliency prediction methods, research on saliency prediction methods based on audiovisual features is still in its infancy. Current research on saliency prediction based on audiovisual features is developed from video saliency prediction methods, but the use of audio information in video saliency tasks is still very limited, and most of the existing attempts simply add or connect audio features and visual features. Therefore, how to reasonably fuse audio features and video features and use audiovisual information for saliency prediction is a challenge. SUMMARY

[0004] Invention purpose: When humans watch a scene, the visual system and the auditory system jointly affect the area of human attention. Most current video saliency prediction methods only process visual information such as video frames. These methods can better predict the area of interest in the scene for the visual system, but it is difficult to process the influence of audio information on human attention. At the same time, the existing video saliency prediction methods that consider audio information simply add or connect audio and video features to realize feature fusion, which cannot effectively fuse audio and video features, resulting in inaccurate predicted saliency regions. The present application provides a saliency prediction method based on audio and video features in view of the present situation.

[0005] The present method is inspired by the dual-flow network framework, and a multi-flow network framework is designed to extract visual spatial saliency features in video frames, visual temporal saliency features between video frames, and audio semantic saliency features synchronized with video frames. The present application proposes an audio and video semantic fusion module at the channel level, which uses the semantic information of the audio and video features in the scene to weight the visual saliency results at the channel level, so as to realize the prediction of the video saliency region using audio and video features. Technical scheme: The video saliency prediction method of the audio and video features proposed in the present application adopts a multi-flow architecture to improve the accuracy of saliency prediction. In order to effectively extract visual temporal features, visual spatial features and audio semantic features related to saliency, a new multi-flow architecture is proposed. The spatial encoder extracts visual spatial features from video frames, the temporal encoder extracts inter-frame visual temporal features from multiple consecutive video frames, and the audio encoder extracts audio semantic features from the log Mel spectrogram synchronized with the video frames. In order to effectively improve the accuracy of visual saliency prediction, a visual saliency basic feature extraction module is proposed, which is improved on the basis of the MobileNet-V2 model and used in the spatial decoder and the temporal decoder to extract multi-level saliency-related basic features of each frame. In order to effectively fuse audio and video features, an audio and video fusion module is proposed, which uses the semantic information in the audio and video features to weight the results of visual saliency prediction at the channel level, so that the more salient channel in the audio and video information obtains a greater weight, and the influence of the audio information on the final saliency prediction result is obtained without introducing audio information interference.

[0006] A video saliency prediction method based on audio and video features, the specific steps are as follows:

[0007] Step 1, data preprocessing, first process the video to be predicted and its time-synchronized audio into continuous video frames and log Mel spectrograms;

[0008] Step 2, first construct a visual saliency basic feature extraction module for extracting features of video frames;

[0009] Step 3, based on the video frame, the spatial saliency features and the temporal saliency features contained in the video frame are extracted, the spatial features and the temporal features are fused to obtain visual saliency features, and the audio semantic features contained in the log mel spectrogram are extracted;

[0010] Step 4, constructing an audio-video feature fusion module, adaptively fusing the visual saliency features and the audio semantic features to obtain audio-video saliency features;

[0011] Step 5, integrating the audio-video saliency features into a single-channel saliency map by using a decoder;

[0012] Step 6, training the overall model constituted by steps 2-5;

[0013] Step 7, using the trained overall model to realize visual saliency prediction.

[0014] Further, the specific implementation of step 1 is as follows:

[0015] Step 1.1, first, processing the video to be predicted, separating the audio and converting it into a wav format, and cutting the video into video frames according to the original video frame rate;

[0016] Step 1.2, processing the video frame includes: reading the picture and converting the picture to RGB format; adjusting the picture resolution to WxH, normalizing the picture, and converting the data type to Tensor type;

[0017] Step 1.3, converting the audio wav file into a log mel spectrogram that is time-synchronized with the video frame, including: resampling the audio wav file to 16000Hz; dividing the resampled audio file into non-overlapping frames with a duration of 960ms; decomposing each 960ms frame through a window with a duration of 25ms and a step of 10ms; generating a frequency spectrum that is integrated into 64 mel interval frequency bands, each frequency band is logarithmically converted after adding a small offset to the amplitude, resulting in a log mel spectrogram with a size of 96x64 and a channel number of 1; obtaining the timestamp of each frame of the video frame, corresponding to the duration covered by the log mel spectrogram and the video frame timestamp; converting the data type to Tensor type.

[0018] Further, the visual saliency base feature extraction module is improved based on the MobileNet-V2 model, and the specific improvements are as follows:

[0019] Firstly, the last spatial pooling layer in the MobileNet-V2 model is replaced with a pyramid hole pooling layer, which is composed of K=4 parallel hole convolution layers. A single convolution layer with a hole size of 1 is used to preserve the original information, and three convolution layers with different hole sizes are used to extract features of different scales. The features obtained through the four hole convolution layers are connected as the output of the last convolution block of the MobileNet-V2 model.

[0020] Secondly, the outputs of the last three convolution blocks of the MobileNet-V2 model are connected. Three convolution layers are used to adjust the channel dimension of the output features of the three convolution blocks. The output spatial resolutions of the three convolution blocks are not the same. The outputs of the last two convolution blocks are bilinearly upsampled to scale the feature maps, so that the sizes of the three feature maps are the same. Then the three features are connected in the channel direction and passed through a convolution layer to obtain the visual saliency base feature F x , where x is the input video frame number.

[0021] Further, in step 3, a spatial encoder is constructed to extract the spatial saliency features contained in the video frames, which is implemented as follows:

[0022] Video frame X n After passing through the visual saliency base feature extraction module, the feature F n is obtained. x After that, a convolution layer is used to integrate the features of different abstraction levels in F x . To improve the performance of the model, a convolutional Gaussian prior layer is added to simulate the center bias phenomenon. The prior feature map is obtained by automatically combining K=8 Gaussian maps with different horizontal and vertical variances through two convolution layers, and then added to the saliency feature. The size of the Gaussian map is the same as the saliency feature, which is The Gaussian map is generated by the following equation:

[0023]

[0024] where f(x,y) is the Gaussian function value of the Gaussian map at position (x,y), and μ x , μ y , σ x , σ y represent the mean and variance of the horizontal and vertical directions, respectively. The n∈{1,2,3,…,8} is set to connect the prior feature map to the saliency feature in the channel direction. The prior feature and the saliency feature are integrated through two convolution layers to output the spatial saliency feature

[0025] Further, in step 3, a temporal encoder is constructed to extract the temporal saliency features contained in the video frames, which is implemented as follows:

[0026] n consecutive video frames (X1, X2, ..., X n ) Through n parallel visual saliency basic feature extraction modules, the features obtained are connected to obtain a temporal feature of size n×45×60×256. The temporal features are extracted and integrated through two convolutional layers. A convolutional Gaussian prior layer is added to the temporal encoder to simulate the center bias phenomenon. The prior feature map is obtained by automatically combining K=8 Gaussian maps with different horizontal and vertical variances through two convolutional layers. All Gaussian maps are set Based on experience n∈{1,2,3,…,8}, connect the obtained prior feature map to the salient features of each frame in the channel direction, integrate the prior features and salient features through two layers of 3D convolutional layers and output the temporal salient features

[0027] Input the logarithmic Mel spectrogram A to the audio encoder, which is synchronized with the input spatial encoder video frame time. n , using the ResNet18 model as the audio encoder from the logarithmic Mel spectrogram A n Obtaining audio semantic features

[0028] Furthermore, in step 3, the spatial and temporal features are connected by channel layer, and two convolutional layers are used to automatically fuse the spatial and temporal features to output visual saliency features.

[0029] Furthermore, in step 3, an audio encoder is constructed to extract the audio semantic features contained in the logarithmic Mel-spectrogram. The specific implementation method is as follows;

[0030] Furthermore, the audio and video fusion module automatically learns the weight parameters between channels based on the channel attention mechanism. The specific implementation method is as follows;

[0031] First, the visual saliency features A global average pooling layer P is used to compress the visual saliency features into channel-level statistical information; in the second step, two nonlinear fully connected layers f are used. V and f A The visual saliency features passed through the global average pooling layer and audio semantic features Adjust the dimension and then add them element by element to get the audio and video semantic features; then use the multi-layer perceptron U with sigmoid activation function to calculate the channel level attention weight W n , the above process is shown in the following equation:

[0032]

[0033] Finally, the obtained attention weight W n Same Element-wise multiplication, get the saliency feature based on audio and video features

[0034] Further, the training in step 6 is carried out in two steps, first, the visual saliency base feature extraction module is trained using SALICON, and the random gradient descent algorithm is used for training, then the overall model is trained using DIEM, and the random gradient descent algorithm is used for training.

[0035] Further, the loss function used in training is as shown in the following formula:

[0036] L(S,M,F)=α1L kl (S,M)+α2L cc (S,M)+α3L nss (S,F)

[0037] Where S represents the single-channel saliency prediction map output by the model, M represents the saliency density map in the data true value, F represents the gaze point map in the data true value, and α1, α2, α3 are artificial weights; L kl , L cc , L nss respectively represent the calculation formulas of KL divergence, linear correlation coefficient and normalized path scanning saliency, as follows:

[0038]

[0039] Where i represents the ith pixel;

[0040]

[0041] Where cov(S,M) and σ(S,M) represent the covariance and standard deviation between the saliency prediction result obtained by the model and the real saliency density map, respectively;

[0042]

[0043] Where μ(S) and σ(S) represent the mean and variance of the saliency prediction result map, respectively, and N represents the number of eye gaze points.

[0044] On the other hand, the present application also provides a visual saliency prediction system based on audio and video features, comprising the following modules:

[0045] The data preprocessing module is used for data preprocessing, and first processes the video to be predicted and its time-synchronized audio into continuous video frames and log mel spectrum maps;

[0046] a basic feature extraction module, configured to construct a visual saliency basic feature extraction module to extract features of a video frame;

[0047] a visual saliency feature and audio semantic feature extraction module, configured to extract spatial saliency features and temporal saliency features contained in the video frame based on the features of the video frame, fuse the spatial features and the temporal features to obtain visual saliency features, and extract audio semantic features contained in a log mel spectrogram;

[0048] a feature fusion module, configured to construct an audio-video feature fusion module to adaptively fuse the visual saliency features and the audio semantic features to obtain audio-video saliency features;

[0049] a single-channel saliency map synthesis module, configured to integrate the audio-video saliency features into a single-channel saliency map by using a decoder;

[0050] a training module, configured to train an overall model composed of the basic feature extraction module, the visual saliency feature and audio semantic feature extraction module, and the feature fusion module;

[0051] a prediction module, configured to realize visual saliency prediction by using the trained overall model.

[0052] The present application has the following advantages:

[0053] The spatial features, the temporal features and the audio features in the video frame and the audio frame are more effectively extracted by using a brand-new multi-stream network framework. The visual and auditory features are more effectively fused by using an audio-video feature fusion module that obtains a channel weighting weight of the visual saliency features by fusing audio-video semantic features. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 which is a general framework diagram of the present application.

[0055] Figure 2 which is a framework diagram of the visual saliency basic feature extraction module of the present application.

[0056] Figure 3 which is a framework diagram of the audio-video feature fusion module of the present application.

[0057] Figure 4 which is a general flowchart of the method. DETAILED DESCRIPTION

[0058] The technical solutions of the present application will be further described below in combination with the drawings and examples.

[0059] The general flowchart of the present application is shown in Figure 4 The specific implementation steps are as follows:

[0060] Step 1: Data preprocessing.

[0061] Step 1.1: First, process the video to be predicted, separate the audio and convert it into WAV format, and crop the video into video frames according to the original video frame rate.

[0062] Step 1.2, processing the video frame includes: reading the image and converting it to RGB format; adjusting the image resolution to 360×640 pixels; normalizing the image; and converting the data type to Tensor.

[0063] Step 1.3: Convert the audio WAV file into a logarithmic Mel spectrogram synchronized with the video frame time. This includes: resampling the audio WAV file to 16000 Hz; dividing the resampled audio file into non-overlapping 960ms frames; decomposing each 960ms frame using a short-time Fourier transform with a window of 25ms and a step size of 10ms; integrating the generated spectrogram into 64 Mel-interval frequency bands, adding a small offset to the amplitude of each frequency band and performing a logarithmic transformation to obtain a logarithmic Mel spectrogram of size 96×64 with 1 channel; obtaining the timestamp of each video frame and mapping the duration covered by the logarithmic Mel spectrogram to the video frame timestamp; converting the data type to Tensor.

[0064] Step 2: Extract the spatial, temporal, and audio features contained in the video frame and the logarithmic Mel-spectrogram. The present invention builds a basic framework based on a multi-stream network structure, which includes a spatial encoder, a temporal encoder, an audio encoder, an audio and video feature fusion module, and a decoder. The model input is n time-synchronized continuous video frames and audio frames, where a single video frame x is input to the spatial encoder. n , input n consecutive video frames (X1, X2, ..., X n ), input the logarithmic Mel spectrum A of the audio encoder that is synchronized with the input spatial encoder video frame time n According to multiple experiments, n=7 can achieve the best results. The spatial features F related to significance are obtained through three encoders. s , time feature F t , audio feature F a For ease of understanding, the convolution layer parameters described in this document are expressed as: number of channels _ convolution kernel size × convolution kernel size _ step size; the pooling layer parameters are expressed as: convolution kernel size × convolution kernel size _ step size. The feature extraction steps are as follows:

[0065] Step 2.1: Constructing the visual saliency base feature extraction module. The present application simulates the process of human visual system processing information, that is, most of the basic visual features are extracted and integrated in the process from the retina to the basic visual cortex. The weight-sharing visual saliency base feature module is used in the spatial encoder and the temporal encoder of the visual. The module is improved on the basis of the MobileNet-V2 model, and the specific improvements are as follows:

[0066] Firstly, the last spatial pooling layer in the MobileNet-V2 model is replaced with a hole pyramid pooling layer. The hole pyramid pooling layer is composed of K=4 parallel hole convolution layers. A single convolution layer with a hole size of 1 (rate=1) is used to preserve the original information. The convolution kernel size is 1x1, and the output feature channel number is 256. Three convolution layers with different hole sizes (rate={6, 12, 18}) and a convolution kernel size of 3x3 are used to extract features of different scales. The features obtained by the four hole convolution layers are connected to serve as the output of the last convolution block of the MobileNet-V2 model.

[0067] Secondly, the outputs of the last three convolution blocks of the MobileNet-V2 are connected. Convolution layers with a convolution kernel size of 1x1 and output feature channel numbers of 64, 128, and 256 are used to adjust the channel dimensions of the output features of the three convolution blocks, respectively. The output spatial resolutions obtained by the three convolution blocks are not the same. The outputs of the last two convolution blocks are bilinearly interpolated and upsampled to scale the feature maps, so that the sizes of the three feature maps are the same, which is (45x80 pixels). Then, the three features are connected in the channel direction, and a (256_3x3_1) convolution layer is used to obtain the visual saliency base feature F x (x is the input video frame sequence number), with a size of 45x80x256.

[0068] Step 2.2: Constructing the spatial encoder. The video frame X n is input into the visual saliency base feature extraction module to obtain the feature F n After that, a convolution layer with a convolution kernel size of 3x3 and an output feature channel number of 256 is used to integrate the features of different abstraction levels in F x To improve the performance of the model, a convolutional Gaussian prior layer is added to simulate the center bias phenomenon. By using two convolution layers with a convolution kernel size of 3x3 and an output feature channel number of 64, K=8 Gaussian maps with different horizontal and vertical variances are automatically combined to obtain a prior feature map, which is added to the saliency feature. The size of the Gaussian map is the same as the saliency feature, which is (45x80 pixels). The Gaussian map is generated by the following equation:

[0069]

[0070] where f(x, y) is the value of the Gaussian function at the position (x, y) of the Gaussian map, and μ x , μ y , σ x , σ y respectively represent the mean and variance of the horizontal and vertical directions. The n e {1, 2, 3, …, 8}. The prior feature map is connected to the saliency feature in the channel direction, and the prior feature and the saliency feature are integrated and output spatial saliency feature by two convolution layers with a kernel size of 3x3 and an output feature channel number of 256

[0071] Step 2.3: Build the temporal encoder. 7 consecutive video frames (X1, X2, …, X n are input into 7 parallel visual saliency base feature extraction modules and the obtained features are connected to obtain a feature with a size of 7x45x60x256. A three-dimensional convolution layer with a kernel size of 3x3 and an output feature channel number of 256 is used to extract and integrate the temporal features. Similar to the spatial encoder, the temporal encoder also adds a convolution Gaussian prior layer to simulate the center bias phenomenon. The prior feature map is obtained by automatically combining K=8 Gaussian maps with different horizontal and vertical variances through two convolution layers (64_3x3_1). The is set according to experience n e {1, 2, 3, …, 8}. The obtained prior feature map is connected to the saliency feature of each frame in the channel direction. The prior feature and the saliency feature are integrated and output temporal saliency feature by two three-dimensional convolution layers with a kernel size of 3x3 and a channel number of 256 Step 2.4: Build the audio encoder. The log mel spectrum graph A n synchronized in time with the input video frame of the spatial encoder is input into the audio encoder. The ResNet18 model is used as the audio encoder to obtain the audio semantic feature n from the log mel spectrum graph A with a size of 1x309.

[0072] Step 2.5: Fuse spatial and temporal features to obtain visual saliency feature. The spatial and temporal features are connected in the channel layer, and two convolution layers are used to automatically fuse the spatial and temporal features. The convolution layer parameters are (1024_3x3_1), (512_3x3_1), and the output visual saliency feature is

[0073] Step 3: Constructing the audio-video feature fusion module. Based on the channel attention mechanism, the purpose of the audio-video feature fusion module is to automatically learn the weight parameters between channels on the channel level, and to selectively enhance or suppress the expression of visual features related or unrelated to audio information using the semantic information of audio-video features. Compared with simply adding or connecting audio features and visual features, the audio-video feature fusion method designed by the present application is more "gentle". It will not cause the loss of visual features or bring a large amount of noise information in the audio features. In other words, it is an audio-guided visual attention fusion method. First, the visual saliency features are compressed to the statistical information on the channel level by a global average pooling layer P, with a size of 1x512. The second step uses two nonlinear fully connected layers f V and f A to adjust the dimension of the visual saliency features and the audio semantic features , and the size is adjusted to 1x512. Then, the element-wise addition of the two is performed to obtain the audio-video semantic features. Then, a multilayer perceptron U with a sigmoid activation function is used to calculate the attention weight W n on the channel level, with a size of 1x512. The above process is shown in the following equation:

[0074]

[0075] Finally, the obtained attention weight W n is multiplied element-wise with to obtain the saliency features based on audio-video features with a size of 45x80x512.

[0076] Step 4: Constructing the decoder. The saliency features based on audio-video features need to be integrated into a single-channel saliency map. The present application has adjusted the weight of the attention between the visual feature channels through the audio-video feature fusion module, and a simple decoder can be used to generate a single-channel saliency map. The present application uses two convolutional layers with a kernel size of 3x3 and an output feature channel number of 128 and 1 to integrate the audio-video saliency features to generate a single-channel saliency map S n .

[0077] Model training details:

[0078] The proposed model is implemented on an NVIDIA 1080 GPU using Pytorch. Among them, the MobileNet-V2 in the visual saliency base feature extraction module uses its public weight initialization, and the ResNet18 in the audio encoder uses the public weight initialization trained on a large audio-visual dataset.

[0079] First, the model uses SALICON to train the visual saliency base feature extraction module. The SALICON dataset is one of the largest image saliency prediction datasets, containing 20,000 image data. By training the visual saliency base feature extraction module on this dataset, the demand for video saliency data can be reduced, and the performance of the model can be improved. During training, the visual saliency base feature extraction module is trained using the stochastic gradient descent algorithm, with an initial learning rate of 10 -3 , momentum set to 0.9, and weight decay coefficient set to 0.0005. The batch size is set to 32, that is, 32 images will be processed for each training iteration.

[0080] The second step is to train the entire model using DIEM. The DIEM dataset is one of the commonly used datasets for video saliency prediction, containing 85 videos with a duration of 27 to 217 seconds. These videos are diverse and provide corresponding audio information. The proposed model uses the stochastic gradient descent algorithm for training, with an initial learning rate of 10 -3 , momentum set to 0.9, and weight decay set to 0.0005. The batch size is set to 4, that is, 4x7=28 images will be processed for each training iteration. The loss function used in the two-step training is the same, as shown in the following formula:

[0081] L(S,M,F)=α1L kl (S,M)+α2L cc (S,M)+α3L nss (S,F)

[0082] Where S represents the single-channel saliency prediction map output by the model, M represents the saliency density map in the data true value, and F represents the fixation point map in the data true value. L kl , L cc , L nss represent the calculation formulas of KL divergence, linear correlation coefficient, and normalized path scanning saliency, respectively, as follows:

[0083]

[0084] KL divergence (Kullback-Leibler divergence, KL) is an asymmetric measure index, and the smaller the value is, the closer the significant prediction result is to the real saliency map. Where i represents the ith pixel.

[0085]

[0086] Linear correlation coefficient (Linear Correlation Coefficient, CC) is commonly used to measure the correlation between two variables, and the larger the value is, the closer the significant prediction result is to the real value, where cov(S, M) and σ(S, M) represent the covariance and standard deviation between the significant prediction result obtained by the model and the real saliency map, respectively.

[0087]

[0088] Normalized scanpath saliency (Normalized Scanpath Saliency, NSS) is an index specially designed for significant prediction result evaluation, which can be regarded as a normalized saliency condition for measuring the gaze point position. The calculation method is to take the average value of the normalized saliency map corresponding to the eye gaze point. Where μ(S) and σ(S) represent the mean and variance of the saliency prediction result map, respectively, i represents the ith pixel, and N represents the number of eye gaze points. The larger the NSS value is, the better the performance of the saliency prediction model is.

[0089] α1, α2, α3 are artificially set weights, which are 1, -0.2, -0.1 respectively in this example.

[0090] On the other hand, the present application also provides a visual saliency prediction system based on audio-video features, comprising the following modules:

[0091] The data preprocessing module is used for data preprocessing, and first processes the video to be predicted and its time-synchronized audio into continuous video frames and log mel spectrum maps;

[0092] The basic feature extraction module is used for constructing a visual saliency basic feature extraction module for extracting features of the video frames.

[0093] The visual saliency feature and audio semantic feature extraction module extracts the spatial saliency features and temporal saliency features contained in the video frames based on the features of the video frames, fuses the spatial features and temporal features to obtain the visual saliency features, and extracts the audio semantic features contained in the log mel spectrum map.

[0094] The feature fusion module is configured to construct an audio-video feature fusion module, to adaptively fuse the visual saliency feature and the audio semantic feature, and to obtain an audio-video saliency feature.

[0095] The single-channel saliency map synthesis module is configured to integrate the audio-video saliency feature into a single-channel saliency map by using a decoder.

[0096] The training module is configured to train the overall model composed of the basic feature extraction module, the visual saliency feature and audio semantic feature extraction module, and the feature fusion module.

[0097] The prediction module is configured to realize visual saliency prediction by using the trained overall model.

[0098] The specific implementation of each module corresponds to each step, and the present application will not be described.

[0099] The specific implementation described herein is only an illustration of the spirit of the present application, and those skilled in the art can make various modifications or supplements to the described specific implementation or replace it with similar ways, without deviating from the spirit of the present application or exceeding the scope defined by the appended claims.

Claims

1. A video saliency prediction method based on audio and video features, characterized in that: The steps include: Step 1: Data preprocessing: First, the video to be predicted and its time-synchronized audio are processed into continuous video frames and logarithmic Mel-spectrograms; Step 2: First, build a visual saliency basic feature extraction module to extract the features of the video frame; Step 3: Extract spatial and temporal salient features from the video frames based on the features of the video frames, fuse the spatial and temporal features to obtain visual salient features, and extract the audio semantic features from the log-mel spectrogram. In step 3, the spatial and temporal features are connected by channel layer, and two convolutional layers are used to automatically fuse the spatial and temporal features to output visual saliency features. In step 3, an audio encoder is constructed to extract the audio semantic features contained in the logarithmic Mel-spectrogram. The specific implementation method is as follows; Input the logarithmic Mel spectrogram A to the audio encoder, which is synchronized with the input spatial encoder video frame time. n , using the ResNet18 model as the audio encoder from the logarithmic Mel spectrogram A n Obtaining audio semantic features Step 4: Construct an audio and video feature fusion module to adaptively fuse visual saliency features with audio semantic features to obtain audio and video saliency features; The audio and video fusion module automatically learns the weight parameters between channels based on the channel attention mechanism. The specific implementation method is as follows; First, the visual saliency features A global average pooling layer P is used to compress the visual saliency features into channel-level statistical information; in the second step, two nonlinear fully connected layers f are used. V and f A The visual saliency features passed through the global average pooling layer and audio semantic features Adjust the dimension and then add them element by element to get the audio and video semantic features; then use the multi-layer perceptron U with sigmoid activation function to calculate the channel level attention weight W n , the above process is shown in the following equation: Finally, the attention weight W n same Multiply element by element to obtain the saliency features based on audio and video features Step 5: Use the decoder to integrate the audio and video saliency features into a single-channel saliency map; Step 6: train the overall model composed of steps 2 to 5; Step 7: Use the trained overall model to achieve visual saliency prediction.

2. The method for predicting video saliency based on audio and video features according to claim 1, wherein: The specific implementation of step 1 is as follows; Step 1.1: First, process the video to be predicted, separate the audio and convert it into WAV format, and crop the video into video frames according to the original video frame rate; Step 1.2, processing the video frame includes: reading the image and converting it to RGB format; adjusting the image resolution to W×H, normalizing the image, and converting the data type to Tensor type; Step 1.3: Convert the audio WAV file into a logarithmic Mel spectrogram synchronized with the video frame time, including: resampling the audio WAV file to 16000 Hz; dividing the resampled audio file into non-overlapping 960ms frames; decomposing each 960ms frame by short-time Fourier transform using a window of 25ms and a step size of 10ms; integrating the generated spectrogram into 64 Mel-interval frequency bands, adding a small offset to the amplitude of each frequency band and performing a logarithmic transformation to obtain a logarithmic Mel spectrogram of size 96×64 and number of channels 1; obtaining the timestamp of each video frame, and corresponding the duration covered by the logarithmic Mel spectrogram to the video frame timestamp; converting the data type to Tensor type.

3. The method for predicting video saliency based on audio and video features according to claim 1, wherein: The visual saliency basic feature extraction module is improved based on the MobileNet-V2 model. The specific improvements are as follows: First, the last spatial pooling layer in the MobileNet-V2 model is replaced with an atrous pyramid pooling layer. This atrous pyramid pooling layer consists of K = 4 parallel atrous convolutional layers. A single convolutional layer with an aperture size of 1 is used to preserve the original information. Three convolutional layers with different aperture sizes are used to extract features at different scales. The features obtained by the four atrous convolutional layers are concatenated as the output of the last convolutional block of the MobileNet-V2 model. The second step is to connect the outputs of the last three convolutional blocks of MobileNet-V2 and use three convolutional layers to adjust the channel dimension of the output features of the three convolutional blocks respectively. The output spatial resolutions of the three convolutional blocks are different. The outputs of the last two convolutional blocks are upsampled by bilinear interpolation and the feature maps are scaled to make the three feature maps of the same size. Then, the three features are connected in the channel direction and passed through a convolutional layer to obtain the visual saliency basic feature F. x , x is the input video frame number.

4. The method for predicting video saliency based on audio and video features according to claim 1, wherein: In step 3, a spatial encoder is constructed to extract spatial saliency features contained in the video frame. The specific implementation method is as follows: Video Frame X n The feature F is obtained by the visual saliency basic feature extraction module n Finally, a convolutional layer is used to integrate F x In order to improve the performance of the model, a convolutional Gaussian prior layer is added to simulate the center bias phenomenon. The two convolutional layers automatically combine K = 8 Gaussian maps with different horizontal and vertical variances to obtain the prior feature map to add to the salient features. The size of the Gaussian map is the same as the salient feature. The Gaussian plot is generated by the following equation: Where f(x,y) is the Gaussian function value of the Gaussian graph at the position (x,y), and μ x 、μ y , σ x , σ y Represents the mean and variance in the horizontal and vertical directions respectively, and sets all Gaussian graphs Connect the prior feature map to the salient features in the channel direction, integrate the prior features and salient features through two convolutional layers and output spatial salient features 5. The method for predicting video saliency based on audio and video features according to claim 1, wherein: In step 3, a temporal encoder is constructed to extract temporal saliency features contained in the video frames. The specific implementation is as follows: n consecutive video frames (X1, X2, ..., X b ) Through n parallel visual saliency basic feature extraction modules, the features obtained are connected to obtain a temporal feature of size n×45×60×256. The temporal features are extracted and integrated through two convolutional layers. A convolutional Gaussian prior layer is added to the temporal encoder to simulate the center bias phenomenon. The prior feature map is obtained by automatically combining K=8 Gaussian maps with different horizontal and vertical variances through two convolutional layers. All Gaussian maps are set Based on experience The obtained prior feature map is connected to the salient features of each frame in the channel direction, and the prior features and salient features are integrated through two layers of three-dimensional convolutional layers to output the temporal salient features.

6. The method for predicting video saliency based on audio and video features according to claim 1, wherein: In step 6, the training is performed in two steps. First, the visual saliency basic feature extraction module is trained using SALICON and the stochastic gradient descent algorithm is used for training. Then, the overall model is trained using DIEM and the stochastic gradient descent algorithm is used for training.

7. The method for predicting video saliency based on audio and video features according to claim 1, wherein: The loss function used in training is shown in the following formula: L(S,M,F)=α1L kl (S,M)+α2L cc (S,M)+α3L nss (S,F) Where S represents the single-channel saliency prediction map output by the model, M represents the saliency density map in the real data value, F represents the fixation point map in the real data value, α1, α2, and α3 are artificially set weights; L kl , L cc , L nss The calculation formulas for KL divergence, linear correlation coefficient, and standardized path scanning significance are as follows: Where i represents the i-th pixel; Where cov(S,M) and σ(S,M) represent the covariance and standard deviation between the saliency prediction results obtained by the model and the true saliency density map, respectively; Where μ(S) and σ(S) represent the mean and variance of the saliency prediction result map, respectively, and N represents the number of gaze points of the human eye.

8. A video saliency prediction system based on audio and video features, used to implement the video saliency prediction method based on audio and video features according to any one of claims 1 to 7, characterized in that: Includes the following modules: The data preprocessing module is used for data preprocessing. First, the video to be predicted and its time-synchronized audio are processed into continuous video frames and logarithmic Mel-spectrograms. Basic feature extraction module, used to build a visual saliency basic feature extraction module, used to extract features of video frames; The visual saliency feature and audio semantic feature extraction module extracts spatial saliency features and temporal saliency features contained in the video frame based on the features of the video frame, fuses the spatial features and temporal features to obtain visual saliency features, and extracts audio semantic features contained in the logarithmic Mel-spectrogram; Feature fusion module: builds an audio and video feature fusion module to adaptively fuse visual saliency features with audio semantic features to obtain audio and video saliency features; A single-channel saliency map synthesis module is used to integrate audio and video saliency features into a single-channel saliency map using a decoder; The training module trains the overall model consisting of the basic feature extraction module, the visual saliency feature and audio semantic feature extraction module, and the feature fusion module; The prediction module is used to achieve visual saliency prediction using the trained overall model.

Images