A video memory prediction method based on a two-stream multi-attention mechanism

Abstract

This invention discloses a video memorability prediction method based on a dual-stream multi-attention mechanism, comprising: encoding visual modal information and text modal information separately using a specific network structure to obtain encoded visual representations and encoded text representations; inputting the encoded visual representations into a hierarchical attention mechanism for deep learning to obtain aggregated visual-level representations; constructing an interactive aggregation network model, inputting the aggregated visual-level representations and encoded text representations into the interactive aggregation network model to obtain high-level semantic representations of the video; inputting the high-level semantic representations of the video into a multi-head attention network module, mapping the high-level semantic representations of the video to different subspaces, obtaining multi-head outputs, and then calculating the memorability score through a fully connected regression network. This invention introduces a hierarchical attention module to process the visual information of the video, obtaining a condensed high-level semantic representation, thereby achieving the effective and simplified processing of visual information.

Description

Technical Field

[0001] This invention belongs to the field of memory prediction technology in the field of multimedia information processing, and particularly relates to a video memory prediction method based on a dual-stream multi-attention mechanism. Background Technology

[0002] With the rapid development of internet communication technology, a massive amount of videos are constantly being created and uploaded to social media platforms. Compared to traditional communication methods such as text and voice, video, by combining subtitles, sound, and visuals to form dynamic sequences of images, conveys rich multimedia information and has become one of the most commonly used social tools for internet users. If a system can predict which visual content will be highly or long-term remembered, it can assess the effective value of a video based on its memorability and compress or discard information deemed unimportant. This research has potential applications in resource management, video production, and education. To enable video creators to produce memorable content or to facilitate the management of massive amounts of video data in databases, video memorability prediction has become a popular research topic. Currently, research on predicting video memorability mainly references existing studies on image memorability. However, the semantic representation and encoding of videos do not consider their unique characteristics compared to other objects. For example, using only static visual information or simple textual information as initial feature representations makes it difficult to uncover the deep, dynamic semantic representations contained in videos. Furthermore, multimodal feature fusion often employs simple front-end or back-end fusion, failing to extract complementary information and potential connections between modalities. Publicly available video datasets with memorability labels contain information in two modalities (textual and visual). To more effectively utilize the original modal information contained in video data and deeply mine enhanced semantic representations related to memorability, multimodal feature learning and interactive fusion centered on visual information will be the focus of future research. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention proposes a video memory prediction method based on a dual-stream multi-attention mechanism. Based on the fundamental structure of correlation between consecutive video frames, a hierarchical attention module is introduced to process the visual information of the video, resulting in a condensed high-level semantic representation, thereby achieving the effective and simplified processing of visual information.

[0004] To achieve the above objectives, this invention provides a video memorability prediction method based on a two-stream multiple attention mechanism, comprising:

[0005] A specific network structure is used to encode visual modal information and text modal information respectively, so as to obtain the encoded visual representation and the encoded text representation;

[0006] The encoded visual representation is input into a hierarchical attention mechanism for deep learning to obtain an aggregated visual-level representation;

[0007] An interactive aggregation network model is constructed, and the visual-level representation of the aggregation and the encoded text representation are input into the interactive aggregation network model to obtain an enhanced semantic representation of the video.

[0008] The enhanced semantic representation of the video is input into a multi-head attention network module, which maps the enhanced semantic representation of the video to different subspaces. After obtaining the multi-head output, the memory score is calculated by a fully connected regression network.

[0009] The video memory prediction method based on a dual-stream multi-attention mechanism provided by this invention employs a specific network structure to encode visual modal information and text modal information separately, and obtains visual representations and text representations as follows:

[0010] x t =TfidfVectorizer(I t )

[0011] x v =AvgPooling(ResNet(I v ;θ))

[0012] Among them, I t For text modal information, I v For visual modal information, x t For the encoded text representation, x v θ represents the encoded visual representation, and θ represents the network parameters to be learned.

[0013] The video memory prediction method based on a two-stream multi-attention mechanism provided by the present invention includes the following method for obtaining aggregated visual-level representations by inputting the encoded visual representations into a hierarchical attention mechanism for deep learning:

[0014] Each individual frame-level feature in the encoded visual representation is encoded using a self-attention mechanism to obtain a self-attention encoded visual representation.

[0015] The visual representation after self-attention encoding is divided into several local temporal blocks of preset length, and inner product attention encoding is applied to the frame-level features within the block to obtain the block-level attention-encoded features within the block.

[0016] The block-level representation, after being encoded with intra-block attention, is aggregated into a visual representation by applying inner product attention modules between different blocks.

[0017] The video memory prediction method based on a dual-stream multi-attention mechanism provided by the present invention encodes each individual frame-level feature in the encoded visual representation using a self-attention mechanism to obtain the self-attention encoded visual representation as follows:

[0018] [Q i ,K i V i ] = f i H i

[0019]

[0020] Among them, f i H represents the features of the i-th frame. i Let [Q] be the projection matrix corresponding to the i-th frame. i ,K i V i [ ] represents the corresponding i-th query, key, and value matrix, where d is the size of each frame dimension. Softmax is for scaling operations, while Softmax is for normalization operations.

[0021] The video memory prediction method based on a dual-stream multi-attention mechanism provided by the present invention divides the self-attention encoded visual representation into several local time blocks of preset length, applies inner product attention encoding to the frame-level features within the block, and obtains the block-level attention-encoded features within the block as follows:

[0022]

[0023]

[0024] in, For block B i The k-th frame-level feature vector in the diagram, where W1 and W2 are the mapping matrices of the network layers, b1 and b2 are the bias vectors, σ is the Sigmoid activation function, and α... ik For the attention score calculated for each frame within the block, ⊙ represents the element-wise product operation, E i This is the block-level representation after aggregation.

[0025] The video memory prediction method based on a dual-stream multi-attention mechanism provided by the present invention obtains an aggregated visual-level representation by applying inner product attention modules between different blocks after the block-level attention encoding is performed as follows:

[0026] β i =W4σ(W3E i +b3)+b4

[0027]

[0028] Among them, E i Let W3, W4 and b3, b4 represent the mapping matrix and bias vector of the network layer, respectively, and β represent the i-th block of the overall video. i To calculate the attention score for each block, This is the aggregated video-level representation.

[0029] The video memorability prediction method based on a two-stream multi-attention mechanism provided by the present invention, wherein the method for obtaining an enhanced semantic representation of the video by inputting the aggregated visual-level representation and the encoded text representation into the interactive aggregation network model includes:

[0030] The correlation matrix between the aggregated visual-level representation and the encoded text representation is calculated by multiplying the feature vectors.

[0031] By transposing the aggregated visual-level representation and the encoded text representation, multiplying them with the correlation matrix, and then performing vector concatenation operations, the resulting data is input into a multi-layer neural network to obtain fused common features.

[0032] By combining the fused common features with the aggregated visual-level representation, an enhanced semantic representation of the video is obtained.

[0033] The video memory prediction method based on a two-stream multi-attention mechanism provided by the present invention calculates the correlation matrix between the aggregated visual-level representation and the encoded text representation by multiplying feature vectors as follows:

[0034]

[0035] in, For the aggregated video-level representation, x t is the encoded text representation, Softmax is the normalization operation, and S is the correlation matrix between the aggregated visual-level representation and the encoded text representation.

[0036] The method for obtaining fused common features in the video memory prediction method based on a two-stream multi-attention mechanism provided by the present invention is as follows:

[0037]

[0038] Where, x com For the common features of the fusion, Concat(·) is the concatenation operation between vectors, Γ(·) is a perceptron composed of multiple linear connection layers and nonlinear activation functions, and S is the correlation matrix between the aggregated visual-level representation and the encoded text representation. For the aggregated video-level representation, xt Θ represents the encoded text representation, and Θ represents the parameters to be learned in the multilayer neural network.

[0039] The video memorability prediction method based on a two-stream multi-attention mechanism provided by the present invention provides a method for obtaining enhanced semantic representations of videos as follows:

[0040]

[0041] Where, x com For the common characteristics of integration, For the aggregated video-level representation, x sum Enhanced semantic representation for videos.

[0042] Technical effects of the invention:

[0043] (1) Based on the basic structure of correlation between consecutive video frames, this invention introduces a hierarchical attention module to process the visual information of the video, and obtains a condensed high-level semantic representation, thereby realizing the effective and simplified visual information.

[0044] (2) This invention utilizes an interactive aggregation network to capture complementary information between text and visual modalities, learns the differences and commonalities between the two stream features, and obtains a more complete common representation of features.

[0045] (3) In building the network framework, this invention introduces different forms of attention modules multiple times, combining deep learning and differential attention to achieve the best prediction effect, which is especially suitable for predicting the memory score of videos. Attached Figure Description

[0046] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0047] Figure 1 This is a flowchart illustrating a video memory prediction method based on a dual-stream multi-attention mechanism according to an embodiment of the present invention.

[0048] Figure 2 This is a schematic diagram of a hierarchical attention module for visual information according to an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the interactive aggregation module for dual-stream features in an embodiment of the present invention. Detailed Implementation

[0050] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0051] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0052] like Figure 1-3 As shown, this embodiment provides a video memorability prediction method based on a dual-stream multi-attention mechanism, including: using a specific network structure to encode visual modal information and text modal information respectively, to obtain encoded visual representations and encoded text representations;

[0053] The encoded visual representation is input into a hierarchical attention mechanism for deep learning to obtain an aggregated visual-level representation;

[0054] An interactive aggregation network model is constructed, and the aggregated visual representation and the encoded text representation are input into the interactive aggregation network model to obtain an enhanced semantic representation of the video.

[0055] The enhanced semantic representation of the video is input into the multi-head attention network module, which maps the enhanced semantic representation of the video to different subspaces. After obtaining the multi-head output, the memory score is calculated by a fully connected regression network.

[0056] Furthermore, a method for encoding visual modal information and text modal information separately using specific network structures to obtain visual representations and text representations is as follows:

[0057] x t =TfidfVectorizer(I t )

[0058] x v =AvgPooling(ResNet(I v ;θ))

[0059] Among them, I t For text modal information, I v For visual modal information, x t For the encoded text representation, x v θ represents the encoded visual representation, and θ represents the network parameters to be learned.

[0060] Furthermore, methods for inputting the encoded visual representation into a hierarchical attention mechanism for deep learning to obtain aggregated visual-level representations include:

[0061] Each individual frame-level feature in the encoded visual representation is encoded using a self-attention mechanism to obtain a self-attention encoded visual representation.

[0062] The visual representation after self-attention encoding is divided into several local temporal blocks of preset length. Inner product attention encoding is applied to the frame-level features within the block to obtain the block-level features after in-block attention encoding.

[0063] The block-level representation, which is encoded with intra-block attention, is then aggregated into a visual representation by applying inner product attention modules between different blocks.

[0064] Furthermore, each individual frame-level feature in the encoded visual representation is encoded using a self-attention mechanism to obtain the self-attention encoded visual representation.

[0065] [Q i ,K i V i ] = f i H i

[0066]

[0067] Among them, f i H represents the features of the i-th frame. i Let [Q] be the projection matrix corresponding to the i-th frame. i ,K i V i [ ] represents the corresponding i-th query, key, and value matrix, where d is the size of each frame dimension. Softmax is for scaling operations, while Softmax is for normalization operations.

[0068] Furthermore, the visual representation after self-attention encoding is divided into several local temporal blocks of preset length. Inner product attention encoding is applied to the frame-level features within each block to obtain the block-level attention-encoded representation.

[0069]

[0070]

[0071] in, For block B i The k-th frame-level feature vector in the diagram, where W1 and W2 are the mapping matrices of the network layers, b1 and b2 are the bias vectors, σ is the Sigmoid activation function, and α... ik For the attention score calculated for each frame within the block, ⊙ represents the element-wise product operation, E i This is the block-level representation after aggregation.

[0072] Furthermore, the method for obtaining the aggregated visual-level representation by applying inner product attention modules between different blocks after encoding the intra-block attention at the block level is as follows:

[0073] β i =W4σ(W3E i +b3)+b4

[0074]

[0075] Among them, E i Let W3, W4 and b3, b4 represent the mapping matrix and bias vector of the network layer, respectively, and β represent the i-th block of the overall video. i To calculate the attention score for each block, This is the aggregated video-level representation.

[0076] Furthermore, methods for obtaining enhanced semantic representations of videos by inputting the aggregated visual-level representations and encoded textual representations into an interactive aggregation network model include:

[0077] The correlation matrix between the aggregated visual-level representation and the encoded text representation is calculated by multiplying the feature vectors.

[0078] By transposing the aggregated visual-level representation and the encoded text representation, multiplying them with the relevant matrix, and then performing vector concatenation operations, the resulting data is input into a multi-layer neural network to obtain the fused common features.

[0079] By combining the fused common features with the aggregated visual-level representation, an enhanced semantic representation of the video is obtained.

[0080] Furthermore, the method for calculating the correlation matrix between the aggregated visual-level representation and the encoded text representation through feature vector multiplication is as follows:

[0081]

[0082] in, For the aggregated video-level representation, x t is the encoded text representation, Softmax is the normalization operation, and S is the correlation matrix between the aggregated visual-level representation and the encoded text representation.

[0083] Furthermore, the method for obtaining the common characteristics of the fusion is as follows:

[0084]

[0085] Where, x com For the common features of the fusion, Concat(·) is the concatenation operation between vectors, Γ(·) is a perceptron composed of multiple linear connection layers and nonlinear activation functions, and S is the correlation matrix between the aggregated visual-level representation and the encoded text representation. For the aggregated video-level representation, x tΘ represents the encoded text representation, and Θ represents the parameters to be learned in the multilayer neural network.

[0086] To ensure that the fused public representation captures semantic information distinct from the visual representation, x is calculated. com , The similarity between these two features, constrained by the F-norm, is expressed as the difference loss.

[0087] Furthermore, the method for obtaining enhanced semantic representations of videos is as follows:

[0088]

[0089] Where, x com For the common characteristics of integration, For the aggregated video-level representation, x sum Enhanced semantic representation for videos.

[0090] After the above series of modeling and encoding processes, the final high-level semantic representation x is obtained. sum Using this as input, the features are mapped to different subspaces through the multi-head attention module, thereby capturing more diverse pattern information in the video data. The formula is as follows:

[0091]

[0092] MultiHead(Q,K,V)=Concat(head1,...,head L )

[0093] in, and Let head be the i-th learnable weight parameter. i For the output of each subspace attention module, L is the number of heads in the multi-head attention module.

[0094] After obtaining the multi-head output, the final memory prediction result is calculated by a fully connected regression network:

[0095]

[0096] Where W5 and b5 are the network parameters to be learned in the regression network layer, and the σ activation function is the ReLU function.

[0097] Based on the loss function generated during the process of acquiring enhanced semantic representations of videos and obtaining memory scores, the entire model is trained end-to-end using stochastic gradient descent optimization, iteratively updating network parameters until the model converges. Specifically, this includes:

[0098] To ensure that the predicted memory score obtained by the method is as close as possible to the true score, the mean squared error between the true and predicted values ​​for each video sample is calculated, and expressed as the prediction loss.

[0099]

[0100] Where N is the number of samples in the training set. and y n This represents the predicted and actual values ​​for the nth video.

[0101] The overall network architecture loss function consists of prediction loss and variance loss, and the overall loss function is expressed as:

[0102]

[0103] Where λ is the balance coefficient between different losses, set to 0.1.

[0104] During network training, the entire model is trained end-to-end using the stochastic gradient descent optimization method, the network parameters are updated iteratively, and the convergence of the model is determined based on the loss value.

[0105] The video samples divided into a test set are input into the trained model. The predictive performance of the proposed method is evaluated based on the Spearman correlation coefficient and compared with some existing memory prediction methods.

[0106] The feasibility of the proposed video memory prediction method based on a dual-stream multi-attention mechanism was verified by designing specific experiments, and the proposed method was quantitatively evaluated by evaluation indicators.

[0107] In designing the experiments for this invention, the publicly available VideoMem dataset was chosen. This dataset contains 8000 publicly available videos with memorization labels, and the model for this invention was built using the PyTorch framework. In the experiments, a 4-fold cross-validation method was used to divide the dataset. During training, the goal was to minimize the loss function, and stochastic gradient descent was used to update the network parameters. The batch size was set to 32, and the number of model iterations was set to 20 epochs. Furthermore, dropout technology was introduced in the multi-layer neural network to reduce computation and overfitting. After training, the set of network parameters with the most accurate prediction performance was selected as the output model based on the Spearman coefficient, a metric for measuring ranking relevance.

[0108] First, through long-term iterative training, the present invention was compared with some existing advanced methods to verify the superiority of the proposed method. The present invention aims to solve the problem of video memory prediction. In designing the comparative experiments, the algorithms referenced mainly came from video feature learning and image memory prediction research. To achieve a more comprehensive comparison, three different types of algorithms were introduced: learning algorithms based on video spatiotemporal features (C3D, HMP), where C3D features are spatiotemporal features extracted using 3D ConvNets, and HMP features are histograms encoding motion patterns of dynamic actions; traditional machine learning algorithms based on image memory (SVR, RFR, Lasso); and deep learning algorithms based on image memory (Hybrid-CNN, AMNet, MemNet).

[0109] Table 1

[0110]

[0111] Comparing the experimental results in Table 1, it can be seen that the method proposed in this invention outperforms other algorithms in predicting video memory scores. Specifically, the learning algorithms utilizing video spatiotemporal features C3D and HMP achieved poor ranking correlation. This is mainly because most videos in the dataset consist of simple scenes with minimal motion changes, making it difficult to extract complex action information. Consequently, the memory score regression network failed to effectively capture features highly correlated with memory scores. In contrast, machine learning and deep learning algorithms based on image memory scores achieved good prediction results, confirming that video memory scores are related to the memory scores of their constituent image frames to some extent. Analysis of the experimental results in Table 1 leads to the conclusion that the method proposed in this invention introduces multiple attention mechanisms, deeply encodes and aggregates the textual and visual modal features of the video, captures rich semantic information highly correlated with memory, and achieves superior prediction performance compared to other algorithms.

[0112] This invention, based on the fundamental structure of correlation between consecutive video frames, introduces a hierarchical attention module to process the visual information of the video, obtaining a condensed high-level semantic representation, thus achieving the effective and simplified processing of visual information. Furthermore, it utilizes an interactive aggregation network to capture complementary information between text and visual modalities, learning the differences and commonalities between dual-stream features to obtain a more complete common feature representation. In constructing the overall network framework, this invention introduces different forms of attention modules multiple times, combining deep learning and differential attention to achieve more accurate prediction results, particularly suitable for predicting video memorability scores, ultimately achieving results superior to existing methods.

[0113] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video memorability prediction method based on a two-stream multi-attention mechanism, characterized in that, include: A specific network structure is used to encode visual modal information and text modal information respectively, so as to obtain the encoded visual representation and the encoded text representation; The encoded visual representation is input into a hierarchical attention mechanism for deep learning to obtain an aggregated visual-level representation; An interactive aggregation network model is constructed, and the visual-level representation of the aggregation and the encoded text representation are input into the interactive aggregation network model to obtain an enhanced semantic representation of the video. The enhanced semantic representation of the video is input into the multi-head attention network module, which maps the enhanced semantic representation of the video to different subspaces. After obtaining the multi-head output, the memory score is calculated by the fully connected layer regression network. The method of inputting the aggregated visual-level representation and the encoded text representation into the interactive aggregation network model to obtain the enhanced semantic representation of the video includes: The correlation matrix between the aggregated visual-level representation and the encoded text representation is calculated by multiplying the feature vectors. By transposing the aggregated visual-level representation and the encoded text representation, multiplying them with the correlation matrix, and then performing vector concatenation operations, the resulting data is input into a multi-layer neural network to obtain fused common features. By combining the fused common features with the aggregated visual-level representation, an enhanced semantic representation of the video is obtained; The method for calculating the correlation matrix between the aggregated visual-level representation and the encoded text representation through feature vector multiplication is as follows: in, For the aggregated video-level representation, is the encoded text representation, Softmax is the normalization operation, and S is the correlation matrix between the aggregated visual-level representation and the encoded text representation; The method for obtaining the common characteristics of the fusion is as follows: in, For the common features of the fusion, Concat(·) is the concatenation operation between vectors, Γ(·) is a perceptron composed of multiple linear connection layers and nonlinear activation functions, and S is the correlation matrix between the aggregated visual-level representation and the encoded text representation. For the aggregated video-level representation, Θ represents the encoded text representation, and Θ represents the parameters to be learned in the multilayer neural network. The method for obtaining enhanced semantic representations of videos is as follows: in, For the common characteristics of integration, For the aggregated video-level representation, Enhanced semantic representation for videos.

2. The video memory prediction method based on a dual-stream multi-attention mechanism as described in claim 1, characterized in that, The method for encoding visual modal information and text modal information separately using specific network structures to obtain visual representations and text representations is as follows: in, For text modal information, For visual modal information, For the encoded text representation, θ represents the encoded visual representation, and θ represents the network parameters to be learned.

3. The video memory prediction method based on a two-stream multiple attention mechanism as described in claim 1, characterized in that, Methods for inputting the encoded visual representation into a hierarchical attention mechanism for deep learning to obtain aggregated visual-level representations include: Each individual frame-level feature in the encoded visual representation is encoded using a self-attention mechanism to obtain a self-attention encoded visual representation. The visual representation after self-attention encoding is divided into several local temporal blocks of preset length, and inner product attention encoding is applied to the frame-level features within the block to obtain the block-level attention-encoded features within the block. The block-level representation, after being encoded with intra-block attention, is aggregated into a visual representation by applying inner product attention modules between different blocks.

4. The video memory prediction method based on a two-stream multiple attention mechanism as described in claim 3, characterized in that, The method for encoding each individual frame-level feature in the encoded visual representation using a self-attention mechanism to obtain the self-attention encoded visual representation is as follows: Among them, f i For the first i Frame feature representation, For the first i The projection matrix corresponding to the frame. The corresponding first i A query, a key-value matrix, d For the dimension size of each frame, 1 / Softmax is for scaling operations, while Softmax is for normalization operations.

5. The video memory prediction method based on a two-stream multiple attention mechanism as described in claim 3, characterized in that, The visual representation after self-attention encoding is divided into several local temporal blocks of preset length. Inner product attention encoding is applied to the frame-level features within each block to obtain the block-level attention-encoded features. in, For block B i The first in k There are several frame-level feature vectors, where W1 and W2 are the mapping matrices of the network layers, and b1 and b2 are the bias vectors. It is the Sigmoid activation function. For the attention score calculated for each frame within the block, ⊙ represents the element-wise product operation, E i This is the block-level representation after aggregation.

6. The video memory prediction method based on a two-stream multiple attention mechanism as described in claim 3, characterized in that, The method for obtaining the aggregated visual-level representation by applying inner product attention modules between different blocks after encoding the intra-block attention at the block level is as follows: Among them, E i Let W3 and W4 represent the i-th block of the overall video, and b3 and b4 represent the mapping matrix and bias vector of the network layer, respectively. To calculate the attention score for each block, This is the aggregated video-level representation.

Images