A natural language guided video summarization method

Abstract

The present invention discloses a natural language guided video summarization method, comprising: decomposing a video file into a frame sequence, extracting frame image features; extracting frame semantic features and text semantic features, performing spatial cosine similarity calculation, and obtaining attention weights; building a natural language guided video summary model; performing model training on an overall network model; and reasonably selecting a video summary according to a frame importance score sequence. The present invention innovatively proposes a natural language guided attention mechanism for the video summary task, introduces the natural language guided attention mechanism into the video summary framework, and ablation experiments show that the natural language guided attention mechanism significantly improves the summary model performance. In addition, the natural language guided attention mechanism proposed in the present invention has stronger objectivity, and can fully focus on video clips related to the title text, and the title text is easily obtained in Internet videos and does not require any cost.

Description

Technical Field

[0001] The invention belongs to the technical field of video summarization, and in particular relates to a video summarization method guided by natural language. Background Art

[0002] With the rapid development of multimedia technology and network information technology in recent years, video has become an increasingly mainstream way of information exchange. How to condense a lengthy video into a few minutes or even a few seconds while maintaining key information - that is, video summarization - has gradually become an important research topic in the field of video technology. Video summarization technology uses computer algorithm programs to automatically select important clips from the video and use these important clips as video summaries, which not only reduces the storage space of the video, but also allows users to quickly browse the video information.

[0003] The idea of ​​​​mimicking human attention first appeared in the field of computer vision. By introducing an attention mechanism that only focuses on part of the image instead of the entire image, the computational complexity of image processing can be reduced and the performance can be improved. Similarly, each frame in a video has its own importance, and some of the clips contain key information about the video content, which we need to focus on when selecting summaries. Therefore, many researchers at home and abroad have introduced the attention mechanism into the video summarization method, assigning different importance weights to different frames in the input sequence, rather than treating all input frames equally, thus providing an intrinsic connection between the input video sequence and the output importance score.

[0004] If a video clip arouses the user's interest, then it is more likely to become an important clip in the entire sequence. Many researchers at home and abroad have modeled based on user attention. For example, Ma et al. proposed in the document [A user attention model for video summarization[C] / / Proceedings of the tenth ACM international conference on Multimedia.2002:533-542] to use low-level features such as motion changes and facial features to score the importance of video clips, and combine these scores to form an attention curve. The part on the peak of the curve is extracted as the key shot to construct the summary. With the development of deep learning technology, some deep video summarization methods based on attention mechanism have been proposed. For example, Zhong et al. proposed the AVS video summarization model in the literature [Video summarization with attention-based encoder–decoder networks[J].IEEE Transactions on Circuits and Systems for Video Technology, 2019, 30(6):1709-1717], which describes supervised video summarization as a sequence-to-sequence learning problem. They explored two decoding networks based on attention mechanism using additive and multiplicative objective functions, obtained attention weights through the properties of the video sequence itself, and learned the attention mechanism in a supervised manner. Apostolidis et al. proposed combining global and local multi-head attention mechanisms to discover frame dependency models of different granularities in the literature [Combining Global and Local Attention with Positional Encoding for Video Summarization[C] / / 2021IEEE International Symposium on Multimedia(ISM).IEEE, 2021:226-234]. The attention mechanism used integrates the components of encoding the temporal position of video frames.

[0005] However, it is not difficult to find that the existing deep video summarization methods based on attention mechanism generally only use video image information to generate summaries without considering information from other modalities. However, descriptive texts such as titles often contain key information that is highly relevant to the video content. Summary of the invention

[0006] In view of the above, the present invention provides a natural language guided video summarization method, and innovatively proposes a natural language guided attention mechanism for the video summarization task. The attention weight is obtained by calculating the similarity between the video sequence and the text, and the natural language guided attention mechanism is introduced into the Encoder-Decoder video summarization framework. The ablation experiment shows that the natural language guided attention mechanism significantly improves the performance of the summary model.

[0007] A natural language guided video summarization method comprises the following steps:

[0008] (1) Decompose the video files in the training set into frame sequences, use the pre-trained deep image network to extract image features from the frame sequences, and obtain the corresponding frame image feature sequences (f1,…,f n );

[0009] (2) The image coding network of the pre-trained multimodal model is used to extract semantic features from the frame sequence to obtain the frame semantic feature sequence (x1,…,x n ); extracting semantic features of the natural language text of the video file using the text encoding network of the pre-trained multimodal model to obtain text semantic features t;

[0010] (3) According to the frame semantic feature sequence (x1,…,x n ) and text semantic features t are calculated by spatial cosine similarity to obtain the attention weight sequence (α1,…,α n );

[0011] (4) Build a video summarization model based on the attention mechanism guided by natural language, whose input is the frame image feature sequence (f1,…,f n ) and the attention weight sequence (α1,…,α n ), the output is a sequence of frame importance scores (y1,…,y n ), and then train the video summary model;

[0012] (5) The frame image feature sequence (f1,…,f n ) and the attention weight sequence (α1,…,α n ) is input into the trained video summary model, and the frame importance score sequence (y1,…,y n );

[0013] (6) Using the frame importance score sequence (y1,…,y n ) Select key shots and synthesize them into video summaries.

[0014] Furthermore, since the natural language texts of a considerable number of video files in the training set cannot reflect their title content well, the natural language texts need to be optimized before semantic feature extraction is performed on the natural language texts of the video files: first, some titles are drafted based on the video content, and then multiple users are asked to watch the videos and select appropriate titles through questionnaires. Finally, the title with the highest user selection ratio is taken as the optimized natural language text.

[0015] Furthermore, the specific implementation of step (3) is as follows: first, the text semantic feature t and the frame semantic feature sequence (x1, ..., x n ) to obtain the cosine similarity between each feature vector in Then, similarity sequence Performing softmax normalization processing gives the attention weight sequence (α1,…,α n ); This attention weight sequence can reflect the importance of the video clip relative to the title text, providing an effective attention mechanism to guide the generation of summaries.

[0016] Furthermore, the video summary model is composed of an Encoder module, an Attention module, and a Decoder module connected in sequence. The Encoder module processes the input frame image feature sequence (f1, ..., f n ) is encoded to obtain the hidden sequence (h1,…,h n ), the Attention module uses the attention weight sequence (α1,…,α n ) for the hidden sequence (h1,…,h n ) is weighted and summed to obtain the fusion variable h. The Decoder module decodes the fusion variable h and outputs a frame importance score sequence (y1,…,y n ).

[0017] Furthermore, the process of training the video summary model in step (4) is as follows: first, the model parameters are initialized, and then the frame image feature sequence (f1, ..., f n ) and the attention weight sequence (α1,…,α n ) is input into the model, and the model predicts the output frame importance score sequence (y1,…,y n ), according to the loss function L, the model parameters are iteratively updated using the gradient descent and back propagation algorithms until the loss function L is minimized and converged or the maximum number of iterations is reached, and the training is completed.

[0018] Furthermore, the expression of the loss function L is as follows:

[0019]

[0020] Where: y i represents the importance score of the i-th video frame, s i represents the annotation score of the i-th video frame, and n is the total number of frames in the video file.

[0021] Furthermore, for the annotation score s of the i-th video frame i , let multiple users label the importance of the video frame, label 1 means important, label 0 means unimportant, then the label score s i That is, the percentage of users with label 1 to all users.

[0022] Furthermore, the specific implementation of step (6) is as follows: first, visually continuous frames are combined into shots, and then the frame importance score sequence (y1, ..., y n ) is converted into a sequence of shot importance scores, and the importance score of the shot can be obtained by averaging the importance scores of each frame in the shot; then the key shots are selected using the 0 / 1 backpack algorithm based on the shot importance score sequence and the shot length. The length of the key shots does not exceed 15% of the entire video length, and finally all the key shots are synthesized into a video summary.

[0023] Based on the above technical solution, the present invention has the following beneficial technical effects:

[0024] 1. After 150 epochs of training, the average F1-score of the five-fold cross-validation of the natural language-guided video summarization model is 47.0%. The present invention conducts an ablation experiment to remove the natural language guidance, and the average F1-score of the five-fold cross-validation of the video summarization model to remove the natural language guidance is 44.5%, indicating that natural language guidance improves the model performance.

[0025] 2. Compared with the user-based attention mechanism, the natural language-guided attention mechanism proposed in the present invention is more objective and can fully focus on the video clips related to the title text; at the same time, the title text is easy to obtain in Internet videos and does not require any cost, while the user's attention needs to be collected and analyzed and is not easy to obtain.

[0026] 3. While learning how humans summarize videos, the summary model pays full attention to the key content related to the natural language text. When a user provides a natural language text and a video, the summary generated by the video summarization method of the present invention has a strong correlation with the natural language text provided by the user, and can fully reflect the user's interests. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of attention weight generation based on natural language guidance in the present invention.

[0028] Figure 2 Schematic diagram of the structure of the Encoder-Decoder video summarization model based on the attention mechanism of the present invention. DETAILED DESCRIPTION

[0029] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0030] like Figure 1 As shown, the video summarization method based on natural language guidance of the present invention comprises the following steps:

[0031] S1: Decompose the video file into frame sequences, use the pre-trained deep image network to extract image features from the frame sequences, and obtain the frame image feature sequence (f1,…,f n ).

[0032] In this embodiment, the video is sampled into a frame sequence at a sampling rate of 2fps, and then the pool5 layer of the GoogLeNet model pre-trained on the large-scale image dataset ImageNet is used to extract the image features of the frame sequence to obtain the frame image feature sequence (f1,…,f n ), n represents the video length, and the feature dimension of each video frame is 1024.

[0033] S2: Use the image encoding network and text encoding network of the pre-trained multimodal model to extract semantic features from the frame sequence and natural language text, and obtain the frame semantic feature sequence (x1,…,x n ) and text semantic features t, and perform spatial cosine similarity calculation to obtain the attention weight sequence (α1,…,α n ).

[0034] This implementation uses the pre-trained CLIP image coding network and text coding network to extract semantic features from frame sequences and natural language text. During the training process, the CLIP model uniformly converts the feature space of the image coding network and the feature space of the text coding network into the semantic space. This good property provides us with the conditions for calculating the similarity between video frame images and text. The frame semantic feature sequence (x1,…,x n ) and text semantic feature t to calculate spatial cosine similarity Similarity_cosine and obtain the similarity sequence Will Doing softmax normalization can get the attention weight sequence (α1,…,α n ), the attention weight sequence obtained at this time can reflect the importance of the video clip relative to the title text, providing an effective attention mechanism to guide the generation of summaries.

[0035]

[0036]

[0037] S3: Build an Encoder-Decoder video summarization model based on the attention mechanism implemented by the LSTM network, and input the frame image feature sequence (f1,…,f n ) and the attention weight sequence (α1,…,α n ), output frame importance score sequence (y1,…,y n ).

[0038] The encoder and decoder in the model adopted in this embodiment can be implemented iteratively by convolutional neural networks, recurrent neural networks, etc. Considering that LSTM can well solve the problem of long-distance dependency and has certain advantages in the problem of long sequence modeling, the present invention selects LSTM for implementation; Figure 2 As shown in the figure, the Encoder-Decoder video summarization model based on the natural guided attention mechanism implemented by the LSTM network is divided into three modules: Encoder, Attention, and Decoder. The specific implementation methods of the three modules are as follows:

[0039] The Encoder module is implemented by n stacked LSTM units. The LSTM unit inputs the memory state c at the previous moment. t-1 and the current input f t , output the hidden state h at that moment t ; Encoder input frame feature sequence (f1,…,f n ), output hidden sequence (h1,…,h n ):

[0040] (h1,…,h n )=Encoder(f1,…,f n )

[0041]

[0042] The Attention module performs a multi-step analysis on the hidden sequence (h1,…,h n ) is weighted and accumulated, and the attention weight sequence is (α1,…,α n ), output variable h:

[0043] h=∑h i *α i

[0044] The decoder module is implemented by n stacked LSTM units, which input the fused variable h obtained above and output the importance score sequence (y1,…,y n ):

[0045] (y1,…,y n )=Decoder(h)

[0046]

[0047] S4: The overall network model is used for model training.

[0048] This implementation method selects the SumMe video summary dataset for training, but the titles of some videos in the SumMe dataset cannot reflect the title content well. For example, "Jumps" cannot fully reflect the video content, which will affect the training effect. Therefore, it is necessary to optimize the video title text of the SumMe dataset. Specifically: some titles are drafted according to the video content, and multiple users are asked to watch the video and select appropriate titles in the form of questionnaires. Finally, the one with the highest user selection ratio is selected as the training text. The original video titles of the SumMe dataset and the optimized titles provided by this method are shown in Table 1:

[0049] Table 1

[0050]

[0051]

[0052] The SumMe dataset has a total of 25 videos. The dataset is small in size and suitable for cross-validation training. This method makes full use of the 25 videos in the dataset and reduces the adverse effects of unbalanced partitioning. This implementation uses 80% of the data as the training set and 20% of the data as the test set. The five-fold cross-validation method is used. In each partitioning method, the number of test sets is 20 and the number of validation sets is 5. The model is trained using mean loss. (y1,…,y n ) is the model prediction score, (s1,…,s n ) is the training set annotation score, and the loss function is as follows:

[0053]

[0054] S5: According to the frame importance score sequence (y1,…,y n ), choose the video summary reasonably.

[0055] This implementation combines visually continuous frames into shots, and converts a frame importance score sequence into a shot importance score sequence, i.e., the importance scores of all frames in a shot are averaged to obtain a shot importance score. The shot importance score and shot length are then used to select key shots using a 0 / 1 backpack algorithm. The length of the key shot needs to be limited to 15% of the original video length, and the key shots are combined into a video summary.

[0056] The above description of the embodiments is to facilitate the understanding and application of the present invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments. Improvements and modifications made by those skilled in the art to the present invention based on the disclosure of the present invention should be within the protection scope of the present invention.

Claims

1. A natural language guided video summarization method, comprising the following steps: (1) Decompose the video files in the training set into frame sequences, use the pre-trained deep image network to extract image features from the frame sequences, and obtain the corresponding frame image feature sequences (f1, ..., f n ); (2) The image encoding network of the pre-trained multimodal model is used to extract semantic features from the frame sequence to obtain the frame semantic feature sequence (x1, ..., x n ); extracting semantic features of the natural language text of the video file using the text encoding network of the pre-trained multimodal model to obtain text semantic features t; (3) According to the frame semantic feature sequence (x1, ..., x n ) and text semantic features t are calculated by spatial cosine similarity to obtain the attention weight sequence (α1, ..., α n ); (4) Build a video summarization model based on the attention mechanism guided by natural language, whose input is the frame image feature sequence (f1, ..., f n ) and the attention weight sequence (α1, ..., α n ), the output is a sequence of frame importance scores (y1, ..., y n ), and then train the video summary model; (5) The frame image feature sequence (f1, ..., f n ) and the attention weight sequence (α1, ..., α n ) is input into the trained video summary model, and the frame importance score sequence (y1, ..., y n ); (6) Using the frame importance score sequence (y1, ..., y n ) Select key shots and synthesize them into video summaries.

2. The video summarization method according to claim 1, characterized in that: Since the natural language texts of a considerable number of video files in the training set cannot reflect their title content well, the natural language texts need to be optimized before semantic feature extraction is performed on the natural language texts of video files: first, some titles are drafted based on the video content, and then multiple users are asked to watch the videos and choose appropriate titles through questionnaires. Finally, the titles with the highest user selection ratio are taken as the optimized natural language texts.

3. The video summarization method according to claim 1, characterized in that: The specific implementation method of step (3) is as follows: firstly, the text semantic feature t and the frame semantic feature sequence (x1, ..., x n ) to obtain the cosine similarity between each feature vector in Then, similarity sequence Doing softmax normalization processing gives the attention weight sequence (α1, ..., α n ).

4. The video summarization method according to claim 1, characterized in that: The video summary model is composed of an Encoder module, an Attention module, and a Decoder module connected in sequence. The Encoder module processes the input frame image feature sequence (f1, ..., f n ) is encoded to obtain the hidden sequence (h1, ..., h n ), the Attention module uses the attention weight sequence (α1, ..., α n ) for the hidden sequence (h1, ..., h n ) is weighted and summed to obtain the fusion variable h. The Decoder module decodes the fusion variable h and outputs a frame importance score sequence (y1, ..., y n ).

5. The video summarization method according to claim 1, characterized in that: The process of training the video summary model in step (4) is as follows: first, the model parameters are initialized, and then the frame image feature sequence (f1, ..., f n ) and the attention weight sequence (α1, ..., α n ) is input into the model, and the model predicts the output frame importance score sequence (y1, ..., y n ), according to the loss function L, the model parameters are iteratively updated using the gradient descent and back propagation algorithms until the loss function L is minimized and converged or the maximum number of iterations is reached, and the training is completed.

6. The video summarization method according to claim 5, characterized in that: The expression of the loss function L is as follows: Where: y i represents the importance score of the i-th video frame, s i represents the annotation score of the i-th video frame, and n is the total number of frames in the video file.

7. The video summarization method according to claim 6, characterized in that: For the i-th video frame, the annotation score s i , let multiple users label the importance of the video frame, label 1 means important, label 0 means unimportant, then the label score s i That is, the percentage of users with label 1 to all users.

8. The video summarization method according to claim 1, characterized in that: The specific implementation of step (6) is as follows: first, visually continuous frames are combined into shots, and then the frame importance score sequence (y1, ..., y n ) is converted into a sequence of shot importance scores, and the importance score of the shot can be obtained by averaging the importance scores of each frame in the shot; then the key shots are selected using the 0 / 1 backpack algorithm based on the shot importance score sequence and the shot length. The length of the key shots does not exceed 15% of the entire video length, and finally all the key shots are synthesized into a video summary.

Images