Transformer-Based Video Summarization Approach

Abstract

The invention provides a converter-based method for extracting video summaries. First, the selected data set is processed to obtain the training data set of the model; then, a video summarization converter neural network model including a self-attention mechanism is constructed, and the self-attention mechanism is used to calculate the similarity between video frames, and through Add the importance score of the previous frame to enhance the ability of the model to capture the global dependency of the video frame sequence, and use the training data set to train the model; finally, use the trained model to process the video data to be processed to obtain the importance of each frame Score, according to the score selection to get the video summary. The present invention can well capture the timing information between video frame sequences, and then can well predict the importance of video frames in the form of scores, and the model network of the present invention can parallelize frame sequences Training has the advantages of fast training timeliness and complete and short video summaries.

Description

technical field [0001] The invention belongs to the technical fields of computer vision and deep learning representation, and in particular relates to a converter-based video summarization method. Background technique [0002] With the rapid development of cameras and video sharing technologies, the number of videos is showing explosive growth. In the face of massive video data, how to efficiently extract useful information from videos has become an important issue. As an important technology to solve this problem, video summarization technology aims to generate a complete and short summary video for the original video. The summary video can convey the information expressed by the original video on the basis of short duration. Hot spots in vision and other fields. Video summary technology comprehensively uses machine learning, artificial intelligence and other technologies, and plays an important role in video retrieval, storage, recommendation and other aspects. [0003]...

AI Technical Summary

Problems solved by technology

For example, two LSTM networks are used in the literature "Ke Zhang, Wei-Lun Chao, FeiSha, et al. Video Summarization with Long Short-Term Memory[C] / / European Conference on Computer Vision. Springer, Cham, 2016.", one from Front to back, one extracts the sequence information of the video frame from the back to the front and predicts the importance score of the video frame. The network structure is simple and can extract key sequence information, but it is difficult for the cyclic neural network to capture long-term dependencies. When processing long video information, it is easy to lose early sequence dependencies; the document "Ji, Zhong, Xiong, Kailin, Pang, Yanwei, etal.Video Summarization with Attention-Based Encoder-Decoder Networks[J].2017." uses codec The codec structure is used to extract video key frames. Although the attention mechanism has been added and good results have been achieved, the codec still uses the LSTM network, and its complexity is related to the length of the video. There are problems of difficulty in parallel training and time-consuming

Images