Full Transform video abstraction method based on local-global attention

Abstract

The invention relates to a full Transform video abstraction method based on local-global attention, and the method employs a full Transform, employs an encoder-decoder structure, employs a supervised mode to carry out video abstraction, regards the video abstraction as a sequence-to-sequence learning problem, directly applies the full Transform to a video abstraction task, and improves the video abstraction efficiency. A full attention mechanism in an original Transform is replaced by a combination of local and global sparse attention, so that the model can capture a long-distance dependency relationship while reducing the calculation cost. The stacking mode of the encoder and the decoder is the same as that of an original Transform, and local-global sparse attention is only used in the encoder end. Experiments on two public multimedia reference data sets TVSum and SumMe show that the method can greatly exceed other video abstraction methods.

Description

Technical Field

[0001] The present invention relates to the technical field of video summary generation, and in particular to a full Transformer video summary method based on local-global attention. Background Art

[0002] With the popularity of various video capture, viewing and storage devices, as well as the widespread use of video sharing platforms (such as YouTube) and social media networks (such as Facebook), the amount of video data is growing at an unprecedented rate. This makes it extremely challenging to browse and find relevant content efficiently. To solve this thorny problem, automated and efficient video summarization techniques are becoming increasingly necessary. The essence of video summarization is to compress the video by selecting the most informative parts to create a summary that represents the original content. The created summary can be a static video storyboard consisting of a set of representative key frames, or a dynamic video preview consisting of key shots. In the work of this invention, we focus on creating summaries based on key shots for three main reasons. First, viewers are more inclined to watch dynamic video clips rather than static storyboards. Second, key shots contain diverse information and can highly represent the original video. Finally, in practice, videos are usually segmented into continuous and non-overlapping shots that maintain intrinsic visual-temporal consistency. These advantages ensure that the selected segments can effectively reflect the overall theme and storyline of the video, even if they account for a small proportion of the entire content, they can also bring a comfortable and interesting user experience. In addition, considering the challenges and inherent characteristics of general video summarization tasks, studying general single video summarization techniques is of great significance to the entire video summarization task. Therefore, we particularly focus on single video summarization techniques that can improve the ability of summarization models without introducing additional relevant signals or domain knowledge from multiple domain-specific videos.

[0003] So far, many influential video summarization techniques have been proposed and achieved remarkable success. These techniques include but are not limited to clustering methods, change detection methods, dictionary methods, and user attention methods, which belong to the scope of traditional non-learning methods. In addition, there are deep learning methods based on convolutional neural networks (CNNs), recurrent neural networks (RNNs), graph convolutional networks (GCNs), Transformers (but only in encoder architectures), attention mechanisms, etc. At present, more and more methods have adopted advanced deep neural network architectures for video summarization, and many empirical results show that their performance is better than traditional methods in most cases. At the same time, compared with unsupervised learning methods, supervised learning methods, especially those using CNNs or RNNs, are usually able to achieve better performance. This is because the supervisory signal from the true label is more beneficial to model training, and CNNs and RNNs have strong frame spatial information extraction capabilities and the ability to model temporal correlations between frames, respectively. However, CNNs have limitations in modeling global dependencies because the convolution operation relies on the deep stacking of multiple convolutional layers to capture long-range dependencies. RNNs themselves are not parallel because they use recursive structures to implement the calculation process in which the current input depends on the previous output.

[0004] At present, there are two main methods for generating video summaries: deep model-based video summaries and attention-based video summaries. Among them, deep model-based video summaries mainly include deep models without Transformer and deep models based on Transformer. Attention-based video summaries mainly include non-Transformer attention and Transformer attention. Specifically:

[0005] 1) Deep Models without Transformer, many techniques using CNN, RNN or their combination have been proposed for video summarization. The pioneering work first explored the bidirectional single-layer long short-term memory (LSTM) network for video summarization and achieved satisfactory results. Due to the limitation that single-layer LSTM cannot handle long videos well, Zhao et al. proposed a hierarchical structure in which the first layer encodes the dependencies between sub-shots, while the second layer uses bidirectional LSTM to better capture contextual information to select key sub-shots. The encouraging results prompted Zhao et al. to further improve the hierarchical structure based on video data. In order to handle large feature-to-implicit mapping matrices and long-range temporal dependencies, Zhao et al. proposed using tensor training embedding layers instead of hierarchical structures for video summarization. The above methods all use RNN or its variants for video summarization without exploring the application of Transformer in this task.

[0006] On the other hand, some methods propose to use CNN for video summarization. Rochan et al. first tried to explore the use of convolutional networks for video summarization and proposed a fully convolutional sequence model that increases the effective context size by stacking convolution operations, enabling the network to model long-distance dependencies and allowing limited parallelization. In addition, some works use a combination of CNN and RNN for video summarization. The key idea of ​​this research direction is how to model the spatiotemporal structure of the video. Although these methods using CNN, RNN or their combination have achieved great success in video summarization, they have inherent limitations in parallel computing.

[0007] 2) Transformer-based deep models: Transformer is usually used in three ways, namely encoder-only, decoder-only, and full encoder-decoder architecture, which are commonly used for understanding, generation, and conversion tasks respectively. Currently, the published methods for applying Transformer for video summarization mainly adopt the encoder-only architecture.

[0008] 3) Non-Transformer Attention: Non-Transformer attention mechanisms usually refer to attention used in conjunction with deep models (especially RNNs). This type of attention is used to focus on previously relevant positions by accumulating vectors in the corresponding task. This mechanism has shown significant advantages not only in natural language processing (NLP) tasks, such as machine translation and text summarization, but also in computer vision tasks such as video summarization.

[0009] 4) Transformer Attention: In this paper, we assume that Transformer attention adopts the self-attention mechanism. Its core feature is parallel computation. Some studies have adopted this attention mechanism for video summarization. Fajtl et al. used the self-attention mechanism to perform the entire sequence-to-sequence conversion in the video summarization task. Ji et al. combined the self-attention mechanism in the encoder to capture short-term contextual information. Li et al. proposed a global diversified attention mechanism, which estimates the diversified attention weights by adjusting the self-attention mechanism and then converts them into importance scores.

[0010] Recently, some works have been proposed that strictly follow the standard self-attention mechanism. Its core properties are scaled dot-product attention and multi-head attention. The former is used to model dependencies between all tokens in a sequence, while the latter allows the model to focus on different representation subspaces. Li et al. use this attention mechanism to explicitly model high-level patterns across videos for video co-summarization. Hsu et al. use temporal and spatial attention to achieve good summarization performance. This mechanism is standard self-attention and is only performed on the encoder side because its model is an encoder-only architecture. Although full attention can capture global dependencies, it will lead to missing local inductive bias. Summary of the invention

[0011] In view of the above problems existing in the prior art, the technical problem to be solved by the present invention is: how to generate a more accurate video summary using a full Transformer architecture.

[0012] To solve the above technical problems, the present invention adopts the following technical solution: a full Transformer video summarization method based on local-global attention, comprising the following steps:

[0013] S1: Obtain a training set, where the training set includes multiple original videos and their corresponding video summaries.

[0014] S2: Construct a summary generation model FullTransNet, which adopts the standard Transformer with a full encoder-decoder structure and uses the local-global sparse multi-head self-attention mechanism LGS-MHSA instead of full attention.

[0015] The encoder consists of N stacked encoder layers. Each encoder layer, except the first one, receives the output of the previous encoder layer as input. Each encoder layer performs a series of transformations on the input sequence to gradually extract higher-level semantic information suitable for summarization.

[0016] The decoder consists of N stacked decoder layers. The number of decoder layers is the same as that of the encoder layers. The input of the first decoder layer is the video summary and its position encoding. Except for the first decoder layer, the input of each decoder layer includes two parts: the output of the previous decoder layer and the context representation from the encoder; the context representation contains the inter-frame relationship information from the original video.

[0017] S3: Train FullTransNet to obtain the optimal FullTransNet. During the training process, the decoder uses the teacher forcing method, and the decoder uses the true value from the training set as input; the parameters of the encoder and decoder are updated by backpropagation according to the loss function, and the training ends when the paper loss no longer decreases.

[0018] S4: Input a new video into the optimal FullTransNet and output the video summary of the new video.

[0019] Furthermore, the encoder in S2 is:

[0020]

[0021] in, represents the positional encoding, which has the same embedding representation Z enc The same shape, representing the i-th encoder layer, represents the output of the i-1th encoder layer, T and d are the number of frames and dimension in the original video, respectively.

[0022] Each encoder layer has two sublayers: one is a multi-head self-attention layer with local-global sparse attention, and the other is a position-by-position fully connected feed-forward network FFN; the outputs of the two sublayers of the i-th encoder layer are denoted as and It is expressed as:

[0023] X′ i =LayerNorm(LGS-MHSA(X i ,X i ,X i )),#(2)

[0024] x″ i =LayerNorm(FFN(X′ i )+X′ i ),#(3)

[0025] in, represents the output of the i-1th encoder layer, that is,

[0026] FFN(X′ i )=max(0,X′ i W1+b1)w2+b2,#(4)

[0027] in, These are all learnable parameters.

[0028] Furthermore, the decoder in S2 is represented as:

[0029]

[0030] in, represents the output of the last encoder layer, represents the i-th decoder layer, represents the output of the i-1th decoder layer, Pdec ∈R L×d represents the positional encoding, whose shape is similar to the embedding representation Z dec same.

[0031] Each decoder layer is mainly composed of three sub-modules: masked multi-head self-attention MMHSA, i.e. causal attention, encoder-decoder attention CA, i.e. cross attention, and position-by-position fully connected feedforward network FFN. The outputs of the three sub-layers in the i-th decoder layer are expressed as:

[0032] S′ i =LayerNorm(MMHSA(S i ,S i ,S i )+S i ),#(6)

[0033]

[0034] S″′ i =LayerNorm(FFN(S″ i )+S″ i ),#(8)

[0035] in, represents the output of the i-1th decoder, that is,

[0036] Furthermore, in S2, the local-global sparse multi-head self-attention mechanism LGS-MHSA is:

[0037] The local-global sparse attention LGA is formulated as the following predefined pattern, and then the similarity score of the query-keyword pair can be calculated as follows:

[0038]

[0039] in, is the unnormalized attention matrix, m and n are the indices of the corresponding tokens, and w is the window size, and Represents the first frame, middle frame and last frame of each shot respectively.

[0040] The output of LGA is expressed as:

[0041]

[0042] in, represents the unnormalized attention matrix and V represents the value matrix.

[0043] LGS-MHSA projects the d-dimensional query Q, key K, and value V into d q d k and d v Dimensions, The LGA attention function, formula (10), is used to calculate the query after each projection. key Sum to generate d v The output value of the dimension, Then, all outputs are concatenated and projected back to the d-dimensional representation as shown in the following formula (11):

[0044] LGS-MHSA(Q,K,V)=Concat(H1,…,H h )W O ,#(11)

[0045] Among them, H1,H h denote the output of the 1st and hth head attentions, respectively, and W O Represents the weight matrix for linear transformation of the splicing result, h is the number of attention heads in FullTransNet,

[0046]

[0047] Furthermore, in S3, the loss function for training FullTransNet is:

[0048]

[0049] Where T represents the total number of original video frames, y n represents the true label of the nth frame, and the importance score is the probability p of selecting the key frame n .

[0050] Compared with the prior art, the present invention has at least the following advantages:

[0051] 1. Compared with existing methods, the full Transformer architecture is applied to video summary generation for the first time. The global self-attention mechanism effectively captures the global context and long-distance dependencies in the video, allowing the model to understand the video content more comprehensively, thereby generating more accurate and informative video summaries. The application of this architecture significantly improves the ability to generate video summaries, and has obvious advantages over traditional methods when processing complex video content.

[0052] 2. The introduction of a sparse attention mechanism (local-global sparse attention) to replace the traditional full attention mechanism significantly reduces the computational burden while maintaining high performance. By focusing attention on local areas and globally important positions, the sparse attention mechanism reduces the complexity of each layer of calculation, thereby reducing the computational resources and time consumption when processing long sequence videos. While maintaining the accuracy and information richness of the generated video summary, the sparse attention mechanism ensures that the performance level is comparable to that of the full attention mechanism. An F-Measure of 54.4% was achieved on the SumMe dataset and an F-Measure of 63.9% was achieved on the TVSum dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A simple diagram showing the motivation and main ideas of the FullTransNet architecture.

[0054] Figure 2 This is an overall overview of the FullTransNet architecture. DETAILED DESCRIPTION

[0055] The present invention is described in further detail below.

[0056] We formulate the video summarization problem as a sequence-to-sequence (seq2seq) learning problem. We address this problem with a standard Transformer with a full encoder-decoder structure. In addition, considering the efficiency of the model, we use local-global sparse attention instead of full attention to strike a balance between performance and efficiency. Our proposed model architecture, FullTransNet, uses these two techniques as main building blocks, such as Figure 2 shown.

[0057] A full Transformer video summarization method based on local-global attention, characterized by comprising the following steps:

[0058] S1: Obtain a training set, where the training set includes multiple original videos and their corresponding video summaries;

[0059] S2: Construct a summary generation model FullTransNet, which uses a standard Transformer with a full encoder-decoder structure and uses a local-global sparse multi-head self-attention mechanism LGS-MHSA instead of full attention;

[0060] The encoder consists of N stacked encoder layers. Each encoder layer, except the first one, receives the output of the previous encoder layer as input. Each encoder layer performs a series of transformations on the input sequence to gradually extract higher-level semantic information suitable for summary. When encoding a video (frame sequence), Transformer not only maps the sequence to a latent space, but also captures the dependencies between frames through an attention mechanism.

[0061] The decoder consists of N stacked decoder layers. The number of decoder layers is the same as that of the encoder layers. The input of the first decoder layer is the video summary and its position encoding. Except for the first decoder layer, the input of each decoder layer includes two parts: the output of the previous decoder layer and the context representation from the encoder; the context representation contains the inter-frame relationship information from the original video, and the decoder uses this information to effectively generate the target sequence.

[0062] There are two main purposes for using local-global sparse attention (LGA) instead of full attention. First, it is used to reduce memory usage and computational requirements and handle longer sequences. Second, it is expected to efficiently select representative frames. Since the computation of full attention grows with the square of the sequence length, which leads to serious scalability issues, we propose to use LGS-MHSA to address this limitation. Considering that most data, especially video data, are characterized by locality, it seems contradictory to directly use a softmax mapping on all tags, that is, assigning weights to all tags, because not all tags are relevant or worthy of attention. Therefore, it is natural to restrict each query to only focus on its neighbors. Based on this, we use local attention to normalize only the neighbor tags in the window w within the row, thereby assigning larger weights to these tags, which is very beneficial for selecting key frames.

[0063] LGA can introduce locality priors to alleviate overfitting on small-scale datasets. The local operation maintains translation invariance, allowing attention to focus on similar or identical information at different positions. In addition, when multiple local window attention layers are stacked, the top layer can access all input information in the entire sequence, forming an almost full receptive field similar to standard self-attention, which makes it closest to standard self-attention in performance. At the same time, the global attention in LGA is specifically designed to learn a global representation of all frames in a sequence, which can alleviate the decline in the ability to model long-distance dependencies when using local attention. LGA is a combination of local attention and global attention. Local attention, global attention, and their combination modes are shown in Figure 2. Figure 1 and Figure 2 shown.

[0064] The attention relationship of the query-keyword pair, and the connectivity matrix between all query-keyword pairs, such as Figure 1 and Figure 2 As shown. Local attention enables the model to focus on the neighborhood of each frame, which helps to quickly identify key frames. The attention calculation for each query only involves the elements within a small window around it, rather than the entire sequence. This means that key-value pairs outside the window can be effectively masked, thereby achieving local attention and greatly reducing computational cost.

[0065] To overcome the limitation of local attention in modeling long-range dependencies, we add global attention to several key positions. In our configured LGA, the first, middle, and last positions are selected as key positions. The elements at these key positions are regarded as global markers and can attend to all elements in the sequence, and vice versa, as Figure 1 and Figure 2 As shown in Figure 2. Global attention enables the model to learn a representation of the entire sequence, which is very helpful for the decoder to predict key frames.

[0066] S3: Train FullTransNet to obtain the optimal FullTransNet. During the training process, the decoder uses the teacher forcing method, and the decoder uses the true value from the training set as input; the parameters of the encoder and decoder are updated by backpropagation according to the loss function, and the training ends when the paper loss no longer decreases.

[0067] S4: Input a new video into the optimal FullTransNet and output the video summary of the new video.

[0068] Specifically, we apply linear projection to the features of each frame of video extracted from GoogLeNet to obtain embedded features. This operation is consistent with the embedding operation in the translation task and reduces the dimension of the video features from 1024 to d. Then, we add position encoding (the same as the encoding in the basic Transformer) to the network to make it aware of spatial position information. The structure of the first encoder layer is as follows Figure 2 (bottom).

[0069] The encoder in S2 is:

[0070]

[0071] in, represents the positional encoding, which has the same embedding representation Z enc The same shape, representing the i-th encoder layer, represents the output of the i-1th encoder layer, T and d are the number of frames and dimension in the original video, respectively.

[0072] Each encoder layer has two sublayers: one is a multi-head self-attention layer with local-global sparse attention, and the other is a position-by-position fully connected feed-forward network FFN; the outputs of the two sublayers of the i-th encoder layer are denoted as and It is expressed as:

[0073] X′ i =LayerNorm(LGS-MHSA(X i ,X i ,X i )),#(2)

[0074] X″ i =LayerNorm(FFN(X′ i )+X′ i ),#(3)

[0075] in, represents the output of the i-1th encoder layer, that is,

[0076] FFN(X′ i )=max(0,X′ i W1+b1)w2+b2,#(4)

[0077] in, These are all learnable parameters.

[0078] In this paper, the multi-head self-attention layer with local-global sparse attention is called local-global sparse multi-head self-attention mechanism (LGS-MHSA). In addition, following the standard Transformer, residual connections are used around each sub-layer and layer normalization is performed; LGS-MHSA refers to multi-head self-attention with local-global sparse mechanism, which is an updated version of the standard multi-head self-attention (MHSA) in which the local-global sparse mechanism is used to add locality priors to the original video and set some positions as global markers so that all markers in the sequence are attended to. In addition to the LGS-MHSA sub-layer, the same position-by-position fully connected feed-forward network FFN is applied to each position. FFN consists of two linear transformations with a ReLU activation function in the middle, which is the same as the standard Transformer.

[0079] Specifically, the decoder in S2 is represented as:

[0080]

[0081] in, represents the output of the last encoder layer, represents the i-th decoder layer, represents the output of the i-1th decoder layer, P dec ∈R L×d represents the positional encoding, whose shape is similar to the embedding representation Z dec The structure of the first layer decoder is as follows Figure 2 (bottom).

[0082] Each decoder layer mainly consists of three sub-modules: masked multi-head self-attention MMHSA, i.e. causal attention, encoder-decoder attention CA, i.e. cross attention, and position-by-position fully connected feedforward network FFN. Similar to the encoder, residual connections are used around each sub-layer, followed by layer normalization. Therefore, the outputs of the three sub-layers in the i-th decoder layer are expressed as:

[0083] S′ i =LayerNorm(MMHSA(S i ,S i ,S i )+S i ),#(6)

[0084]

[0085] S″′ i =LayerNorm(FFN(S″ i )+S″ i ),#(8)

[0086] in, represents the output of the i-1th decoder, that is, CA(.) receives input from the encoder and decoder, and MMHSA(.) allows each query Q to focus only on all keys k and values ​​V at the current query position and its previous positions. This is usually done by applying an unnormalized attention matrix This is done by applying a masking function, which allows for parallel computation during training. All values ​​at positions that do not require attention are set to -∞, i.e., when m <n, This masked self-attention ensures that information after the current position is not passed to the decoder, thus maintaining the autoregressive property of the video summarization task.

[0087] FullTransNet allows the decoder to access the entire output of the encoder, using the context of the input sequence, the original video, to help generate the next most likely key frame. The output of the last decoder layer is passed to a linear transformation to project its dimension d to the length T of the original video. Subsequently, a softmax function is applied to these linear projection values ​​to convert them into a probability distribution, thereby determining which frames are most likely to be selected as key frames.

[0088] Specifically, in S2, the local-global sparse multi-head self-attention mechanism LGS-MHSA is:

[0089] The local-global sparse attention LGA is formulated as the following predefined pattern, and then the similarity score of the query-keyword pair can be calculated as follows:

[0090]

[0091] in is the unnormalized attention matrix, m and n are the indices of the corresponding tokens, and w is the window size, and Represents the first frame, middle frame and last frame of each shot respectively.

[0092] The output of LGA is expressed as:

[0093]

[0094] in, represents the unnormalized attention matrix and V represents the value matrix.

[0095] LGS-MHSA allows the model to simultaneously focus on information from different representation subspaces at different locations. LGS-MHSA projects the d-dimensional query Q, key K, and value V into d q d k and d v Dimensions, The LGA attention function, formula (10), is used to calculate the query after each projection. key Sum to generate d v The output value of the dimension, Then, all outputs are concatenated and projected back to the d-dimensional representation as shown in the following formula (11):

[0096] LGS-MHSA(Q,K,V)=Concat(H1,…,H h )W O ,#(11)

[0097] Among them, H1,H h denote the output of the 1st and hth head attentions, respectively, and W O Represents the weight matrix for linear transformation of the concatenation result. h is the number of attention heads in FullTransNet,

[0098]

[0099] In the experiment, we set h = 8, and

[0100] In addition, it is worth noting that sparse attention is only implemented on the encoder side, since the length of the summary sequence is usually very small relative to the original input. The computational complexity of the local attention mode is O(n×m), which is linear in the length of the input frame sequence n. Although global attention is adopted in this work, the computational complexity of LGA is still O(n) because the number of these global tokens is relatively small and independent of n.

[0101] Specifically, in S3, the loss function for training FullTransNet is:

[0102]

[0103] Where T represents the total number of original video frames, y n represents the true label of the nth frame, which is obtained by converting the importance score into a shot-level binary label (0 and 1), where the importance score is the probability p of selecting a keyframe. n .

[0104] During training, the decoder uses a teacher forcing approach as the decoder leverages it to process summary sequences. At each time step, the decoder does not directly use its previous predictions, but instead considers the target output sequence from the training set, i.e., the true value, as input. This approach allows FullTransNet to effectively and accurately update its learning parameters based on the correct error backpropagation, which helps train a good model in a fast-converging manner so that the trained model generates the desired video summary. Teacher forcing helps the model learn how to generate the next token, while the mask mechanism ensures that the model does not "cheat" by peeking at future tokens during training. The combination of these two techniques enables the model to learn how to generate accurate target sequences more efficiently. In addition, implementing the LGA sparse mechanism is not trivial because local attention requires a form of banded matrix multiplication. To this end, we adopted a custom CUDA kernel to achieve this.

[0105] Experimental setup

[0106] 1. Dataset

[0107] We used two public benchmark datasets: SumMe and TVSum. SumMe contains 25 user videos ranging from 1 to 6 minutes in length, each of which provides a summary with multiple user annotations (provided by 15-18 different users) in the form of key shots. The dataset covers a variety of events, including first-person and third-person shots, such as holidays, cooking, and sports. TVSum contains 50 videos in 10 categories (5 videos per category), ranging from 2 to 10 minutes in length, annotated by 20 users in the form of frame-level importance scores. Similar to SumMe, this dataset also covers a variety of genres, such as beekeeping, sandwich making, and animal care. Since our method is based on supervised learning, more annotated data is helpful for training. Therefore, two annotated datasets with key frames are also used as augmented training datasets: OVP and YouTube. OVP contains 50 videos covering a variety of genres and ranging from 1 to 10 minutes in length, while YouTube consists of 39 videos with a variety of visual styles (excluding animations) and ranging from 1 to 4 minutes in length. The descriptions of the four datasets are shown in Table 1.

[0108] Table 1: Description of the datasets used in the experiments. “Frames” refers to the number of frames in a single shot.

[0109]

[0110] We apply three common settings, namely standard setting, enhanced setting and migration setting, to train and evaluate the model of the present invention. Specifically, in the standard setting, the training set and test set of SumMe and TVSum are all from the same dataset. Each dataset is randomly divided into 5 non-overlapping parts, 80% for training and the remaining 20% ​​for evaluation. In the enhanced setting, YouTube and OVP are added to the training set, and the test set remains unchanged. As for the migration setting, YouTube, OVP and TVSum (SumMe) are used as training sets, and SumMe (TVSum) is used as a test set.

[0111] Since our model requires real key shots, for the TVSum dataset which only provides frame-level importance scores, we convert the frame-level importance scores into key shots.

[0112] 2 Evaluation Metrics

[0113] F-Measure (i.e., F-score) is used as the evaluation indicator. Given a video V, V gt and V gs Represent the true summary and the generated summary respectively. Then according to V gt and V gs The precision P and recall R are calculated based on the time overlap length between them. The calculation method is as follows:

[0114]

[0115] Among them, V gt ∩V gs represents the time overlap between them, and represents the time duration. Then, the calculation formula of the harmonic mean F-Measure is as follows:

[0116]

[0117] A higher F-Measure indicates that the generated summary has more temporal overlap with the true summary while maintaining less redundancy.

[0118] 3. Implementation Settings

[0119] We train the FullTransNet of the present invention and its various ablation variants on an NVIDIA3090 graphics card equipped with 24GB of memory. The method of the present invention is implemented using Python 3.10 and PyTorch 2.0. We train all models using the Adam optimizer, with 300 training rounds, training from scratch for the corresponding datasets with different settings, with a batch size of 1, a learning rate of 1e-3, a momentum of 0.9, and a weight decay of 1e-4. It takes 5-8 hours to train a model. We set the size of the sliding window to 17, that is, each token will pay attention to 8 tokens on both sides of itself. Empirically, tokens in this attention span have been shown to be the most relevant. In addition, we set the feature dimension of the token to 64. The main purpose of doing this is to achieve a trade-off between computational efficiency and model complexity. In this task, since the maximum length of the video in all datasets is 1,513 frames, in order to accommodate all video frames and calculate the weight matrix, we set the sequence length to 1,536. For these sequences whose length is less than the maximum value, we fill “0” into the corresponding positions of these sequences to keep the length of each video sequence consistent.

[0120] In addition, all videos are downsampled from 30fps to 2fps to remove redundancy. We use the output of the pool5 layer of GoogLeNet pre-trained on ImageNet as the feature descriptor for each video frame. A five-fold cross validation is performed for each setting to obtain the average performance.

[0121] 4 Comparison with Existing Methods

[0122] The proposed FullTransNet is compared with several existing methods in terms of F-Measure on SumMe and TVSum datasets. Among these representative works, vsLSTM, dppLSTM, H-RNN, HSA-RNN, TTH-RNN and SUM-FCN mainly use RNN or CNN as the core construction technology, while A-AVS, M-AVS, vsLSTM+Att, dppLSTM+Att, SUM-GAN-AAE, DASP, VASNet, SUM-GDAsup, H-MAN, DMASum, VJMHT and STVT use attention mechanism as the key construction unit.

[0123] The quantitative comparison results are shown in Table 2. It can be seen that in almost all three common settings (i.e., canonical, augmented, and transfer settings), the summary performance of the model using the attention mechanism is always better than that of other models that do not use attention. This shows that the attention mechanism can effectively model long-range dependencies in video sequence data, which is crucial for improving the performance of video summarization tasks. In addition, a similar phenomenon is observed between non-Transformer and Transformer attention, where Transformer attention generally outperforms non-Transformer attention in summary performance. We believe that the reason may be that the self-attention mechanism does not require recursion and convolution operations, and can better learn representations in sequences than other attention mechanisms. More specifically, we can see that the FullTransNet of the present invention surpasses almost all existing methods on almost both datasets in all three settings, except DASP and DMASum. DASP only slightly improves by 0.4% on the TVSum dataset in the augmented setting, while the method of the present invention significantly surpasses DASP on SumMe in the canonical and augmented settings, with improvements of 8.9% and 7.6% respectively, and even improves by 0.3% in the augmented setting of TVSum. This clearly shows that the method of the present invention can achieve a good balanced performance on both datasets, verifying its effectiveness and advancement, while DASP may be a method for a specific dataset. DMASum improves by 1.2% and 1.3% on SumMe and TVSum respectively in the transfer setting. We speculate that FullTransNet may have over-learned these different types of video data due to its relatively high network capacity, resulting in lower generalization ability.

[0124] Table 2: F-Measure (%) of different methods under three settings. The best and second best results are marked in bold and underlined respectively. * indicates that these results are reproduced using the public code with the same GoogLeNet feature descriptor and downsampling strategy settings as other methods for fair comparison. Indicates that since we did not find the corresponding code, the results listed are from a published paper whose method uses the VGG-16 feature descriptor but adopts the same downsampling strategy as all other methods

[0125]

[0126] 4.5 Ablation Experiment

[0127] We conduct extensive ablation studies under canonical settings on both datasets to comprehensively evaluate the impact of each component in FullTransNet on the performance. Specifically, we study and analyze the impact of the local-global sparse attention (LGA) module, the number of encoder layers in LGA, the dimensionality of hidden layers in FFN, the dimensionality of frame embeddings, the number of heads in LGS-MHSA, the window size (WS) in local attention (LA), and the number of global tokens in global attention (GA). Unless otherwise stated, all settings are the same as mentioned in Section IV-C.

[0128] 1) Impact of Local-Global Sparse Attention Module: The main purpose of conducting this ablation study is to test the impact of different sparse attention modes (including LA, GA, and LGA) and compare the performance of sparse attention with full attention (FA) on summarization performance. The results are shown in Table 3. It can be seen that compared with LA and GA, FA generally produces better results, but its computational complexity is higher, resulting in significant computational and memory overhead. In addition, as a combination of LA and GA, LGA achieves an F-Measure of 54.4% and 63.9% on the SumMe and TVSum datasets, respectively, and has lower computational cost, shorter inference time, and less memory usage than FA. The excellent performance shows that the LGA module of the present invention has significant advantages in modeling long-term dependencies and saving computational and memory costs. This is very beneficial for improving the performance of long sequence modeling tasks such as video summarization and text summarization. This observation and finding is also consistent with the results of BigBird for text summarization. It is worth noting that FLOPs, running time, and memory are calculated on the 18th video of SumMe and the 45th video of TVSum, respectively. The former contains 149 frames, of which 44 frames perform global attention, while the latter contains 166 frames, of which 51 frames perform global attention.

[0129] Table 3: Comparison of F-Measure (%) of different attention modes on SumMe and TVSum. FA stands for global attention, LA and GA stand for local attention and global attention, respectively. LGA refers to the combination of local attention and global attention. Note that all attention modes are only adopted on the encoder side; all evaluations are performed under standard settings. Other parameters are the same as FullTransNet in Table 4.

[0130]

[0131] 2) Effect of the number of encoder layers on LGA: Empirically, the more encoder layers there are, the stronger the representation power of the model is generally, because more encoder layers are able to capture more complex and advanced features, while more decoder layers enable the model to consider more contextual information, leading to more accurate results. However, larger models require more computational cost and may lead to overfitting problems. Therefore, we conduct this ablation study to show how different numbers of layers affect model performance. In this ablation, the number of encoder and decoder layers is set to 2, 4, 6, and 8, respectively. From row (A) of Table 4, it can be observed that as the number of encoder and decoder layers increases, the performance on the SumMe dataset gradually improves, and the best result can be achieved by using FullTransNet with 6 layers in both the encoder and decoder. However, with the 8-layer setting, the performance drops significantly, indicating possible overfitting. Similar observations are made for the TVSum dataset, with the only exception that the model with 2 layers is slightly improved by 0.01% compared to the model with 4 layers. We believe this may be related to the dataset itself. The data distribution on the SumMe dataset may be more uniform, which is beneficial to model training. Therefore, a 6-layer encoder is used in all experiments.

[0132] Table 4: Comparison of F-Measure (%) of different attention modes on SumMe and TVSum. FA stands for global attention, LA and GA stand for local attention and global attention, respectively. LGA refers to the combination of local attention and global attention. Note that all attention modes are only adopted on the encoder side; all evaluations are performed under standard settings. Other parameters are the same as FullTransNet in Table 4.

[0133]

[0134] 3) Impact of frame embedding dimension: The dimension of frame embedding is critical to the performance of the model. We conducted an ablation study on the dimension of token embedding to evaluate the performance of the model under different dimension settings. Table 4 row (B) shows the results generated using five different dimensions (32, 64, 256, 512, and 1,024). It can be seen that the performance on both datasets is best when the dimension is set to 64, and after that, the performance gradually decreases as the dimension increases. We believe that this situation may lead to overfitting, so we set the dimension to 64 and use this setting in all experiments.

[0135] 4) Impact of hidden layer dimensions in FFN: In this ablation study, the main purpose of this paper is to explore the influence of different dimensions d ff How does the intermediate hidden layer of the feedforward network affect the performance of the network? We measure the performance by changing the number of neurons in the hidden layer. Table 4 row (C) shows the performance comparison of models with different hidden layer dimensions. It can be observed that when d ff The performance continues to improve when the dimension increases from 512 to 1,024, and the performance continues to improve when the dimension increases from 1,024 to 2,048. ff When d is fixed to 1024, other parameters may not be the best hyperparameters, causing the model to fall into a local optimum. On the contrary, when d ff When set to 2048, combined with other parameters, the model can learn the optimal model with good generalization ability. Therefore, in the FullTransNet model of the present invention, the hidden layer dimension of FFN is set to 2,048.

[0136] 5) Impact of the number of heads in LGS-MHSA: We will explore the impact of the number of heads in the local-global sparse multi-head self-attention (LGS-MHSA) mechanism on the performance. The relevant results are shown in row (D) of Table 4. It can be seen that as the number of heads increases, the performance on both datasets gradually improves, and the best results are achieved when 8 heads are used. This may be because more heads in LGS-MHSA can focus on information at different locations from different representation subspaces, thereby enhancing the representation ability of the model.

[0137] 6) Effect of Window Size in Local Attention: To demonstrate the importance of window size in local attention, we study the effect of different window size configurations (9, 17, 65, and 129) and report the experimental results in row (E) of Table 4. The results show that the best results are achieved on both SumMe and TVSum datasets using a window size of 17; the performance seems to gradually decrease when the window size is 65 and 129, while the best performance is only achieved on TVSum when the window size is 9. This observation suggests that modeling short-term dependencies is more critical for video summarization than modeling long-term dependencies. The reason may be that if the window size is too large, the attention may be scattered to many irrelevant tokens, thereby distracting from the key local information. This scattered attention will hinder the model from effectively capturing the information of important tokens, resulting in a decrease in accuracy. This hypothesis can be verified from the evidence in Table 2. It can be seen that the results of the local attention (LA) of the present invention on both datasets are comparable to or better than those of the full attention (FA). This is consistent with the common sense that for video summarization tasks, a keyframe is more correlated with its neighboring frames than with all frames in the entire sequence, especially for extractive summarization tasks. Therefore, in all experiments, we use a fixed configuration of window size WS=9.

[0138] 7) Effect of the number of tokens in global attention: The effect of the number of tokens in global attention is shown in row (F) of Table 4. It can be seen that as the number of tokens increases, the performance also improves. However, this does not mean that more tokens always lead to better performance. The evidence in Table 2 supports this view. We only use the first frame, the middle frame, and the last frame of the shot as global tokens. Here, the number of tokens is 1, 2, and 3, respectively, referring to using only the first frame, the first frame and the middle frame, and all three frames.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A full Transformer video summarization method based on local-global attention, characterized by: The steps include: S1: Obtain a training set, where the training set includes multiple original videos and their corresponding video summaries; S2: Construct a summary generation model FullTransNet, which uses a standard Transformer with a full encoder-decoder structure and uses a local-global sparse multi-head self-attention mechanism LGS-MHSA instead of full attention; The encoder consists of N stacked encoder layers. Each encoder layer, except the first one, receives the output of the previous encoder layer as input. Each encoder layer performs a series of transformations on the input sequence to gradually extract higher-level semantic information suitable for summarization. The decoder consists of N stacked decoder layers. The number of decoder layers is the same as that of encoder layers. The input of the first decoder layer is the video summary and its position encoding. Except for the first decoder layer, the input of each decoder layer includes two parts: the output of the previous decoder layer and the context representation from the encoder; the context representation contains the inter-frame relationship information from the original video. S3: Train FullTransNet to get the optimal FullTransNet. During the training process, the decoder uses the teacher forcing method, and the decoder uses the true value from the training set as input. The parameters of the encoder and decoder are updated according to the back propagation of the loss function. The training ends when the paper loss no longer decreases. S4: Input a new video into the optimal FullTransNet and output the video summary of the new video.

2. A full Transformer video summarization method based on local-global attention as claimed in claim 1, characterized in that: The encoder in S2 is: in, represents the positional encoding, which has the same embedding representation Z enc The same shape, representing the i-th encoder layer, represents the output of the i-1th encoder layer, T and d are the number of frames and dimensions in the original video respectively; Each encoder layer has two sublayers: one is a multi-head self-attention layer with local-global sparse attention, and the other is a position-by-position fully connected feed-forward network FFN; the outputs of the two sublayers of the i-th encoder layer are denoted as and It is expressed as: X′ i =LayerNorm(LGS-MHSA(X i ,X i ,X i )),#(2) X″ i =LayerNorm(FFN(X′ i )+X′ i ),#(3) in, represents the output of the i-1th encoder layer, that is, FFN(X′ i )=max(0,X′ i W1+b1)W2+b2, #(4) in, These are all learnable parameters.

3. A full Transformer video summarization method based on local-global attention as claimed in claim 2, characterized in that: The decoder in S2 is represented as: in, represents the output of the last encoder layer, represents the i-th decoder layer, represents the output of the i-1th decoder layer, P dec ∈R L×d represents the positional encoding, whose shape is similar to the embedding representation Z dec same; Each decoder layer is mainly composed of three sub-modules: masked multi-head self-attention MMHSA, i.e. causal attention, encoder-decoder attention CA, i.e. cross attention, and position-by-position fully connected feedforward network FFN. The outputs of the three sub-layers in the i-th decoder layer are expressed as: S′ i =LayerNorm(MMHSA(S i ,S i ,S i )+S i ),#(6) S″′ i =LayerNorm(FFN(S″ i )+S″ i ),#(8) in, represents the output of the i-1th decoder, that is, 4. The method for full Transformer video summarization based on local-global attention as claimed in claim 3, characterized in that: In S2, the local-global sparse multi-head self-attention mechanism LGS-MHSA is: The local-global sparse attention LGA is formulated as the following predefined pattern, and then the similarity score of the query-keyword pair can be calculated as follows: in, is the unnormalized attention matrix, m and n are the indices of the corresponding tokens, and w is the window size, and Respectively represent the first frame, middle frame and last frame of each shot; The output of LGA is expressed as: in, represents the unnormalized attention matrix, V represents the value matrix; LGS-MHSA projects the d-dimensional query Q, key K, and value V into d q d k and d v Dimensions, The LGA attention function, formula (10), is used to calculate the query after each projection. key Sum to generate d v The output value of the dimension, Then, all outputs are concatenated and projected back to the d-dimensional representation as shown in the following formula (11): LGS-MHSA(Q,K,V)=Concat(H1,…,H h )W O ,#(11) Among them, H1,H h denote the output of the 1st and hth head attentions, respectively, and W O Represents the weight matrix for linear transformation of the splicing result, h is the number of attention heads in FullTransNet, 5. The method for full Transformer video summarization based on local-global attention as claimed in claim 4, characterized in that: In S3, the loss function for training FullTransNet is: Where T represents the total number of original video frames, y n represents the true label of the nth frame, and the importance score is the probability p of selecting the key frame n .

Images