Video summary generation method based on global multi-scale coding and local sparse attention

Abstract

The present invention discloses a video summary generation method based on global multi-scale coding and local sparse attention, which belongs to the field of computer vision technology. The method comprises: reading an input video, extracting a frame-level feature vector of the input video; constructing a video summary generation model, and inputting the frame-level feature vector into the video summary model; adaptively fusing the outputs of the global multi-scale coding and local sparse attention modules to generate fusion features; the fusion features are output through a regression network to output frame importance scores, representative frames are selected, and a video summary is generated. The video summary method proposed in the present invention can accurately identify key segments in a video by combining global multi-scale coding with a local sparse attention mechanism, thereby significantly improving video browsing efficiency and user experience. Experiments were conducted on the benchmark datasets SumMe and TVSum, and the experimental results fully demonstrated the effectiveness of the method.

Description

Technical Field

[0001] The present invention relates to a video summary generation method based on global multi-scale coding and local sparse attention, and belongs to the technical field of computer vision. Background Art

[0002] With the rapid development of multimedia technology, video data has exploded in numerous fields, including social media, surveillance and security, education and training, medical imaging, and news media. Users are faced with the challenge of quickly extracting key information from this massive amount of video data. Manually viewing and filtering long videos is not only time-consuming and labor-intensive, but also prone to missing important segments. Consequently, automated video summarization has become a key research direction in computer vision in recent years. It aims to automatically extract important frames or segments from videos through algorithms, generating concise and representative summaries to help users quickly understand the video content.

[0003] Early video summarization methods primarily relied on low-level visual features, such as color, texture, edges, and optical flow, selecting key frames by analyzing the distribution and changes of these features. However, these methods based on manual features lack an understanding of high-level semantics and contextual relationships, and can often only capture surface-level information, resulting in the generated summaries lacking content completeness and expressiveness. With the rise of deep learning, researchers have introduced models such as convolutional neural networks, recurrent neural networks, and long short-term memory networks to automatically learn frame importance scores, achieving more accurate summary generation through end-to-end training. These methods have significantly improved the model's ability to model high-level semantics, but still face many challenges when processing long videos.

[0004] First, existing deep learning-based methods are deficient in modeling long-range dependencies. Video is a time-series data, often with complex long-term dependencies between frames. Focusing solely on local information cannot fully understand the overall narrative structure of a video. While the self-attention mechanism offers a solution, its computational complexity increases quadratically with the length of the sequence, creating a computational bottleneck when processing long videos. Second, to improve efficiency, some models employ methods such as sampling or dimensionality reduction, but these methods can easily lose key local details, leading to reduced summary quality, especially in scenarios sensitive to short-term events, such as sporting events and security monitoring.

[0005] Therefore, how to comprehensively model global and local features, effectively capture long-term and short-term dependencies, and retain rich context and detail information while ensuring computational efficiency has become a key issue that urgently needs to be broken through in the current field of video summarization. Summary of the Invention

[0006] The purpose of this invention is to provide a video summary generation method based on global multi-scale coding and local sparse attention, aiming to solve the problem in the prior art that there is a trade-off between global dependency modeling and local detail capture, which leads to high computational complexity in long video processing and unsatisfactory summary quality.

[0007] To achieve the above objectives, the present invention provides a video summary generation method based on global multi-scale coding and local sparse attention, comprising the following steps:

[0008] S1: Read the input video and extract the frame-level feature vector of the video frame;

[0009] S2: Input the frame-level feature vector into the video summary generation model to predict the frame-level importance score. The video summary generation model includes:

[0010] Video feature optimizer: The video feature optimizer receives a frame-level feature vector, generates a global context vector by performing global average pooling, performs attention weighting on the global context vector and the frame-level feature vector, and outputs an optimized feature sequence;

[0011] Global multi-scale encoding module: The global multi-scale encoding module receives the optimized feature sequence, extracts global feature representation through a multi-head attention mechanism, and extracts local feature representation through a multi-scale depthwise separable convolution operation. The global semantic information in the video is extracted by fusing the global feature representation and the local feature representation.

[0012] Local block diagonal sparse attention module: The output of the global multi-scale encoding module is used as the input of the local block diagonal sparse attention module. The local block diagonal sparse attention module sparsifies the self-attention matrix into a block diagonal structure and models local dependencies by splicing attention weighted features, frame-level unique features, and inter-block diversity features.

[0013] S3: Adaptively weighted fusion of the output of the global multi-scale encoding module and the output of the local block diagonal sparse attention module to generate a fused feature representation;

[0014] S4: Input the fused feature representation into the regression network, output the frame-level importance score, select the most representative frame, and generate the final video summary.

[0015] Preferably, the video feature optimizer includes:

[0016] Perform global average pooling on the frame-level feature vector to generate a global context vector C. The specific calculation formula is as follows:

[0017] ,

[0018] in, represents the feature vector of the i-th frame, n is the number of video frames;

[0019] Perform attention weighting on the global context vector and the frame-level feature vector to generate a weighted feature sequence Y;

[0020] Perform layer normalization processing on the weighted feature sequence and output the optimized feature sequence Z. The specific calculation formula is: Z = LayerNorm(Y).

[0021] Preferably, the global multi-scale encoding module includes:

[0022] The multi-head attention mechanism is used to calculate the global feature representation G of the optimized feature sequence. The specific calculation formula is:

[0023] G = Concat(head1,…,head h )W G

[0024] Among them, head i = A i ’ V i ’ , A i ’ is the attention weight obtained by the global multi-scale encoding module by modeling the global inter-frame dependency through the multi-head self-attention mechanism, V ’ is the value matrix obtained by linear transformation, h is the number of attention heads, W G is the output mapping matrix;

[0025] A multi-scale depth-separable convolution operation is used to extract the local feature representation F. The specific calculation formula is as follows: F = F3+F5

[0026] Where F3 = Pointwise(D3(V ’ ))、F5 = Pointwise(D5(V ’ )), D k Represents a depthwise convolution operation with a kernel size of k, and Pointwise represents point-by-point convolution;

[0027] Finally, the global feature representation G and the local feature representation F are adaptively fused to generate the final global multi-scale encoding representation F global .

[0028] Preferably, the local block diagonal sparse attention module comprises:

[0029] Divide the self-attention matrix into non-overlapping local blocks, each of which contains a preset number of consecutive frames;

[0030] Splicing attention-weighted features, frame-level uniqueness features, and inter-block diversity features;

[0031] The concatenated features are fused through a linear layer and the local importance representation of the frame is output. The local importance representation of the frame is F local Calculated according to the following formula:

[0032] F local = Linear(Concat(A f ,U f ,D f ))

[0033] Among them, A f represents the attention weighted feature, U f represents the frame-level unique feature, D f Represents inter-block diversity features, Concat represents feature concatenation operations, and Linear represents linear layer fusion operations.

[0034] Preferably, the preset number of frames contained in each local block in the local block diagonal sparse attention module is 60 frames.

[0035] Preferably, the frame-level unique feature U in the local block diagonal sparse attention module f It is calculated by attention entropy, and the calculation formula is as follows:

[0036] ,

[0037] in, p i Indicates the i The attention weight of each frame, N represents the number of frames in the local block; the inter-block diversity feature D f It is calculated by cosine similarity, and the calculation formula is as follows:

[0038] ,

[0039] in, F i and F j denote the feature vectors of the i-th and j-th frames in the local block, respectively, (•) denotes the vector dot product, and || denotes the Euclidean norm of the vector.

[0040] Preferably, the calculation formula of the adaptive weighted fusion is as follows:

[0041] F fused = β × F global + (1-β) × F local

[0042] Among them, F fused represents the fusion feature representation, F global represents the output of the global multi-scale encoding module, F local represents the output of the local block diagonal sparse attention module, β is the adaptive weight coefficient, and satisfies 0<β<1.

[0043] Preferably, the regression network comprises the following layers: a first Dropout layer, a first normalization layer, a first fully connected layer, a ReLU activation layer, a second Dropout layer, a second normalization layer, a second fully connected layer and a Sigmoid activation layer.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. This invention introduces a global context vector through a video feature optimizer and dynamically adjusts the weights of frame-level feature vectors using an attention-weighted mechanism, effectively enhancing the representation of important frames and key features in a video. This design significantly improves the model's ability to focus on core content, allowing video summaries to be more focused on key information, thereby improving the representativeness and accuracy of the summaries.

[0046] 2. The global multi-scale encoding module of this invention combines a multi-head attention mechanism with multi-scale depthwise separable convolution operations to simultaneously capture global long-range dependencies and local short-term details in a video. This integrated modeling approach enables the generated summary to fully and meticulously reflect the semantic content of the video, avoiding the information loss associated with relying solely on a single feature extraction method.

[0047] 3. The local block diagonal sparse attention module of this invention significantly reduces computational complexity by sparsifying the self-attention matrix into a block diagonal structure, focusing only on non-overlapping local frame blocks. Furthermore, the introduction of frame-level uniqueness features and inter-block diversity features enhances the discriminative power of frame importance prediction, significantly improving the diversity and accuracy of summaries while also increasing computational efficiency.

[0048] 4. This paper employs an adaptive weighted fusion mechanism to fuse the outputs of the global multi-scale encoding module and the local block diagonal sparse attention module, and outputs frame-level importance scores through an efficient regression network. This design ensures that the generated summary retains important information while maintaining high compression and expressiveness, meeting the dual requirements of video summarization quality and efficiency in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0050] Figure 1 Flowchart for the implementation of the video summary generation method based on global multi-scale coding and local sparse attention provided by the present invention;

[0051] Figure 2 The overall framework diagram of the video summary generation method based on global multi-scale coding and local sparse attention provided by an embodiment of the present invention;

[0052] Figure 3 1 is a schematic structural diagram of a video feature optimizer provided by an embodiment of the present invention;

[0053] Figure 4 is a schematic structural diagram of a global multi-scale encoding module provided by an embodiment of the present invention;

[0054] Figure 5 2 is a schematic diagram of the structure of the local block diagonal sparse attention module provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several variations and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0056] like Figure 1 As shown in FIG, a flow chart of the implementation of the video summary generation method based on global multi-scale coding and local sparse attention provided by the present invention includes the following steps:

[0057] S1, reads the input video and extracts the frame-level feature vector of the input video;

[0058] S2, construct a video summary generation model and input the frame-level feature vector into the video summary model;

[0059] S3, adaptively fuses the output of the global multi-scale encoding and the local sparse attention module to generate fused features;

[0060] In S4, the fused features are passed through a regression network to output frame importance scores, representative frames are selected, and a video summary is generated.

[0061] Example 1:

[0062] The present invention provides a preferred embodiment for executing S1, which is applicable to video data of any length and type. The specific steps are as follows:

[0063] First, the original video is pre-sampled at a sampling rate of two frames per second to obtain a video frame sequence V={v1,v2,…, v n}, where v i represents the i-th frame image in the video, and n is the total number of frames in the video. Next, the GoogLeNet network pre-trained on ImageNet is used as a feature extractor to encode the visual features of each frame. Specifically, the 1024-dimensional feature vector output by the pool5 layer of the GoogLeNet network is extracted as the feature representation of the corresponding frame. Thus, the frame feature sequence X = [x1, x2,…, x n ], where x i The feature vector representing the i-th frame, n is the number of video frames, and is used to characterize the visual content contained in the frame.

[0064] Example 2:

[0065] The present invention provides a preferred embodiment to perform S2. The purpose of this embodiment is to use the designed video summary generation model to effectively balance the trade-off between global dependency modeling and local detail capture, thereby solving the problems of high computational complexity and poor summary quality in the existing technology when processing long videos, while improving the model's comprehensive understanding and representation capabilities of video content, thereby more accurately predicting the importance score of each frame. Figure 2 The figure shows the overall framework of the network model of this embodiment. The network consists of three parts: video feature optimizer, global multi-scale encoding module and local block diagonal sparse attention module. The specific construction steps of the three parts are as follows:

[0066] S21, construct a video feature optimizer. Figure 3 Specifically, the optimizer first receives the input feature sequence X = [x1, x2,…, x n ], where x i represents the feature vector of the i-th frame, and n is the number of video frames. The module first generates a global context vector C by performing global average pooling on all frame feature vectors. The calculation method is:

[0067] .

[0068] This global context vector captures the overall semantic information of the video and serves as the query vector in the subsequent attention mechanism to guide the modeling of inter-frame relationships.

[0069] Next, the module inputs the input feature sequence and the global context vector into the linear transformation layer through the attention mechanism to generate the key vector K, query vector Q and value vector V, specifically: K = W K X, Q = W Q C, V = W V X, where W K 、W Q 、W Vis a learnable weight matrix. To calculate the attention weight, the module uses the scaled dot product attention method, multiplying the query vector Q with the key vector K, scaling it, and normalizing it through the softmax function, that is:

[0070] ,in d Represents the dimensions of the query vector Q and key vector K.

[0071] In the implementation, the module has a fixed scaling factor of 0.03125.

[0072] After completing the attention weighting, the module calculates the weighted feature output Y, which is calculated as: Y = AV. To further improve the stability of the feature distribution, the module applies layer normalization to the weighted features. The normalization formula is: Z = LayerNorm(Y), and the final output is the optimized feature sequence Z with dimension n × d. This optimized feature sequence contains dynamic weighting information guided by the global context, which can effectively highlight important frame features in the video, providing more discriminative input for the subsequent global multi-scale encoding module and local block diagonal sparse attention module.

[0073] S22, construct a global multi-scale encoding module. Figure 4 The following is a schematic diagram of the global multi-scale encoding module. This module takes the optimized feature sequence Z as input, first undergoes a linear transformation operation, and generates query matrices for the multi-head self-attention mechanism. , key matrix Sum Matrix , specifically defined as follows:

[0074] .

[0075] Subsequently, the module uses a multi-head self-attention mechanism to model the global inter-frame dependencies, and the calculation formula of its attention weight is:

[0076] ,

[0077] in, d k For each attention head’s key vector dimension, the softmax operation is normalized along the last dimension to obtain the relative importance distribution between different frames. The output of the multi-head attention is integrated as follows:

[0078] G = Concat(head1,…,head h )W G

[0079] Among them head i = A i’ V i ’ , h represents the number of attention heads, W G is the output mapping matrix.

[0080] The module further uses multi-scale depth-separable convolution operation to extract local feature representation F. The specific calculation formula is as follows: F = F3+F5

[0081] Where F3 = Pointwise(D3(V ’ ))、F5 = Pointwise(D5(V ’ )), D k Represents a depthwise convolution operation with a kernel size of k, and Pointwise represents point-by-point convolution;

[0082] Finally, the global feature representation G and the local feature representation F are adaptively fused to generate the final global multi-scale encoding representation F global The specific calculation formula is F global = AFF(G,F), where AFF() represents the adaptive feature fusion function.

[0083] S23, build a local block diagonal sparse attention module. Figure 5 Figure 1 shows a schematic diagram of the local block diagonal sparse attention module. Specifically, the module divides the self-attention matrix into non-overlapping local blocks, each of which contains a preset number of consecutive frames.

[0084] In this embodiment, the preset number of frames contained in each local block is 60 frames, which is the optimal number of frames obtained based on experimental verification. It can fully capture the feature correlation between local frames while ensuring computational efficiency. Then, the attention weighted features, frame-level unique features and inter-block diversity features are spliced ​​together, where the frame-level unique features U f It is calculated by attention entropy, and the calculation formula is as follows:

[0085] ,

[0086] in, p i Indicates the i The attention weight of each frame, N represents the number of frames in the local block; the inter-block diversity feature D f It is calculated by cosine similarity, and the calculation formula is as follows:

[0087] ,

[0088] in, F i and Fj denote the feature vectors of the i-th and j-th frames in the local block, respectively, (•) denotes the vector dot product, and || denotes the Euclidean norm of the vector.

[0089] Finally, the concatenated features are fused through a linear layer to output the frame local importance representation, which is F local Calculated according to the following formula:

[0090] F local = Linear(Concat(A f ,U f ,D f ))

[0091] Among them, A f represents the attention weighted feature, U f represents the frame-level unique feature, D f Represents the inter-block diversity feature, Concat represents the feature concatenation operation, and Linear represents the linear layer fusion operation.

[0092] Example 3:

[0093] The present invention provides a preferred embodiment for performing S3. The output of the global multi-scale encoding module and the output of the local block diagonal sparse attention module are adaptively weighted fused to generate a fused feature representation. The specific calculation formula is as follows:

[0094] F fused = β × F global + (1-β) × F local

[0095] Among them, F fused represents the fusion feature representation, F global represents the output of the global multi-scale encoding module, F local represents the output of the local block diagonal sparse attention module, β is the adaptive weight coefficient, and satisfies 0<β<1.

[0096] Example 4:

[0097] The present invention provides a preferred embodiment for performing S4. In this embodiment, the regression network sequentially connects the following layers: a first dropout layer, a first normalization layer, a first fully connected layer, a ReLU activation layer, a second dropout layer, a second normalization layer, a second fully connected layer, and a Sigmoid activation layer. The specific calculation process is as follows:

[0098] ,

[0099] in, yrepresents the input features, Dropout_1 and Dropout_2 represent the first and second Dropout layers respectively, Norm_1 and Norm_2 represent the first and second normalization layers respectively, Linear_1 and Linear_2 represent the first and second fully connected layers respectively, ReLU represents the ReLU activation function, σ represents the Sigmoid activation function, Represents the frame-level importance score of the output.

[0100] To verify the effectiveness of the above embodiments, the present invention is applied in practice and compared with other advanced methods by calculating the F score (%). Specifically, the benchmark datasets SumMe and TVSum are used to evaluate the network.

[0101] The SumMe dataset contains 25 videos ranging in length from 1 to 6 minutes, covering a variety of scenarios and topics. Each video is manually annotated by 15 to 18 users, providing a high-quality reference standard for generating video summaries. The TVSum dataset is even larger, containing 50 videos ranging in length from 2 to 10 minutes. Each video in this dataset is annotated with frame-level importance scores by 20 users, providing rich detail and diversity for model training and evaluation.

[0102] During the experiments, we employed a standard 5-fold cross-validation (5FCV) method to ensure the robustness and reliability of our results. Specifically, all videos were evenly divided into five subsets. Four of these subsets were selected from each experiment as training sets for model training and optimization; the remaining subset was used as a test set to evaluate model performance. In this way, we conducted five independent experiments, each using a different subset as the test set, to ensure comprehensiveness and objectivity in our evaluation results. Finally, we took the average of these five experimental results as the final result to accurately reflect the performance of our method.

[0103] Table 1 Compared with the advanced methods Comparison results

[0104]

[0105] Experiments on the SumMe and TVSum datasets yielded experimental results, as shown in Table 1. Comparisons of the proposed method with other advanced methods demonstrate significant improvements in F-score, fully demonstrating its effectiveness and superiority in the task of video summarization. By introducing global multi-scale encoding and a local sparse attention mechanism, the proposed method can more accurately capture global semantic information and local detail features in a video, thereby generating a more representative and accurate video summary. Furthermore, the proposed model exhibits good stability and generalization capabilities during training and testing, and is adaptable to different types of video data, providing a reliable solution for video summary generation in practical applications.

Claims

1. A video summary generation method based on global multi-scale coding and local sparse attention, characterized by: The following steps are involved: S1: Read the input video and extract the frame-level feature vector of the video frame; S2: Input the frame-level feature vector into the video summary generation model to predict the frame-level importance score. The video summary generation model includes: Video feature optimizer: The video feature optimizer receives a frame-level feature vector, generates a global context vector by performing global average pooling, performs attention weighting on the global context vector and the frame-level feature vector, and outputs an optimized feature sequence; Global multi-scale encoding module: The global multi-scale encoding module receives the optimized feature sequence, extracts global feature representation through a multi-head attention mechanism, and extracts local feature representation through a multi-scale depthwise separable convolution operation. The global semantic information in the video is extracted by fusing the global feature representation and the local feature representation. Local block diagonal sparse attention module: The output of the global multi-scale encoding module is used as the input of the local block diagonal sparse attention module. The local block diagonal sparse attention module sparsifies the self-attention matrix into a block diagonal structure and models local dependencies by splicing attention weighted features, frame-level unique features, and inter-block diversity features. S3: Adaptively weighted fusion of the output of the global multi-scale encoding module and the output of the local block diagonal sparse attention module to generate a fused feature representation; S4: Input the fused feature representation into the regression network, output the frame-level importance score, select the most representative frame, and generate the final video summary.

2. The video summary generation method based on global multi-scale coding and local sparse attention according to claim 1, characterized in that The video feature optimizer comprises: Perform global average pooling on the frame-level feature vector to generate a global context vector C. The specific calculation formula is as follows: ， in, x i Indicates the i The feature vector of the frame, n is the number of video frames; Perform attention weighting on the global context vector and the frame-level feature vector to generate a weighted feature sequence Y; Perform layer normalization on the weighted feature sequence and output the optimized feature sequence Z. The specific calculation formula is: Z = LayerNorm(Y).

3. The video summary generation method based on global multi-scale coding and local sparse attention according to claim 1, characterized in that The global multi-scale encoding module includes: The multi-head attention mechanism is used to calculate the global feature representation G of the optimized feature sequence. The specific calculation formula is: G = Concat(head1,…,head h )W G Among them, head i = A i ’ V ’ , A i ’ is the attention weight obtained by the global multi-scale encoding module by modeling the global inter-frame dependency through the multi-head self-attention mechanism, V ’ is the value matrix obtained by linear transformation, h is the number of attention heads, W G is the output mapping matrix; A multi-scale depth-separable convolution operation is used to extract the local feature representation F. The specific calculation formula is as follows: F = F3+F5 Where F3 = Pointwise(D3(V ’ ))、F5 = Pointwise(D5(V ’ )), D k Represents a depthwise convolution operation with a kernel size of k, and Pointwise represents point-by-point convolution; Finally, the global feature representation G and the local feature representation F are adaptively fused to generate the final global multi-scale encoding representation F global .

4. The video summary generation method based on global multi-scale coding and local sparse attention according to claim 1, characterized in that: The local block diagonal sparse attention module divides the self-attention matrix into non-overlapping local blocks, each of which contains a preset number of consecutive frames; splices the attention weighted features, frame-level uniqueness features and inter-block diversity features; The concatenated features are fused through a linear layer and the local importance representation of the frame is output. The local importance representation of the frame is F local Calculated according to the following formula: F local = Linear(Concat(A f ,U f ,D f )) Among them, A f represents the attention weighted feature, U f represents the frame-level unique feature, D f Represents inter-block diversity features, Concat represents feature concatenation operations, and Linear represents linear layer fusion operations.

5. The video summary generation method based on global multi-scale coding and local sparse attention according to claim 4, characterized in that: The preset number of frames contained in each local block in the local block diagonal sparse attention module is 60 frames.

6. The video summary generation method based on global multi-scale coding and local sparse attention according to claim 4, characterized in that: The frame-level unique features U in the local block diagonal sparse attention module f It is calculated by attention entropy, and the calculation formula is as follows: ， in, p i Indicates the i The attention weight of each frame, N represents the number of frames in the local block; the inter-block diversity feature D f It is calculated by cosine similarity, and the calculation formula is as follows: ， in, F i and F j denote the feature vectors of the i-th and j-th frames in the local block, respectively, (•) denotes the vector dot product, and || denotes the Euclidean norm of the vector.

7. The video summary generation method based on global multi-scale coding and local sparse attention according to claim 1, characterized in that: The calculation formula of the adaptive weighted fusion is as follows: F fused =β×F global +(1-β)×F local Among them, F fused represents the fusion feature representation, F global represents the output of the global multi-scale encoding module, F local represents the output of the local block diagonal sparse attention module, β is the adaptive weight coefficient, and satisfies 0<β<1.

8. The video summary generation method based on global multi-scale coding and local sparse attention according to claim 1, characterized in that: The regression network includes the following layers: Dropout layer, normalization layer, fully connected layer, ReLU activation layer, Dropout layer, normalization layer, fully connected layer and Sigmoid activation layer.

Images