Video summary generation method based on global memory retrieval and dual-stream dynamic fusion

Abstract

The present invention discloses a method for generating a video summary based on global memory retrieval and dual-stream dynamic fusion, which belongs to the field of computer vision technology. The method comprises: extracting features from an input video frame sequence to obtain temporal video features; inputting the temporal video features into a video summary generation model, and predicting a frame-level importance score through the model; weightedly fusing the frame-level importance score with the center confidence to obtain a final frame-level importance score; performing temporal non-maximum suppression processing on the frame-level importance score to suppress redundant frames in adjacent high-scoring frames, selecting key frames based on the processed results, and generating a final video summary. The experimental results obtained on the benchmark datasets SumMe and TVSum demonstrate the advanced nature of the method. The video summary generation method proposed in the present invention can effectively improve the temporal modeling capability of long videos and the accuracy of key frame selection.

Description

Technical Field

[0001] The present invention relates to a video summary generation method based on global memory retrieval and dual-stream dynamic fusion, and belongs to the technical field of computer vision. Background Art

[0002] With the explosive growth of video data, video summarization has become a key research area in multimedia processing. In applications such as security surveillance, film and television production, and online education, traditional video browsing methods are no longer sufficient for efficient information acquisition, necessitating the development of intelligent summarization techniques. However, existing methods still suffer from significant deficiencies in keyframe selection accuracy and temporal modeling integrity.

[0003] Early video summarization methods were primarily based on hand-crafted features and heuristic rules. While simple to implement, such methods, such as color histogram-based keyframe selection and motion energy-based segment extraction, rely heavily on artificial prior knowledge and struggle to adapt to complex and changing video content. In particular, when processing videos containing multiple semantic events, they often fail to accurately identify key content, resulting in incomplete and inadequate summaries.

[0004] In recent years, deep learning-based video summarization methods have made significant progress. Traditional convolutional neural network (CNN) methods extract spatial features through local receptive fields but lack effective modeling of long-term temporal dependencies. Subsequent recurrent neural network (RNN) architectures, while improving temporal modeling capabilities, are limited by the vanishing gradient problem and struggle to capture long-range dependencies in videos. Experiments have shown that the temporal modeling efficiency of LSTM-based methods drops by over 35% when the video length exceeds 500 frames.

[0005] However, existing technologies still face several key challenges: First, when modeling very long videos (over 1,000 frames), it is difficult to ensure the integrity and continuity of temporal information; second, for complex scenes containing multiple parallel events, the semantic understanding depth of existing methods is still insufficient; finally, how to further improve processing efficiency while ensuring summary quality requires further research. Solving these problems will drive video summarization technology towards a smarter and more efficient direction. Summary of the Invention

[0006] In response to the problems existing in the existing methods, the purpose of the present invention is to provide a video summary generation method based on global memory retrieval and dual-stream dynamic fusion, aiming to solve the problems of the existing video summary methods in terms of the accuracy of key frame selection and the integrity of temporal modeling.

[0007] The present invention provides a video summary generation method based on global memory retrieval and dual-stream dynamic fusion, the method comprising the following steps:

[0008] S1: Extract features from the input video frame sequence to obtain video temporal features;

[0009] S2: Input the video temporal features into the video summary generation model, and use the model to predict the frame-level importance score. The video summary generation model includes:

[0010] Global memory retrieval module: The global memory retrieval module takes the video temporal features as input, performs attention retrieval through a learnable global memory matrix, and generates enhanced semantic features that integrate global semantics;

[0011] Dual-stream dynamic fusion module: The dual-stream dynamic fusion module takes the enhanced semantic features as input, extracts temporal saliency features through the forward convolution path and the backward convolution path respectively, and obtains temporal fusion features based on dynamic gating fusion;

[0012] Multi-task decision module: The multi-task decision module takes the temporal fusion features as input and generates frame-level importance scores, segment boundary offsets and center confidences in parallel through a multi-task prediction head;

[0013] S3: performing weighted fusion of the frame-level importance score and the center confidence to obtain a final frame-level importance score;

[0014] S4: performing temporal non-maximum suppression processing on the frame-level importance scores, suppressing redundant frames in adjacent high-scoring frames, selecting key frames based on the processed results, and generating a final video summary.

[0015] Preferably, the global memory retrieval module includes:

[0016] Constructing a learnable global memory matrix , where K is the memory capacity, is the memory dimension, Represents a real matrix with dimensions of m rows and n columns;

[0017] The video temporal features are transformed into Mapping to query vector , where T is the time series length and D is the input feature dimension;

[0018] Calculate the similarity matrix between the query vector Q and the global memory matrix M, and generate the attention weight through Softmax normalization ;

[0019] Perform feature retrieval on the global memory matrix M according to the attention weight A to generate global semantic features ;

[0020] Global semantic features By linear projection to dimension D, we get , and perform residual connection with video temporal features to output enhanced semantic features , the specific calculation formula is as follows:

[0021] ,

[0022] Where LayerNorm(·) represents the layer normalization operation, F g ' represents the feature of the global semantic feature after linear projection, F' represents the video temporal feature, and α is the preset residual weight with a value range of 0.1≤α≤0.5.

[0023] Preferably, the dual-stream dynamic fusion module includes:

[0024] Forward convolution path: Perform a one-dimensional convolution Conv1D operation on the input feature, and then pass through the first convolution layer with a convolution kernel size of 3 and half the number of channels, the ReLU activation function, and the second convolution layer to restore the original number of channels to obtain the feature F fw ;

[0025] Reverse convolution path: After flipping the input features along the time dimension, the convolution structure is processed using the same steps as the forward convolution path, and then flipped back to the original time order to obtain the feature F bw ;

[0026] Dynamic gate fusion: first combine the bidirectional feature F fw With F bw Splicing along the channel dimension , and then generate the time-adaptive gating weights through the fully connected layer and Sigmoid function ,in is a learnable parameter, T is the time series length, represents the Sigmoid function, b g represents the bias term;

[0027] Perform bidirectional feature fusion according to the gate weight to obtain the fusion feature ,in represents element-by-element multiplication, G is the gate weight;

[0028] The fusion feature F bi and enhanced semantic features F e Perform residual connection and output temporal fusion feature F t , the specific calculation formula is as follows:

[0029] ,

[0030] Where LayerNorm(·) represents the layer normalization operation, is the preset residual weight, the value range is 0.1≤ ≤0.5.

[0031] Preferably, the multi-task decision module includes:

[0032] Classification branch: Map the features to 1D through the fully connected layer, and output the frame-level importance score through the Sigmoid function , where T is the time series length;

[0033] Regression branch: Map the features to 2D through the fully connected layer, and output the non-negative segment boundary offset through the exponential function ;

[0034] Center branch: Map the features to 1D through the fully connected layer, and output the center confidence through the Sigmoid function .

[0035] Preferably, the weighted fusion satisfies:

[0036] ,

[0037] in is the final frame-level importance score, T is the time sequence length, is the frame-level importance score, is the central confidence, Represents element-wise multiplication.

[0038] Preferably, the global memory matrix is initialized as follows:

[0039] The dataset is randomly initialized using a normal distribution with a mean of 0 and a standard deviation of 0.02, and is layer-normalized.

[0040] Preferably, the similarity matrix calculation operation is as follows:

[0041] The query vector With the global memory matrix The transposed matrix of Perform matrix multiplication to generate a similarity matrix , where S t,k represents the similarity score between the t-th query vector and the k-th memory block;

[0042] The similarity score is calculated by vector dot product, i.e. ,in is the memory dimension.

[0043] Preferably, the Softmax normalization satisfies:

[0044] Apply the Softmax function to each row of the similarity matrix S so that the weight of each memory block is satisfy , the Softmax function is defined as:

[0045] ,

[0046] where exp(·) represents the natural exponential function, S t,k represents the similarity score between the t-th query vector and the k-th memory block, where K is the memory capacity.

[0047] Preferably, the feature retrieval operation is as follows:

[0048] The attention weight With the global memory matrix Perform matrix multiplication to obtain global semantic features ;

[0049] The feature retrieval is calculated according to the following rules:

[0050] ,

[0051] in represents the feature vector of the k-th memory block, is the memory dimension, K is the memory capacity, t represents the t-th query vector, Represents the weight of the t-th query vector and the k-th memory block.

[0052] Preferably, the weight generation method of the dynamic gating fusion is:

[0053] The input features are mapped to 1 dimension through the fully connected layer, and then the gate value in the [0,1] interval is generated by the Sigmoid function.

[0054] Compared with the prior art, the present invention has at least the following beneficial effects:

[0055] 1. The present invention provides a video summary generation method based on global memory retrieval and dual-stream dynamic fusion. The global memory retrieval module stores and retrieves semantic prototype features across video clips, and combines the dual-stream dynamic fusion module to adaptively fuse forward and reverse temporal features, significantly improving the temporal modeling capability of long videos and the accuracy of key frame selection.

[0056] 2. This paper proposes a global memory retrieval module that efficiently models the global semantics of a video using a learnable memory matrix and an attention mechanism. Compared to traditional local attention methods, this module is able to capture long-range cross-segment dependencies, reduce the false selection of redundant frames, and enhance the semantic coherence of the summary.

[0057] 3. This paper designs a two-stream dynamic fusion module that simultaneously models the local temporal dependencies and global context of the video through dynamic gated fusion of the forward and backward convolutional paths. This module significantly improves the model's robustness to complex scenes (such as rapid shot cuts and motion blur), ensuring that the generated summary covers the core content of the video.

[0058] 4. Experiments on the SumMe and TVSum benchmark datasets show that the proposed method has significantly improved the performance of video summary generation compared with the existing state-of-the-art methods, verifying its comprehensive advantages in summary quality, computational efficiency and generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0060] Figure 1 A flowchart of the video summary generation method based on global memory retrieval and dual-stream dynamic fusion provided by the present invention;

[0061] Figure 2 This is a diagram showing the overall framework of the video summary generation method based on global memory retrieval and dual-stream dynamic fusion provided by an embodiment of the present invention;

[0062] Figure 3 A schematic diagram of a global memory retrieval module provided by an embodiment of the present invention;

[0063] Figure 4 A schematic diagram of a dual-stream dynamic fusion module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several variations and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0065] like Figure 1 The figure shows a flow chart of the implementation of the video summary generation method based on global memory retrieval and dual-stream dynamic fusion provided by the present invention. This embodiment provides a video summary generation method based on global memory retrieval and dual-stream dynamic fusion, including the following steps:

[0066] S1: Extract features from the input video frame sequence to obtain video temporal features;

[0067] S2: Input the video temporal features into the video summary generation model, and use the model to predict the frame-level importance score;

[0068] S3: performing weighted fusion of the frame-level importance score and the center confidence to obtain a final frame-level importance score;

[0069] S4: performing temporal non-maximum suppression processing on the frame-level importance scores, suppressing redundant frames in adjacent high-scoring frames, selecting key frames based on the processed results, and generating a final video summary.

[0070] Example 1:

[0071] The present invention provides a preferred embodiment to perform S1 to extract features from the input video frame sequence to obtain video temporal features. The present invention can process videos of any length and any type of video, such as surveillance, movies, etc. The specific steps are as follows:

[0072] First, the original video is pre-sampled at a sampling rate of 2FPS to obtain a video frame sequence V = {v1,v2,..., v T}, where v i Denotes the i-th video frame, and T denotes the temporal length, i.e., the total number of video frames. Then, the GoogLeNet network pre-trained on the ImageNet dataset is used as a feature extractor to extract visual features for each frame. Specifically, the 1024-dimensional feature vector output by the pool5 layer of the GoogLeNet network is used as the feature representation of each frame, denoted as ,in The feature vector representing the i-th frame is used to characterize the visual content within the frame. These features can capture key information in the image, such as objects, scenes, and actions.

[0073] The extracted frame-level feature sequence Perform preprocessing:

[0074] 1. Layer Normalization LayerNorm: For each feature vector f i Normalize along the channel dimension so that the feature distribution satisfies the mean of 0 and the variance of 1;

[0075] 2. Dropout: Randomly set the elements in the feature vector to zero with a probability of 0.1 to suppress overfitting of the model.

[0076] The temporal features of the preprocessed video are recorded as , as the input of the subsequent global memory retrieval module.

[0077] Example 2:

[0078] The present invention provides a preferred embodiment for performing S2, inputting the video temporal features into a video summary generation model, and predicting the frame-level importance score through the model.

[0079] like Figure 2 The figure below shows the overall framework of the network model in this embodiment. The network consists of three parts: a global memory retrieval module, a dual-stream dynamic fusion module, and a multi-task decision module. The specific construction steps of each module are described below:

[0080] S21, build a global memory retrieval module, the schematic diagram of which is as follows Figure 3 Specifically, a learnable "memory" stores typical video scene and object features. For each frame's features, the module retrieves the most relevant information from the memory and fuses it to generate enhanced semantic features. For example, if a video contains "football match," the module retrieves features related to "sports" and "field" from the memory to enhance the representation of the current frame. This includes the following steps:

[0081] Construct memory matrix: Initialize the learnable global memory matrix , where K=128 is the memory capacity, is the memory dimension, sampled from a normal distribution N(0,0.02) and layer-normalized.

[0082] Attention Retrieval:

[0083] 1. Video temporal features are transformed into Mapping to query vector , where T is the time series length and D is the input feature dimension;

[0084] 2. The similarity matrix calculation between the query vector Q and the global memory matrix M is realized by matrix multiplication. With the global memory matrix The transposed matrix of Perform matrix multiplication to generate a similarity matrix , where S t,k represents the similarity score between the t-th query vector and the k-th memory block.

[0085] The similarity score is calculated by vector dot product, i.e. ,in This computation method is efficiently implemented through parallel matrix operations and is suitable for long video time series modeling.

[0086] Generate attention weights by Softmax normalization of the similarity matrix S , that is, applying the Softmax function to each row of the similarity matrix S, the Softmax function is defined as:

[0087] ,

[0088] where exp(·) represents the natural exponential function, S t,k represents the similarity score between the t-th query vector and the k-th memory block, where K is the memory capacity.

[0089] The weight of each memory block after normalization satisfy ,This step ensures that the model focuses on the most relevant memory blocks and ,suppresses irrelevant noise.

[0090] 3. Perform feature retrieval on the global memory matrix M based on the attention weight A, that is, perform matrix multiplication to generate global semantic features ,in represents the feature vector of the k-th memory block, is the memory dimension, K is the memory capacity, t represents the t-th query vector, Represents the weight of the t-th query vector and the k-th memory block.

[0091] This operation is essentially feature interpolation based on attention weights, which can dynamically fuse information from multiple memory blocks.

[0092] Residual fusion: The global semantic feature F g By linear projection to dimension D, we get , and perform residual connection with video temporal features to output enhanced semantic features , the specific calculation formula is as follows:

[0093] ,

[0094] Where LayerNorm(·) represents the layer normalization operation, F g ' represents the feature of the global semantic feature after linear projection, F' represents the video temporal feature, and α is the preset residual weight with a value range of 0.1≤α≤0.5.

[0095] S22, build a dual-stream dynamic fusion module, the schematic diagram of which is as follows Figure 4 Specifically, the forward convolutional path analyzes the relationship between frames according to the normal time sequence of the video to capture continuous actions, such as people walking and objects moving; the reverse convolutional path analyzes the video in reverse order to discover potential contextual connections, such as the causal relationship in dialogue scenes; and dynamic gated fusion automatically adjusts the weights of the forward and reverse paths based on the content. For example, for fast-action scenes, the forward path is given a higher weight, while for complex dialogue scenes, the reverse path is more important. The specific steps are as follows:

[0096] Forward convolution path: Perform a one-dimensional convolution Conv1D operation on the input feature, and then pass through the first convolution layer with a convolution kernel size of 3 and half the number of channels, the ReLU activation function, and the second convolution layer to restore the original number of channels to obtain the feature F fw ;

[0097] Reverse convolution path: After flipping the input features along the time series dimension, a one-dimensional convolution Conv1D operation is performed, which passes through the first convolution layer with a convolution kernel size of 3 and half the number of channels, the ReLU activation function, and the second convolution layer to restore the original number of channels, and then flips back to the original time series order to obtain the feature F bw ;

[0098] Dynamic Gating Fusion:

[0099] 1. Gating weight generation: First, the bidirectional feature F fw With F bw Splicing along the channel dimension , and then generate the time-adaptive gating weights through the fully connected layer and Sigmoid function ,in is a learnable parameter, T is the time series length, represents the Sigmoid function, represents the bias term.

[0100] 2. Feature fusion: Perform bidirectional feature fusion according to the gate weight to obtain fusion features , the formula is:

[0101] ,

[0102] in Represents element-by-element multiplication, G is the gate weight, when G t ≈1, the current time step is mainly forward features, which is suitable for continuous action scenes; when G t When ≈0, the backward features are dominant, which is suitable for scenarios that require reverse context understanding.

[0103] 3. Residual fusion: Fusion feature F bi Enhanced semantic features F with input e Perform residual connection and output temporal fusion feature F t , the specific calculation formula is as follows:

[0104] ,

[0105] Where LayerNorm(·) represents the layer normalization operation, and α is the preset residual weight, with a value range of 0.1≤α≤0.5.

[0106] S23 builds a multi-task decision module. This module is the final prediction layer of the video summarization system. Its core function is to predict three types of key information in parallel by sharing underlying features:

[0107] Classification branch: Map the features to 1D through the fully connected layer, and output the frame-level importance score through the Sigmoid function , represents the probability of each frame being selected as a key frame, where T is the time series length;

[0108] Regression branch: Map the features to 2D through the fully connected layer, and output the non-negative segment boundary offset through the exponential function , marking the start and end positions of key segments, supporting temporal non-maximum suppression NMS, helping to determine the window range of NMS processing, providing the candidate segment window range, avoiding the limitations of fixed window size, and adapting to the changes in segment length in different videos;

[0109] Center branch: Map the features to 1D through the fully connected layer, and output the center confidence through the Sigmoid function , indicating the credibility of the current frame as the core of the fragment.

[0110] This module achieves a balance between feature representation sharing and task-specific learning through multi-task collaborative training, significantly improving the model's understanding of video content. The specific multi-task loss function is designed as follows:

[0111] 1. Classification task: Use improved Focal Loss to solve category imbalance:

[0112] ,

[0113] in Manually label the importance of the labels.

[0114] 2. Regression task: Using DIoU Loss to enhance position sensitivity:

[0115] ,

[0116] in ρ is the Euclidean distance between the center of the predicted box and the true box, and c is the minimum diagonal length of the external box.

[0117] 3. Central task: using Gaussian weighted BCE Loss:

[0118] ,

[0119] in is the normalized distance from the current frame to the center of the fragment, .

[0120] Then perform dynamic weight adjustment and use Uncertainty Weighting to automatically adjust the loss weight:

[0121] ,

[0122] in are learnable parameters.

[0123] Example 3:

[0124] The present invention provides a preferred embodiment to perform S3, performing weighted fusion of the frame-level importance score and the center confidence to obtain a final frame-level importance score. The weighted fusion satisfies:

[0125] ,

[0126] in is the final frame-level importance score, T is the time sequence length, is the frame-level importance score, is the central confidence, Represents element-wise multiplication.

[0127] High-scoring frames typically represent the most representative content in the video, such as a highlight or important conversation. Temporal consistency processing is then performed to smooth the scores to avoid large fluctuations between adjacent frames. For example, if several consecutive frames have high scores, the system ensures that they are all retained.

[0128] Example 4:

[0129] The present invention provides a preferred embodiment of performing S4, performing temporal non-maximum suppression on the frame-level importance scores, suppressing redundant frames in adjacent high-scoring frames, selecting key frames based on the processed results, and generating a final video summary. Specifically, the following steps are included:

[0130] 1. Temporal non-maximum suppression (NMS) processing: An improved adaptive NMS algorithm is used to remove redundant frames. The specific operations are as follows:

[0131] Sliding window mechanism: setting dynamic window size , where T is the total number of video frames, and local extreme value detection is performed within the window.

[0132] Score the frames in each window , calculate the mean and standard deviation ;

[0133] Dynamic suppression threshold: ,in is the adjustment coefficient;

[0134] Keep only the score The rest are regarded as redundant frames.

[0135] 2. Keyframe selection optimization:

[0136] Clustering screening: After NMS processing, the candidate frames {v1,...,v m We extract ResNet-50 features and perform K-means clustering (K = number of summary frames / 2). We select the two frames with the highest scores from each cluster to ensure content diversity.

[0137] Knapsack problem modeling: Use the time series segmentation detection algorithm (KTS algorithm) to divide the video into N shots {shot1,...,shot N Each shot consists of a set of consecutive frames. Specifically, the maximum length of the summary cannot exceed 15% of the total length of the input video. A knapsack problem is then created and solved using a dynamic programming algorithm to obtain the optimal summary result. This problem can be mathematically expressed as:

[0138] ,

[0139] in Indicates whether to select the i-th lens, represents the length of the i-th lens, Indicates the 90th percentile of the frame score within the shot, is the total length of the video. Then, the selected shots are reorganized in chronological order to output the final dynamic video summary.

[0140] In order to verify the effectiveness of the above embodiments, the present invention is applied in practice, and the F score (%) is calculated to compare and analyze with existing advanced methods. Experiments are conducted on two benchmark datasets, SumMe and TVSum, to evaluate the performance of the present invention in terms of temporal modeling capabilities of long videos and the accuracy of key frame selection. The SumMe dataset contains 25 video samples covering multiple categories of scenes. Each video has a frame-level importance score provided by at least 15 annotators, which can effectively evaluate the consistency between the algorithm-generated summary and manual annotation. The TVSum dataset contains 50 selected videos from YouTube, covering 10 different categories such as news, documentaries, and travel. Each video is annotated by 20 independent annotators, which can comprehensively test the generalization ability of the algorithm under different video types.

[0141] To ensure the reliability of the experimental results, this study adopted a strict 5-fold cross-validation scheme, evenly dividing the data into 5 mutually exclusive subsets and performing rotation tests. Finally, the average value of the 5 experiments was taken as the evaluation result. This method significantly reduced the impact of data division bias on the experimental results.

[0142] Table 1 Compared with the advanced methods Comparison results

[0143]

[0144] In comparative experiments, this study selected FCSN, DR-DSN, A-AVS, M-AVS, and VASNet, currently mainstream video summarization methods, as baseline models. Under standard evaluation settings, as shown in Table 1, the proposed method demonstrated the best performance on both the SumMe and TVSum datasets. In particular, key metrics such as content coverage and temporal coherence were significantly improved compared to the baseline models, fully demonstrating the advanced nature and effectiveness of the proposed algorithm in video summarization tasks. Experimental results demonstrate that the proposed method not only accurately captures the key content of a video, but also adapts to the characteristics of different video types, generating summary results with stable quality.

[0145] The experimental results obtained by the method of the present invention on the SumMe and TVSum datasets demonstrate the advanced nature of the method, which is mainly due to the following innovative designs: First, the global memory retrieval module achieves enhanced representation of video semantic features through a learnable memory matrix, enabling the model to accurately identify key content from a global perspective; second, the two-stream dynamic fusion module captures local temporal patterns through the forward convolution path, combined with the reverse attention path to model long-range dependencies, significantly improving the model's ability to understand complex video structures. Compared with traditional methods such as VASNet and FCSN, the memory retrieval mechanism of the present invention can effectively alleviate the problem of information attenuation in long videos, while the dynamic gated fusion strategy realizes the intelligent weighting of multi-scale features, achieving significant breakthroughs in both key frame selection accuracy and summary coherence. Through end-to-end joint optimization, the model demonstrates strong adaptability to diverse video content while maintaining a real-time processing speed of 30fps.

Claims

1. A video summary generation method based on global memory retrieval and dual-stream dynamic fusion, characterized in that: The following steps are involved: S1: Extract features from the input video frame sequence to obtain video temporal features; S2: Input the video temporal features into the video summary generation model, and use the model to predict the frame-level importance score. The video summary generation model includes: Global memory retrieval module: The global memory retrieval module takes the video temporal features as input, performs attention retrieval through a learnable global memory matrix, and generates enhanced semantic features that integrate global semantics. The specific operation steps are as follows: Constructing a learnable global memory matrix , where K is the memory capacity, D m is the memory dimension, Represents a real matrix with dimensions of m rows and n columns; Attention Retrieval: The video temporal features are transformed into Mapping to query vector , where T is the time series length and D is the input feature dimension; Calculate the similarity matrix between the query vector Q and the global memory matrix M, and generate the attention weight through Softmax normalization ; Perform feature retrieval on the global memory matrix M according to the attention weight A to generate global semantic features ; The feature retrieval operation is as follows: The attention weight With the global memory matrix Perform matrix multiplication to obtain global semantic features ; The feature retrieval is calculated according to the following rules: ， in Denotes the feature vector of the kth memory block, D m is the memory dimension, K is the memory capacity, and t represents the t-th query vector; The global semantic feature F g By linear projection to dimension D, we get , and perform residual connection with the video temporal features to output enhanced semantic features F e , the specific calculation formula is as follows: ， Where LayerNorm(·) represents the layer normalization operation, F g ' represents the feature of the global semantic feature after linear projection, F' represents the video temporal feature, α is the preset residual weight, and the value range is 0.1≤α≤0.5; Dual-stream dynamic fusion module: The dual-stream dynamic fusion module takes the enhanced semantic features as input, extracts temporal saliency features through the forward convolution path and the backward convolution path respectively, and obtains temporal fusion features based on dynamic gating fusion; Multi-task decision module: The multi-task decision module takes the temporal fusion features as input and generates frame-level importance scores, segment boundary offsets and center confidences in parallel through a multi-task prediction head; S3: performing weighted fusion of the frame-level importance score and the center confidence to obtain a final frame-level importance score; S4: performing temporal non-maximum suppression processing on the frame-level importance scores, suppressing redundant frames in adjacent high-scoring frames, selecting key frames based on the processed results, and generating a final video summary.

2. The video summary generation method based on global memory retrieval and dual-stream dynamic fusion according to claim 1 is characterized in that: The dual-stream dynamic fusion module includes: Forward convolution path: Perform a one-dimensional convolution Conv1D operation on the input feature, and then pass through the first convolution layer with a convolution kernel size of 3 and half the number of channels, the ReLU activation function, and the second convolution layer to restore the original number of channels to obtain the feature F fw ; Reverse convolution path: After flipping the input features along the time dimension, the convolution structure is processed using the same steps as the forward convolution path, and then flipped back to the original time order to obtain the feature F bw ; Dynamic gate fusion: first combine the bidirectional feature F fw With F bw Splicing along the channel dimension , and then generate the time-adaptive gating weights through the fully connected layer and Sigmoid function ,in is a learnable parameter, T is the time series length, represents the Sigmoid function, b g represents the bias term; Perform bidirectional feature fusion according to the gate weight to obtain the fusion feature , in represents element-by-element multiplication, G is the gate weight; The fusion feature F bi and enhanced semantic features F e Perform residual connection and output temporal fusion feature F t , the specific calculation formula is as follows: ， Where LayerNorm(·) represents the layer normalization operation, and α is the preset residual weight, with a value range of 0.1≤α≤0.

5.

3. The video summary generation method based on global memory retrieval and dual-stream dynamic fusion according to claim 1 is characterized in that: The multi-task decision module includes: Classification branch: Map the features to 1D through the fully connected layer, and output the frame-level importance score through the Sigmoid function , where T is the time series length; Regression branch: Map the features to 2D through the fully connected layer, and output the non-negative segment boundary offset through the exponential function ; Center branch: Map the features to 1D through the fully connected layer, and output the center confidence through the Sigmoid function .

4. The video summary generation method based on global memory retrieval and dual-stream dynamic fusion according to claim 1 is characterized in that: The weighted fusion satisfies: ， in is the final frame-level importance score, T is the time series length, ⊙ represents element-by-element multiplication, P cls is the frame-level importance score, P ctr is the center confidence.

5. The video summary generation method based on global memory retrieval and dual-stream dynamic fusion according to claim 2 is characterized in that: The global memory matrix is initialized as follows: The dataset is randomly initialized using a normal distribution with a mean of 0 and a standard deviation of 0.02, and is layer-normalized.

6. The video summary generation method based on global memory retrieval and dual-stream dynamic fusion according to claim 2 is characterized in that: The similarity matrix calculation operation is as follows: The query vector With the global memory matrix The transposed matrix M T Perform matrix multiplication to generate a similarity matrix , where S t,k represents the similarity score between the t-th query vector and the k-th memory block; The similarity score is calculated by vector dot product, i.e. , where D m is the memory dimension.

7. The video summary generation method based on global memory retrieval and dual-stream dynamic fusion according to claim 2 is characterized in that: The Softmax normalization satisfies: Apply the Softmax function to each row of the similarity matrix S so that the weight A of each memory block is t,k satisfy , the Softmax function is defined as: ， where exp(·) represents the natural exponential function, S t,k represents the similarity score between the t-th query vector and the k-th memory block, where K is the memory capacity.

Images