Video abstract generation method and system based on graph-guided adaptive key frame sampling

Abstract

The invention discloses a video abstract generation method and system based on graph-guided adaptive key frame sampling, and relates to the technical field of video abstract generation, and the method comprises the steps: obtaining and extracting different scale features of each video frame, and generating multi-scale fusion features through the weighted fusion of scale attention weights; calculating inter-frame similarity to construct a weighted undirected graph, obtaining global structured frame features of each video frame through multi-layer feature propagation, and calculating graph guide weight of each frame; introducing a learnable distribution focusing parameter to perform power scaling on the graph guide weight to obtain an adaptive sampling weight, and calculating a final comprehensive weight; and according to the video frame length of the original video, calculating the number of sampling frames through a video length adaptive function, and according to the comprehensive weight and the number of sampling frames, sampling key frames by adopting a frame screening strategy based on sorting to generate a video abstract. According to the method, the video abstract with better coverage, diversity and stability balance can be generated.

Description

Technical Field

[0001] This invention relates to the field of video summarization technology, and in particular to a video summarization method and system based on graph-guided adaptive keyframe sampling. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of short video platforms, the amount of video data has exploded. How to extract key information from massive amounts of long videos and generate concise and accurate video summaries has become one of the most pressing technological challenges. Video summarization technology belongs to the fields of Natural Language Processing (NLP) and Multimedia Information Processing. It is a technology that identifies and extracts the core content of a video, compressing long videos into short video (or text) summaries, enabling users to understand the main information of a video in a shorter time.

[0004] Video summarization aims to extract a set of keyframes from long videos to preserve core semantics and narrative, while maintaining stability and efficiency under varying video lengths, scene changes, and computational budget constraints (i.e., frame count constraints). Existing video summarization technologies have certain shortcomings in keyframe selection: relying on a fixed sampling ratio, while simple, cannot adapt to videos of different lengths and content complexities, easily leading to oversampling in short videos and undersampling in long videos; non-uniform sampling methods driven by single-frame scores easily ignore the content and semantic relationships between frames, resulting in redundant keyframes and fragmented narratives. In other words, under different video lengths and variable scenes, it is necessary to simultaneously determine "how much to sample" and "which to sample" to achieve a stable trade-off between coverage, diversity, and coherence. This presents two challenges: first, it is necessary to adaptively determine the sampling budget and distribution pattern according to video length; second, it is necessary to characterize the global relationships between frames with a sense of relationship to avoid repetition and maintain coherence, and the output should also be stable and adjustable to meet the latency and resource constraints of practical systems.

[0005] Existing video summarization technologies largely rely on frame-by-frame scoring and ranking to determine "which to sample," combined with redundancy removal strategies such as similarity penalties, clustering, or coverage losses, supplemented by local temporal tools like non-maximum suppression, score smoothing, and self-attention. Regarding "how much to sample," they often employ random sampling methods such as fixed proportions, simple time-based rules, greedy or knapsack selection under a given budget, temperature or threshold adjustment, or reinforcement learning. Overall, existing methods tend to treat the budget as an external hyperparameter and limit relationship modeling to local or post-hoc stages. This budget is independent of the selection process, leading to inconsistent choices, a lack of unified control mechanisms, and difficulty in achieving a stable balance between coverage, diversity, and coherence. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a video summarization method and system based on graph-guided adaptive keyframe sampling. It utilizes global content and semantic relationships to generate a coherent importance distribution, and combines a sampling rate function that varies with video length and a controllable distribution focusing mechanism to unify the budget and selection process. This solves the problems of existing video summarization technologies, such as the inability to adapt to videos of different durations with a fixed sampling ratio, insufficient inter-frame correlation modeling leading to fragmented or repetitive keyframe selection, and the difficulty in balancing coverage, diversity, and coherence due to the independence of the sampling budget and keyframe selection process.

[0007] In a first aspect, the present invention provides a video summarization method based on graph-guided adaptive keyframe sampling.

[0008] A video summarization method based on graph-guided adaptive keyframe sampling includes: Obtain the raw video to extract the video frame sequence; Features at different scales are extracted from each video frame, and then weighted and fused by scale attention weights to generate multi-scale fused features. Based on the fusion features of video frame sequences, inter-frame similarity is calculated to construct a weighted undirected graph. Through multi-layer feature propagation, global structured frame features of each video frame are obtained, and graph guidance weights for each frame are calculated. A learnable distribution focusing parameter is introduced to scale the graph-guided weights exponentially to obtain adaptive sampling weights, which are then combined with the graph-guided weights to calculate the final comprehensive weights. Based on the video frame length of the original video, the number of sampled frames is calculated using a video length adaptive function. Then, based on the comprehensive weight and the number of sampled frames, a ranking-based frame filtering strategy is used to sample key frames and generate a video summary.

[0009] A further technical solution involves extracting different scale features from each video frame, weighting and fusing them using scale attention weights to generate multi-scale fused features, including: Low-level and high-level features at different scales are extracted from each video frame in the video frame sequence. Based on features at different scales, the scale attention weights for each scale feature are calculated using a lightweight mapping function. Based on the scale attention weight, features at different scales are weighted and fused to generate multi-scale fused features.

[0010] A further technical solution is that the calculation process of the graph guidance weight is as follows: Using each video frame as a node and the multi-scale fused features of each video frame as the initial features of the node, the inter-frame similarity is calculated to construct an adjacency matrix. Construct a weighted undirected graph based on the adjacency matrix. Among them, the set of nodes Let E represent video frames and E be the set of edges. The formula for calculating the similarity between frames is: ; Multi-layer feature propagation is performed on the graph structure of a weighted undirected graph to aggregate contextual information and learn global dependencies, thereby obtaining global structured frame features for each video frame. Based on the global structured frame features, the structured importance score for each video frame is calculated as follows: ; Based on the importance score, the probability distribution, i.e., the graph-guided weights, is obtained by normalization using the Sigmoid function: ; in, Indicates the first i , j Initial characteristics of video frames; This represents the global structured frame features of the i-th video frame. L Indicates the number of feature propagation layers; Indicates the first i The importance score of each video frame; , For hyperparameters; T This represents the number of video frames in the original video.

[0011] A further technical solution involves calculating the comprehensive weight as follows: Introducing learnable distribution focusing parameters Graph guided weights Power-law scaling yields adaptive sampling weights, expressed as: Where T is the number of video frames in the original video; Guided weights for graphs With adaptive sampling weights We perform weighted fusion to obtain a comprehensive weight that takes into account both global and local features.

[0012] A further technical solution involves smoothing the overall weights through a time-neighborhood averaging operation, resulting in smoothed weights, expressed as follows: ; in, The radius of the time window. This is the overall weight.

[0013] A further technical solution is that the video length adaptive function is: ; The formula for calculating the number of sampling frames is: ; in, For the Sigmoid function, , , , These are configurable hyperparameters. T This represents the number of video frames in the original video.

[0014] Secondly, the present invention provides a video summarization system based on graph-guided adaptive keyframe sampling.

[0015] A video summarization system based on graph-guided adaptive keyframe sampling includes: The video acquisition module is used to acquire raw video to extract video frame sequences; The video frame feature extraction module is used to extract features at different scales for each video frame, and then generate multi-scale fused features by weighted fusion of scale attention weights. The graph-guided keyframe modeling module is used to calculate the inter-frame similarity based on the fusion features of video frame sequences to construct a weighted undirected graph. Through multi-layer feature propagation, the global structured frame features of each video frame are obtained, and the graph-guided weights of each frame are calculated. The weight optimization module is used to introduce learnable distribution focusing parameters to scale the graph-guided weights exponentially to obtain adaptive sampling weights, and then combine them with the graph-guided weights to calculate the final comprehensive weights. The adaptive sampling module is used to calculate the number of sampled frames based on the video frame length of the original video using a video length adaptive function. Based on the comprehensive weight and the number of sampled frames, it uses a sorting-based frame filtering strategy to sample key frames and generate a video summary.

[0016] Thirdly, the present invention also provides an electronic device, comprising: a memory for storing executable instructions; and a processor for implementing the above-described video summarization method based on graph-guided adaptive keyframe sampling when executing the executable instructions stored in the memory.

[0017] Fourthly, the present invention also provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-described video summarization method based on graph-guided adaptive keyframe sampling.

[0018] Fifthly, the present invention also provides a computer program product comprising executable instructions stored in a computer-readable storage medium; wherein, when a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, it implements the above-described video summarization method based on graph-guided adaptive keyframe sampling.

[0019] The above one or more technical solutions have the following beneficial effects: 1. This invention proposes a video summarization method and system based on graph-guided adaptive keyframe sampling. The method adaptively determines the sampling quantity based on video duration, avoiding oversampling of short videos and undersampling of long videos. It utilizes global content and semantic association to guide keyframe selection, ensuring summary coverage while suppressing redundancy and maintaining plot coherence. Through a controllable distributed focusing mechanism, it achieves a flexible balance between distributed and concentrated sampling, adapting to different application scenarios. This invention solves the problems of existing video summarization technologies, such as the inability to adapt to videos of different durations with a fixed sampling ratio, insufficient inter-frame association modeling leading to fragmented or repetitive keyframe selection, and the difficulty in balancing coverage, diversity, and coherence due to the independence of the sampling budget and keyframe selection process. It enables the generated video summaries to achieve a stable and adjustable balance between coverage, diversity, and coherence.

[0020] 2. The graph-guided adaptive keyframe sampling mechanism proposed in this invention obtains the relation-aware importance distribution using graph-guided methods, combined with... The strategy integrates distributed focusing with adaptive budgeting for video length, providing a unified answer to the questions of "what to collect" and "how much to collect." This method can balance coverage, redundancy removal, and narrative coherence, and meets different scenarios and resource constraints through controllable focusing intensity. It is simple, end-to-end, easy to integrate with multiple features / backbone, and can also be transferred to tasks such as shot-level summarization, video retrieval, and highlight generation, making it more applicable.

[0021] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0023] Figure 1 This is a schematic diagram of the graph-guided adaptive keyframe sampling process proposed in an embodiment of the present invention; Figure 2The graphs show the prediction performance of the G-AFS model proposed in this invention under different scenarios; where (a) represents a long-term scenario and (b) represents a sudden change scenario. Figure 3 This is a schematic diagram illustrating the results of keyframe sampling using the G-AFS model and the existing score-based AFS strategy in an embodiment of the present invention. Detailed Implementation

[0024] It should be noted that the following detailed descriptions are exemplary and are intended only to describe specific embodiments and to provide further explanation of the invention, and are not intended to limit the scope of exemplary embodiments of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0025] Terminology Explanation NLP (Natural Language Processing): The core tasks of natural language processing include natural language understanding and natural language generation. AFS (Adaptive Keyframe Sampling): Adaptive keyframe sampling; LSTM (Long Short-Term Memory): Long Short-Term Memory network; MLP (Multilayer Perceptron): Multilayer perceptron; GGKM (Graph-Guided Keyframe Modeling): Graph-guided keyframe modeling; Google Net (Going deeper with convolutions google): A type of deep convolutional neural network.

[0026] Example 1 To achieve better video summarization, it is necessary to simultaneously determine "how much to sample" and "which to sample" to achieve a stable trade-off between coverage, diversity, and coherence. To this end, this embodiment provides a video summarization method based on graph-guided adaptive keyframe sampling (G-AFS). It utilizes global content and semantic relationships to generate a coherent importance distribution and combines a sampling rate function that varies with video length and a controllable distribution focusing mechanism to unify the budget and selection process. This effectively avoids oversampling of short videos and undersampling of long videos, and supports the sampling concentration requirements of different applications. It achieves a stable and adjustable balance between coverage, diversity, and coherence, resulting in better performance. This solves the problems of existing video summarization methods in keyframe selection, such as the fixed sampling ratio being difficult to adapt to different video lengths, insufficient inter-frame correlation modeling, and poor redundancy suppression.

[0027] The video summarization method based on graph-guided adaptive keyframe sampling (G-AFS) proposed in this embodiment, such as... Figure 1 As shown, this method follows a main thread that runs from "frame features - relationship modeling - adaptive sampling" to the end. First, the input video frames are processed by a CNN to extract time-aligned frame feature sequences. Then, a sparse graph is constructed based on inter-frame similarity and aggregated using SAGE Conv (spatial graph convolution), outputting a normalized importance distribution. This global relationship answers the question of "which to sample." Finally, an adaptive sampler processes this importance distribution... Sharpening, and through length perception rate Given a budget K (i.e., the number of sampling frames), keyframe selection is completed within the same pipeline. By linking "distribution + budget," the system answers "which to sample" and "how many to sample," forming a compact, learnable, and adjustable keyframe generation path. Specifically, the method proposed in this embodiment includes the following steps: Step S1: Obtain the original video to extract the video frame sequence; Step S2: Extract different scale features from each video frame, and generate multi-scale fusion features by weighted fusion of scale attention weights. Step S3: Based on the fusion features of the video frame sequence, calculate the inter-frame similarity to construct a weighted undirected graph. Through multi-layer feature propagation, obtain the global structured frame features of each video frame and calculate the graph guidance weight for each frame. Step S4: Introduce learnable distribution focusing parameters to scale the graph-guided weights exponentially to obtain adaptive sampling weights, and then combine them with the graph-guided weights to calculate the final comprehensive weights. Step S5: Based on the video frame length of the original video, calculate the number of sampled frames using a video length adaptive function. Based on the comprehensive weight and the number of sampled frames, use a sorting-based frame filtering strategy to sample key frames and generate a video summary.

[0028] The following content provides a more detailed introduction to the video summarization method based on graph-guided adaptive keyframe sampling proposed in this embodiment.

[0029] It needs to be clarified that the goal of video summarization is to select a subset from T frames within a limited budget to maximize the overall utility of "coverage, diversity, and coherence," which can be expressed as: Given a sequence of frame features: ; in, express d 3D feature space, d Dimension of frame features. Within the limited frame budget The next goal is to select a subset of indices from all frames: ; To maximize the overall utility function: ; Among them, coverage Ensure the summary includes key semantic events and exhibits diversity. Reduce information redundancy and improve consistency Maintaining chronological order and natural transitions in the plot. , , Indicates the weight.

[0030] Based on this, this embodiment proposes an adaptive keyframe sampling mechanism to address the problems of fixed sampling ratios being difficult to adapt to different video lengths and the lack of controllability in sampling distribution. Specifically, firstly, the initial importance score of the video frame is calculated through the aforementioned steps S2 and S3. The normalized weights of the frames are obtained through the Sigmoid function.

[0031] To further enhance the semantic integrity and robustness of inter-frame feature representation, this embodiment proposes a multi-scale semantic feature fusion module. This module aims to integrate visual features from different levels and receptive fields to simultaneously capture local details and global semantics of the video, providing more discriminative feature representations for subsequent keyframe modeling and sampling. Furthermore, to comprehensively model the semantic dependencies and content relationships between video frames and improve the global consistency and semantic coverage of keyframe selection, this embodiment proposes a Graph-Guided Keyframe Modeling (GGKM) module. This module constructs an inter-frame relationship graph and performs feature propagation, transforming local visual features into a global structured representation, providing a reliable weight distribution basis for subsequent adaptive sampling. The specific steps are as follows: Step S2.1: Extract low-level and high-level features at different scales from each video frame in the video frame sequence.

[0032] Specifically, let the video frame feature sequence be... We use existing feature extraction networks, such as Google Net, as the backbone network to extract low-level features at different scales from each video frame. and high-level characteristics .

[0033] Based on this, considering the low-level features in the video frame sequence High-level features typically contain rich texture, edge, and local motion information, while high-level features This approach focuses more on objects, scenes, and semantic structures. Single-level features often struggle to simultaneously capture local temporal details and global contextual semantics. Therefore, a multi-scale fusion mechanism is needed to obtain frame feature representations that combine precision and semantic consistency. To this end, this embodiment performs weighted fusion of features from different scales to obtain the fused semantic features, namely: ; in, Represents global features (such as the output of deep convolutions or graph embeddings). This represents local features (such as shallow convolution or optical flow features). and The scale weight matrix is ​​a learnable matrix. This is the bias term. This linear fusion process can be viewed as a joint reconstruction of multi-scale semantic channels, capable of adaptively allocating the importance of features at different scales during training.

[0034] As one implementation method, scale attention weights are also introduced to further enhance the dynamic selection capability of features. and The system achieves adaptive weighting of features at different scales through soft normalization. Specifically, in step S2.2, based on features at different scales, the scale attention weight of each scale feature is calculated using a lightweight mapping function. Based on the scale attention weight, the features at different scales are weighted and fused to generate multi-scale fused features.

[0035] The formula for calculating the scale attention weight is as follows: ; In the above formula, It is a lightweight mapping function, such as a linear transformation or a 1×1 convolution, and satisfies .

[0036] Based on this, the final multi-scale fusion feature representation can be obtained as follows: ; The above formula not only achieves feature weighted fusion at the channel level, but also introduces dynamic adaptive selection at the semantic level, enabling the model to automatically adjust the proportion of attention to global and local features according to the complexity of video content.

[0037] In step S3, a graph-guided keyframe modeling module is introduced to fuse features. Input is fed into the Graph Guided Keyframe Modeling (GGKM) module to generate structured embeddings. This allows the model to maintain consistency and complementarity of multi-scale information in the frame-level semantic space. Experiments show that this module can significantly improve the model's adaptability to scene changes, object motion, and semantic transitions, providing a more robust feature foundation for subsequent adaptive keyframe sampling.

[0038] In the Graph Guided Keyframe Modeling (GGKM) module, to capture the content associations between frames, each video frame is first used as a node, and the multi-scale fused features of each video frame are used as the initial features of the nodes. The inter-frame similarity is then calculated to construct an adjacency matrix. Construct a weighted undirected graph based on the adjacency matrix. , where the set of nodes Let E represent video frames, E be the set of edges, and E be the adjacency matrix. Defined by inter-frame similarity, it is expressed as: ; Secondly, to enhance the sparsity of the graph structure, a threshold or - The nearest neighbor strategy retains only inter-frame connections with high similarity. In this embodiment, multi-layer feature propagation is performed on the graph structure to aggregate contextual information and learn global dependencies. Let the initial features of each node be... Then the first The layer update formula is: ; in, Represents a node The neighborhood group, , For learnable parameters, This is a nonlinear activation function (such as ReLU). After... After layer propagation, a globally structured frame feature representation is obtained, as follows: ; in, This represents the graph convolution operation in layer L.

[0039] Based on the above global representation, the structured importance score for each frame is calculated as follows: ; Based on this importance score, the probability distribution, i.e., the graph-guided weights, is obtained by normalization using the Sigmoid function: ; in, Indicates the first i, j Initial characteristics of video frames; Indicates the first i Global structured frame features of video frames. L Indicates the number of feature propagation layers; Indicates the first i The importance score of each video frame; , For hyperparameters; T This represents the number of video frames in the original video.

[0040] The aforementioned weight distribution comprehensively considers global semantic dependence, structural centrality, and local saliency between frames, thus providing semantically consistent and hierarchical weight priors for the subsequent adaptive keyframe sampling module.

[0041] After completing frame-level weight prediction and temporal constraint optimization, the model needs to convert the generated frame importance distribution into the final video summary result. That is, based on the fused weight distribution, key frames are selected, their temporal structure is smoothed, and a video summary that meets the duration constraint is generated.

[0042] Specifically, to achieve flexible adjustment of the sampling distribution, a learnable distribution focusing parameter is introduced through step S4 above. The normalized weights are then scaled exponentially to obtain the adaptive sampling weights, which are: ; Among them, when When sampling is concentrated on high-importance frames, a stronger focusing effect is achieved; when At that time, the sampling distribution is smoother, promoting increased diversity; this parameter It can learn automatically during training, enabling the model to adaptively balance coverage and diversity in different scenarios.

[0043] Then guide the weights of the graph With adaptive sampling weights The fusion process is performed to obtain a final weight distribution that takes into account both global and local features, as follows: ; in, This is a balancing parameter used to control the proportion of global semantic information and local saliency in the final selection. When When the size is large, the abstract places more emphasis on semantic consistency; when When the size is smaller, more emphasis is placed on salience and diversity.

[0044] In step S5, to address the mismatch in fixed budget caused by different video lengths, a video length adaptive function is introduced to calculate the number of sampling frames, expressed as: ; in, For the Sigmoid function, , , , This is a configurable hyperparameter, where T is the number of video frames in the original video.

[0045] Subsequently, the number of frames selected is determined based on the video length adaptive budget function. According to The number of samples, i.e. the number of frames, is dynamically determined to ensure that short videos are not oversampled and long videos are not undersampled, thus achieving length-adaptive sampling control.

[0046] Finally, based on the comprehensive weight and the number of sampled frames, a sorting-based frame filtering strategy is used to sample key frames, as follows: ; in, This indicates selecting the one with the highest weight. Count the frame indices to obtain the candidate keyframe set. Generate video summaries.

[0047] As one implementation method, to improve the visual coherence and natural playback of the summary, a post-processing mechanism of temporal smoothing and redundancy suppression can be introduced. This involves smoothing the weight distribution through temporal neighborhood averaging, as shown below: ; in, For comprehensive weighting, The time window radius (typically 3–5 frames) can reduce the impact of isolated frames or local noise on the abstracting results, making the abstract frame sequence more coherent.

[0048] Finally, the smoothed weights are reordered and keyframes are selected: ; And based on the index set Extract the corresponding frame sequence to form the final summary, which is: ; Preferably, to generate a user-visual summary video, adjacent keyframes can be further merged into consecutive segments. Specifically, if adjacent frames are like the first... i, j The spacing between frames is less than a set threshold. If we consider them as the same segment, it can be represented as: ; The same segment mentioned above can be merged into a continuous segment, thereby reducing the problem of excessive fragmentation while maintaining semantic integrity.

[0049] By employing the aforementioned fusion weights, temporal smoothing, and fragmentation optimization, the coherence of the summaries at both the visual and semantic levels is effectively improved. Experiments show that this approach can significantly enhance the readability and consistency of the model's summaries across multiple datasets, achieving a superior balance in structure, temporal sequence, and information representation in the final generated video summaries.

[0050] As one implementation method, during the training process of the above video summarization generation, soft sampling is performed using a multinomial distribution without replacement: To maintain gradient propagation; while in the inference phase, the previous value is directly taken. The frame with the highest weight: This adaptive keyframe sampling mechanism effectively suppresses redundancy while ensuring coverage and consistency, and can adaptively adjust the sampling density according to different video features, providing a stable and high-quality candidate frame distribution for subsequent graph-guided optimization.

[0051] In video summarization tasks, keyframe selection needs to ensure not only semantic coverage and content diversity but also natural temporal continuity. Overly concentrated keyframe distribution or semantic repetition can lead to information redundancy, plot breaks, or abrupt transitions in the summary. Redundancy suppression and temporal consistency constraints can address these issues, ensuring that the selection results are both representative and maintain a reasonable temporal distribution. Therefore, this embodiment designs a Redundancy Suppression and Temporal Consistency (RSTC) module. First, considering that the keyframe score sequence should exhibit a smooth change in the temporal dimension, a temporal consistency constraint is introduced to penalize drastic fluctuations in scores between adjacent frames. This constraint is as follows: ; in, Indicates the first t The sampling weights of frames. This constraint encourages the model to maintain the continuity of the prediction distribution within the temporal neighborhood, thereby avoiding overly discrete or unstable keyframe selection results.

[0052] Secondly, to prevent content redundancy, a redundancy suppression term is designed based on inter-frame feature similarity, as follows: ; in, This represents the cosine similarity between frame features. If two frames are semantically similar and are selected simultaneously, a higher penalty value will be generated, thus prompting the model to be more inclined to select frames with complementary semantics.

[0053] To comprehensively consider both temporal smoothness and semantic redundancy removal, the above two constraints are jointly defined as the temporal consistency loss, which is: ; in, and This is a balancing factor used to control the relative importance of temporal smoothing and redundancy suppression. By adjusting the relative weights of these two factors, a flexible trade-off can be struck between diversity and coherence in the abstract.

[0054] During training, the RSTC module, acting as a regularization constraint, is jointly optimized with the main loss function (coverage, diversity, and coherence) to enable the model to adapt to the temporal structure and frame distribution characteristics of different video types during end-to-end learning. This constraint effectively reduces keyframe repetition and temporal jumps, making the generated summaries visually more natural and semantically more coherent. Experimental results show that the RSTC module significantly improves the temporal smoothness and content diversity of summaries on different datasets, providing a strong structural constraint for the overall model's summarization quality.

[0055] like Figure 3 As shown, each row represents the video time from left to right, and the colored blocks are the selected keyframes: white represents the Ground Truth, red represents the G-AFS filtering results, and beige represents the score-based AFS filtering results, all within the same budget. The G-AFS proposed in this embodiment leverages global relationships and distributed abrasion. The selection of more dispersed and coherent keyframes, while existing score-based AFS often clusters in local segments, making it difficult to balance coverage and redundancy suppression, can further illustrate the difference caused by the joint decision of "which to select" and "how much to select". The G-AFS model proposed in this embodiment can achieve wider temporal coverage and lower redundancy, maintain narrative coherence, and achieve better video summarization generation results.

[0056] Furthermore, to verify the superiority of the proposed method and to comprehensively evaluate the video summarization performance of the proposed model, the following specific experiments were conducted in this embodiment.

[0057] In this embodiment, the experiments were conducted on a server equipped with an NVIDIA Quadro RTX 6000 GPU, CUDA version 11.3, and cuDNN version 8.3.2. All experiments were implemented using the PyTorch framework, with fixed random seeds (seed=1, 2, 3, 4, 5, 6) to ensure the reproducibility of the results. The model used GoogleNet as the backbone network, with a learning rate of 1×10⁻⁶. - ³, weight decay is 1×10 -7 The batch size was 1, and the training epochs were 2. Experiments were conducted on existing datasets SumMe, TVSum, and VideoXum, with an input sampling rate of 15fps and feature types of [missing information]. blip The total number of parameters in the model is approximately 6.24M.

[0058] Experiments were conducted on three publicly available video summarization datasets: TVSum, SumMe, and a self-built VideoXum dataset. The TVSum dataset contains 50 everyday life videos from YouTube, covering various topics such as news, sports, and travel. Each video has frame-level importance scores provided by multiple annotators. The SumMe dataset contains 25 user-shot videos of real-world scenes, such as holiday activities, performances, and competitions, and provides manually annotated summaries, often used to evaluate a model's summarization ability under subjective semantic understanding. To further verify the model's generalization performance in complex scenarios, the VideoXum dataset was also used. This dataset contains approximately 14,000 video samples, covering various scenarios such as urban life, natural scenery, social activities, sports events, and film clips, with individual video lengths ranging from tens of seconds to over ten minutes. All of the above videos provide frame-level visual features and corresponding summary annotations, exhibiting higher content diversity and semantic complexity, effectively used to test the model's summarization robustness and generalization ability under large-scale cross-scene video conditions.

[0059] To ensure the fairness and stability of the results, the experiments on the three existing datasets used—SumMe, TVSum, and VideoXum—all employed 5-fold cross-validation, and the average result of all splits was used as the final evaluation score. This evaluation method not only reduces the fluctuations caused by data splits but also more accurately reflects the overall generalization performance of the model.

[0060] To comprehensively evaluate the performance of the G-AFS model in video summarization generation tasks and verify the effectiveness of the proposed method, several representative supervised and self-supervised video summarization models were selected as comparison baselines, specifically including the following six methods: (1) TL;DW (Gyglietal., 2014): An early supervised video summarization method that learns frame-level importance through low-level visual features (such as motion, scene changes, and object salience) and human preference labels, representing the traditional handmade feature-driven summarization approach.

[0061] (2) CLIP-it (Leietal., 2023): A cross-modal video summarization method based on the CLIP visual language model, which uses text prompts and visual alignment mechanisms to extract semantic associations and achieve summarization generation in zero-sample or few-sample scenarios.

[0062] (3) A2Summ (Zhang et al., 2021): An end-to-end video summarization model based on attention mechanism, which strengthens the modeling of inter-frame dependencies through adaptive weighting mechanism and achieves a dynamic balance between global semantic consistency and local saliency.

[0063] (4) DMASum (Fengetal., 2022): A video summarization framework that introduces a dual-attention mechanism at multiple scales, which can model the semantic relationships between frames at different time scales, thereby improving the semantic coverage and temporal consistency of the summary.

[0064] (5) SSPVS (Huang et al., 2023): A self-supervised predictive video summarization model that achieves key frame selection under unlabeled conditions through frame order prediction and reconstruction constraints.

[0065] (6) CSTA (Fuetal., 2024): A supervised model based on spatio-temporal attention mechanism (CNN-based Spatio-TemporalAttention, CSTA), which captures the inter-frame contextual relationship through joint modeling of channel and temporal dimensions, and is currently the best performing attention-based baseline.

[0066] The aforementioned models cover everything from traditional feature-driven methods to advanced deep learning-based architectures. They include both early supervised video summarization frameworks and the latest methods that introduce multi-scale modeling and self-supervised mechanisms in recent years, and can comprehensively reflect the development trends and technological progress in the field of video summarization.

[0067] Furthermore, the comparison methods employ uniform feature and budget settings, covering methods such as uniformly spaced fixed-ratio sampling (Uniform-K), frame-by-frame scoring Top-K selection (Score-TopK), scoring selection with temporal non-maximum suppression (Score+NMS), representative frame selection based on feature clustering (K-means / Medoids), and key frame selection methods with diversity constraints based on the determinantal point process (DPP) of similarity kernel matrix. Moreover, no additional supervision or prior is introduced for any baseline to ensure the fairness of the comparison.

[0068] Furthermore, this embodiment employs several commonly used evaluation metrics in the current video summarization field, including F-score (F1 value), Kendall's τ (rank correlation coefficient), and Spearman's ρ (spearman's metric), to evaluate the experimental results of the aforementioned models. These multiple metrics can measure the comprehensive performance of the model-generated summaries in terms of information coverage, content diversity, and temporal consistency from different perspectives. The multi-metric evaluation system allows for a comprehensive assessment of the model from multiple dimensions, including accuracy, coverage, diversity, and temporal consistency, providing a reliable basis for subsequent comparative experiments and performance analysis. In addition, during the experiments, all metrics were calculated based on the frame-level alignment results of manually annotated summaries and model-generated summaries.

[0069] Table 1. Experimental results of different models on the TVSum and SumMe datasets.

[0070] The final experimental results are shown in Table 1 above. In Table 1, τ and ρ are two correlation coefficients that measure the consistency of ranking. τ is used to compare the pairwise consistency between two ranking sequences and is suitable for measuring the stability of ranking. ρ is calculated based on the rank values ​​of two variables and reflects the overall consistency between rankings. It is suitable for evaluating the strength of the correlation between variables.

[0071] As can be seen from the above, the advantages of the G-AFS proposed in this embodiment mainly benefit from the synergistic design of its graph guidance and adaptive sampling mechanism. By introducing inter-frame semantic relationship modeling into the feature space, the model can capture more fine-grained global structural information. Simultaneously, the adaptive sampling strategy dynamically adjusts the number of keyframes according to the video length and content complexity, achieving efficient frame selection in dynamic and complex scenes. Compared with traditional methods that rely solely on temporal attention, G-AFS has advantages in capturing visual associations, maintaining semantic consistency, and suppressing inter-frame redundancy. Furthermore, G-AFS has a simple and efficient structure, does not rely on additional modal information or complex feature engineering, and can achieve high-quality video summarization with only a single visual feature input. Its consistently leading performance in both τ and ρ metrics fully verifies the effectiveness and innovation of this model in inter-frame relationship modeling and summarization stability, demonstrating the robustness and generalization ability of G-AFS in video summarization tasks.

[0072] To further verify the generalization ability of the proposed G-AFS model in large-scale and complex scenarios, supplementary experiments were conducted on the VideoXum dataset. The experimental results are shown in Table 2 below.

[0073] Table 2 Experimental results of different models on the VideoXum dataset

[0074] As shown above, G-AFS achieved an F1 score of 22.5% on the VideoXum dataset, significantly outperforming the other two existing comparison methods, Frozen-BLIP (F1 score of 16.1%) and VSUM-BLIP (F1 score of 21.7%). This result demonstrates that G-AFS can maintain stable summarization performance across domains and diverse video scenarios. Thanks to its graph-guided semantic modeling and adaptive sampling strategy, the model effectively captures global relationships between frames and dynamically adjusts the keyframe distribution according to video complexity, thus exhibiting stronger generalization and robustness on large-scale video data.

[0075] In addition, to further analyze the impact of each core module on the overall performance of the model, systematic ablation experiments were conducted on the TVSum and SumMe datasets. The experiments mainly explored the independent and joint effects of two key components: (1) the Graph-Guided modeling module, which is used to capture global semantic relationships and contextual dependencies between frames; and (2) the Adaptive Keyframe Sampling (AFS) module, which is used to adaptively adjust the number of keyframe samples and the distribution focus according to the complexity and duration of the video content. The experiments compared the Spearman ρ and Kendall τ correlation coefficients under different combinations by enabling or disabling these two modules respectively, in order to evaluate the model's performance in terms of frame-level ordering consistency and structural stability.

[0076] As shown in Table 3, √ indicates that this module is added to the complete model, and × indicates that this module is removed from the complete model. When both the Graph-Guided and AFS modules are turned off, the model performance is the weakest, indicating that relying solely on the backbone network for feature extraction is insufficient to capture the temporal dependencies and semantic structures between keyframes. When only the Graph-Guided module is enabled, the model's performance improvement on both datasets is limited, indicating that although this module can capture global correlations between frames to some extent, it is still difficult to obtain stable summaries under different video lengths and content complexities due to the lack of sampling adjustment mechanisms.

[0077] Table 3. Experimental results of models composed of different modules on the TVSum and SumMe datasets.

[0078] In contrast, enabling only the AFS module significantly improves model performance. This demonstrates that the adaptive sampling mechanism can effectively adjust frame selection probabilities dynamically based on video structure, thereby improving the coverage of the summary and the consistency of frame ordering. The AFS module, by introducing a learnable distribution focusing parameter α and a video length adaptive function φ(T), achieves dynamic keyframe distribution optimization under different scenes and video durations, giving the model good flexibility and generalization ability.

[0079] When the Graph-Guided and AFS modules are used together, the model achieves state-of-the-art results on both datasets, improving performance by approximately +0.08–0.10 compared to the individual modules. This significant improvement validates the complementarity of the two modules: the Graph-Guided module provides global semantic constraints, suppressing redundancy and maintaining semantic coherence during frame selection; while the AFS module further implements temporal adaptation and focus adjustment at the sampling level, enabling the model to achieve a dynamic balance between local temporal order and global distribution.

[0080] Therefore, ablation experiments show that the combination of Graph-Guided and AFS is key to improving the performance of the G-AFS model. The former is responsible for constructing semantic relation priors and strengthening global dependency modeling, while the latter optimizes frame selection distribution through an adaptive sampling strategy. The synergistic effect of the two makes the model significantly better than the single-module configuration in terms of frame-level ranking consistency, summarization coherence, and cross-scene robustness.

[0081] To verify the predictive performance of the G-AFS model, such as Figure 2 As shown, the G-AFS model's predicted curve (blue dashed line) exhibits a trend characteristic that more closely matches the actual value (i.e., the green solid line) compared to the CSTA model's predicted curve (red dotted line). Figure 2 As shown in (a), in the long-term scene (video_23), G-AFS exhibits excellent stability. When the video content changes gradually, the CSTA predicted trajectory shows obvious fluctuations and deviations, while G-AFS maintains a smooth transition of the prediction curve through dynamic adaptive keyframe sampling, and its fluctuation amplitude is highly synchronized with the rhythm of the actual value change. Especially in the later part of the video 23, the CSTA prediction shows a continuous deviation, while G-AFS accurately captures the gradual change pattern through long-range dependency modeling, and the predicted trajectory always stays within a reasonable error range.

[0082] like Figure 2 As shown in Figure (b), G-AFS demonstrates a fast response advantage for the abrupt change scenario (video_45). During the drastic change event in the middle of the video, the CSTA predicted trajectory exhibits a lagging correction, requiring significantly more frames to recover the true trajectory than G-AFS. This difference validates the effectiveness of G-AFS's two-stage correction mechanism in real-time adjustment of predicted keyframes. Furthermore, it was observed that in the stable phase after the abrupt change event, the oscillations of the G-AFS prediction curve decay faster, indicating that the model has better dynamic equilibrium capabilities.

[0083] In summary, the video summarization method proposed in this embodiment can adaptively determine the sampling quantity based on the video duration, avoiding oversampling of short videos and undersampling of long videos. It utilizes global content and semantic association to guide keyframe selection, ensuring summary coverage while suppressing redundancy and maintaining plot coherence. Through a controllable distributed focusing mechanism, G-AFS can achieve a flexible balance between distributed and centralized sampling, adapting to different application scenarios. Experimental results also show that the proposed G-AFS method exhibits superior prediction consistency under different complexity scenarios. Its curve shape significantly outperforms existing baseline methods in terms of temporal stability, abrupt change response speed, and anti-interference ability, achieving a better balance between coverage, diversity, and stability.

[0084] Example 2 This embodiment provides a video summarization system based on graph-guided adaptive keyframe sampling, specifically including: The video acquisition module is used to acquire raw video to extract video frame sequences; The video frame feature extraction module is used to extract features at different scales for each video frame, and then generate multi-scale fused features by weighted fusion of scale attention weights. The graph-guided keyframe modeling module is used to calculate the inter-frame similarity based on the fusion features of video frame sequences to construct a weighted undirected graph. Through multi-layer feature propagation, the global structured frame features of each video frame are obtained, and the graph-guided weights of each frame are calculated. The weight optimization module is used to introduce learnable distribution focusing parameters to scale the graph-guided weights exponentially to obtain adaptive sampling weights, and then combine them with the graph-guided weights to calculate the final comprehensive weights. The adaptive sampling module is used to calculate the number of sampled frames based on the video frame length of the original video using a video length adaptive function. Based on the comprehensive weight and the number of sampled frames, it uses a sorting-based frame filtering strategy to sample key frames and generate a video summary.

[0085] Example 3 This embodiment provides an electronic device, including: a memory for storing executable instructions; and a processor for executing the executable instructions stored in the memory to implement the method provided in this embodiment.

[0086] Example 4 This embodiment also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, will cause the processor to execute the method described above in this embodiment.

[0087] Example 5 This embodiment provides a computer program product including executable instructions, which are computer instructions; the executable instructions are stored in a computer-readable storage medium. When the processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the electronic device performs the method described in this embodiment.

[0088] The steps and methods involved in Embodiments 2 to 5 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0089] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0090] The above description is only a preferred embodiment of the present invention. Although the specific implementation of the present invention has been described in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative effort are still within the scope of protection of the present invention.

Claims

1. A video summarization method based on graph-guided adaptive keyframe sampling, characterized in that, include: Obtain the raw video to extract the video frame sequence; Features at different scales are extracted from each video frame, and then weighted and fused by scale attention weights to generate multi-scale fused features. Based on the fusion features of video frame sequences, inter-frame similarity is calculated to construct a weighted undirected graph. Through multi-layer feature propagation, global structured frame features of each video frame are obtained, and graph guidance weights for each frame are calculated. A learnable distribution focusing parameter is introduced to scale the graph-guided weights exponentially to obtain adaptive sampling weights, which are then combined with the graph-guided weights to calculate the final comprehensive weights. Based on the video frame length of the original video, the number of sampled frames is calculated using a video length adaptive function. Then, based on the comprehensive weight and the number of sampled frames, a ranking-based frame filtering strategy is used to sample key frames and generate a video summary.

2. The video summarization method based on graph-guided adaptive keyframe sampling as described in claim 1, characterized in that, The process of extracting different scale features from each video frame, weighting and fusing them with scale attention weights, and generating multi-scale fused features includes: Low-level and high-level features at different scales are extracted from each video frame in the video frame sequence. Based on features at different scales, the scale attention weights for each scale feature are calculated using a lightweight mapping function. Based on the scale attention weight, features at different scales are weighted and fused to generate multi-scale fused features.

3. The video summarization method based on graph-guided adaptive keyframe sampling as described in claim 1, characterized in that, The calculation process for the graph guidance weight is as follows: Using each video frame as a node and the multi-scale fused features of each video frame as the initial features of the node, the inter-frame similarity is calculated to construct an adjacency matrix. Construct a weighted undirected graph based on the adjacency matrix. Among them, the set of nodes Let E represent video frames and E be the set of edges. The formula for calculating the similarity between frames is: ； Multi-layer feature propagation is performed on the graph structure of a weighted undirected graph to aggregate contextual information and learn global dependencies, thereby obtaining global structured frame features for each video frame. Based on the global structured frame features, the structured importance score for each video frame is calculated as follows: ； Based on the importance score, the probability distribution, i.e., the graph-guided weights, is obtained by normalization using the Sigmoid function: ； in, Indicates the first i, j Initial characteristics of video frames; Indicates the first i Global structured frame features of video frames. L Indicates the number of feature propagation layers; Indicates the first i The importance score of each video frame; , For hyperparameters; T This represents the number of video frames in the original video.

4. The video summarization method based on graph-guided adaptive keyframe sampling as described in claim 1, characterized in that, The calculation process for the comprehensive weight is as follows: Introducing learnable distribution focusing parameters Graph guiding weights Power-law scaling yields adaptive sampling weights, expressed as: ;in, T The number of video frames in the original video; Guided weights for graphs With adaptive sampling weights We perform weighted fusion to obtain a comprehensive weight that takes into account both global and local features.

5. The video summarization method based on graph-guided adaptive keyframe sampling as described in claim 1, characterized in that, The overall weights are smoothed by averaging over time, resulting in the smoothed weights, which are expressed as follows: ； in, The radius of the time window. This is the overall weight.

6. The video summarization method based on graph-guided adaptive keyframe sampling as described in claim 1, characterized in that, The video length adaptive function is: ； The formula for calculating the number of sampling frames is: ； in, For the Sigmoid function, , , , This is a configurable hyperparameter, where T is the number of video frames in the original video.

7. A video summarization method system based on graph-guided adaptive keyframe sampling, characterized in that, include: The video acquisition module is used to acquire raw video to extract video frame sequences; The video frame feature extraction module is used to extract features at different scales for each video frame, and then generate multi-scale fused features by weighted fusion of scale attention weights. The graph-guided keyframe modeling module is used to calculate the inter-frame similarity based on the fusion features of video frame sequences to construct a weighted undirected graph. Through multi-layer feature propagation, the global structured frame features of each video frame are obtained, and the graph-guided weights of each frame are calculated. The weight optimization module is used to introduce learnable distribution focusing parameters to scale the graph-guided weights exponentially to obtain adaptive sampling weights, and then combine them with the graph-guided weights to calculate the final comprehensive weights. The adaptive sampling module is used to calculate the number of sampled frames based on the video frame length of the original video using a video length adaptive function. Based on the comprehensive weight and the number of sampled frames, it uses a sorting-based frame filtering strategy to sample key frames and generate a video summary.

8. An electronic device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the video summarization method based on graph-guided adaptive keyframe sampling as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The device stores executable instructions that, when executed by a processor, implement the video summarization method based on graph-guided adaptive keyframe sampling as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes executable instructions stored in a computer-readable storage medium; When the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, it implements the video summarization method based on graph-guided adaptive keyframe sampling as described in any one of claims 1-6.

Images