Lightweight key frame extraction algorithm based on double attention fusion mechanism

Abstract

The invention discloses a lightweight key frame extraction algorithm based on a double attention fusion mechanism, and belongs to the crossing field of computer vision and Internet of Things. The end layer calculates the information amount of adjacent frames based on the image difference, and the redundant data amount is reduced by screening candidate frames; the edge layer constructs a double attention fusion mechanism, extracts image space features through a space attention module, captures time sequence association through a time attention module, and dynamically fuses the two types of features in combination with a gating mechanism to generate a key frame score; and further designing a reinforcement learning algorithm optimization model, and adopting a composite reward function containing a diversity reward and a representative reward to ensure the difference and representativeness of the key frame set. According to the method, optimal allocation of computing resources is achieved through hierarchical processing, a lightweight attention mechanism and reinforcement learning training are combined, the balance problem of redundant frame processing and semantic recognition in video analysis is effectively solved, and the real-time processing efficiency of a garbage classification system is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the intersection of computer vision and the Internet of Things, and specifically to a lightweight keyframe extraction algorithm based on a dual attention fusion mechanism. Background Art

[0002] With the acceleration of urbanization and population growth, the amount of urban domestic waste has exploded, placing tremendous pressure on the environment and resources. Effective waste sorting has become a key component of environmental protection and resource recycling, and is of great significance to promoting sustainable development. Traditional waste sorting methods rely primarily on manual supervision, which is not only inefficient but also susceptible to subjective factors and misclassification. In recent years, machine learning technology has been gradually revolutionizing the field of waste sorting. By collecting and analyzing large amounts of data, building intelligent recognition models, and automatically distinguishing waste types, it has significantly improved sorting efficiency and reduced the need for human intervention.

[0003] However, processing, storing, and analyzing waste sorting data presents significant challenges. To address this dilemma, the device-edge-cloud collaborative architecture emerged. As an innovative distributed computing paradigm, it effectively addresses the bottleneck of big data processing by rationally partitioning and allocating computing tasks. It also achieves a balanced distribution of computing resources, reduces network transmission latency, and improves overall system efficiency, providing strong technical support for intelligent waste sorting in the big data era.

[0004] However, in an end-edge-cloud collaborative architecture, edge devices have limited computing resources, while video stream data contains a large number of redundant frames and invalid frames that are irrelevant to the waste sorting task. Processing all video frames without screening not only consumes valuable computing resources and increases system load, but can also significantly increase communication latency, thereby affecting model training quality and overall system performance. Therefore, effectively reducing the processing of redundant frames while ensuring the accuracy of key frame extraction has become a key challenge for optimizing the performance of end-edge-cloud collaborative waste sorting systems and improving resource utilization efficiency.

[0005] Frame redundancy refers to the fact that due to the temporal continuity of video streams, adjacent frames often exhibit extremely high levels of information redundancy and visual similarity. This is particularly true when the scene is relatively static or slowly changing. The system will continually capture redundant frames with extremely low information gain. This data redundancy not only consumes significant storage resources but also significantly increases the computational load of subsequent processing, reducing overall system efficiency.

[0006] The invalid frame problem refers to the fact that due to the complex and variable nature of waste sorting scenarios, cameras may not only capture images of incoming waste but also pedestrians, fallen leaves blown by the wind, or other non-waste objects. Therefore, keyframe extraction algorithms must not only eliminate redundant frames but also accurately identify valid frames relevant to the waste sorting task.

[0007] In response to the above challenges, existing technical routes are mainly divided into two categories:

[0008] Keyframe extraction algorithms based on image information differences can effectively remove redundant frames and identify scene changes by calculating the differences between adjacent frames. However, these algorithms struggle to semantically distinguish whether a frame's content is relevant to the garbage classification task, and therefore are unable to effectively select valid frames that are actually valuable for the task.

[0009] Keyframe extraction methods based on deep learning can focus more on valid frames relevant to the garbage classification task and achieve higher-level semantic understanding by learning feature representations of garbage targets. However, deep learning models generally have high requirements for hardware performance. Especially in video analysis tasks, these models need to process large amounts of data and perform complex calculations such as convolution operations, feature extraction, and multi-layer nonlinear transformations. These computational operations consume a large amount of computing resources. The demand for computing power is particularly urgent when processing video streams in real time. When deployed directly on resource-limited edge devices, they often face problems such as insufficient computing resources and long response times, making it difficult to meet the real-time requirements of garbage classification systems. Summary of the Invention

[0010] To address these issues, this paper designs a lightweight keyframe extraction algorithm based on a dual-attention fusion mechanism, drawing on the unique characteristics of a device-edge-cloud collaborative framework. This algorithm leverages the computing resources of devices at different levels. This strategy employs a layered design approach to achieve a rational distribution of computing tasks.

[0011] First, on resource-constrained end devices, traditional image-based candidate frame extraction techniques are used to perform preliminary screening of the video stream, quickly removing highly similar redundant frames. This phase of the design takes into account the limited computing power of the end devices, so traditional image feature analysis methods with lower computational complexity are selected. This significantly reduces the amount of data transmitted to the edge layer while maintaining a certain level of screening quality.

[0012] These filtered candidate frames are then fed into edge devices with greater computing power for more refined keyframe identification. At the edge layer, this paper proposes a lightweight keyframe extraction algorithm based on a dual attention fusion (DAF) mechanism. By fusing feature representations from both spatial and temporal attention dimensions, this algorithm can more comprehensively capture the semantic information of video content, enabling more accurate keyframe identification.

[0013] To this end, the technical solution adopted by the present invention is as follows: a lightweight key frame extraction algorithm based on a dual attention fusion mechanism, specifically,

[0014] The algorithm includes an end layer and an edge layer;

[0015] Deploy high-definition cameras to collect video streams and transmit them to the end layer;

[0016] In the end layer, the information content of each frame image in the video stream and the adjacent frame images is calculated based on the image difference; if the information content is greater than the set first threshold, it is determined to be a candidate frame;

[0017] Inputting the candidate frame into the edge layer, and constructing a dual attention fusion mechanism in the edge layer, including a spatial attention module, a temporal attention module and a gating mechanism;

[0018] The spatial attention module identifies and outputs spatial attention features in the candidate frame image;

[0019] The temporal attention module identifies and outputs temporal attention features between candidate frame images;

[0020] Using a gating mechanism to fuse the spatial attention features and the temporal attention features to obtain a key frame score; if the key frame score is greater than a set second threshold, it is determined to be a key frame;

[0021] A reinforcement learning algorithm is designed to optimize the dual attention fusion mechanism. The reward function of the reinforcement learning algorithm includes: a diversity reward based on the difference between key frames and a representative reward based on the difference between the key frames and the video stream sequence.

[0022] Furthermore, the calculation process of the information volume is as follows:

[0023] For the video stream sequence {F1, F2, ..., F n}, where F i Represents the i-th frame image, and defines the information volume of the i-th frame image I(F i )for:

[0024] I(F1)=0

[0025]

[0026] Among them G i It's F i Grayscale image, W and H are the width and height of the frame image, respectively, |G i (x,y)-G i-1 (x,y)| is the absolute value of the grayscale value of the pixel (x,y).

[0027] Furthermore, the calculation process of the spatial attention feature is as follows:

[0028] First, the query vector Q, key vector K, and value vector V of the candidate frame X are calculated as follows:

[0029] Q=W q ·F r (X),K=W k ·F r (X),V=W v ·F r (X)

[0030] Among them, W q is the query weight matrix, W k is the key weight matrix, W v is the value weight matrix, F r (X) is the vector representation of the candidate frame after dimensionality reduction projection;

[0031] Then, the original attention weight matrix A is calculated by the softmax function raw , as shown below:

[0032]

[0033] Next, the original attention weight matrix is operated through the Threshold function, retaining only the largest k attention weights of the original attention weights to generate the top-k weight matrix, as shown below:

[0034] A sparse =Th resh old(A raw ,top-k)

[0035] Finally, the top-k weight matrix is dot-producted with the value vector to generate the spatial attention feature A of the candidate frame image. s , the calculation formula is: A s =A sparse ·V.

[0036] Furthermore, the calculation process of the temporal attention feature is as follows:

[0037] First, the depth convolution function DepthwiseConv is used to perform a depth convolution operation on the transposed candidate frame image, and the pointwise convolution function PointwiseConv is used to perform a depth convolution operation on the depth convolution operation result Z d Perform point-by-point convolution, expressed as follows:

[0038] Z d =Depth wiseConv(X T )

[0039] Z p =PointwiseConv(Z d )

[0040] Among them, Z p is the point-by-point convolution result;

[0041] Then, the temporal attention feature C is calculated based on the point-by-point convolution result, which is expressed as follows:

[0042]

[0043] Where W is the weight, represents the point-by-point convolution result at time step t, and σ1 is the normalization function. Furthermore, the fusion process of the gating mechanism is as follows:

[0044] G=σ2(W g ·[A s ;C])

[0045] F fused =G·A s +(aG)·C

[0046] Among them, σ2 is the activation function, [A s ; C] represents the concatenation of spatial attention features and temporal attention features, G is the gating signal, a represents the dynamic fusion weight of spatial attention features and temporal attention features, and the sum of the fusion weight of spatial attention features and the fusion weight of temporal attention features is a, that is, G+(aG)=a;

[0047] F fused Score keyframes.

[0048] Furthermore, the diversity reward R div The calculation formula is as follows:

[0049]

[0050] Where m is the number of key frames; f p ,f q are the pth and qth key frames respectively, sim(f i ,fj ) is the cosine similarity between keyframes.

[0051] Furthermore, the representative reward R rep The calculation formula is as follows:

[0052]

[0053] Among them, d(F i ,f j ) is the key frame f j With frame image F i min(·) is the minimum function, and S is the set of key frames.

[0054] Compared with the prior art, the advantages of the present invention are:

[0055] (1) The present invention uses traditional candidate frame extraction technology based on image information on resource-constrained terminal devices to perform preliminary screening of video streams and quickly remove highly similar redundant frames. This can significantly reduce the amount of data that needs to be transmitted to the edge layer while ensuring a certain screening quality.

[0056] (2) The present invention inputs the filtered candidate frames into an edge device with stronger computing power for more refined key frame identification. A lightweight key frame extraction algorithm based on a dual attention fusion mechanism is used at the edge layer. By fusing the feature representations of spatial attention and temporal attention, the algorithm can more comprehensively capture the semantic information of the video content, thereby achieving more accurate key frame identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0058] Figure 1 This is the overall framework diagram of this embodiment;

[0059] Figure 2 This is a diagram of the key frame extraction architecture based on device-edge collaboration in this embodiment;

[0060] Figure 3 Schematic diagram of a partial frame sequence of an original video according to this embodiment;

[0061] Figure 4 This is a schematic diagram of random video key frame acquisition in this embodiment;

[0062] Figure 5Schematic diagram of the reward coefficient of this embodiment;

[0063] Figure 6 This is a comparison of the indicators of each algorithm in this embodiment on different data sets. DETAILED DESCRIPTION

[0064] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0065] This embodiment provides a lightweight key frame extraction algorithm based on a dual attention fusion mechanism. Figure 1 shown.

[0066] The end layer is primarily responsible for data collection and candidate frame extraction. High-definition cameras are deployed at waste sorting sites to collect real-time video streams from waste disposal points and perform preliminary key frame screening. By quantifying the "information content" of each frame and calculating the differences between adjacent frames, the difference values can be used to determine whether the current frame has changed. The specific algorithm is as follows:

[0067] For a video sequence {F1,F2,…,F n}, where F i Represents the i-th frame, and defines the information volume of the i-th frame I(F i )for:

[0068] I(F1)=0# (1)

[0069]

[0070] Among them G i It's F i Grayscale image, W and H are the width and height of the image respectively, |G i (x,y)-G i-1 (x,y)| is the absolute value of the grayscale value of the pixel point (x,y). The information difference between adjacent frames is calculated and compared with the pre-set threshold. If it exceeds the threshold, it is identified as a candidate frame.

[0071] The above algorithm does not require high-performance computing equipment. By simply calculating the difference in image information, the algorithm can effectively filter out candidate frames with significant changes. In the garbage sorting scenario, the changes of interest primarily occur when the garbage is dropped off. Therefore, this algorithm helps remove redundant frames when there is no garbage and garbage frames that remain static for a long time.

[0072] The selected candidate frames are transmitted to the edge layer for further processing.

[0073] While simple and efficient, the candidate frame extraction algorithm based on frame pixel differences cannot identify dynamic frames with clear information. The candidate frames passed by the peer layer require further determination to determine whether they are truly garbage. Therefore, this embodiment also deploys a lightweight key frame extraction algorithm based on the Dual Attention Fusion (DAF) mechanism at the edge layer, prioritizing computational efficiency while maintaining high performance.

[0074] The DAF model uses an innovative dual-attention architecture that effectively captures both spatial relationships within video frames and temporal relationships between frames. The spatial attention module focuses on identifying key regions within a frame, while the spatial attention module captures semantic changes between frames. The fusion of the two enables a comprehensive assessment of the information content of candidate frames, significantly improving the accuracy of keyframe recognition.

[0075] To further reduce the complexity of the model, this embodiment conducts targeted optimization of the spatial attention module: a dimensionality reduction projection mechanism is introduced to significantly reduce the number of calculation parameters, and combined with a sparse attention calculation strategy, while maintaining the expressive power of the attention mechanism, a significant reduction in computational complexity is achieved.

[0076] The dimensionality reduction projection refers to reducing the dimension of a higher-dimensional candidate frame into a lower-dimensional vector using a deep learning method or a linear transformation method, which is not limited in this embodiment.

[0077] The specific calculation formula of the spatial attention module is as follows:

[0078] First, the query vector Q, key vector K, and value vector V of the candidate frame X are calculated as follows:

[0079] Q=W q ·F r (X),K=W k ·F r (X),V=W v ·F r (X)# (3)

[0080] Among them, W q is the query weight matrix, W k is the key weight matrix, W v is the value weight matrix, F r (X) is the vector representation of the candidate frame after dimensionality reduction projection;

[0081] Then, the original attention weight matrix A is calculated by the softmax function raw , as shown below:

[0082]

[0083] Next, the original attention weight matrix is operated by the Threshold function, retaining only the largest k attention weights of the original attention weights to generate the top-k weight matrix, as shown below:

[0084] A sparse =Th resh old(A raw ,top-k)# (5)

[0085] Finally, the top-k weight matrix is dot-producted with the value vector to generate the spatial attention feature A of the candidate frame image. s , the calculation formula is: A s =A sparse ·V# (6).

[0086] The temporal attention module is subjected to deep convolution separation to reduce parameters and computational complexity. The specific formula is as follows:

[0087] Z d =Depth wiseConv(X T )# (7)

[0088] Z p =PointwiseConv(Z d )# (8)

[0089]

[0090] Wherein formula (7) is used for the depthwise separable convolution of the transposed input candidate frame, and formula (8) is used for the pointwise convolution of the depthwise convolution output.

[0091] The two attention outputs are then fused through a gating mechanism:

[0092] G=σ2(W g ·[A s ;C])# (10)

[0093] F fused =G·A s +(aG)·C# (11)

[0094] This embodiment uses reinforcement learning to optimize the DAF model. Its main advantages are: it can directly optimize the final goal of key frame extraction rather than the intermediate proxy goal; it has sequential decision-making capabilities and can consider global information to make better choices; it does not require a large amount of labeled data and can achieve autonomous learning through reward signals, which is particularly suitable for edge computing environments with limited resources.

[0095] Based on these advantages, we adopt a reinforcement learning framework to dynamically adjust the DAF model and design a lightweight but comprehensive reward function, focusing on the following two key aspects:

[0096] Diversity reward: Diversity reward measures the differences between the selected keyframes to ensure that the selected frames are not too similar. The calculation formula is as follows:

[0097]

[0098] Where m is the number of key frames, f p ,f q is the selected key frame feature, sim(f p ,f q ) is the cosine similarity between features,

[0099] Representativeness reward: The representativeness reward measures the degree to which the selected keyframe represents the entire sequence, ensuring that the selected frame can summarize the entire video content. The calculation formula is as follows:

[0100]

[0101] Where n is the total length of the video stream sequence, S is the selected key frame index set, d(F i ,f j ) is the key frame f j With frame image F i The Euclidean distance of .

[0102] The above embodiment is experimentally analyzed below:

[0103] Experimental Environment: The hardware used was an NVIDIA RTX A4000 GPU, a 12th-generation Intel(R) Core(TM) i5-12600K processor, and 64GB of RAM. The software used the PyTorch 2.5 framework and the PyCharm development tool, using Python 3.11. Libraries such as NumPy, Matplotlib, and OpenCV were also installed for data processing and image analysis.

[0104] Dataset: This experiment uses three datasets to evaluate the key frame extraction algorithm, namely the SumMe dataset, the TVSum dataset, and a self-made dataset.

[0105] SumMe dataset: This dataset is a public dataset widely used for video summarization tasks. It contains 25 videos with a length of 3 to 6 minutes, covering multiple fields such as sports, news, and entertainment.

[0106] The TVSum dataset is a key public dataset for video summarization tasks, consisting of 50 videos from YouTube, covering a variety of fields such as sports, news, and entertainment. Each video is approximately 2 to 6 minutes long and has been manually annotated with keyframes that represent the most representative and informative parts of the video.

[0107] A self-made dataset: To better meet the needs of waste sorting scenarios, the experiment also used a self-made dataset. This dataset consists of 21 videos of waste sorting scenes, covering different types of waste and environmental conditions. The purpose is to simulate video data from actual scenarios and further verify the performance of the keyframe extraction algorithm in real-world applications.

[0108] Parameter settings: By default, all model training rounds are uniformly set to 200 rounds, 80% of the data set is used as the training set, 20% is used as the test set, and the learning rate is set to 0.00001.

[0109] Comparison method: DAF is compared with the following four keyframe extraction algorithms.

[0110] Motion optical flow: This method analyzes image changes by calculating the pixel motion vector field between adjacent frames in a video. Based on the assumption of constant image brightness, this method solves the optical flow equation to obtain motion information for each pixel, thereby quantifying the degree of change in visual content between frames. In keyframe detection, when the optical flow field shows significant changes, it usually indicates a significant scene transition, and these frames can be marked as keyframes.

[0111] RS-SUM is an unsupervised video summarization method based on self-supervised learning. Its core concept is to identify keyframes using a "restorative score." The algorithm first uses a Transformer network structure to process video frame features, then masks some frames using a random masking machine, and then trains the model to recover the masked content. By calculating the difficulty of restoring a frame, the algorithm assigns an importance score to each frame. Frames that are difficult to restore are considered to contain more unique information and are therefore more suitable as keyframes.

[0112] VAsNet is an attention-based video summarization algorithm that utilizes a self-attention module to capture global correlations between video frames. The algorithm first represents the input video as a sequence of features. Then, using the self-attention mechanism, it calculates the correlation weights of each frame with all other frames, enabling the network to automatically focus on the most important regions in the video. After the attention calculation, VASNet applies residual connections and a two-layer feedforward network, ultimately generating an importance score between 0 and 1 for each frame using a sigmoid activation function.

[0113] VSSUM is an unsupervised video summarization algorithm based on reinforcement learning. It processes video feature sequences using a bidirectional LSTM / GRU network to generate selection probabilities for each frame. The algorithm is trained using the REINFORCE framework and incorporates a reward mechanism that combines diversity and representativeness to ensure that summaries contain frames with significant visual differences while also being representative of the overall content.

[0114] Experimental results: Figure 3 As shown in the figure, the original video stream consists of a large number of frame sequences. By comparing the image information differences between two random frames, the similarity is as high as 97.7%. It can be seen that there is a significant similar frame redundancy problem in the garbage classification monitoring video.

[0115] Figure 4 This figure shows the overall performance of the keyframe extraction algorithm in selecting keyframes. The curve represents the amount of information change in the image frame, reflecting the degree of information difference between adjacent frames in a video sequence. When the video scene changes, this curve shows a clear peak. The dots mark the locations of the keyframes selected by the algorithm. Experiments show that these keyframes are mainly distributed at the peak of the curve, indicating that the algorithm successfully captures significant changes in the video.

[0116] As shown in Table 1, the original video contains 19,671 frames. After processing, 5,957 frames were selected as candidate frames, a 69.7% reduction in processing time of 0.78 seconds. Experiments demonstrate that the algorithm not only accurately identifies scene changes but also significantly reduces data volume, improving storage and transmission efficiency. It is therefore suitable for real-time or large-scale video processing scenarios.

[0117] Table 1 Video statistics

[0118]

[0119] A simple keyframe extraction algorithm deployed at the edge layer has achieved promising results, achieving a 69.7% data reduction. To further optimize system performance and meet the strict resource constraints of edge computing environments, this study deployed a lightweight model for keyframe monitoring at the edge layer.

[0120] As shown in Table 2, the lightweight model proposed in this embodiment significantly reduces the model size while maintaining functional integrity. Compared with existing methods such as VASNet (28.03MB), rs-sum (15.03MB), and vssum (10.02MB), this model only occupies 1.79MB of storage space, a volume reduction of approximately 82% to 94%. This optimization greatly reduces the storage and computing pressure on edge devices, allowing the entire video processing pipeline to run more efficiently on resource-constrained edge devices.

[0121] Table 2 Total volume of different algorithm models

[0122]

[0123] In order to ensure the effectiveness of key frame detection in lightweight models, this embodiment uses reinforcement learning to improve model performance. Figure 5 As shown in the figure, the reward coefficient of the model trained using reinforcement learning shows a clear upward trend with each training round. It gradually increases from approximately 0.74 in the initial stage to around 0.87 after 200 rounds, an overall improvement of approximately 17.6%. This demonstrates that the reinforcement learning strategy can effectively compensate for the performance loss caused by model lightweighting.

[0124] Finally, this example applies the algorithm to three data sets and verifies the performance using the F-Score indicator. Figure 6 As shown, the algorithm in this embodiment demonstrates good performance on three datasets with different characteristics. On the SumMe dataset, this method achieves a score of 46.2, slightly lower than several other methods; on the TVSum dataset, it scores 59.5, approaching the level of mainstream methods; and on the self-built dataset, this method achieves an F-Score of 60.1, demonstrating particularly outstanding performance. Although performance is slightly lower than that of large models, this slight performance sacrifice yields a significant reduction in model size and a significant increase in inference speed, demonstrating the advantages of lightweight design in scenario adaptability.

[0125] Table 3 Comparison of F-Score indicators of different algorithms

[0126]

[0127]

[0128] This example introduces the design of a federated garbage classification architecture that collaborates across devices, edges, and clouds, focusing on a key strategy: keyframe extraction. To address the communication overhead and computational costs associated with garbage classification, a three-layer collaborative architecture involving devices, edges, and clouds is proposed to improve data processing efficiency and classification accuracy. Furthermore, a lightweight keyframe extraction algorithm based on a dual attention fusion mechanism is designed, leveraging reinforcement learning to ensure performance while reducing model size. Experiments conducted on both public and self-developed datasets demonstrate that the algorithm can consistently extract valid keyframes across multiple datasets.

[0129] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A lightweight keyframe extraction algorithm based on a dual attention fusion mechanism, characterized by: The algorithm includes an end layer and an edge layer; Deploy high-definition cameras to collect video streams and transmit them to the end layer; In the end layer, the information content of each frame image and the adjacent frame images of the video stream is calculated based on the image difference; If the amount of information is greater than the set first threshold, it is determined to be a candidate frame; Inputting the candidate frame into the edge layer, and constructing a dual attention fusion mechanism in the edge layer, including a spatial attention module, a temporal attention module and a gating mechanism; The spatial attention module identifies and outputs spatial attention features in the candidate frame image; The temporal attention module identifies and outputs temporal attention features between candidate frame images; Using a gating mechanism to fuse the spatial attention features and the temporal attention features to obtain a keyframe score; If the key frame score is greater than the set second threshold, it is considered to be a key frame; A reinforcement learning algorithm is designed to optimize the dual attention fusion mechanism. The reward function of the reinforcement learning algorithm includes: a diversity reward based on the difference between key frames and a representative reward based on the difference between the key frames and the video stream sequence.

2. The algorithm according to claim 1, characterized in that The calculation process of the information volume is as follows: For the video stream sequence {F1, F2, ..., F n }, where F i Represents the i-th frame image, and defines the information volume of the i-th frame image I(F i )for: I(F1)=0 Among them G i It's F i Grayscale image, W and H are the width and height of the frame image, respectively, |G i (x,y)-G i-1 (x,y)| is the absolute value of the grayscale value of the pixel (x,y).

3. The algorithm according to claim 1, characterized in that The calculation process of the spatial attention feature is as follows: First, the query vector Q, key vector K, and value vector V of the candidate frame X are calculated as follows: Q=W q ·F r (X),K=W k ·F r (X),V=W v ·F r (X) Among them, W q is the query weight matrix, W k is the key weight matrix, W v is the value weight matrix, F r (X) is the vector representation of the candidate frame after dimensionality reduction projection; Then, the original attention weight matrix A is calculated by the softmax function raw , as shown below: Next, the original attention weight matrix is operated by the Threshold function, retaining only the largest k attention weights of the original attention weights to generate the top-k weight matrix, as shown below: A sparse =Th resh old(A raw ,top-k) Finally, the top-k weight matrix is dot-producted with the value vector to generate the spatial attention feature A of the candidate frame image. s , the calculation formula is: A s =A sparse ·V.

4. The algorithm according to claim 3, characterized in that The calculation process of the temporal attention feature is as follows: First, the depth convolution function DepthwiseConv is used to perform a depth convolution operation on the transposed candidate frame image, and the pointwise convolution function PointwiseConv is used to perform a depth convolution operation on the depth convolution operation result Z d Perform point-by-point convolution, expressed as follows: Z d =Depth wiseConv(X T ) WITH p =PointwiseConv(Z d ) Among them, Z p is the point-by-point convolution result; Then, the temporal attention feature C is calculated based on the point-by-point convolution result, which is expressed as follows: Where W is the weight, represents the point-by-point convolution result of t time steps, and σ1 is the normalization function.

5. The algorithm according to claim 4, characterized in that The fusion process of the gating mechanism is as follows: G=σ2(W g ·[A s ;C]) F fused =G·A s +(a-G)·C Among them, σ2 is the activation function, [A s ; C] represents the concatenation of spatial attention features and temporal attention features, G is the gating signal, a represents the dynamic fusion weight of spatial attention features and temporal attention features, and the sum of the fusion weight of spatial attention features and the fusion weight of temporal attention features is a, that is, G+(aG)=a; F fused Score keyframes.

6. The algorithm according to claim 1, characterized in that The Diversity Award R div The calculation formula is as follows: Where m is the number of key frames; f p ,f q are the pth and qth key frames respectively, sim(f i ,f j ) is the cosine similarity between keyframes.

7. The algorithm according to claim 1 or 2, characterized in that The representative reward R rep The calculation formula is as follows: Among them, d(F i ,f j ) is the key frame f j With frame image F i min(·) is the minimum function, and S is the set of key frames.

Images