Video key frame extraction method and device, equipment and medium

Abstract

The invention relates to the field of artificial intelligence, the scheme can be applied to the fields of medical treatment and finance, and the invention provides a video key frame extraction method, device, equipment and medium, and the method comprises the steps: converting an input video frame sequence into a feature vector sequence; calculating pre-selected scene metadata and the feature vector sequence through a dynamic sparse attention mechanism, and outputting a corresponding importance weight; determining a key frame set based on the importance weight; and optimizing the key frame set, and outputting a key frame sequence with time sequence coherence. According to the method, attention calculation overhead is remarkably reduced to improve processing efficiency, scene adaptive key frame selection is realized, core information omission and redundancy are avoided, representativeness and coherence of an output sequence are guaranteed, and accuracy, high efficiency and practicability of key frame extraction are considered.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and the solution can be applied to the medical and financial fields, particularly to a method, apparatus, device, and medium for extracting keyframes from videos. Background Technology

[0002] Video keyframe extraction, a core foundational task in computer vision, aims to select the most representative frames from continuous video sequences to achieve efficient representation of video content, supporting downstream applications such as video summarization, surveillance analysis, medical image analysis, and financial security monitoring. Traditional methods often rely on manually designed features and heuristic rules, such as calculating inter-frame similarity through color histogram differences. While computationally efficient, these methods lack semantic understanding in complex scenes, are limited by dynamic backgrounds and sudden changes in lighting, and struggle to meet the refined requirements of specialized scenarios like healthcare and finance. In recent years, deep learning technology has driven performance improvements. Convolutional Neural Network (CNN) methods achieve frame-level importance assessment through spatial feature modeling, while Recurrent Neural Network (RNN) methods further incorporate temporal dependency analysis, strengthening inter-frame association modeling. However, these methods still have obvious drawbacks: the computational complexity of the fully connected attention mechanism increases quadratically with the video length, resulting in a surge in resource consumption when processing long videos; the fixed architecture lacks flexibility and cannot adapt to differentiated needs, such as monitoring that needs to focus on moving targets, education that needs to retain whiteboard notes, medical imaging videos that need to accurately capture key operations or lesion dynamics, and financial monitoring that needs to focus on transaction behavior and abnormal operations, all of which are difficult for existing methods to adapt to.

[0003] While sparse attention mechanisms alleviate efficiency issues, static modes are prone to losing long-range dependencies, and dynamic sparse modes are mostly optimized for single tasks, lacking cross-scenario generalization ability. Furthermore, the independent task training paradigm fails to utilize shared knowledge across related tasks, further limiting its application in specialized scenarios. Therefore, existing video keyframe extraction methods suffer from the problem of balancing computational efficiency and scene adaptability, as well as insufficient utilization of cross-task knowledge. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and medium for extracting video keyframes, aiming to solve the problems of existing video keyframe extraction methods, such as difficulty in balancing computational efficiency and scene adaptability, and insufficient utilization of cross-task knowledge.

[0005] In a first aspect, embodiments of the present invention provide a method for extracting keyframes from a video, the method comprising: Convert the input video frame sequence into a feature vector sequence; The dynamic sparse attention mechanism is used to calculate the importance weights of the pre-selected scene metadata and the feature vector sequence. The set of keyframes is determined based on the aforementioned importance weights; The keyframe set is optimized to output a keyframe sequence with temporal coherence.

[0006] Secondly, embodiments of the present invention also provide a video keyframe extraction device, the device comprising: The conversion unit is used to convert the input video frame sequence into a feature vector sequence; The computing unit is used to calculate the pre-selected scene metadata and the feature vector sequence through a dynamic sparse attention mechanism, and output the corresponding importance weights; A determining unit is used to determine a set of keyframes based on the importance weights; An optimization unit is used to optimize the set of keyframes and output a keyframe sequence with temporal coherence.

[0007] Thirdly, embodiments of the present invention also provide an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect above.

[0008] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, can implement the method described in the first aspect.

[0009] This invention provides a method, apparatus, device, and medium for extracting keyframes from videos. The method includes: converting an input video frame sequence into a feature vector sequence; calculating the importance weights of pre-selected scene metadata and the feature vector sequence using a dynamic sparse attention mechanism; determining a set of keyframes based on the importance weights; and optimizing the set of keyframes to output a keyframe sequence with temporal coherence. This invention significantly reduces the attention computation overhead and improves processing efficiency by converting the video frame sequence into a feature vector sequence, accurately calculating the importance weights using a dynamic sparse attention mechanism in conjunction with scene metadata, and then outputting a keyframe sequence through keyframe filtering and temporal optimization. Simultaneously, it achieves scene-adaptive keyframe selection, avoiding the omission and redundancy of core information, ensuring the representativeness and coherence of the output sequence, and balancing the accuracy, efficiency, and practicality of keyframe extraction. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating the video keyframe extraction method provided in an embodiment of the present invention. Figure 2 A schematic block diagram of a video keyframe extraction device provided in an embodiment of the present invention; Figure 3 A schematic block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0014] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0015] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. Embodiments of this invention provide a method, apparatus, device, and medium for extracting video keyframes. The video keyframe extraction method can be applied to a terminal device or server. The terminal device can be an electronic device such as a mobile phone, tablet computer, laptop computer, desktop computer, personal digital assistant, and wearable device; the server can be a single server or a server cluster composed of multiple servers. The invention will now be described in detail through specific embodiments.

[0016] Figure 1 This is a flowchart illustrating the video keyframe extraction method provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110-S140.

[0017] S110. Convert the input video frame sequence into a feature vector sequence.

[0018] In this embodiment, the Vision Transformer (ViT-Large) is used as the backbone network to convert the input video frame sequence into a high-dimensional feature representation (i.e., a feature vector sequence), and the output is a frame-level feature vector sequence. Where T is the number of video frames, and each feature vector , where represents the d-dimensional depth feature of the t-th frame.

[0019] S120. Calculate the pre-selected scene metadata and the feature vector sequence through a dynamic sparse attention mechanism, and output the corresponding importance weights.

[0020] In this embodiment, scene metadata is... and the feature vector sequence As input, cross-task sparse attention computation is performed through a dynamic sparse attention mechanism, and the final output is frame-level importance weights. ,in, The importance weight of the first frame in the corresponding video frame sequence.

[0021] In medical settings, scene metadata can be set to information such as the digestive tract examination requirements of capsule endoscopy and the location of suspected lesions. Combined with video frame sequences, it can quickly focus on key frames such as mucosal abnormalities and bleeding points, helping doctors to filter core information from tens of thousands of images. In financial scenarios, scene metadata can be configured as ATM transaction scenarios and counter service types. Combined with video frame sequences, it can accurately locate key scenes such as abnormal withdrawals and transaction disputes, taking into account both recognition accuracy and real-time risk control requirements.

[0022] In one embodiment, step S120 includes: dynamically generating a structured sparse pattern based on the feature vector sequence and the scene metadata; and performing cross-task sparse attention calculation on the feature vector sequence and the scene metadata under the constraints of the structured sparse pattern, and outputting the corresponding importance weights.

[0023] In this embodiment, a structured sparse pattern (such as an N:M sparse pattern) is dynamically generated based on the feature vector sequence and the scene metadata. Under the constraints of the structured sparse pattern, cross-task sparse attention is calculated on the feature vector sequence and the scene metadata, and the corresponding importance weights are output.

[0024] In one embodiment, the step of dynamically generating a structured sparse pattern based on the feature vector sequence and the scene metadata includes: calculating a gating vector for each frame using a gating network based on the feature vector sequence and the scene metadata; grouping each gating vector according to a preset structured sparse rule to obtain sparse gating vectors corresponding to each frame; and deriving a corresponding mask matrix based on all the sparse gating vectors to generate the structured sparse pattern based on the mask matrix.

[0025] In this embodiment, the feature vector sequence and the scene metadata are used as input, and the gating network is based on the formula... Calculate the gating vector for each frame; in, Let represent the gating vector of frame t. Each element of the gate vector takes a value between 0 and 1, representing the activation intensity of the corresponding feature dimension. σ is the sigmoid activation function. This represents a vector concatenation operation. and All are learnable parameters. , is the weight matrix of the gated network. Its dimension is line × ( )List, It is the dimension of the feature vector. It is a dimension of scene metadata. , is the bias vector of the gated network; Each gate vector is grouped according to a preset structured sparsity rule (such as a dynamic N:M sparsity mode) to obtain the sparse gate vectors corresponding to each frame. , That is, in each group of M consecutive elements of the gating vector, after sorting the elements from largest to smallest according to the activation intensity of each element, the first N elements are retained and the rest are set to zero, thereby reducing the overhead of subsequent attention calculations. The corresponding mask matrix is ​​derived based on all the sparse gating vectors. The structured sparse pattern is generated based on the mask matrix; specifically, based on all the sparse gate vectors, a corresponding binary mask matrix is ​​formed according to the rule of "non-zero elements are mapped to 1, and zero elements are mapped to 0". .

[0026] In one embodiment, after dynamically generating a structured sparse pattern based on the feature vector sequence and the scene metadata, the method further includes: projecting the feature vector sequence into different spaces, and calculating the context representation of each attention head under the constraints of the structured sparse pattern; concatenating each context representation into a vector to obtain a corresponding joint weight matrix; performing a linear transformation on the joint weight matrix based on a preset weight matrix to obtain a feature importance weight vector; and calculating the importance weight based on the feature importance weight vector.

[0027] In this embodiment, the feature vector sequence is projected into three spaces: query (Q), key (K), and value (V). ,in, It is a learnable projection matrix. It is a dimension; in which each attention head has its own independent set of learnable projection matrices; Based on the mask matrix The structural rules are solidified into structured sparse patterns that can participate in computation. Under the constraints of these structured sparse patterns, the context representation (Head) of each attention head is computed. i i = 1, 2, ..., H, where H is the total number of attention points; Specifically, , It is a mask matrix, ⊙ represents element-wise multiplication, T is the video frame number, and Head i Each of the aforementioned context representations is concatenated into a vector to obtain the corresponding joint weight matrix [Head1,Head2,...,Head...]. H Based on the preset weight matrix W, a linear transformation is performed on the joint weight matrix to obtain the feature importance weight vector. features , features =W×[Head1,Head2,...,Head H Based on the feature importance weight vector features Calculate the importance weight , .

[0028] To support multi-task learning, a shared-private attention head mechanism is adopted. Some attention heads (i.e., shared attention heads) are shared across all tasks to capture general temporal patterns, while other attention heads (i.e., task attention heads) are task-specific and used to learn scenario-specific feature representations. The ratio of the total number of shared attention heads to the total number of task attention heads can be set according to the scenario, with a default ratio of 3:1.

[0029] In one embodiment, after calculating the importance weights based on the feature importance weight vector, the method further includes: obtaining shared parameters common to all tasks and task-specific parameters for each task; for each task, fixing the shared parameters and minimizing the training loss of the task to obtain the optimal specific parameters for each task; fixing the optimal specific parameters of the task and minimizing the sum of the validation losses of all tasks to obtain the optimal shared parameters, wherein the optimal specific parameters and the optimal shared parameters are used to iteratively optimize the accuracy of subsequent importance weight generation.

[0030] In this embodiment, a two-layer optimized meta-learning framework is used to handle shared parameters common to all tasks. and task-specific parameters for each task Optimize; share parameters as learnable parameters ( and ) and the learnable projection matrix of all shared attention heads, with task-specific parameters being the learnable projection matrix of the task attention head.

[0031] The optimization process is performed through two loops, inner and outer, as shown in the following expression: ; in, and They represent the first The training loss and validation loss for each task. Indicates the total number of tasks. Represents the set of all optimal specific parameters; The specific optimization process is as follows: Inner layer optimization: For each task, fix the shared parameters. Without changing the training loss for that task, we can obtain the optimal specific parameters for that task. This allows us to obtain the optimal specific parameters for each task. ; Outer layer optimization: Optimal specific parameters for a fixed task Without changing the parameters, the optimal shared parameters are obtained by minimizing the sum of the validation losses of all tasks, thereby improving its cross-task generalization ability. By iteratively optimizing the inner and outer layers, the optimal shared parameters and the optimal specific parameters for each task are finally obtained. The optimal specific parameters and the optimal shared parameters are used to iteratively optimize the accuracy of subsequent importance weight generation.

[0032] S130. Determine the set of keyframes based on the importance weights.

[0033] In this embodiment, based on the importance weight , The set of keyframes is determined by a differentiable Top-K selector.

[0034] In one embodiment, step S130 includes: determining a set of keyframes based on the importance weights using a differentiable Top-K selector; wherein the differentiable Top-K selector is based on temperature-controlled softmax relaxation.

[0035] In this embodiment, the keyframe set is determined based on the importance weights using a differentiable Top-K selector; the differentiable Top-K selector is based on temperature-controlled softmax relaxation, and its specific expression is as follows: ;in, Let t be the probability value (selection probability) of frame t being selected as a keyframe. As the importance weight of the t-th frame, For adjustable temperature parameters, ,when When, the probability distribution approaches the one-hot form, when As the temperature parameter increases, the selection becomes smoother, the probability differences between frames decrease, and the selection becomes more exploratory. In the early stages of training, a larger temperature parameter is used. To encourage exploration, the temperature parameter was gradually reduced as training progressed. To increase the certainty of choices.

[0036] In one embodiment, the keyframe set is determined based on the importance weights using a differentiable Top-K selector; wherein the differentiable Top-K selector is based on temperature-controlled softmax relaxation and includes: generating temperature parameters using an annealing strategy based on the importance weights; calculating the selection probability of each frame using softmax relaxation with the temperature parameters and the importance weights as inputs; binarizing the selection probabilities based on a preset indicator function and an adjustable threshold to obtain a keyframe indicator value for each frame; and determining the keyframe set based on each keyframe indicator value.

[0037] In this embodiment, an annealing strategy is used to generate temperature parameters based on the importance weights. The annealing strategy uses a larger temperature parameter in the early stages of training. To encourage exploration, the temperature parameter was gradually reduced as training progressed. To improve the certainty of selection; with the temperature parameter and the importance weights As input, the selection probability of each frame is calculated using softmax relaxation, with the specific expression being: ;in, Let t be the probability value (selection probability) of frame t being selected as a keyframe. As the importance weight of the t-th frame, For adjustable temperature parameters, ,when When, the probability distribution approaches the one-hot form, when As the threshold increases, the selection becomes smoother, the probability differences between frames decrease, and the selection becomes more exploratory; based on a preset indicator function and an adjustable threshold. The selection probability is binarized to obtain the keyframe indication value for each frame, specifically expressed as follows: , The adjustable threshold determines the sparsity of the final output keyframes. This sparsity can be controlled by adjusting the adjustable threshold. This results in the final selection of keyframes accounting for approximately 10% to 20% of the total video frames. For the characteristic function, when Output 1 if the condition is met, otherwise output 0; determine the keyframe set based on the keyframe indicator values, i.e., the indicator value of a certain frame. When that happens, the frame is included in the keyframe set.

[0038] S140. Optimize the set of keyframes and output a keyframe sequence with temporal coherence.

[0039] In this embodiment, the selected keyframes are subjected to temporal consistency adjustment and redundancy elimination (such as removing frames with highly similar or repeated content) to ensure that the output keyframe sequence has temporal coherence and representativeness.

[0040] In summary, this invention transforms video frame sequences into feature vector sequences, combines scene metadata with a dynamic sparse attention mechanism to accurately calculate importance weights, and then outputs a keyframe sequence through keyframe filtering and temporal optimization. This significantly reduces the attention computation overhead to improve processing efficiency, while achieving scene-adaptive keyframe selection, avoiding the omission and redundancy of core information, ensuring the representativeness and coherence of the output sequence, and balancing the accuracy, efficiency, and practicality of keyframe extraction.

[0041] Figure 2 This is a schematic block diagram of a video keyframe extraction device provided in an embodiment of the present invention. Figure 2 As shown, corresponding to the above video keyframe extraction method, this invention also provides a video keyframe extraction device. The device is configured in a terminal device or server. The terminal device can be an electronic device such as a mobile phone, tablet computer, laptop computer, desktop computer, personal digital assistant, or wearable device. The server can be a single server or a server cluster composed of multiple servers. For details, please refer to... Figure 2 The video keyframe extraction device 700 includes: The conversion unit 701 is used to convert the input video frame sequence into a feature vector sequence; The computing unit 702 is used to calculate the pre-selected scene metadata and the feature vector sequence through a dynamic sparse attention mechanism, and output the corresponding importance weights; Determining unit 703 is used to determine a set of keyframes based on the importance weights; The optimization unit 704 is used to optimize the key frame set and output a key frame sequence with temporal coherence.

[0042] In some embodiments, when the computing unit 702 performs the step of calculating the pre-selected scene metadata and the feature vector sequence through a dynamic sparse attention mechanism and outputting the corresponding importance weights, it is specifically used for: A structured sparse pattern is dynamically generated based on the feature vector sequence and the scene metadata; under the constraints of the structured sparse pattern, cross-task sparse attention is calculated on the feature vector sequence and the scene metadata, and the corresponding importance weights are output.

[0043] In some embodiments, when the computing unit 702 performs the step of dynamically generating a structured sparse pattern based on the feature vector sequence and the scene metadata, it is specifically used for: Based on the feature vector sequence and the scene metadata, a gating vector for each frame is calculated using a gating network; each gating vector is grouped according to a preset structured sparse rule to obtain sparse gating vectors corresponding to each frame; and a corresponding mask matrix is ​​derived based on all the sparse gating vectors to generate the structured sparse pattern according to the mask matrix.

[0044] In some embodiments, after performing the step of dynamically generating a structured sparse pattern based on the feature vector sequence and the scene metadata, the computing unit 702 is further configured to: The feature vector sequence is projected into different spaces, and under the constraint of the structured sparse pattern, the context representation of each attention head is calculated; each context representation is concatenated into a vector to obtain the corresponding joint weight matrix; the joint weight matrix is ​​linearly transformed based on the preset weight matrix to obtain the feature importance weight vector; the importance weight is calculated based on the feature importance weight vector.

[0045] In some embodiments, after performing the step of calculating the importance weights based on the feature importance weight vector, the calculation unit 702 is further configured to: Obtain shared parameters common to all tasks and task-specific parameters for each task; for each task, keep the shared parameters unchanged and obtain the optimal task-specific parameters by minimizing the training loss of the task; keep the optimal task-specific parameters unchanged and obtain the optimal shared parameters by minimizing the sum of the validation losses of all tasks. The optimal specific parameters and the optimal shared parameters are used to iteratively optimize the accuracy of subsequent importance weight generation.

[0046] In some embodiments, when performing the step of determining the keyframe set based on the importance weight, the determining unit 703 is specifically used for: Based on the importance weights, a set of keyframes is determined by a differentiable Top-K selector; wherein, the differentiable Top-K selector is based on temperature-controlled softmax relaxation.

[0047] In some embodiments, the determining unit 703 determines the set of keyframes based on the importance weights using a differentiable Top-K selector; wherein, when the differentiable Top-K selector uses temperature-controlled softmax relaxation as its core step, it is specifically used for: Based on the importance weights, an annealing strategy is used to generate temperature parameters; using the temperature parameters and the importance weights as inputs, the selection probability of each frame is calculated through softmax relaxation; based on a preset indicator function and an adjustable threshold, the selection probability is binarized to obtain the keyframe indicator value of each frame; and a keyframe set is determined based on each keyframe indicator value.

[0048] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned video keyframe extraction device and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0049] The aforementioned video keyframe extraction device can be implemented as a computer program, which can perform tasks such as... Figure 3 It runs on the electronic device shown.

[0050] Please see Figure 3 , Figure 3 This is a schematic block diagram of an electronic device provided in an embodiment of the present invention. The electronic device 800 can be a terminal or a server. The terminal can be an electronic device with communication functions. The server can be a standalone server or a server cluster composed of multiple servers.

[0051] See Figure 3The electronic device 800 includes a processor 802, a memory, and a network interface 805 connected via a system bus 801. The memory may include a non-volatile storage medium 803 and internal memory 804.

[0052] The non-volatile storage medium 803 may store an operating system 8031 ​​and a computer program 8032. The computer program 8032 includes program instructions that, when executed, cause the processor 802 to perform a video keyframe extraction method.

[0053] The processor 802 provides computing and control capabilities to support the operation of the entire electronic device 800.

[0054] The internal memory 804 provides an environment for the execution of the computer program 8032 in the non-volatile storage medium 803. When the computer program 8032 is executed by the processor 802, the processor 802 can execute a video keyframe extraction method.

[0055] This network interface 805 is used for network communication with other devices. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the electronic device 800 to which the present invention is applied. The specific electronic device 800 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0056] The processor 802 is used to run a computer program 8032 stored in the memory to perform the following steps: The input video frame sequence is converted into a feature vector sequence; the pre-selected scene metadata and the feature vector sequence are calculated using a dynamic sparse attention mechanism to output the corresponding importance weights; a set of key frames is determined based on the importance weights; the set of key frames is optimized to output a key frame sequence with temporal coherence.

[0057] In some embodiments, when the processor 802 calculates the pre-selected scene metadata and the feature vector sequence through a dynamic sparse attention mechanism and outputs the corresponding importance weights, the following steps are specifically implemented: A structured sparse pattern is dynamically generated based on the feature vector sequence and the scene metadata; under the constraints of the structured sparse pattern, cross-task sparse attention is calculated on the feature vector sequence and the scene metadata, and the corresponding importance weights are output.

[0058] In some embodiments, when implementing the step of dynamically generating a structured sparse pattern based on the feature vector sequence and the scene metadata, the processor 802 specifically implements the following steps: Based on the feature vector sequence and the scene metadata, a gating vector for each frame is calculated using a gating network; each gating vector is grouped according to a preset structured sparse rule to obtain sparse gating vectors corresponding to each frame; and a corresponding mask matrix is ​​derived based on all the sparse gating vectors to generate the structured sparse pattern according to the mask matrix.

[0059] In some embodiments, after implementing the step of dynamically generating a structured sparse pattern based on the feature vector sequence and the scene metadata, the processor 802 further implements the following steps: The feature vector sequence is projected into different spaces, and under the constraint of the structured sparse pattern, the context representation of each attention head is calculated; each context representation is concatenated into a vector to obtain the corresponding joint weight matrix; the joint weight matrix is ​​linearly transformed based on the preset weight matrix to obtain the feature importance weight vector; the importance weight is calculated based on the feature importance weight vector.

[0060] In some embodiments, after implementing the step of calculating the importance weight based on the feature importance weight vector, the processor 802 further implements the following steps: Obtain shared parameters common to all tasks and task-specific parameters for each task; for each task, keep the shared parameters unchanged and obtain the optimal task-specific parameters by minimizing the training loss of the task; keep the optimal task-specific parameters unchanged and obtain the optimal shared parameters by minimizing the sum of the validation losses of all tasks. The optimal specific parameters and the optimal shared parameters are used to iteratively optimize the accuracy of subsequent importance weight generation.

[0061] In some embodiments, when implementing the step of determining the keyframe set based on the importance weights, the processor 802 specifically implements the following steps: Based on the importance weights, a set of keyframes is determined by a differentiable Top-K selector; wherein, the differentiable Top-K selector is based on temperature-controlled softmax relaxation.

[0062] In some embodiments, when the processor 802 determines the set of keyframes based on the importance weights using a differentiable Top-K selector, wherein the differentiable Top-K selector uses temperature-controlled softmax relaxation as its core step, the processor specifically implements the following steps: Based on the importance weights, an annealing strategy is used to generate temperature parameters; using the temperature parameters and the importance weights as inputs, the selection probability of each frame is calculated through softmax relaxation; based on a preset indicator function and an adjustable threshold, the selection probability is binarized to obtain the keyframe indicator value of each frame; and a keyframe set is determined based on each keyframe indicator value.

[0063] It should be understood that, in this embodiment of the invention, the processor 802 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0064] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0065] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When executed by a processor, the program instructions cause the processor to perform the following steps: The input video frame sequence is converted into a feature vector sequence; the pre-selected scene metadata and the feature vector sequence are calculated using a dynamic sparse attention mechanism to output the corresponding importance weights; a set of key frames is determined based on the importance weights; the set of key frames is optimized to output a key frame sequence with temporal coherence.

[0066] In one embodiment, when the processor executes the program instructions to calculate the pre-selected scene metadata and the feature vector sequence using a dynamic sparse attention mechanism, and outputs the corresponding importance weights, the specific steps are as follows: A structured sparse pattern is dynamically generated based on the feature vector sequence and the scene metadata; under the constraints of the structured sparse pattern, cross-task sparse attention is calculated on the feature vector sequence and the scene metadata, and the corresponding importance weights are output.

[0067] In one embodiment, when the processor executes the program instructions to dynamically generate a structured sparse pattern based on the feature vector sequence and the scene metadata, it specifically implements the following steps: Based on the feature vector sequence and the scene metadata, a gating vector for each frame is calculated using a gating network; each gating vector is grouped according to a preset structured sparse rule to obtain sparse gating vectors corresponding to each frame; and a corresponding mask matrix is ​​derived based on all the sparse gating vectors to generate the structured sparse pattern according to the mask matrix.

[0068] In one embodiment, after executing the program instructions to dynamically generate a structured sparse pattern based on the feature vector sequence and the scene metadata, the processor further implements the following steps: The feature vector sequence is projected into different spaces, and under the constraint of the structured sparse pattern, the context representation of each attention head is calculated; each context representation is concatenated into a vector to obtain the corresponding joint weight matrix; the joint weight matrix is ​​linearly transformed based on the preset weight matrix to obtain the feature importance weight vector; the importance weight is calculated based on the feature importance weight vector.

[0069] In one embodiment, after executing the program instructions to calculate the importance weights based on the feature importance weight vector, the processor further implements the following steps: Obtain shared parameters common to all tasks and task-specific parameters for each task; for each task, keep the shared parameters unchanged and obtain the optimal task-specific parameters by minimizing the training loss of the task; keep the optimal task-specific parameters unchanged and obtain the optimal shared parameters by minimizing the sum of the validation losses of all tasks. The optimal specific parameters and the optimal shared parameters are used to iteratively optimize the accuracy of subsequent importance weight generation.

[0070] In one embodiment, when the processor executes the program instructions to implement the step of determining the keyframe set based on the importance weight, it specifically implements the following steps: Based on the importance weights, a set of keyframes is determined by a differentiable Top-K selector; wherein, the differentiable Top-K selector is based on temperature-controlled softmax relaxation.

[0071] In one embodiment, the processor, upon executing the program instructions, determines the keyframe set based on the importance weights using a differentiable Top-K selector; wherein, when the differentiable Top-K selector employs temperature-controlled softmax relaxation as its core step, it specifically implements the following steps: Based on the importance weights, an annealing strategy is used to generate temperature parameters; using the temperature parameters and the importance weights as inputs, the selection probability of each frame is calculated through softmax relaxation; based on a preset indicator function and an adjustable threshold, the selection probability is binarized to obtain the keyframe indicator value of each frame; and a keyframe set is determined based on each keyframe indicator value.

[0072] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0073] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0074] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0075] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0076] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0077] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for extracting keyframes from a video, characterized in that, The method includes: Convert the input video frame sequence into a feature vector sequence; The dynamic sparse attention mechanism is used to calculate the importance weights of the pre-selected scene metadata and the feature vector sequence. The set of keyframes is determined based on the aforementioned importance weights; The keyframe set is optimized to output a keyframe sequence with temporal coherence.

2. The video keyframe extraction method according to claim 1, characterized in that, The step involves calculating the importance weights of pre-selected scene metadata and the feature vector sequence using a dynamic sparse attention mechanism, including: A structured sparse pattern is dynamically generated based on the feature vector sequence and the scene metadata; Under the constraints of the structured sparse pattern, cross-task sparse attention is calculated on the feature vector sequence and the scene metadata, and the corresponding importance weights are output.

3. The video keyframe extraction method according to claim 2, characterized in that, The step of dynamically generating a structured sparse pattern based on the feature vector sequence and the scene metadata includes: Based on the feature vector sequence and the scene metadata, a gating vector for each frame is calculated using a gating network; Each gate vector is grouped according to a preset structured sparsity rule to obtain the sparse gate vector corresponding to each frame. Based on all the sparse gate vectors, the corresponding mask matrix is ​​derived to generate the structured sparse pattern according to the mask matrix.

4. The video keyframe extraction method according to claim 2, characterized in that, After dynamically generating the structured sparse pattern based on the feature vector sequence and the scene metadata, the method further includes: The feature vector sequence is projected into different spaces, and the contextual representation of each attention head is calculated under the constraints of the structured sparse pattern. Each of the aforementioned context representations is concatenated into a vector to obtain the corresponding joint weight matrix; A linear transformation is performed on the joint weight matrix based on a preset weight matrix to obtain a feature importance weight vector; The importance weights are calculated based on the feature importance weight vector.

5. The video keyframe extraction method according to claim 4, characterized in that, After calculating the importance weight based on the feature importance weight vector, the method further includes: Retrieve shared parameters common to all tasks and task-specific parameters for each task; For each task, the shared parameters are kept constant, and the optimal specific parameters for each task are obtained by minimizing the training loss of the task. With the optimal specific parameters of the fixed task unchanged, the optimal shared parameters are obtained by minimizing the sum of the verification losses of all tasks. The optimal specific parameters and the optimal shared parameters are used to iteratively optimize the accuracy of subsequent importance weight generation.

6. The video keyframe extraction method according to claim 1, characterized in that, The determination of the keyframe set based on the importance weight includes: Based on the importance weights, a set of keyframes is determined by a differentiable Top-K selector; wherein, the differentiable Top-K selector is based on temperature-controlled softmax relaxation.

7. The video keyframe extraction method according to claim 6, characterized in that, The keyframe set is determined based on the importance weights using a differentiable Top-K selector; wherein the differentiable Top-K selector is based on temperature-controlled softmax relaxation and includes: Based on the aforementioned importance weights, an annealing strategy is used to generate temperature parameters; Using the temperature parameter and the importance weight as input, the selection probability of each frame is calculated through softmax relaxation; Based on a preset indicator function and an adjustable threshold, the selection probability is binarized to obtain the keyframe indicator value for each frame. The keyframe set is determined based on the keyframe indication values.

8. A video keyframe extraction device, characterized in that, The device includes: The conversion unit is used to convert the input video frame sequence into a feature vector sequence; The computing unit is used to calculate the pre-selected scene metadata and the feature vector sequence through a dynamic sparse attention mechanism, and output the corresponding importance weights; A determining unit is used to determine a set of keyframes based on the importance weights; An optimization unit is used to optimize the set of keyframes and output a keyframe sequence with temporal coherence.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the video keyframe extraction method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions that, when executed by a processor, cause the processor to perform the video keyframe extraction method as described in any one of claims 1-7.

Images