Key frame extraction method and device based on few-sample learning, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, provides a key frame extraction method and device based on few sample learning, equipment and a medium, is applied to financial and medical health care service scenes, can construct a loss function based on adversarial loss, perception loss and time sequence consistency loss, realizes collaborative optimization of feature generation, purification and selection, and improves the accuracy of feature extraction. The error accumulation problem of staged training is avoided; a BEGAN generator based on boundary balance generative adversarial network can supplement a large number of high-quality synthetic frames under the condition of few samples, and the problem of scarcity of training data is relieved; interframe time sequence dependence is captured through a space-time attention mechanism based on a time sequence perception discriminator, and the space-time consistency of features can be improved; and noise introduced by adversarial training can be eliminated based on the feature purification sub-module, and the purity of features is ensured, so that high-quality and accurate key frame extraction is realized under the condition of few samples.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for keyframe extraction based on few-shot learning. Background Technology

[0002] Video keyframe extraction is one of the core tasks of video analysis and processing. Its goal is to select the most representative frames from a video sequence to efficiently represent the video content. For example, in the financial field, keyframe extraction from bank branch surveillance videos can identify abnormal behavior, while in the medical field, keyframe extraction from operating room videos can assist in medical teaching.

[0003] Traditional methods primarily rely on manually designed features (such as optical flow and color histograms) or clustering-based methods to extract keyframes. However, these methods have limited generalization ability in complex scenes and are difficult to adapt to few-sample learning conditions. In recent years, deep learning technology has significantly improved keyframe extraction performance through end-to-end training, but it still relies on large-scale labeled data and performs poorly in data-scarce scenarios.

[0004] To address the few-shot learning problem mentioned above, existing technologies have proposed the following strategies: (1) Data augmentation techniques: Generative Adversarial Networks (GANs) are used to generate synthetic samples to expand the training set, but there are still shortcomings in video temporal feature modeling; (2) Feature alignment and purification: For example, based on the generative model, the robustness of adversarial examples is improved by denoising and scoring matching, but the spatiotemporal consistency of the video is not fully considered; (3) Improvement of temporal modeling: Feature purification and generation processes are often separated, leading to information loss.

[0005] It is evident that existing methods have the following limitations: (1) Data dependency: Most methods require a large number of labeled video frames, while synthetic samples generated under conditions of few samples are prone to mode collapse or temporal breakage; (2) Feature inconsistency: Traditional adversarial training generates large differences in inter-frame features, making it difficult to maintain the continuity of video content. Summary of the Invention

[0006] In view of the above, it is necessary to provide a keyframe extraction method, apparatus, device and medium based on few-shot learning, which aims to solve the problem that the keyframe extraction process is highly dependent on data and difficult to maintain temporal continuity.

[0007] A keyframe extraction method based on few-shot learning, the keyframe extraction method based on few-shot learning includes: Collect video data below a preset amount and preprocess the video to obtain a standard frame sequence; The initial network is constructed based on the boundary-balanced generative adversarial network generator BEGAN generator, the temporal-aware discriminator, and the feature purification submodule. A loss function is constructed based on adversarial loss, perceptual loss, and time-series consistency loss. Using the standard frame sequence as training samples, the initial network is trained based on the loss function to obtain the keyframe extraction network; In response to a keyframe extraction instruction for a target video, the target video is processed using the keyframe extraction network to obtain target keyframes.

[0008] A keyframe extraction device based on few-shot learning, the keyframe extraction device based on few-shot learning includes: The preprocessing unit is used to acquire video data below a preset amount and preprocess the video to obtain a standard frame sequence. The building unit is used to construct the initial network based on the boundary-balanced generative adversarial network generator BEGAN generator, the temporal-aware discriminator, and the feature purification submodule. The construction unit is also used to construct a loss function based on adversarial loss, perceptual loss, and time-series consistency loss; The training unit is used to train the initial network based on the loss function using the standard frame sequence as training samples to obtain the keyframe extraction network. The processing unit is configured to process the target video using the keyframe extraction network in response to a keyframe extraction instruction for the target video, thereby obtaining the target keyframes.

[0009] A computer device, the computer device comprising: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the keyframe extraction method based on few-shot learning.

[0010] A computer-readable storage medium storing at least one instruction, which is executed by a processor in a computer device to implement the keyframe extraction method based on few-shot learning.

[0011] As can be seen from the above technical solutions, this invention can construct a loss function based on adversarial loss, perceptual loss, and temporal consistency loss, thereby achieving collaborative optimization of feature generation, purification, and selection, and avoiding the error accumulation problem of staged training. The BEGAN generator based on boundary balance can supplement a large number of high-quality synthetic frames under conditions of few samples, alleviating the problem of scarce training data. The temporal perceptual discriminator captures the temporal dependencies between frames through a spatiotemporal attention mechanism, which can improve the spatiotemporal consistency of features. The feature purification submodule can eliminate the noise introduced by adversarial training and ensure the purity of features, thus achieving high-quality and accurate keyframe extraction under conditions of few samples. Attached Figure Description

[0012] Figure 1 This is a flowchart of a preferred embodiment of the keyframe extraction method based on few-shot learning of the present invention.

[0013] Figure 2 This is a functional block diagram of a preferred embodiment of the keyframe extraction device based on few-sample learning of the present invention.

[0014] Figure 3 This is a schematic diagram of the structure of a computer device that implements a preferred embodiment of the keyframe extraction method based on few-shot learning according to the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the keyframe extraction method based on few-shot learning according to the present invention. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.

[0017] The keyframe extraction method based on few-shot learning is applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0018] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0019] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0020] The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0021] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0022] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0023] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0024] S10: Collect video data below a preset amount and preprocess the video to obtain a standard frame sequence.

[0025] In this embodiment, the preset data volume can correspond to a small-scale data volume.

[0026] In this embodiment, the video can be from any field, such as surveillance video of a bank branch in the financial field, or video from an operating room in the medical field.

[0027] In this embodiment, the preprocessing of the video to obtain a standard frame sequence includes: Extract a video frame sequence consisting of a pixel matrix for each frame from the video; The standard frame sequence is obtained by mapping the pixel matrix of each frame to a preset pixel value range and a preset size.

[0028] The preset dimensions correspond to the specified width and height.

[0029] The preset pixel value range and the preset size can be configured according to the actual system processing requirements and performance.

[0030] The above embodiments can eliminate the differences in size and pixel distribution between different video sources, provide input data with a uniform format for subsequent processing, and reduce the risk of distribution shift during model training.

[0031] S11. The initial network is constructed based on the Boundary Equilibrium Generative Adversarial Networks (BEGAN) generator, the temporal-aware discriminator, and the feature purification submodule.

[0032] In this embodiment, the construction of the initial network based on the boundary-balanced generative adversarial network (BEGAN) generator, the time-aware discriminator, and the feature purification submodule includes: The BEGAN generator is constructed using an encoder-decoder structure with residual connections; wherein the BEGAN generator takes Gaussian noise as input. A multi-head self-attention layer is embedded in an encoder-decoder structure with residual connections to obtain the time-aware discriminator. The last layer of the encoder-decoder structure with residual connections is replaced with a Sigmoid activation function layer to obtain a scoring network, and the feature purification submodule is constructed based on the scoring network. The initial network is obtained by sequentially connecting the BEGAN generator, the time-aware discriminator, and the feature purification submodule.

[0033] The initial network may further include a keyframe selection module, which is used to select a subset of keyframes based on the importance score of video frames and an adaptive threshold method.

[0034] The keyframe selection module can be connected to the output of the feature purification submodule.

[0035] S12, construct a loss function based on adversarial loss, perceptual loss, and time-series consistency loss.

[0036] In this embodiment, the construction of the loss function based on adversarial loss, perceptual loss, and time-series consistency loss includes: The video frames input to the BEGAN generator are obtained as real frames, and the video frames output by the BEGAN generator are obtained as composite frames. The real frame is input to the time-aware discriminator to obtain the real frame reconstruction result, and the synthesized frame is input to the time-aware discriminator to obtain the synthesized frame reconstruction result; The L1 norm of the real frame and the reconstruction result of the real frame is calculated as the first L1 norm, and the L1 norm of the composite frame and the reconstruction result of the composite frame is calculated as the second L1 norm. The difference between the first L1 norm and the second L1 norm is calculated to obtain the adversarial loss; The real frames are input into the VGG-19 network (Visual Geometry Group 19-layer network), and the feature activation values ​​of a specified network layer in the VGG-19 network are extracted as the first feature activation values; wherein, the specified network layer is a configured set of multiple feature extraction layers that contain local image details and reflect global semantics. The synthesized frame is input into the VGG-19 network, and the feature activation values ​​of the specified network layer in the VGG-19 network are extracted as the second feature activation values. The perceptual loss is obtained by summing the L2 norms of the first feature activation value and the second feature activation value corresponding to each specified network layer. Obtain multiple clean spatiotemporal features output by the feature purification submodule; Calculate the L2 norm of two adjacent clean spatiotemporal features sequentially according to the time order of the video frames; The temporal consistency loss is obtained by summing the L2 norms of all adjacent pure spatiotemporal features. Obtain the first hyperparameter corresponding to the adversarial loss, obtain the second hyperparameter corresponding to the perceptual loss, and obtain the third hyperparameter corresponding to the temporal consistency loss; The loss function is obtained by weighting the first hyperparameter, the second hyperparameter, the third hyperparameter, the adversarial loss, the perceptual loss, and the temporal consistency loss.

[0037] The adversarial loss is used to drive the distribution of generated frames to approximate real frames, the perceptual loss is used to constrain the semantic similarity between generated frames and real frames in a specified feature space of VGG-19, and the temporal consistency loss is used to ensure smooth transition of features between adjacent frames.

[0038] The first hyperparameter, the second hyperparameter, and the third hyperparameter are used to control the relative importance of each loss.

[0039] By constructing the above loss function, we can achieve coordinated optimization of feature generation, purification, and selection, avoid the error accumulation problem of traditional staged training, and improve the overall stability of the model.

[0040] S13, using the standard frame sequence as training samples, train the initial network based on the loss function to obtain the keyframe extraction network.

[0041] In this embodiment, the loss gradient can be used to achieve end-to-end optimization of the entire module through backpropagation in order to train the keyframe extraction network.

[0042] S14, in response to the keyframe extraction instruction for the target video, the target video is processed using the keyframe extraction network to obtain the target keyframe.

[0043] In this embodiment, the keyframe extraction instruction can be automatically triggered when the target video is detected to have been uploaded to a designated platform.

[0044] In this embodiment, processing the target video using the keyframe extraction network to obtain target keyframes includes: The target video is preprocessed to obtain a target frame sequence; A potential noise vector is sampled from a Gaussian distribution and then input into the BEGAN generator. Obtain the frame sequence output by the BEGAN generator; The union of the frame sequence output by the BEGAN generator and the target frame sequence is calculated to obtain the key frame candidate set; The time-aware discriminator is used to process the keyframe candidate set to obtain spatiotemporal correlation features of the target with spatiotemporal awareness; The feature purification submodule is used to denoise and optimize the target spatiotemporal correlation features to obtain clean spatiotemporal features. The semantic importance score of each frame in the keyframe candidate set is calculated based on the pure spatiotemporal features. The target keyframe is selected from the keyframe candidate set based on the adaptive thresholding method and the semantic importance score of each frame.

[0045] The target video can be preprocessed using the same preprocessing method as described above, which will not be repeated here.

[0046] Among them, it can be derived from the Gaussian distribution. z ~ N (0,I Sampling in) d The BEGAN generator generates a potential noise vector and maps the random noise vector to a synthetic frame, so as to supplement a large number of high-quality synthetic frames under the condition of few samples, alleviate the problem of scarce training data, and the balanced training strategy of the BEGAN generator can effectively avoid the mode collapse problem of traditional adversarial generative networks.

[0047] In this embodiment, the step of processing the keyframe candidate set using the time-aware discriminator to obtain spatiotemporal correlation features of the target with spatiotemporal awareness includes: The time-aware discriminator is used to extract single-frame features from the keyframe candidate set to obtain the depth features of each frame; The multi-head self-attention layer of the time-aware discriminator processes the depth features of each frame to obtain inter-frame association weights. The depth features of all frames are weighted and summed according to the inter-frame association weights to obtain the weighted features; The weighted features are subjected to layer normalization to obtain the target spatiotemporal correlation features.

[0048] In the multi-head self-attention layer, inter-frame attention weights can be calculated based on the multi-head self-attention mechanism to serve as the inter-frame association weights.

[0049] In the above embodiments, the temporal-aware discriminator can capture inter-frame temporal dependencies through a spatiotemporal attention mechanism, allowing the model to adaptively focus on key areas such as areas with intense motion, thereby improving the spatiotemporal consistency of features.

[0050] In this embodiment, the step of using the feature purification submodule to denoise and optimize the target spatiotemporal correlation features to obtain clean spatiotemporal features includes: Gaussian noise is added to the target spatiotemporal correlation features to obtain initial noise features; Starting with the initial noise features, iterative denoising is performed based on the Langevin dynamics algorithm; wherein, in each iteration, the gradient of the log probability density of the noise features in each round is estimated using the scoring network of the feature purification submodule as the score for each round. After a preset number of iterations, the currently obtained feature is determined as the pure spatiotemporal feature.

[0051] The score output by the scoring network can be represented as follows: ; in, Indicates the first k The score of the noisy feature at the next iteration; Indicates spatiotemporal correlation characteristics; This represents the true probability distribution followed by spatiotemporal correlation features; This indicates that the Hamiltonian operator is applied to spatiotemporal correlation features.

[0052] The formula for iterative denoising based on the Langevin dynamics algorithm can be expressed as follows: ; in, Indicates the first k Features after +1 iterations; Indicates the first k Features after the next iteration; Indicates the learning step size; Indicates standard Gaussian noise; This represents a standard normal distribution.

[0053] The preset number of times can be configured according to actual accuracy requirements.

[0054] In the above embodiments, the noise introduced by adversarial training can be eliminated based on the feature purification submodule, thereby improving the robustness of features under interference such as occlusion and changes in lighting, and ensuring the purity of features.

[0055] Furthermore, based on the pure spatiotemporal features, the semantic importance score of each frame is calculated, and then an adaptive threshold method is used to select a subset of key frames. Without the need to manually design selection rules, the selection of key frames is completed directly based on the semantic value of deep features, which improves the extraction accuracy under conditions of few samples.

[0056] For example, consider a 10-second surveillance video from a bank lobby in the financial sector. This video contains human movements, with an input frame rate of 30fps. After inputting 300 frames of original video containing three main actions—walking, jumping, and sitting—into the keyframe extraction network (the original video suffers from motion blur and uneven lighting), traditional video frame extraction methods would select frames [25, 150, 275] as keyframes. However, frames 150 would be blurry, making action recognition difficult. Using the video frame extraction method of this embodiment, a clear action transition frame (such as the jump sequence in frames 148-152) is first generated by the BEGAN generator. Then, a temporal-aware discriminator assigns higher weights to the motion region, and a feature purification submodule eliminates noise interference from blurry frames. Finally, frames [30, 155, 280] are selected to accurately capture the moment of action transition, significantly improving image quality.

[0057] For example, consider a 20-second video of a laparoscopic cholecystectomy in the medical field. This video includes surgical instrument manipulation and has an input frame rate of 25fps. After inputting 500 frames of original video containing the three core operations of incision establishment, gallbladder separation, and hemostasis suturing into the keyframe extraction network (this original video has local blurring problems caused by blood occlusion and lens reflection), traditional video frame extraction methods would select frames [80, 250, 420] as keyframes. However, frames 250 are difficult to identify due to blood occlusion, making it difficult to distinguish the boundary between the gallbladder and the tissue. The video frame extraction method in this embodiment can first generate clear operation transition frames (such as the gallbladder separation sequence in frames 245-255) by the BEGAN generator, then assign higher weights to the "instrument-tissue contact area" by the temporal awareness discriminator, and eliminate noise interference from occluded frames by the feature purification submodule. Finally, frames [85, 255, 425] are selected to accurately capture the key interactive moments of the surgical operation. The clarity of tissue boundaries and instrument positions in the frames is significantly improved, which is convenient for subsequent surgical procedure quality control and teaching review.

[0058] As can be seen from the above technical solutions, this invention can construct a loss function based on adversarial loss, perceptual loss, and temporal consistency loss, thereby achieving collaborative optimization of feature generation, purification, and selection, and avoiding the error accumulation problem of staged training. The BEGAN generator based on boundary balance can supplement a large number of high-quality synthetic frames under conditions of few samples, alleviating the problem of scarce training data. The temporal perceptual discriminator captures the temporal dependencies between frames through a spatiotemporal attention mechanism, which can improve the spatiotemporal consistency of features. The feature purification submodule can eliminate the noise introduced by adversarial training and ensure the purity of features, thus achieving high-quality and accurate keyframe extraction under conditions of few samples.

[0059] like Figure 2 The diagram shown is a functional block diagram of a preferred embodiment of the keyframe extraction device based on few-shot learning of the present invention. The keyframe extraction device 11 based on few-shot learning includes a preprocessing unit 110, a construction unit 111, a training unit 112, and a processing unit 113. The module / unit referred to in this invention is a series of computer program segments that can be executed by a processor and perform a fixed function, and which are stored in memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0060] The preprocessing unit 110 is used to acquire video with a data volume lower than a preset value and preprocess the video to obtain a standard frame sequence.

[0061] In this embodiment, the preset data volume can correspond to a small-scale data volume.

[0062] In this embodiment, the video can be from any field, such as surveillance video of a bank branch in the financial field, or video from an operating room in the medical field.

[0063] In this embodiment, the preprocessing unit 110 preprocesses the video to obtain a standard frame sequence, including: Extract a video frame sequence consisting of a pixel matrix for each frame from the video; The standard frame sequence is obtained by mapping the pixel matrix of each frame to a preset pixel value range and a preset size.

[0064] The preset dimensions correspond to the specified width and height.

[0065] The preset pixel value range and the preset size can be configured according to the actual system processing requirements and performance.

[0066] The above embodiments can eliminate the differences in size and pixel distribution between different video sources, provide input data with a uniform format for subsequent processing, and reduce the risk of distribution shift during model training.

[0067] The building unit 111 is used to construct an initial network based on the Boundary Equilibrium Generative Adversarial Networks (BEGAN) generator, the temporal-aware discriminator, and the feature purification submodule.

[0068] In this embodiment, the construction unit 111 constructs an initial network based on a boundary-balanced generative adversarial network (BEGAN) generator, a temporal-aware discriminator, and a feature purification submodule, including: The BEGAN generator is constructed using an encoder-decoder structure with residual connections; wherein the BEGAN generator takes Gaussian noise as input. A multi-head self-attention layer is embedded in an encoder-decoder structure with residual connections to obtain the time-aware discriminator. The last layer of the encoder-decoder structure with residual connections is replaced with a Sigmoid activation function layer to obtain a scoring network, and the feature purification submodule is constructed based on the scoring network. The initial network is obtained by sequentially connecting the BEGAN generator, the time-aware discriminator, and the feature purification submodule.

[0069] The initial network may further include a keyframe selection module, which is used to select a subset of keyframes based on the importance score of video frames and an adaptive threshold method.

[0070] The keyframe selection module can be connected to the output of the feature purification submodule.

[0071] The construction unit 111 is also used to construct a loss function based on adversarial loss, perceptual loss, and time-series consistency loss.

[0072] In this embodiment, the construction unit 111 constructs a loss function based on adversarial loss, perceptual loss, and time-series consistency loss, including: The video frames input to the BEGAN generator are obtained as real frames, and the video frames output by the BEGAN generator are obtained as composite frames. The real frame is input to the time-aware discriminator to obtain the real frame reconstruction result, and the synthesized frame is input to the time-aware discriminator to obtain the synthesized frame reconstruction result; The L1 norm of the real frame and the reconstruction result of the real frame is calculated as the first L1 norm, and the L1 norm of the composite frame and the reconstruction result of the composite frame is calculated as the second L1 norm. The difference between the first L1 norm and the second L1 norm is calculated to obtain the adversarial loss; The real frames are input into the VGG-19 network (Visual Geometry Group 19-layer network), and the feature activation values ​​of a specified network layer in the VGG-19 network are extracted as the first feature activation values; wherein, the specified network layer is a configured set of multiple feature extraction layers that contain local image details and reflect global semantics. The synthesized frame is input into the VGG-19 network, and the feature activation values ​​of the specified network layer in the VGG-19 network are extracted as the second feature activation values. The perceptual loss is obtained by summing the L2 norms of the first feature activation value and the second feature activation value corresponding to each specified network layer. Obtain multiple clean spatiotemporal features output by the feature purification submodule; Calculate the L2 norm of two adjacent clean spatiotemporal features sequentially according to the time order of the video frames; The temporal consistency loss is obtained by summing the L2 norms of all adjacent pure spatiotemporal features. Obtain the first hyperparameter corresponding to the adversarial loss, obtain the second hyperparameter corresponding to the perceptual loss, and obtain the third hyperparameter corresponding to the temporal consistency loss; The loss function is obtained by weighting the first hyperparameter, the second hyperparameter, the third hyperparameter, the adversarial loss, the perceptual loss, and the temporal consistency loss.

[0073] The adversarial loss is used to drive the distribution of generated frames to approximate real frames, the perceptual loss is used to constrain the semantic similarity between generated frames and real frames in a specified feature space of VGG-19, and the temporal consistency loss is used to ensure smooth transition of features between adjacent frames.

[0074] The first hyperparameter, the second hyperparameter, and the third hyperparameter are used to control the relative importance of each loss.

[0075] By constructing the above loss function, we can achieve coordinated optimization of feature generation, purification, and selection, avoid the error accumulation problem of traditional staged training, and improve the overall stability of the model.

[0076] The training unit 112 is used to train the initial network based on the loss function using the standard frame sequence as training samples to obtain the keyframe extraction network.

[0077] In this embodiment, the loss gradient can be used to achieve end-to-end optimization of the entire module through backpropagation in order to train the keyframe extraction network.

[0078] The processing unit 113 is configured to process the target video using the keyframe extraction network in response to a keyframe extraction instruction for the target video, thereby obtaining target keyframes.

[0079] In this embodiment, the keyframe extraction instruction can be automatically triggered when the target video is detected to have been uploaded to a designated platform.

[0080] In this embodiment, the processing unit 113 uses the keyframe extraction network to process the target video to obtain target keyframes, including: The target video is preprocessed to obtain a target frame sequence; A potential noise vector is sampled from a Gaussian distribution and then input into the BEGAN generator. Obtain the frame sequence output by the BEGAN generator; The union of the frame sequence output by the BEGAN generator and the target frame sequence is calculated to obtain the key frame candidate set; The time-aware discriminator is used to process the keyframe candidate set to obtain spatiotemporal correlation features of the target with spatiotemporal awareness; The feature purification submodule is used to denoise and optimize the target spatiotemporal correlation features to obtain clean spatiotemporal features. The semantic importance score of each frame in the keyframe candidate set is calculated based on the pure spatiotemporal features. The target keyframe is selected from the keyframe candidate set based on the adaptive thresholding method and the semantic importance score of each frame.

[0081] The target video can be preprocessed using the same preprocessing method as described above, which will not be repeated here.

[0082] Among them, it can be derived from the Gaussian distribution. z ~ N (0, I Sampling in) d The BEGAN generator generates a potential noise vector and maps the random noise vector to a synthetic frame, so as to supplement a large number of high-quality synthetic frames under the condition of few samples, alleviate the problem of scarce training data, and the balanced training strategy of the BEGAN generator can effectively avoid the mode collapse problem of traditional adversarial generative networks.

[0083] In this embodiment, the step of processing the keyframe candidate set using the time-aware discriminator to obtain spatiotemporal correlation features of the target with spatiotemporal awareness includes: The time-aware discriminator is used to extract single-frame features from the keyframe candidate set to obtain the depth features of each frame; The multi-head self-attention layer of the time-aware discriminator processes the depth features of each frame to obtain inter-frame association weights. The depth features of all frames are weighted and summed according to the inter-frame association weights to obtain the weighted features; The weighted features are subjected to layer normalization to obtain the target spatiotemporal correlation features.

[0084] In the multi-head self-attention layer, inter-frame attention weights can be calculated based on the multi-head self-attention mechanism to serve as the inter-frame association weights.

[0085] In the above embodiments, the temporal-aware discriminator can capture inter-frame temporal dependencies through a spatiotemporal attention mechanism, allowing the model to adaptively focus on key areas such as areas with intense motion, thereby improving the spatiotemporal consistency of features.

[0086] In this embodiment, the step of using the feature purification submodule to denoise and optimize the target spatiotemporal correlation features to obtain clean spatiotemporal features includes: Gaussian noise is added to the target spatiotemporal correlation features to obtain initial noise features; Starting with the initial noise features, iterative denoising is performed based on the Langevin dynamics algorithm; wherein, in each iteration, the gradient of the log probability density of the noise features in each round is estimated using the scoring network of the feature purification submodule as the score for each round. After a preset number of iterations, the currently obtained feature is determined as the pure spatiotemporal feature.

[0087] The score output by the scoring network can be represented as follows: ; in, Indicates the first k The score of the noisy feature at the next iteration; Indicates spatiotemporal correlation characteristics; This represents the true probability distribution followed by spatiotemporal correlation features; This indicates that the Hamiltonian operator is applied to spatiotemporal correlation features.

[0088] The formula for iterative denoising based on the Langevin dynamics algorithm can be expressed as follows: ; in, Indicates the first k Features after +1 iterations; Indicates the first k Features after the next iteration; Indicates the learning step size; Indicates standard Gaussian noise; This represents a standard normal distribution.

[0089] The preset number of times can be configured according to actual accuracy requirements.

[0090] In the above embodiments, the noise introduced by adversarial training can be eliminated based on the feature purification submodule, thereby improving the robustness of features under interference such as occlusion and changes in lighting, and ensuring the purity of features.

[0091] Furthermore, based on the pure spatiotemporal features, the semantic importance score of each frame is calculated, and then an adaptive threshold method is used to select a subset of key frames. Without the need to manually design selection rules, the selection of key frames is completed directly based on the semantic value of deep features, which improves the extraction accuracy under conditions of few samples.

[0092] For example, consider a 10-second surveillance video from a bank lobby in the financial sector. This video contains human movements, with an input frame rate of 30fps. After inputting 300 frames of original video containing three main actions—walking, jumping, and sitting—into the keyframe extraction network (the original video suffers from motion blur and uneven lighting), traditional video frame extraction methods would select frames [25, 150, 275] as keyframes. However, frames 150 would be blurry, making action recognition difficult. Using the video frame extraction method of this embodiment, a clear action transition frame (such as the jump sequence in frames 148-152) is first generated by the BEGAN generator. Then, a temporal-aware discriminator assigns higher weights to the motion region, and a feature purification submodule eliminates noise interference from blurry frames. Finally, frames [30, 155, 280] are selected to accurately capture the moment of action transition, significantly improving image quality.

[0093] For example, consider a 20-second video of a laparoscopic cholecystectomy in the medical field. This video includes surgical instrument manipulation and has an input frame rate of 25fps. After inputting 500 frames of original video containing the three core operations of incision establishment, gallbladder separation, and hemostasis suturing into the keyframe extraction network (this original video has local blurring problems caused by blood occlusion and lens reflection), traditional video frame extraction methods would select frames [80, 250, 420] as keyframes. However, frames 250 are difficult to identify due to blood occlusion, making it difficult to distinguish the boundary between the gallbladder and the tissue. The video frame extraction method in this embodiment can first generate clear operation transition frames (such as the gallbladder separation sequence in frames 245-255) by the BEGAN generator, then assign higher weights to the "instrument-tissue contact area" by the temporal awareness discriminator, and eliminate noise interference from occluded frames by the feature purification submodule. Finally, frames [85, 255, 425] are selected to accurately capture the key interactive moments of the surgical operation. The clarity of tissue boundaries and instrument positions in the frames is significantly improved, which is convenient for subsequent surgical procedure quality control and teaching review.

[0094] As can be seen from the above technical solutions, this invention can construct a loss function based on adversarial loss, perceptual loss, and temporal consistency loss, thereby achieving collaborative optimization of feature generation, purification, and selection, and avoiding the error accumulation problem of staged training. The BEGAN generator based on boundary balance can supplement a large number of high-quality synthetic frames under conditions of few samples, alleviating the problem of scarce training data. The temporal perceptual discriminator captures the temporal dependencies between frames through a spatiotemporal attention mechanism, which can improve the spatiotemporal consistency of features. The feature purification submodule can eliminate the noise introduced by adversarial training and ensure the purity of features, thus achieving high-quality and accurate keyframe extraction under conditions of few samples.

[0095] like Figure 3The diagram shown is a schematic representation of the structure of a computer device that implements a keyframe extraction method based on few-shot learning according to a preferred embodiment of the present invention.

[0096] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a keyframe extraction program based on few-shot learning.

[0097] Those skilled in the art will understand that the schematic diagram is merely an example of computer device 1 and does not constitute a limitation on computer device 1. Computer device 1 can be either a bus topology or a star topology. Computer device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, computer device 1 may also include input / output devices, network access devices, etc.

[0098] It should be noted that the computer device 1 described is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0099] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a portable hard drive of the computer device 1. In other embodiments, the memory 12 can be an external storage device of the computer device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device 1. Furthermore, the memory 12 can include both internal and external storage units of the computer device 1. The memory 12 can be used not only to store application software and various types of data installed on the computer device 1, such as the code of a keyframe extraction program based on few-shot learning, but also to temporarily store data that has been output or will be output.

[0100] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the computer device 1, connecting various components of the computer device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing a keyframe extraction program based on few-shot learning) and calls data stored in the memory 12 to perform various functions of the computer device 1 and process data.

[0101] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes these applications to implement the steps in the various embodiments of the keyframe extraction method based on few-shot learning described above, for example... Figure 1 The steps are shown.

[0102] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into a preprocessing unit 110, a construction unit 111, a training unit 112, and a processing unit 113.

[0103] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the keyframe extraction method based on few-shot learning described in the various embodiments of the present invention.

[0104] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0105] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0106] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0107] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0108] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The bus is represented by only one straight line, but this does not mean that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0109] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0110] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish a communication connection between the computer device 1 and other computer devices.

[0111] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the computer device 1 and to display a visual user interface.

[0112] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0113] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0114] Combination Figure 1 The memory 12 in the computer device 1 stores multiple instructions to implement a keyframe extraction method based on few-shot learning, and the processor 13 can execute the multiple instructions to achieve the following: Collect video data below a preset amount and preprocess the video to obtain a standard frame sequence; The initial network is constructed based on the boundary-balanced generative adversarial network generator BEGAN generator, the temporal-aware discriminator, and the feature purification submodule. A loss function is constructed based on adversarial loss, perceptual loss, and time-series consistency loss. Using the standard frame sequence as training samples, the initial network is trained based on the loss function to obtain the keyframe extraction network; In response to a keyframe extraction instruction for a target video, the target video is processed using the keyframe extraction network to obtain target keyframes.

[0115] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0116] It should be noted that all data involved in this case was legally obtained. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0117] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0118] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0119] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0120] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0121] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0122] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0123] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this invention can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A keyframe extraction method based on few-shot learning, characterized in that, The keyframe extraction method based on few-shot learning includes: Collect video data below a preset amount and preprocess the video to obtain a standard frame sequence; The initial network is constructed based on the boundary-balanced generative adversarial network generator BEGAN generator, the temporal-aware discriminator, and the feature purification submodule. A loss function is constructed based on adversarial loss, perceptual loss, and time-series consistency loss. Using the standard frame sequence as training samples, the initial network is trained based on the loss function to obtain the keyframe extraction network; In response to a keyframe extraction instruction for a target video, the target video is processed using the keyframe extraction network to obtain target keyframes.

2. The keyframe extraction method based on few-shot learning as described in claim 1, characterized in that, The preprocessing of the video to obtain a standard frame sequence includes: Extract a video frame sequence consisting of a pixel matrix for each frame from the video; The standard frame sequence is obtained by mapping the pixel matrix of each frame to a preset pixel value range and a preset size.

3. The keyframe extraction method based on few-shot learning as described in claim 1, characterized in that, The initial network constructed using the boundary-balanced generative adversarial network (BEGAN) generator, the time-aware discriminator, and the feature purification submodule includes: The BEGAN generator is constructed using an encoder-decoder structure with residual connections; wherein the BEGAN generator takes Gaussian noise as input. A multi-head self-attention layer is embedded in an encoder-decoder structure with residual connections to obtain the time-aware discriminator. The last layer of the encoder-decoder structure with residual connections is replaced with a Sigmoid activation function layer to obtain a scoring network, and the feature purification submodule is constructed based on the scoring network. The initial network is obtained by sequentially connecting the BEGAN generator, the time-aware discriminator, and the feature purification submodule.

4. The keyframe extraction method based on few-shot learning as described in claim 1, characterized in that, The loss function constructed based on adversarial loss, perceptual loss, and time-series consistency loss includes: The video frames input to the BEGAN generator are obtained as real frames, and the video frames output by the BEGAN generator are obtained as composite frames. The real frame is input to the time-aware discriminator to obtain the real frame reconstruction result, and the synthesized frame is input to the time-aware discriminator to obtain the synthesized frame reconstruction result; The L1 norm of the real frame and the reconstruction result of the real frame is calculated as the first L1 norm, and the L1 norm of the composite frame and the reconstruction result of the composite frame is calculated as the second L1 norm. The difference between the first L1 norm and the second L1 norm is calculated to obtain the adversarial loss; The real frame is input into the VGG-19 network, and the feature activation value of a specified network layer in the VGG-19 network is extracted as the first feature activation value; wherein, the specified network layer is a configured set of multiple feature extraction layers that contain local image details and reflect global semantics; The synthesized frame is input into the VGG-19 network, and the feature activation values ​​of the specified network layer in the VGG-19 network are extracted as the second feature activation values. The perceptual loss is obtained by summing the L2 norms of the first feature activation value and the second feature activation value corresponding to each specified network layer. Obtain multiple clean spatiotemporal features output by the feature purification submodule; Calculate the L2 norm of two adjacent clean spatiotemporal features sequentially according to the time order of the video frames; The temporal consistency loss is obtained by summing the L2 norms of all adjacent pure spatiotemporal features. Obtain the first hyperparameter corresponding to the adversarial loss, obtain the second hyperparameter corresponding to the perceptual loss, and obtain the third hyperparameter corresponding to the temporal consistency loss; The loss function is obtained by weighting the first hyperparameter, the second hyperparameter, the third hyperparameter, the adversarial loss, the perceptual loss, and the temporal consistency loss.

5. The keyframe extraction method based on few-shot learning as described in claim 3, characterized in that, The process of using the keyframe extraction network to process the target video to obtain target keyframes includes: The target video is preprocessed to obtain a target frame sequence; A potential noise vector is sampled from a Gaussian distribution and then input into the BEGAN generator. Obtain the frame sequence output by the BEGAN generator; The union of the frame sequence output by the BEGAN generator and the target frame sequence is calculated to obtain the key frame candidate set; The time-aware discriminator is used to process the keyframe candidate set to obtain spatiotemporal correlation features of the target with spatiotemporal awareness; The feature purification submodule is used to denoise and optimize the target spatiotemporal correlation features to obtain clean spatiotemporal features. The semantic importance score of each frame in the keyframe candidate set is calculated based on the pure spatiotemporal features. The target keyframe is selected from the keyframe candidate set based on the adaptive thresholding method and the semantic importance score of each frame.

6. The keyframe extraction method based on few-shot learning as described in claim 5, characterized in that, The step of processing the keyframe candidate set using the time-aware discriminator to obtain spatiotemporal correlation features of the target with spatiotemporal awareness includes: The time-aware discriminator is used to extract single-frame features from the keyframe candidate set to obtain the depth features of each frame; The multi-head self-attention layer of the time-aware discriminator processes the depth features of each frame to obtain inter-frame association weights. The depth features of all frames are weighted and summed according to the inter-frame association weights to obtain the weighted features; The weighted features are subjected to layer normalization to obtain the target spatiotemporal correlation features.

7. The keyframe extraction method based on few-shot learning as described in claim 5, characterized in that, The step of using the feature purification submodule to denoise and optimize the target spatiotemporal correlation features to obtain clean spatiotemporal features includes: Gaussian noise is added to the target spatiotemporal correlation features to obtain initial noise features; Starting with the initial noise features, iterative denoising is performed based on the Langevin dynamics algorithm; wherein, in each iteration, the gradient of the log probability density of the noise features in each round is estimated by the scoring network of the feature purification submodule as the score for each round. After a preset number of iterations, the currently obtained feature is determined as the pure spatiotemporal feature.

8. A keyframe extraction device based on few-shot learning, characterized in that, The keyframe extraction device based on few-shot learning includes: The preprocessing unit is used to acquire video data below a preset amount and preprocess the video to obtain a standard frame sequence. The building unit is used to construct the initial network based on the boundary-balanced generative adversarial network generator BEGAN generator, the temporal-aware discriminator, and the feature purification submodule. The construction unit is also used to construct a loss function based on adversarial loss, perceptual loss, and time-series consistency loss; The training unit is used to train the initial network based on the loss function using the standard frame sequence as training samples to obtain the keyframe extraction network. The processing unit is configured to process the target video using the keyframe extraction network in response to a keyframe extraction instruction for the target video, thereby obtaining the target keyframes.

9. A computer device, characterized in that, The computer device includes: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the keyframe extraction method based on few-shot learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in a computer device to implement the keyframe extraction method based on few-shot learning as described in any one of claims 1 to 7.

Images