Video key frame extraction method and device with enhanced prototype memory, equipment and medium

Abstract

The invention relates to the technical field of data analysis, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses a prototype memory enhanced video key frame extraction method, device, equipment and medium, the method comprises the following steps: carrying out time sequence pulse convolution on each video frame in a time sequence pulse sequence converted from original video data; performing correlation matching on the pulse feature sequence and differentiable prototype features in a preset prototype memory bank to obtain a related memory feature sequence, and performing dynamic fusion on the pulse feature sequence and the related memory feature sequence to obtain memory enhanced pulse features; and performing key analysis on the video frames with the memory enhanced pulse features to obtain key frame discrimination scores, and screening out video key frames. According to the method, overfitting is effectively suppressed in a few-sample scene, video dynamic time sequence information is accurately captured, and the efficiency and accuracy of key frame recognition are improved while the calculation complexity and memory overhead are reduced.

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, and in particular to a method, apparatus, device, and medium for extracting keyframes from video with prototype memory enhancement. Background Technology

[0002] Video keyframe extraction is an important task in video content analysis and retrieval. However, existing technologies mainly rely on manual feature extraction combined with clustering algorithms or fully supervised deep learning models, which suffer from problems such as overfitting in few-shot scenarios and the difficulty of static convolutional networks in capturing dynamic temporal information. Although few-shot learning and spiking neural networks have provided new ideas for temporal feature modeling, existing methods have limited memory capacity and high energy consumption. While attention-based methods can improve discrimination ability, they have high computational complexity and lack few-shot optimization, making it difficult to achieve efficient and accurate keyframe extraction.

[0003] In the medical and health field, such as in endoscopic or ultrasound image analysis, existing keyframe extraction methods rely on static image processing or large-scale depth models, making it difficult to capture key dynamic change information. Furthermore, they are prone to overfitting in cases with few samples, which may lead to the omission of important lesions or abnormal frames, thus affecting diagnostic efficiency and accuracy.

[0004] In the field of fintech business, such as transaction monitoring or risk control video analysis, existing methods have high computational complexity when processing long time-series monitoring videos, and at the same time, they are difficult to effectively capture the temporal characteristics of a small number of abnormal events. This may lead to key abnormal behaviors being ignored or misjudged, which limits the reliability of real-time risk warnings and decision support.

[0005] Therefore, existing video keyframe extraction methods are prone to overfitting in scenarios with few samples, have difficulty capturing dynamic temporal information, have high computational complexity and limited memory capacity, resulting in insufficient efficiency and accuracy in keyframe recognition. Summary of the Invention

[0006] This invention provides a prototype memory-enhanced video keyframe extraction method, apparatus, device, and medium. Its main purpose is to solve the problems of existing video keyframe extraction methods being prone to overfitting in scenarios with few samples, difficulty in capturing dynamic temporal information, high computational complexity, and limited memory capacity, resulting in insufficient efficiency and accuracy in keyframe recognition.

[0007] In a first aspect, to achieve the above objectives, the present invention provides a prototype memory-enhanced video keyframe extraction method, comprising: Convert the pre-acquired raw video data into a time-series pulse sequence; Perform temporal pulse convolution on each video frame in the temporal pulse sequence to obtain a pulse feature sequence; The pulse feature sequence is correlated with the differentiable prototype features in the preset prototype memory bank to obtain the relevant memory feature sequence. The pulse feature sequence and the related memory feature sequence are dynamically fused to obtain memory-enhanced pulse features; Keyframes are analyzed on the video frames corresponding to the memory-enhanced pulse features to obtain keyframe discrimination scores, and video keyframes are selected from the original video data based on the keyframe discrimination scores.

[0008] Secondly, the present invention also provides a prototype memory-enhanced video keyframe extraction device, comprising: The data conversion module is used to convert the pre-acquired raw video data into a time-series pulse sequence; The pulse convolution module is used to perform temporal pulse convolution on each video frame in the temporal pulse sequence to obtain a pulse feature sequence. The feature matching module is used to perform correlation matching between the pulse feature sequence and the differentiable prototype features in the preset prototype memory bank to obtain the relevant memory feature sequence; The feature fusion module is used to dynamically fuse the pulse feature sequence and the related memory feature sequence to obtain memory-enhanced pulse features; The keyframe analysis module is used to perform keyframe analysis on the video frames corresponding to the memory-enhanced pulse features, obtain keyframe discrimination scores, and filter out video keyframes from the original video data based on the keyframe discrimination scores.

[0009] Thirdly, the present invention also provides an electronic device, the electronic device comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the prototype memory-enhanced video keyframe extraction method described above.

[0010] Fourthly, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the above-described prototype memory-enhanced video keyframe extraction method.

[0011] This invention converts pre-acquired raw video data into a temporal pulse sequence, mapping continuous, highly redundant video pixel changes into sparse and time-directed event representations. Furthermore, the temporal pulse sequence naturally conforms to the information transmission mechanism of biological neurons. Temporal pulse convolution is performed on each video frame in the temporal pulse sequence to obtain a pulse feature sequence, enabling simultaneous modeling of the video's spatial structure and temporal evolution within the pulse domain. This effectively preserves and highlights key motion changes and temporal dependencies. The pulse feature sequence is then correlated with differentiable prototype features in a pre-defined prototype memory bank to obtain a related memory feature sequence. This fully utilizes typical temporal patterns accumulated from historical videos and cross-tasks to enhance the knowledge of the current input features, thereby improving feature representation capabilities under limited sample conditions. The pulse feature sequence and the related memory... The feature sequences are dynamically fused to obtain memory-enhanced pulse features. By adaptively adjusting the fusion ratio of the two types of features, the system can flexibly emphasize current perception or historical memory under different scenarios and content changes. Keyframe analysis is performed on the video frames corresponding to the memory-enhanced pulse features to obtain keyframe discrimination scores. Based on these scores, video keyframes are selected from the original video data. This means that the selection of video keyframes no longer relies solely on the static appearance information of a single frame, but comprehensively integrates the high-level semantics and dynamic change features represented by temporal pulse coding and historical prototype memory. This allows for a more accurate depiction of representative and discriminative key moments in the video. In scenarios with few samples, overfitting is effectively suppressed, and dynamic temporal information of the video is accurately captured. Furthermore, the efficiency and accuracy of keyframe recognition are improved while reducing computational complexity and memory overhead. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of an application environment for a prototype memory-enhanced video keyframe extraction method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a prototype memory-enhanced video keyframe extraction method according to an embodiment of the present invention. Figure 3 This is a flowchart illustrating the feature correlation matching process in a prototype memory-enhanced video keyframe extraction method according to an embodiment of the present invention. Figure 4This is a schematic diagram of a prototype memory-enhanced video keyframe extraction device according to an embodiment of the present invention; Figure 5 A schematic diagram of an electronic device for implementing a video keyframe extraction method with prototype memory enhancement according to an embodiment of the present invention; Figure 6 This is another structural schematic diagram of an electronic device that implements a prototype memory-enhanced video keyframe extraction method according to an embodiment of the present invention.

[0014] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0015] To enable those skilled in the art to better understand the technical solutions of this disclosure, and to fully understand and implement the process of how this disclosure applies technical means to solve technical problems and achieve corresponding technical effects, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. The embodiments of this disclosure and the various features within them can be combined with each other without conflict, and the resulting technical solutions are all within the protection scope of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort should fall within the protection scope of this disclosure.

[0016] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0017] This application provides a prototype memory-enhanced video keyframe extraction method. The execution subject of this method includes, but is not limited to, at least one electronic device configured to execute the device provided in this application, such as a server or a terminal. In other words, the prototype memory-enhanced video keyframe extraction method can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0018] This invention provides a prototype memory-enhanced video keyframe extraction method, which can be applied to applications such as... Figure 1In this application environment, the client communicates with the server via a network. The server can convert pre-acquired raw video data into a temporal pulse sequence through the client, mapping continuous, highly redundant video pixel changes into sparse and time-directed event representations. Furthermore, the temporal pulse sequence naturally conforms to the information transmission mechanism of biological neurons. Temporal pulse convolution is performed on each video frame in the temporal pulse sequence to obtain a pulse feature sequence, which can simultaneously model the spatial structure and temporal evolution of the video within the pulse domain, effectively preserving and highlighting key motion changes and temporal dependencies. The pulse feature sequence is then correlated with differentiable prototype features in a preset prototype memory bank to obtain a related memory feature sequence. This fully utilizes typical temporal patterns accumulated from historical videos and cross-tasks to enhance the knowledge of the current input features, thereby improving feature expression capabilities under few-sample conditions. The pulse feature sequence and the related memory feature sequence are then subjected to dynamic... The system employs temporal fusion to obtain memory-enhanced pulse features. By adaptively adjusting the fusion ratio of the two types of features, the system can flexibly prioritize current perception or historical memory under different scenarios and content changes. Keyframe analysis is performed on the video frames corresponding to the memory-enhanced pulse features to obtain keyframe discrimination scores. Based on these scores, video keyframes are selected from the original video data. This means that the selection of video keyframes no longer relies solely on the static appearance information of a single frame, but comprehensively integrates the high-level semantics and dynamic change features represented by temporal pulse coding and historical prototype memory. This allows for a more accurate depiction of representative and discriminative key moments in the video. In scenarios with few samples, overfitting is effectively suppressed, and dynamic temporal information of the video is accurately captured. While reducing computational complexity and memory overhead, the efficiency and accuracy of keyframe recognition are improved. Finally, the video keyframes are output and fed back to the user client. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0019] The following explanation of this invention relates to the present invention specification. By drawing on pulse coding, temporal information transmission, and memory enhancement mechanisms in the nervous system, the present invention converts raw video data into temporal pulse sequences, enabling the perception of the dynamic evolution process of raw video data in a way that is more in line with biological cognitive laws. At the same time, by introducing a differentiable prototype memory bank to simulate the storage and retrieval of typical patterns in biological memory, the present invention achieves selective enhancement and rapid response to key semantic information. This not only effectively improves the ability to express complex temporal features and key events, but also reduces the risk of overfitting under conditions of few samples, balancing computational efficiency and memory utilization, thereby significantly improving the accuracy, robustness, and overall practicality of keyframe extraction.

[0020] Reference Figure 2 The diagram shown is a flowchart illustrating a prototype memory-enhanced video keyframe extraction method according to an embodiment of the present invention. In this embodiment, the prototype memory-enhanced video keyframe extraction method includes: S1. Convert the pre-acquired raw video data into a time-series pulse sequence.

[0021] In this embodiment of the invention, the raw video data is a collection of continuous video frames directly acquired by the camera device, typically containing image frames arranged in chronological order and their corresponding pixel information. The temporal pulse sequence is a sequence of pulse signals arranged in chronological order, formed by discretizing and encoding the continuous video frames in the time dimension, and is used to characterize the changing features of the video content at different times.

[0022] In specific healthcare scenarios, it can be applied to monitor patient behavior and physiological status. For example, in wards or rehabilitation training settings, pulse-encoded videos of patients' gait, limb movements, or facial expressions can be mapped into temporal pulse features to characterize movement rhythm, amplitude, and abnormal changes, thereby assisting in fall risk warning, rehabilitation progress assessment, or postoperative recovery status analysis.

[0023] In specific fintech scenarios, it can be applied to video-based identity verification and behavioral risk control. For example, during remote account opening or transaction confirmation, the video of the user's operation process and facial behavior can be pulse-encoded to convert dynamic behavioral features such as blink frequency, head micro-movements, and operation rhythm into pulse features, which can be used to identify the consistency of liveness features and abnormal operation patterns.

[0024] In this embodiment of the invention, the step of converting the pre-acquired raw video data into a time-series pulse sequence includes: The pre-acquired raw video data is used to extract video frames according to a preset frame rate, and the video frames are then aggregated into a video frame sequence. The video frame sequence is subjected to noise suppression to form a denoised video frame sequence; The video frame denoising sequence is pixel normalized to obtain a video frame normalized sequence; Perform inter-frame variation analysis on adjacent video frames in the normalized video frame sequence to obtain the inter-frame variation amplitude; When the inter-frame variation amplitude is greater than a preset amplitude threshold, the trigger events of the corresponding adjacent video frames in the normalized video frame sequence are extracted; A pulse signal is generated based on the triggering event, and the pulse signal is encoded to obtain a video frame pulse representation; The video frame pulse representations are combined in the order of the video frame sequence to obtain a time-series pulse sequence.

[0025] In this embodiment of the invention, the original video data is sampled according to a preset target frame rate, and discrete video frames are extracted from the continuous original video data in chronological order to avoid the problem of timing inconsistency caused by the difference in the original video frame rate. The extracted video frames are arranged and summarized according to the sampling time order to form a structured video frame sequence.

[0026] Furthermore, sensor noise, compression noise, or environmental interference are introduced, and noise suppression processing is performed frame by frame on the video frame sequence. High-frequency random noise components are weakened through spatial filtering or temporal smoothing, while the main structure and motion information in the video frame are not destroyed, thus obtaining a more stable video frame denoising sequence.

[0027] In this embodiment of the invention, the original pixel intensity of the video frame denoising sequence is mapped to a preset standard value range to eliminate pixel distribution differences caused by different lighting conditions, shooting devices or exposure parameters, so that each video frame in the video frame denoising sequence is consistent on the numerical scale, forming a normalized video frame sequence.

[0028] Furthermore, pixel-by-pixel or region-by-region comparative analysis is performed on adjacent video frames in the normalized sequence of video frames to obtain the degree of change in pixel intensity, structural features or overall energy between adjacent video frames, and the degree of change is quantified into the inter-frame change amplitude to characterize the dynamic change characteristics of video content in the time dimension.

[0029] In this embodiment of the invention, the amplitude of the inter-frame change is compared with a preset amplitude threshold. When the amplitude of the change between adjacent video frames exceeds the amplitude threshold, it is determined that there is a significant content change at the time position corresponding to the amplitude of the change exceeding the amplitude threshold. This significant content change is determined as a trigger event to indicate the occurrence of potential important motion, scene switching or key behavior in the video.

[0030] Furthermore, the corresponding video frame position is determined on the time axis based on the triggering event, and a discrete pulse signal is generated when the triggering condition is met. The pulse signal is encoded according to a preset pulse coding rule, and the event intensity, occurrence time and duration characteristics are mapped into a unified pulse representation form, thereby obtaining a video frame pulse representation that can characterize the temporal characteristics and dynamic change information of the video frame.

[0031] In this embodiment of the invention, the video frame pulse representations corresponding to each video frame are sequentially arranged and combined according to the time order of the original video frames, forming a temporal pulse sequence with a clear temporal sequence relationship, so that the dynamic changes of the video content can be continuously expressed in the form of pulses on the time axis.

[0032] This invention can map continuous, highly redundant video pixel changes into sparse and time-directed event representations, thereby significantly reducing irrelevant static information and redundant computation. The time-series pulse sequence naturally conforms to the information transmission mechanism of biological neurons, improving the response sensitivity to rapidly changing scenes. It also lays the foundation for achieving low-energy, highly robust video feature extraction and few-sample learning.

[0033] S2. Perform temporal pulse convolution on each video frame in the temporal pulse sequence to obtain a pulse feature sequence.

[0034] In this embodiment of the invention, temporal pulse convolution is a convolution operation that performs weighted integration and temporal correlation calculations on pulse inputs at different times within the pulse signal domain, used to extract spatiotemporal joint features from videos. The pulse feature sequence is a sequence of pulse feature representations containing temporal dependencies and structural information, obtained after temporal pulse convolution processing, used to characterize key dynamic patterns in video content.

[0035] In specific healthcare scenarios, it can be applied to endoscopic videos, ultrasound videos, or surgical monitoring videos. It performs temporal pulse convolution on each video frame to highlight subtle dynamic features of lesion morphology changes, instrument movement trajectories, or tissue structures, thereby forming a pulse feature sequence with temporal semantics. This can be used to assist in keyframe screening, abnormal segment localization, or rapid backtracking of lesion areas. Even with limited labeled samples, it can still improve the stability and accuracy of medical video analysis.

[0036] In specific fintech scenarios, it can be applied to surveillance video analysis, counter operation behavior recognition, or remote identity verification processes. By extracting dynamic features of personnel movement changes, interaction rhythms, and abnormal behaviors through temporal pulse convolution, it generates pulse feature sequences for subsequent keyframe extraction, risk behavior identification, or compliance auditing, thereby improving the video intelligent analysis capabilities of financial businesses in complex environments and under conditions of few samples.

[0037] In this embodiment of the invention, the step of performing temporal pulse convolution on each video frame in the temporal pulse sequence to obtain a pulse feature sequence includes: The temporal pulse sequence is subjected to synaptic weighted integration using a preset temporal pulse convolutional network to obtain the integration result; The membrane potential of the spiking neurons in the temporal spiking convolutional network is dynamically updated using the integral result to obtain the updated membrane potential; When the updated membrane potential is greater than a preset potential threshold, the spiking neuron is triggered to fire a pulse, and the pulse output after the pulse firing is obtained. The pulse output is used to attenuate the updated membrane potential, thereby generating an attenuated membrane potential; The decaying membrane potential is returned to the integral result to dynamically update the membrane potential of the spiking neurons in the temporal pulse convolutional network, thereby obtaining the step of updating the membrane potential, and the total number of pulse outputs is counted. When the total number of outputs equals the number of video frames in the time-series pulse sequence, the return process stops, and all the pulse outputs are aggregated into a pulse feature sequence.

[0038] In this embodiment of the invention, by assigning corresponding synaptic weights to pulse signals at different times and spatial locations in a temporal pulse sequence, and accumulating and summing the temporal pulse sequence according to the synaptic weights in the time dimension, the pulse responses at adjacent times are fused to form an integral result that reflects the local temporal correlation, which is used to characterize the comprehensive input stimulus intensity at the current time.

[0039] Furthermore, by using the integral result as a temporal input, the membrane potential of each spiking neuron in the temporal spiking convolutional network is dynamically updated time-by-time. Through the accumulation, leakage, and modulation of historical pulse responses, the comprehensive response of the spiking neuron to the input pulse intensity and temporal relationship is reflected in real time, thereby obtaining the updated membrane potential state.

[0040] In this embodiment of the invention, when the updated membrane potential accumulates to exceed the set potential threshold, it is determined that the spiking neuron has reached the activation state, thereby generating a discrete pulse firing, and the result after the pulse firing is recorded in the form of pulse output to represent the important temporal characteristic response at the current moment.

[0041] In this embodiment of the invention, the membrane potential update of each spiking neuron at time t follows the improved Hodgkin-Huxley equation:

[0042] in, Indicates the time of the spiking neuron The rate of change of membrane potential, Indicates the time of the spiking neuron The membrane potential state, Indicates the first The connection weights between each presynaptic neuron and the current spiking neuron Indicates the first A presynaptic neuron in Pulse output at any moment This represents the conductivity of the sodium ion channel. This represents the conductivity of the potassium ion channel. This represents the reversal potential of sodium ions. This represents the reversal potential of potassium ions. This indicates the external input current.

[0043] Furthermore, when the membrane potential exceeds the potential threshold, the spiking neuron generates a pulse output:

[0044] in, Indicates the time of the spiking neuron pulse output, Indicates the time of the spiking neuron The membrane potential state.

[0045] Furthermore, after the pulse is fired, a decay or reset mechanism is applied to the neuronal membrane potential to bring it back to the preset potential range, thereby simulating the recovery process of biological neurons, preventing continuous invalid activation, and providing a stable initial state for the response to subsequent time-series inputs.

[0046] In this embodiment of the invention, in subsequent time steps, the decaying membrane potential is used as the initial state to continue participating in the synaptic weighted integral-driven membrane potential update cycle, while the pulse output of each trigger is counted to fully record the neuron's response throughout the entire video time sequence.

[0047] Furthermore, after the neuron has completed its response to the temporal pulse inputs corresponding to all video frames, the neuron membrane potential update cycle is terminated, and all pulse outputs are collected in chronological order to form a pulse feature sequence that characterizes the temporal dynamic features of the video.

[0048] This invention can simultaneously model the spatial structure and temporal evolution of video in the pulse domain, effectively preserving and highlighting key motion changes and temporal dependencies.

[0049] S3. The pulse feature sequence is correlated with the differentiable prototype features in the preset prototype memory bank to obtain the relevant memory feature sequence.

[0050] In this embodiment of the invention, the preset prototype memory is a feature storage structure that is pre-constructed and can be updated during training. It is used to store representative prototype features from different videos or tasks. Differentiable prototype features are feature vectors stored in the prototype memory that can be jointly optimized through gradient backpropagation, and are used to characterize typical video temporal patterns. Correlation matching determines the correspondence between the current input features and existing knowledge by analyzing the similarity or correlation between the impulse feature sequence and each prototype feature in the prototype memory.

[0051] The relevant memory feature sequence is a sequence of memory features that are most relevant to the current pulse feature sequence in terms of semantics or temporal pattern, obtained by retrieving and weighting the relevant memory from the prototype memory bank based on the relevance matching results.

[0052] In specific healthcare scenarios, it can be applied to video analysis of endoscopic examinations, ultrasound diagnosis, or surgical procedures. By matching the pulse feature sequences extracted from the current patient video with the typical lesion morphology, tissue movement patterns, or key surgical operation prototypes learned in the prototype memory bank, the relevant memory feature sequences that are closest to the current examination process are obtained, thereby helping to highlight key frames or abnormal segments with clinical significance.

[0053] In specific fintech scenarios, it can be applied to surveillance video analysis or business operation process auditing. By matching the real-time acquired video pulse feature sequences with the prototype features of normal operating procedures, compliant behaviors, or historical risk events stored in the prototype memory bank, corresponding relevant memory feature sequences are obtained. This enhances the ability to perceive abnormal behavior, illegal operations, or risk events, thereby improving the accuracy and reliability of video intelligent risk control and post-event auditing in complex scenarios and with limited samples.

[0054] Figure 3 This is a flowchart illustrating the feature correlation matching process in a prototype memory-enhanced video keyframe extraction method provided in an embodiment of the present invention.

[0055] In this embodiment of the invention, the step of performing correlation matching between the pulse feature sequence and differentiable prototype features in a preset prototype memory bank to obtain a related memory feature sequence includes: Extract multiple differentiable prototype features from a pre-defined prototype memory; The pulse feature sequence is mapped into the feature space according to the differentiable prototype feature to generate a pulse feature mapping sequence; The correlation degree is obtained by performing correlation analysis between each pulse feature in the pulse feature mapping sequence and the differentiable prototype feature; An attention weight is generated for each of the differentiable prototype features based on the relevance. The attention weights are used to perform a weighted summation of the differentiable prototype features to obtain a relevant memory feature sequence.

[0056] In this embodiment of the invention, multiple representative prototype features are read from a pre-built and continuously updated prototype memory. These prototype features are used to characterize the typical structure of different video content or temporal patterns.

[0057] Furthermore, by using linear transformation or nonlinear projection, the pulse feature sequence is mapped to a feature space consistent with the differentiable prototype features, allowing the two to be compared under the same metric, thus providing a unified feature representation basis for subsequent correlation analysis.

[0058] In this embodiment of the invention, for each pulse feature mapping vector at any given time, a similarity or relevance index is calculated with each differentiable prototype feature to quantify the degree of matching between the current input feature and different prototype features at the temporal or semantic level. Furthermore, the relevance is normalized or nonlinearly transformed and mapped to corresponding attention weights, giving greater weight to prototype features with higher correlation to the current pulse feature.

[0059] In this embodiment of the invention, the attention weight of each differentiable prototype feature is calculated using the following formula:

[0060] in, This indicates that the differentiable prototype feature is at time 1 / 2. Attention weights This represents the Sigmoid activation function. This represents the learnable projection matrix. This indicates a modified linear unit activation function. This represents a differentiable prototype feature. Indicates time The pulse features in the pulse feature mapping sequence, This indicates the bias term.

[0061] Furthermore, the retrieved relevant memory feature sequences are as follows:

[0062] in, Indicates at time Related memory characteristics, This represents the total number of differentiable prototype features. Indicates at time Next Attention weights for each differentiable prototype feature. Indicates the first A differentiable prototype feature.

[0063] In this embodiment of the invention, multiple differentiable prototype features are weighted and combined using the generated attention weights to form a memory feature sequence that is highly correlated with the current pulse feature sequence in terms of temporal structure and semantic information, which is used to enhance the effect of subsequent feature representation and keyframe analysis.

[0064] This invention can effectively introduce prior memory constraints on typical temporal patterns and key semantic features, enabling current video features to be quickly aligned and complementarily enhanced with historical high-value prototypes, thereby improving the discriminativeness and stability of feature representation and reducing the learning difficulty under few sample conditions.

[0065] S4. Dynamically fuse the pulse feature sequence and the related memory feature sequence to obtain memory-enhanced pulse features.

[0066] In this embodiment of the invention, the memory-enhanced pulse feature is a feature representation formed by organically combining the current pulse feature sequence with the relevant memory feature sequence through a dynamic fusion mechanism. It retains the real-time dynamic information of the current video and incorporates prior knowledge across videos.

[0067] In specific healthcare scenarios, it can be applied to endoscopic examinations, intraoperative video analysis, or image follow-up. By dynamically fusing the pulse feature sequence extracted during the current examination with the relevant memory feature sequences corresponding to typical lesion evolution patterns, tissue motion characteristics, or standard operating procedures learned in the memory bank, a memory-enhanced pulse feature is formed. This allows for more stable highlighting of key clinically significant images and abnormal changes even under conditions of high noise interference and limited labeled samples.

[0068] In specific fintech scenarios, it can be applied to business monitoring videos, counter operation audits, or remote identity verification processes. By dynamically fusing the pulse feature sequence extracted from real-time video with the memory feature sequence corresponding to historical compliant operations and typical risk events, memory-enhanced pulse features are generated to strengthen the ability to identify abnormal behavior, operational deviations, or potential risk scenarios, thereby improving the accuracy and robustness of keyframe screening and risk analysis in complex business environments.

[0069] In this embodiment of the invention, the step of dynamically fusing the pulse feature sequence and the related memory feature sequence to obtain memory-enhanced pulse features includes: The pulse feature sequence and the related memory feature sequence are aligned by time steps to obtain the pulse aligned feature sequence and the memory aligned feature sequence, respectively. The pulse alignment feature sequence and the memory alignment feature sequence are concatenated to obtain the concatenated feature sequence; A linear transformation is performed on the spliced ​​feature sequence to generate fusion gating coefficients; The fusion gate coefficients are nonlinearly mapped to obtain the gate weights; The pulse alignment feature sequence and the memory alignment feature sequence are summed element-wise using the gating weights to form a fused alignment feature. The fused alignment features are temporally smoothed to obtain memory-enhanced pulse features.

[0070] In this embodiment of the invention, the pulse feature sequence and the related memory feature sequence are aligned step by step according to the time axis. By unifying the time index and the frame-level correspondence, the pulse feature at each moment is matched with the corresponding memory feature at the same time step, thereby obtaining a pulse-aligned feature sequence and a memory-aligned feature sequence that are structurally consistent and temporally synchronized.

[0071] Furthermore, based on the completion of time step alignment, the pulse alignment features and memory alignment features corresponding to the same time step are spliced ​​and fused in the feature dimension, so that the current time-series pulse information and historical memory information are associated in a unified representation space, thereby forming a spliced ​​feature sequence containing instantaneous dynamic features and long-term memory features.

[0072] In this embodiment of the invention, a linear transformation is applied to the spliced ​​feature sequence. By weighting and biasing the feature dimensions, the spliced ​​feature sequence is mapped to a gated representation, thereby obtaining fusion gating coefficients that characterize the relative contributions of impulse features and memory features. Furthermore, the fusion gating coefficients in the linear space are compressed and constrained to a specified range of values, thereby obtaining numerically stable and more adjustable gating weights.

[0073] In this embodiment of the invention, the gating weights are multiplied element-wise with the pulse-aligned feature sequence and the memory-aligned feature sequence at the time step and feature dimension, respectively, and the weighted results at the corresponding positions are summed and fused, so that the two types of features can be adaptively combined in the same representation space, thereby obtaining a fused aligned feature that takes into account both instantaneous dynamic information and long-term memory information.

[0074] Furthermore, by weakening the abrupt noise between adjacent time steps in the fusion alignment feature and strengthening the stable change trend, the fusion alignment feature expression is made more consistent in time, thereby forming a memory-enhanced pulse feature that has both dynamic response capability and temporal stability.

[0075] In this embodiment of the invention, the dynamic fusion calculation formula for the pulse feature sequence and the related memory feature sequence is as follows:

[0076]

[0077] in, Indicates at time The memory-enhancing pulse characteristics, Indicates the gating weight, Indicates time The pulse features in the pulse feature mapping sequence, Indicates at time Related memory characteristics, This represents the Sigmoid activation function. This represents a learnable gated projection matrix.

[0078] This invention can adaptively balance the contributions of current temporal dynamic information and historical prototype memory information in a unified representation, thereby strengthening key semantic patterns and suppressing redundant or noisy features.

[0079] S5. Perform key analysis on the video frames corresponding to the memory-enhanced pulse features to obtain key frame discrimination scores, and filter out video key frames from the original video data based on the key frame discrimination scores.

[0080] In this embodiment of the invention, the keyframe discrimination score is a numerical index obtained through keyness analysis, used to represent the probability that a corresponding video frame will become a keyframe. A video keyframe is a video frame selected from the original video data based on its keyframe discrimination score, representing the main content or important moments of the video.

[0081] In specific healthcare scenarios, it can be applied to endoscopic examinations, intraoperative video recording, or remote consultation video analysis. By performing key analysis on video frames corresponding to memory-enhanced pulse characteristics, key frame discrimination scores are obtained for each frame, thereby automatically filtering out key frames containing lesion exposure, key instrument operations, or significant changes in tissue morphology. This helps doctors quickly review the core diagnosis and treatment process, reduces the burden of manually browsing long videos, and improves diagnostic and teaching efficiency under conditions of few samples.

[0082] In specific fintech scenarios, it can be applied to branch monitoring, business operation video auditing, or remote identity verification. By performing critical analysis on video frames corresponding to memory-enhanced pulse features and generating key frame discrimination scores, key video frames involving key operational steps, abnormal behaviors, or risk events can be selected for subsequent compliance review, risk evidence collection, or intelligent alarms, thereby improving the accuracy and traceability of video risk control and auditing in complex business environments.

[0083] In this embodiment of the invention, the step of performing keyframe analysis on the video frames corresponding to the memory-enhanced pulse features to obtain keyframe discrimination scores includes: The memory-enhanced pulse features are matched one-to-one with the sequential video frames of the original video data to form a feature sample set. The target learning model is obtained by optimizing the preset few-shot learning model using the feature sample set. The memory-enhanced pulse features are temporally aggregated to generate pulse aggregated features, and the temporal discriminative features in the pulse aggregated features are extracted. The target learning model is used to perform key score analysis on the temporal discrimination features to generate key frame discrimination scores.

[0084] In this embodiment of the invention, each video frame of the original video data is mapped one-to-one with the corresponding memory-enhanced pulse feature according to the time order and frame-level indexing relationship of the original video data, so that each video frame is associated with the corresponding enhanced feature representation, thereby constructing a feature sample set containing the correspondence between video frames and memory-enhanced pulse features, so that each feature sample simultaneously contains the original video temporal position and the enhanced pulse feature information.

[0085] Furthermore, within the few-shot learning framework, a feature sample set composed of memory-enhanced spiking features is used as the model input. The feature sample set is divided into a support set and a query set. The support set guides the few-shot learning model to quickly adapt to the current video task, and the meta-learning loss is calculated based on the query set. This meta-learning loss, through a backpropagation mechanism, jointly updates the attention parameters in the few-shot learning model and the differentiable prototype features in the prototype memory. Simultaneously, combined with the temporal plasticity rules of spiking neurons, the synaptic weights of the temporal spiking convolutional network are updated using Local Pulse Temporal Dependency Plasticity (STDP) based on the spiking temporal relationship. On this basis, the meta-learning controller dynamically maintains and adjusts the prototype memory according to the global optimization signal, enabling the few-shot learning model to continuously enhance its representation ability of key temporal features under limited sample conditions, ultimately forming a target learning model optimized for the current task.

[0086] In this embodiment of the invention, the calculation formula for the meta-learning loss based on the query set is as follows:

[0087] in, This represents the meta-learning loss calculated using the meta-learning loss function. This represents the sample set of the query set. Indicates the first in the query set Input data for each sample, Indicates the first in the query set The true label of each sample Indicates in the query sample Predicted memory-enhancing pulse characteristics Indicates query sample In an ideal scenario, the target features are represented as follows: Represents the balance coefficient. Indicates the Kullback–Leibler divergence. Indicating query samples in a few-shot learning model Attention parameters, Indicating query samples in a few-shot learning model The target attention parameters.

[0088] Furthermore, the formula for calculating the local impulse time-dependent plasticity (STDP) update is as follows:

[0089] in, Indicates the first The increment of synaptic weight, Indicates the learning rate. Indicates the time of the spiking neuron pulse output, Indicates the first At time 1, presynaptic neurons pulse output, This represents the exponential decay factor.

[0090] In this embodiment of the invention, the memory-enhanced pulse features are organized and aligned in chronological order. A temporal modeling mechanism is introduced in the time dimension to perform weighted accumulation and correlation fusion of pulse responses at different time steps, thereby characterizing the dynamic law of pulse feature evolution over time. During the aggregation process, the ability of the memory-enhanced unit to retain and update historical pulse information is fully utilized to suppress the interference of instantaneous noise pulses, highlight the temporal response pattern with discriminative significance, and finally form a pulse aggregation feature that can comprehensively reflect global temporal context information.

[0091] Furthermore, the change trends, temporal correlations, and evolutionary consistency of impulse responses at different time points are measured to mine key time segments and corresponding dynamic patterns that can distinguish different states or events, suppress temporal redundancy and weak discriminative information, and thus extract temporal discriminative features with significant discriminative ability for the target task from the impulse aggregation features.

[0092] In this embodiment of the invention, the extracted temporal discrimination features are input into the optimized target learning model. The contribution of the temporal discrimination features corresponding to each video frame is analyzed through the feature mapping and discriminant function within the target learning model to obtain their relative importance in the overall temporal discrimination. The target learning model comprehensively considers the temporal context relationship and feature discrimination intensity, and performs normalization and weighted analysis on the temporal discrimination feature responses at different times, thereby generating a corresponding key score for the temporal discrimination features of each video frame. Finally, the key frame discrimination score reflecting the discrimination value of each video frame is obtained, and the calculation formula is as follows:

[0093] in, Indicates the keyframe discrimination score. This represents the Sigmoid function. This represents the learnable weight matrix. Indicates at time The memory-enhancing pulse characteristics, This indicates the bias term.

[0094] In this embodiment of the invention, the step of filtering video keyframes from the original video data based on the keyframe discrimination score includes: A set of candidate video frames corresponding to keyframe discrimination scores greater than a preset score threshold is selected from the original video data. The inter-frame similarity of adjacent video frames in the candidate video frame set is calculated based on the memory-enhanced pulse features. The video frame with the smallest inter-frame similarity is selected from the candidate video frame set as the video keyframe.

[0095] In this embodiment of the invention, the key frame discrimination score of each video frame in the original video data is compared with a pre-set score threshold. Video frames with key frame discrimination scores higher than the score threshold are selected, thereby eliminating ordinary frames with low keyness or high redundancy, forming a candidate video frame set with high potential information value.

[0096] In this embodiment of the invention, for adjacent video frames arranged in chronological order in the candidate video frame set, memory-enhanced pulse feature representations corresponding to adjacent video frames are extracted, and feature distance metrics or similarity functions are used to analyze the feature differences between adjacent video frames in order to quantify the similarity of consecutive frames in temporal semantics and dynamic changes, thereby obtaining the inter-frame similarity that reflects the magnitude of content changes between candidate frames.

[0097] Furthermore, the inter-frame similarity between adjacent candidate video frames is compared and analyzed. The candidate video frame with the greatest difference from the preceding and following frames, i.e. the video frame with the smallest inter-frame similarity, is selected as the video key frame to highlight the key moments in the video with significant semantic changes and the greatest information increment.

[0098] This invention enables the selection of video keyframes to no longer rely solely on the static appearance information of a single frame, but rather integrates the high-level semantics and dynamic change features represented by temporal pulse coding and historical prototype memory, thereby more accurately depicting the key moments in the video that are representative and of discriminative value.

[0099] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0100] like Figure 4 The diagram shown is a functional block diagram of a prototype memory-enhanced video keyframe extraction device provided in an embodiment of the present invention.

[0101] In this embodiment of the disclosure, a prototype memory-enhanced video keyframe extraction device is provided, which corresponds one-to-one with the prototype memory-enhanced video keyframe extraction method described in the above embodiment. For example... Figure 4 As shown, the prototype memory-enhanced video keyframe extraction device 100 can be installed in an electronic device. According to its functions, the prototype memory-enhanced video keyframe extraction device 100 includes a data conversion module 101, a pulse convolution module 102, a feature matching module 103, a feature fusion module 104, and a keyframe analysis module 105. Detailed descriptions of each functional module are as follows: The data conversion module 101 is used to convert the pre-acquired raw video data into a time-series pulse sequence; The pulse convolution module 102 is used to perform temporal pulse convolution on each video frame in the temporal pulse sequence to obtain a pulse feature sequence. Feature matching module 103 is used to perform correlation matching between the pulse feature sequence and the differentiable prototype features in the preset prototype memory bank to obtain a related memory feature sequence; Feature fusion module 104 is used to dynamically fuse the pulse feature sequence and the related memory feature sequence to obtain memory-enhanced pulse features; The keyframe analysis module 105 is used to perform keyframe analysis on the video frames corresponding to the memory-enhanced pulse features, obtain keyframe discrimination scores, and filter out video keyframes from the original video data based on the keyframe discrimination scores.

[0102] In one embodiment, the data conversion module 101 performs the conversion of pre-acquired raw video data into a time-series pulse sequence, including: The pre-acquired raw video data is used to extract video frames according to a preset frame rate, and the video frames are then aggregated into a video frame sequence. The video frame sequence is subjected to noise suppression to form a denoised video frame sequence; The video frame denoising sequence is pixel normalized to obtain a video frame normalized sequence; Perform inter-frame variation analysis on adjacent video frames in the normalized video frame sequence to obtain the inter-frame variation amplitude; When the inter-frame variation amplitude is greater than a preset amplitude threshold, the trigger events of the corresponding adjacent video frames in the normalized video frame sequence are extracted; A pulse signal is generated based on the triggering event, and the pulse signal is encoded to obtain a video frame pulse representation; The video frame pulse representations are combined in the order of the video frame sequence to obtain a time-series pulse sequence.

[0103] In one embodiment, the pulse convolution module 102 performs temporal pulse convolution on each video frame in the temporal pulse sequence to obtain a pulse feature sequence, including: The temporal pulse sequence is subjected to synaptic weighted integration using a preset temporal pulse convolutional network to obtain the integration result; The membrane potential of the spiking neurons in the temporal spiking convolutional network is dynamically updated using the integral result to obtain the updated membrane potential; When the updated membrane potential is greater than a preset potential threshold, the spiking neuron is triggered to fire a pulse, and the pulse output after the pulse firing is obtained. The pulse output is used to attenuate the updated membrane potential, thereby generating an attenuated membrane potential; The decaying membrane potential is returned to the integral result to dynamically update the membrane potential of the spiking neurons in the temporal pulse convolutional network, thereby obtaining the step of updating the membrane potential, and the total number of pulse outputs is counted. When the total number of outputs equals the number of video frames in the time-series pulse sequence, the return process stops, and all the pulse outputs are aggregated into a pulse feature sequence.

[0104] In one embodiment, the feature matching module 103 performs correlation matching between the pulse feature sequence and differentiable prototype features in a preset prototype memory bank to obtain a related memory feature sequence, including: Extract multiple differentiable prototype features from a pre-defined prototype memory; The pulse feature sequence is mapped into the feature space according to the differentiable prototype feature to generate a pulse feature mapping sequence; The correlation degree is obtained by performing correlation analysis between each pulse feature in the pulse feature mapping sequence and the differentiable prototype feature; An attention weight is generated for each of the differentiable prototype features based on the relevance. The attention weights are used to perform a weighted summation of the differentiable prototype features to obtain a relevant memory feature sequence.

[0105] In one embodiment, the feature fusion module 104 performs dynamic fusion of the pulse feature sequence and the related memory feature sequence to obtain memory-enhanced pulse features, including: The pulse feature sequence and the related memory feature sequence are aligned by time steps to obtain the pulse aligned feature sequence and the memory aligned feature sequence, respectively. The pulse alignment feature sequence and the memory alignment feature sequence are concatenated to obtain the concatenated feature sequence; A linear transformation is performed on the spliced ​​feature sequence to generate fusion gating coefficients; The fusion gate coefficients are nonlinearly mapped to obtain the gate weights; The pulse alignment feature sequence and the memory alignment feature sequence are summed element-wise using the gating weights to form a fused alignment feature. The fused alignment features are temporally smoothed to obtain memory-enhanced pulse features.

[0106] In one embodiment, the keyframe analysis module 105 performs keyframe analysis on the video frames corresponding to the memory-enhanced pulse features to obtain keyframe discrimination scores, including: The memory-enhanced pulse features are matched one-to-one with the sequential video frames of the original video data to form a feature sample set. The target learning model is obtained by optimizing the preset few-shot learning model using the feature sample set. The memory-enhanced pulse features are temporally aggregated to generate pulse aggregated features, and the temporal discriminative features in the pulse aggregated features are extracted. The target learning model is used to perform key score analysis on the temporal discrimination features to generate key frame discrimination scores.

[0107] In one embodiment, the keyframe analysis module 105 performs the following steps when filtering video keyframes from the original video data based on the keyframe discrimination score: A set of candidate video frames corresponding to keyframe discrimination scores greater than a preset score threshold is selected from the original video data. The inter-frame similarity of adjacent video frames in the candidate video frame set is calculated based on the memory-enhanced pulse features. The video frame with the smallest inter-frame similarity is selected from the candidate video frame set as the video keyframe.

[0108] In this invention, the specific limitations of the prototype memory-enhanced video keyframe extraction device can be found in the above-described limitations of the prototype memory-enhanced video keyframe extraction method, and will not be repeated here. Each module in the aforementioned prototype memory-enhanced video keyframe extraction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0109] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a prototype memory-enhanced video keyframe extraction method on the server side.

[0110] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a prototype memory-enhanced video keyframe extraction method.

[0111] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Convert the pre-acquired raw video data into a time-series pulse sequence; Perform temporal pulse convolution on each video frame in the temporal pulse sequence to obtain a pulse feature sequence; The pulse feature sequence is correlated with the differentiable prototype features in the preset prototype memory bank to obtain the relevant memory feature sequence. The pulse feature sequence and the related memory feature sequence are dynamically fused to obtain memory-enhanced pulse features; Keyframes are analyzed on the video frames corresponding to the memory-enhanced pulse features to obtain keyframe discrimination scores, and video keyframes are selected from the original video data based on the keyframe discrimination scores.

[0112] In the several embodiments provided by this invention, it should be understood that the disclosed devices and apparatuses can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0113] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0114] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0115] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0116] In some embodiments of this example, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the steps of the method described in the above embodiments.

[0117] The readable storage medium of the present invention stores a computer program, which, when executed by a processor of an electronic device, can perform the following: Convert the pre-acquired raw video data into a time-series pulse sequence; Perform temporal pulse convolution on each video frame in the temporal pulse sequence to obtain a pulse feature sequence; The pulse feature sequence is correlated with the differentiable prototype features in the preset prototype memory bank to obtain the relevant memory feature sequence. The pulse feature sequence and the related memory feature sequence are dynamically fused to obtain memory-enhanced pulse features; Keyframes are analyzed on the video frames corresponding to the memory-enhanced pulse features to obtain keyframe discrimination scores, and video keyframes are selected from the original video data based on the keyframe discrimination scores.

[0118] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0119] Computer-readable storage media may also store at least one computer-executable program / instruction, such as computer-readable instructions. Computer-readable storage media include, but are not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Computer-readable storage media may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions stored on the computer-readable storage medium, the various methods described above can be performed.

[0120] In addition, the computer device may include (but is not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (e.g., keyboard, mouse, speakers, etc.).

[0121] The processor can communicate with external devices via the I / O bus through wired or wireless networks.

[0122] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product / computer program product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0123] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0124] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0125] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0126] It should be noted that, in this disclosure, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element limited by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0127] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

[0128] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely for illustrative purposes and do not represent actual use.

Claims

1. A method for extracting keyframes from video with prototype memory enhancement, characterized in that, The method includes: Convert the pre-acquired raw video data into a time-series pulse sequence; Perform temporal pulse convolution on each video frame in the temporal pulse sequence to obtain a pulse feature sequence; The pulse feature sequence is correlated with the differentiable prototype features in the preset prototype memory bank to obtain the relevant memory feature sequence. The pulse feature sequence and the related memory feature sequence are dynamically fused to obtain memory-enhanced pulse features; Keyframes are analyzed on the video frames corresponding to the memory-enhanced pulse features to obtain keyframe discrimination scores, and video keyframes are selected from the original video data based on the keyframe discrimination scores.

2. The video keyframe extraction method with prototype memory enhancement as described in claim 1, characterized in that, The step of converting pre-acquired raw video data into a time-series pulse sequence includes: The pre-acquired raw video data is used to extract video frames according to a preset frame rate, and the video frames are then aggregated into a video frame sequence. The video frame sequence is subjected to noise suppression to form a denoised video frame sequence; The video frame denoising sequence is pixel normalized to obtain a video frame normalized sequence; Perform inter-frame variation analysis on adjacent video frames in the normalized video frame sequence to obtain the inter-frame variation amplitude; When the inter-frame variation amplitude is greater than a preset amplitude threshold, the trigger events of the corresponding adjacent video frames in the normalized video frame sequence are extracted; A pulse signal is generated based on the triggering event, and the pulse signal is encoded to obtain a video frame pulse representation; The video frame pulse representations are combined in the order of the video frame sequence to obtain a time-series pulse sequence.

3. The video keyframe extraction method with prototype memory enhancement as described in claim 1, characterized in that, The step of performing temporal pulse convolution on each video frame in the temporal pulse sequence to obtain a pulse feature sequence includes: The temporal pulse sequence is subjected to synaptic weighted integration using a preset temporal pulse convolutional network to obtain the integration result; The membrane potential of the spiking neurons in the temporal spiking convolutional network is dynamically updated using the integral result to obtain the updated membrane potential; When the updated membrane potential is greater than a preset potential threshold, the spiking neuron is triggered to fire a pulse, and the pulse output after the pulse firing is obtained. The pulse output is used to attenuate the updated membrane potential, thereby generating an attenuated membrane potential; The decaying membrane potential is returned to the integral result to dynamically update the membrane potential of the spiking neurons in the temporal pulse convolutional network, thereby obtaining the step of updating the membrane potential, and the total number of pulse outputs is counted. When the total number of outputs equals the number of video frames in the time-series pulse sequence, the return process stops, and all the pulse outputs are aggregated into a pulse feature sequence.

4. The video keyframe extraction method with prototype memory enhancement as described in claim 1, characterized in that, The step of performing correlation matching between the pulse feature sequence and differentiable prototype features in a preset prototype memory bank to obtain a related memory feature sequence includes: Extract multiple differentiable prototype features from a pre-defined prototype memory; The pulse feature sequence is mapped into the feature space according to the differentiable prototype feature to generate a pulse feature mapping sequence; The correlation degree is obtained by performing correlation analysis between each pulse feature in the pulse feature mapping sequence and the differentiable prototype feature; An attention weight is generated for each of the differentiable prototype features based on the relevance. The attention weights are used to perform a weighted summation of the differentiable prototype features to obtain a relevant memory feature sequence.

5. The video keyframe extraction method with prototype memory enhancement as described in claim 1, characterized in that, The dynamic fusion of the pulse feature sequence and the related memory feature sequence to obtain memory-enhanced pulse features includes: The pulse feature sequence and the related memory feature sequence are aligned by time steps to obtain the pulse aligned feature sequence and the memory aligned feature sequence, respectively. The pulse alignment feature sequence and the memory alignment feature sequence are concatenated to obtain the concatenated feature sequence; A linear transformation is performed on the spliced ​​feature sequence to generate fusion gating coefficients; The fusion gate coefficients are nonlinearly mapped to obtain the gate weights; The pulse alignment feature sequence and the memory alignment feature sequence are summed element-wise using the gating weights to form a fused alignment feature. The fused alignment features are temporally smoothed to obtain memory-enhanced pulse features.

6. The video keyframe extraction method with prototype memory enhancement as described in claim 1, characterized in that, The keyframe discrimination score is obtained by performing keyframe analysis on the video frames corresponding to the memory-enhanced pulse features, including: The memory-enhanced pulse features are matched one-to-one with the sequential video frames of the original video data to form a feature sample set. The target learning model is obtained by optimizing the preset few-shot learning model using the feature sample set. The memory-enhanced pulse features are temporally aggregated to generate pulse aggregated features, and the temporal discriminative features in the pulse aggregated features are extracted. The target learning model is used to perform key score analysis on the temporal discrimination features to generate key frame discrimination scores.

7. The video keyframe extraction method with prototype memory enhancement as described in claim 6, characterized in that, The step of filtering video keyframes from the original video data based on the keyframe discrimination score includes: A set of candidate video frames corresponding to keyframe discrimination scores greater than a preset score threshold is selected from the original video data. The inter-frame similarity of adjacent video frames in the candidate video frame set is calculated based on the memory-enhanced pulse features. The video frame with the smallest inter-frame similarity is selected from the candidate video frame set as the video keyframe.

8. A prototype memory-enhanced video keyframe extraction device, characterized in that, The device includes: The data conversion module is used to convert the pre-acquired raw video data into a time-series pulse sequence; The pulse convolution module is used to perform temporal pulse convolution on each video frame in the temporal pulse sequence to obtain a pulse feature sequence. The feature matching module is used to perform correlation matching between the pulse feature sequence and the differentiable prototype features in the preset prototype memory bank to obtain the relevant memory feature sequence; The feature fusion module is used to dynamically fuse the pulse feature sequence and the related memory feature sequence to obtain memory-enhanced pulse features; The keyframe analysis module is used to perform keyframe analysis on the video frames corresponding to the memory-enhanced pulse features, obtain keyframe discrimination scores, and filter out video keyframes from the original video data based on the keyframe discrimination scores.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a prototype memory-enhanced video keyframe extraction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements a prototype memory-enhanced video keyframe extraction method as described in any one of claims 1 to 7.

Images