Key frame extraction method and device based on prototype network, equipment and medium

Abstract

The invention relates to the technical field of video processing, can be applied to business system platforms such as medical health and financial science and technology, and discloses a key frame extraction method based on a prototype network, which comprises the following steps: acquiring original input data, and performing frame extraction on the original input data to obtain an RGB frame sequence; constructing a visual language embedding generator by a visual encoder and a text encoder; inputting the RGB frame sequence and the text input data into a visual language embedding generator, and outputting a visual feature vector and a text feature vector; constructing a text prototype based on the text feature vector, generating a target visual frame in combination with the RGB frame sequence, and training a visual encoder through the target visual frame; calculating a visual prototype according to the RGB frame sequence through a visual encoder, constructing a hybrid prototype by the visual prototype and the text prototype, and calculating an adaptive dynamic threshold according to the hybrid prototype; and calculating the similarity between the RGB frame sequence and the hybrid prototype through a visual encoder, and screening video key frames according to the similarity and an adaptive dynamic threshold.

Description

Keyframe extraction method, apparatus, equipment, and medium based on prototype network Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a method, apparatus, device, and storage medium for keyframe extraction based on a prototype network. Background Technology

[0002] Video keyframe extraction is a core task in video analysis and retrieval. Its core objective is to select the most representative frames from a video sequence to achieve efficient representation of the video content. Traditional solutions mainly rely on manually designed features (such as optical flow and color histograms) or fully supervised training based on deep learning. However, these methods often exhibit significant limitations in scenarios with scarce data. In recent years, few-shot learning techniques, with their advantage of enabling rapid model adaptation using only a small number of labeled samples, have provided a novel approach to overcoming this challenge.

[0003] Existing few-shot keyframe extraction methods can be divided into two main categories: one is based on unimodal features, such as unsupervised clustering in some schemes, but these methods generally ignore the semantic guidance value of auxiliary modalities such as text. The other is cross-modal alignment methods, such as the cross-modal soft label alignment (CSA) technique proposed in existing research. Although it can enhance visual feature representation with the help of text information, the alignment process depends on a large amount of labeled data and is not optimized for the temporal characteristics unique to video. In addition, some methods use multi-level alignment strategies to decompose the relationship between video and text, but their keyframe selection is based solely on visual similarity and lacks a dynamic threshold adjustment mechanism. At the application level of prototype networks, existing related methods combine keyframe selection with anomaly detection, but feature alignment is limited to within the visual modality. It is worth noting that although visual language pre-trained models (such as CLIP) can establish a cross-modal shared embedding space, they still face two major challenges when directly applied to keyframe extraction: first, the fine-grained alignment relationship between video frames and text is not explicitly modeled during the pre-training stage; second, there is a lack of dynamic optimization mechanisms for cross-modal prototypes during the few-shot adaptation stage.

[0004] In the healthcare field, technologies related to few-shot keyframe extraction face significant challenges in practical application. Medical videos (such as surgical procedures, endoscopic examinations, and medical image sequences) are often characterized by scarce and extremely costly data annotation. Professional physicians need to invest a significant amount of time annotating keyframes (such as frames showing lesions or key surgical steps), which is insufficient to meet the needs of traditional fully supervised methods. Furthermore, the limitations of existing few-shot methods further restrict their application effectiveness: single-modal feature methods ignore the semantic guidance of text modalities such as electronic medical records and diagnostic reports, making it impossible to accurately associate visual frames with medical terms (such as the image frame corresponding to "polypoid lesion"); cross-modal alignment methods rely on a large amount of labeled data, which conflicts with the scarcity of medical data, and they are not optimized for the temporal continuity of medical videos (such as the development process of lesions and the sequential logic of surgical procedures), making it difficult for keyframe extraction to meet the precise needs of clinical diagnosis or surgical debriefing. Meanwhile, pre-trained models such as CLIP lack fine-grained frame-text alignment in medical scenarios (such as accurate mapping of medical image details and professional terms), and lack cross-modal prototype dynamic optimization mechanisms during the few-sample adaptation stage. This makes it difficult for the models to quickly adapt to medical videos taken by different diseases and different devices, and thus cannot provide reliable support for scenarios such as clinical auxiliary diagnosis, surgical teaching, and medical dispute tracing.

[0005] In the fintech business sector, the limitations of related technologies also hinder their application in video-based financial scenarios. Video data in the financial field (such as customer identity verification videos, financial product presentation videos, and transaction scenario monitoring videos) has both scarce annotations and high security requirements, highlighting the shortcomings of existing technologies: Single-modal methods rely solely on visual features to extract keyframes, failing to incorporate semantic information such as customer-entered identity information and product contract text. This results in keyframe extraction (such as clear facial frames in identity verification and abnormal behavior frames in transaction monitoring) lacking guidance from financial business logic, leading to insufficient accuracy. Cross-modal alignment methods rely on annotation data, which contradicts the current situation of financial data privacy protection and high annotation costs. Furthermore, they are not optimized for the temporal characteristics of financial videos (such as the sequence of transaction processes and the continuity of identity verification steps), making it difficult to adapt to the standardized process requirements of financial businesses. Furthermore, models like CLIP lack fine-grained alignment optimization for financial scenarios (such as the correlation between customer signature actions in videos and contract text terms, and the correspondence between abnormal transaction behaviors in surveillance videos and transaction records). The lack of dynamic optimization mechanisms during the few-sample adaptation phase makes it difficult for these models to quickly adapt to different financial scenarios (such as online loan verification and offline branch monitoring). They fail to meet core fintech needs such as customer identity authentication compliance review, financial risk monitoring, and business process backtracking, and may even lead to business risks such as identity theft and missed risk assessments due to errors in keyframe extraction. Summary of the Invention

[0006] The main objective of this invention is to provide a keyframe extraction method, apparatus, device, and storage medium based on a prototype network, aiming to solve the technical problems in video keyframe extraction where existing technologies rely on a large amount of labeled data in low-sample scenarios, have inaccurate cross-modal alignment, and lack dynamic threshold adjustment mechanisms, resulting in insufficient accuracy and recall in keyframe extraction.

[0007] To achieve the above objectives, this invention provides a keyframe extraction method based on a prototype network, comprising: acquiring raw input data and extracting frames from the raw input data to obtain an RGB frame sequence; constructing a visual language embedding generator using a visual encoder and a text encoder; inputting the RGB frame sequence and text input data into the visual language embedding generator for parsing, and outputting visual feature vectors and text feature vectors respectively; constructing a text prototype based on the text feature vectors, generating a target visual frame by combining the visual feature vectors of the RGB frame sequence, and training the visual encoder of the visual language embedding generator using the target visual frame; calculating a visual prototype based on the RGB frame sequence using the trained visual encoder of the visual encoder of the visual language embedding generator, fusing the visual prototype with the text prototype to construct a hybrid prototype, and calculating an adaptive dynamic threshold based on the hybrid prototype; calculating the similarity between the RGB frame sequence and the hybrid prototype using the trained visual encoder of the visual encoder of the visual language embedding generator, and filtering video keyframes based on the similarity and the adaptive dynamic threshold.

[0008] Furthermore, to achieve the above objectives, the present invention provides a keyframe extraction device based on a prototype network, comprising: a data processing module for acquiring raw input data and extracting frames from the raw input data to obtain an RGB frame sequence; a model building module for constructing a visual language embedding generator using a visual encoder and a text encoder; a feature generation module for inputting the RGB frame sequence and text input data into the visual language embedding generator for parsing, and outputting visual feature vectors and text feature vectors respectively; a text prototype module for constructing a text prototype based on the text feature vectors, generating a target visual frame by combining the visual feature vectors of the RGB frame sequence, and training the visual encoder of the visual language embedding generator using the target visual frame; a hybrid prototype module for calculating a visual prototype based on the RGB frame sequence using the trained visual encoder of the visual encoder of the visual language embedding generator, fusing the visual prototype with the text prototype to construct a hybrid prototype, and calculating an adaptive dynamic threshold based on the hybrid prototype; and a keyframe module for calculating the similarity between the RGB frame sequence and the hybrid prototype using the trained visual encoder of the visual encoder of the visual language embedding generator, and filtering video keyframes based on the similarity and the adaptive dynamic threshold.

[0009] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a keyframe extraction program based on a prototype network stored in the memory and executable on the processor, wherein when the keyframe extraction program based on the prototype network is executed by the processor, it implements the steps of the keyframe extraction method based on the prototype network as described above.

[0010] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a keyframe extraction program based on a prototype network, wherein the keyframe extraction program based on the prototype network, when executed by a processor, implements the steps of the keyframe extraction method based on the prototype network as described above.

[0011] Beneficial Effects: This invention relates to the field of video processing technology and can be applied to business system platforms such as healthcare and fintech. It discloses a keyframe extraction method based on a prototype network, comprising: acquiring raw input data and extracting frames from the raw input data to obtain an RGB frame sequence; constructing a visual language embedding generator using a visual encoder and a text encoder; inputting the RGB frame sequence and text input data into the visual language embedding generator for parsing, outputting visual feature vectors and text feature vectors respectively; constructing a text prototype based on the text feature vectors, generating a target visual frame by combining the visual feature vectors of the RGB frame sequence, and training the visual encoder of the visual language embedding generator using the target visual frame; calculating the visual prototype based on the RGB frame sequence using the trained visual encoder of the visual encoder of the visual language embedding generator, fusing the visual prototype with the text prototype to construct a hybrid prototype, and calculating an adaptive dynamic threshold based on the hybrid prototype; calculating the similarity between the RGB frame sequence and the hybrid prototype using the trained visual encoder of the visual encoder of the visual language embedding generator, and filtering video keyframes based on the similarity and the adaptive dynamic threshold. This invention first acquires video and text input, extracts the RGB frame sequence, constructs an embedding generator using visual and text encoders, and outputs corresponding feature vectors. A prototype is built based on textual features, and a target visual frame is generated by combining visual features to train the encoder. Then, the visual and textual prototypes are fused to obtain a hybrid prototype and a dynamic threshold. Keyframes are selected based on similarity. This method requires only a small number of labeled frames, improving the recall rate of keyframes in low-sample scenarios. Attached Figure Description

[0012] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the drawings: Figure 1 is a schematic diagram of an application environment of the keyframe extraction method based on a prototype network in an embodiment of the present invention; Figure 2 is a schematic flowchart of an embodiment of the keyframe extraction method based on a prototype network in the present invention; Figure 3 is a schematic diagram of the functional modules of a preferred embodiment of the keyframe extraction device based on a prototype network in the present invention; Figure 4 is a schematic diagram of the structure of a computer device in an embodiment of the present invention; Figure 5 is another schematic diagram of the structure of a computer device in an embodiment of the present invention. Detailed Implementation

[0013] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0014] The keyframe extraction method based on prototype networks provided in this invention can be applied in the application environment shown in Figure 1, where the user terminal communicates with the server via a network. The server can obtain raw input data from the user terminal and extract frames from the raw input data to obtain an RGB frame sequence. A visual language embedding generator is constructed using a visual encoder and a text encoder. The RGB frame sequence and text input data are input into the visual language embedding generator for parsing, outputting visual feature vectors and text feature vectors respectively. A text prototype is constructed based on the text feature vectors, and a target visual frame is generated by combining the visual feature vectors of the RGB frame sequence. The visual encoder of the visual language embedding generator is trained using the target visual frame. The trained visual encoder of the visual language embedding generator calculates the visual prototype based on the RGB frame sequence, merges the visual prototype with the text prototype to construct a hybrid prototype, and calculates an adaptive dynamic threshold based on the hybrid prototype. The trained visual encoder of the visual language embedding generator calculates the similarity between the RGB frame sequence and the hybrid prototype, and selects video keyframes based on the similarity and the adaptive dynamic threshold. This invention first acquires video and text input, extracts RGB frame sequences, constructs an embedding generator using visual and text encoders, and outputs corresponding feature vectors. A prototype is constructed based on textual features, and a target visual frame is generated by combining visual features to train the encoder. Then, the visual and textual prototypes are fused to obtain a hybrid prototype and a dynamic threshold, and keyframes are selected based on similarity. This method requires only a small number of labeled frames, improving the keyframe recall rate in low-sample scenarios. The user end can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server end can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0015] Please refer to Figure 2, which is a flowchart illustrating an embodiment of the keyframe extraction method based on a prototype network provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0016] As shown in Figure 2, the keyframe extraction method based on prototype networks proposed in this invention includes the following steps: S100, acquiring raw input data and extracting frames from the raw input data to obtain an RGB frame sequence; S200, constructing a visual language embedding generator using a visual encoder and a text encoder; S300, inputting the RGB frame sequence and text input data into the visual language embedding generator for parsing, and outputting visual feature vectors and text feature vectors respectively; S400, constructing a text prototype based on the text feature vectors, generating a target visual frame by combining the visual feature vectors of the RGB frame sequence, and training the visual encoder of the visual language embedding generator using the target visual frame; S500, calculating the visual prototype based on the RGB frame sequence using the trained visual encoder of the visual language embedding generator, fusing the visual prototype with the text prototype to construct a hybrid prototype, and calculating an adaptive dynamic threshold based on the hybrid prototype; S600, calculating the similarity between the RGB frame sequence and the hybrid prototype using the trained visual encoder of the visual encoder of the visual language embedding generator, and filtering video keyframes based on the similarity and the adaptive dynamic threshold.

[0017] In this embodiment, the raw input data consists of two parts: first, the raw video stream in MP4 format encoded with H.264; and second, optional text description data, supporting SRT subtitle files or manually annotated text content. In the frame extraction stage, a continuous RGB frame sequence is extracted from the video stream at a fixed rate of 24 fps (frames per second) using a video decoder, providing basic visual data for subsequent feature extraction.

[0018] The visual language embedding generator is built upon a dual-tower CLIP (Contrastive Language-Image Pre-trained) model, with its core consisting of a visual encoder and a text encoder. The visual encoder employs a ViT-B / 32 architecture (a basic visual feature extraction model based on Transformer, where 32 represents the size of the input image patch), while the text encoder is a natural language processing encoder based on the Transformer architecture. The extracted RGB frame sequence is input into the visual encoder, which then outputs a 768-dimensional normalized visual feature vector. Similarly, the text input data, after being parsed by the text encoder, also outputs a 768-dimensional normalized text feature vector. This unification of the dimensions of the two feature vectors lays the foundation for cross-modal alignment.

[0019] The construction of text prototypes requires first determining the number of keyframe categories K, then generating K text prototypes, and maintaining the stability of the prototypes through a momentum update mechanism. Subsequently, pseudo-labels are generated using these text prototypes, and semantic association is performed on unlabeled RGB frames. Visual frames corresponding to pseudo-labels with high confidence (e.g., a confidence threshold of 0.8) are selected as target visual frames. Then, the visual encoder is trained using an alignment loss function (used to optimize the semantic consistency between visual features and text prototypes) to continuously improve the semantic discrimination ability of visual features, forming a positive feedback between the visual and language modalities.

[0020] The visual prototype is calculated based on the trained visual encoder, while the hybrid prototype is obtained by linearly fusing the visual and textual prototypes. The adaptive dynamic threshold is calculated using a two-layer perceptron (MLP, a simple deep learning model) and a sigmoid activation function. This threshold can adaptively adjust according to the distribution characteristics of different video content, thereby improving the flexibility of keyframe selection.

[0021] Finally, the visual feature vectors of all frames in the RGB frame sequence are extracted using the trained visual encoder. The similarity between each frame's features and the hybrid prototype is calculated (usually using cosine similarity). Frames with similarity exceeding the corresponding category's dynamic threshold are selected as candidate keyframes. To remove redundant frames, a temporal NMS (non-maximum suppression) algorithm is used, with a time threshold of 0.5 seconds to eliminate duplicate candidate frames with too close time intervals. The final output is the filtered keyframe index set, completing the entire keyframe extraction process.

[0022] For example, in the fintech field, this technology can be widely applied to the efficient analysis and value mining of financial video data. For instance, in scenarios such as live-streaming financial product promotions, online financial lectures, and stock market analysis videos, traditional keyframe extraction requires a large amount of labeled data to accurately capture core information, while this technology only requires 5-10 labeled keyframes (such as "product yield announcement," "risk warning explanation," and "market turning point analysis") to achieve efficient extraction. In financial regulatory scenarios, regulatory agencies can use this technology to process massive amounts of online meeting videos from financial institutions and customer service recordings, quickly extracting keyframes related to compliance statements, explanations of trading rules, and risk disclosures, forming core evidence fragments for compliance review and significantly improving regulatory efficiency. In the field of robo-advisors, by extracting keyframes from financial news videos and accompanying videos of industry analysis reports (such as "key points of policy releases," "core data display of corporate financial reports," and "analyst's core viewpoints"), combined with cross-modal alignment of textual and visual features, video summaries can be quickly generated, providing investors with accurate decision-making reference information while reducing the time cost of information acquisition. Furthermore, in anti-fraud scenarios, this technology can quickly capture key frames such as false promises and inaccurate data displays in false advertising videos related to financial fraud, providing technical support for fraud identification and evidence preservation.

[0023] In the healthcare field, this technology provides an efficient solution for the intelligent processing of medical video data. In scenarios such as telemedicine consultation videos, surgical teaching videos, and medical academic lectures, keyframe extraction can facilitate the rapid retrieval and sharing of medical information. For example, in surgical teaching videos, by labeling 5-10 keyframes such as "surgical incision location," "key operation steps," and "postoperative suturing," this technology can automatically extract the core operation frame sequence throughout the entire surgical procedure, forming standardized teaching segments that facilitate rapid learning of key surgical techniques by medical staff. For medical imaging-related videos (such as ultrasound dynamic videos and endoscopic examination videos), combined with text descriptions (such as "lesion area annotation" and "organ structure description"), this technology can accurately extract keyframes containing lesion characteristics and key organ structures, assisting doctors in diagnosing conditions. Especially in primary healthcare institutions, it can reduce the time doctors spend browsing massive amounts of video data and improve diagnostic efficiency. In terms of medical supervision and quality control, for videos of medical institutions' diagnosis and treatment processes, keyframes such as "patient informed consent confirmation," "treatment plan notification," and "key medical operation records" can be extracted as important evidence for medical quality assessment and dispute resolution. Furthermore, in the field of health science videos, by extracting key frames such as "core health knowledge points" and "key steps for operation demonstration," a summary of the science video can be quickly generated, helping the public to efficiently acquire health knowledge and improve the effectiveness of science communication.

[0024] In one embodiment, S100 includes: S101, acquiring video input data and text input data, and preprocessing the video input data and text input data; S102, extracting frames from the preprocessed video input data using a video decoder to obtain frame sampling data; S103, combining the frame sampling data in chronological order to obtain an RGB frame sequence.

[0025] In this embodiment, the video input data specifically refers to MP4 files in H.264 encoding format. This encoding standard is widely used, offering both high compression ratios and good compatibility, effectively reducing data storage and transmission overhead while ensuring video quality. Text input data is optional, supporting SRT subtitle files (a common format containing subtitle text and corresponding timeline information) or manually annotated text descriptions. The text content must be relevant to the video theme to provide semantic guidance information. The preprocessing stage primarily standardizes two types of data: video data undergoes format verification to ensure compliance with the H.264 encoding standard, preventing decoding failures due to format abnormalities; text data undergoes cleaning (e.g., removing redundant characters, standardizing encoding formats) and semantic normalization (e.g., standardizing descriptive sentence structures) to ensure the accuracy and effectiveness of text information, laying a solid foundation for subsequent cross-modal alignment work.

[0026] The video decoder is the core tool for frame extraction. Its core function is to parse the pre-processed MP4 video file and extract consecutive image frames from the video stream at a fixed rate of 24 fps (frames per second). This process is called frame sampling, and the final single-frame image data is called frame sampled data. A sampling rate of 24 fps is the optimal choice considering both visual continuity and data processing efficiency. It can fully preserve the key visual information in the video while avoiding data redundancy caused by an excessively high sampling rate or information loss caused by an excessively low sampling rate.

[0027] The combination of frame sampling data must strictly follow the original temporal order of the video. All extracted single-frame images are arranged sequentially according to their time in the video, forming an ordered RGB frame sequence. This sequence fully preserves the temporal characteristics and visual information of the video. Each frame is stored as red (R), green (G), and blue (B) pixel data, which is the basic data format for subsequent visual feature extraction and cross-modal processing. Its orderliness ensures that subsequent models accurately capture the dynamic changes and temporal relationships in the video.

[0028] For example, in the fintech field, this technical process provides crucial support for the efficient processing of video-based financial data. For scenarios such as live-streaming financial product sales and online financial lectures, video input data can be H.264 encoded MP4 files of the live stream playback, while text input data can be SRT subtitles from the live stream (containing product introductions, profit explanations, risk warnings, etc.) or manually compiled core knowledge point text. After preprocessing to standardize video and text formats, the video decoder extracts frame sampling data at 24fps and then combines it into an RGB frame sequence in chronological order. Subsequently, it can combine semantic information to accurately extract key frames such as "product yield announcement," "risk clause interpretation," and "customer Q&A," which can be used to generate video summaries, compliance review evidence, or product promotional materials. In financial regulatory scenarios, for online meeting videos and customer service videos of financial institutions, this process extracts RGB frame sequences, which, combined with text descriptions (such as meeting agendas and service specification texts), can quickly locate frame data involving key aspects such as compliance operations, transaction confirmation, and information disclosure, improving the efficiency and accuracy of regulatory verification. In addition, for financial videos such as stock market analysis and industry analysis, this process can transform continuous video data into an ordered sequence of frames, providing a foundation for subsequent extraction of key frames such as market turning points and core data charts by combining text features (such as market data descriptions and analyst opinion texts), helping investors quickly obtain effective information.

[0029] In the healthcare field, this technological workflow can efficiently process various types of medical video data, empowering medical services and medical research. For example, surgical instruction videos (H.264 encoded MP4 files) can be paired with SRT subtitles (explanation of surgical steps, precautions) or doctor-annotated text (such as "incision location," "instrument operation," "suture steps") as input. After preprocessing, the video decoder extracts frame sampling data at 24fps and combines it into an RGB frame sequence. Subsequently, key surgical step frames can be extracted based on this sequence and text semantics to form standardized teaching segments, facilitating learning and training for medical staff. For telemedicine consultation videos, the video input is an MP4 recording of the consultation process, and the text input is manually annotated text such as patient case summaries and consultation discussion points. The RGB frame sequence obtained through this workflow can help doctors quickly locate key frames such as patient condition descriptions, examination results displays, and treatment plan discussions, improving consultation efficiency. In medical image analysis, H.264 encoded files such as ultrasound dynamic videos and endoscopic examination videos, after preprocessing, are converted into RGB frame sequences through frame extraction and sequence combination. Combined with text descriptions (such as "lesion area annotation" and "organ structure description"), this helps doctors accurately locate frame data containing lesion features and key organ structures, providing support for disease diagnosis. Furthermore, health education videos can use this process to extract frame sequences, which, along with educational text (such as "health knowledge points" and "operation demonstration instructions"), quickly select core teaching frames to generate concise and easy-to-understand educational content, improving the effectiveness of health knowledge dissemination.

[0030] In one embodiment, step S300 includes: S3011, parsing the RGB frame sequence to obtain single-frame visual data; S3012, inputting the single-frame visual data into the visual encoder of the visual language embedding generator; and S3013, processing the single-frame visual data through the temporal attention enhancement mechanism of the visual encoder to obtain a visual feature vector.

[0031] In this embodiment, the RGB frame sequence is a continuous collection of images arranged in chronological order. The core of parsing this sequence is to extract single-frame visual data frame by frame, that is, to separate the pixel matrix information of each independent frame from the ordered frame sequence. These single-frame visual data completely preserve the visual details of the video at the corresponding time point and serve as the basic input for subsequent feature extraction. The visual encoder of the visual language embedding generator adopts the ViT-B / 32 architecture, and its core function is to transform two-dimensional image data into high-dimensional semantic feature vectors.

[0032] The temporal attention enhancement mechanism is a key improvement module in the visual encoder. It is implemented by adding a lightweight Temporal Transformer to the final layer of the encoder. This mechanism effectively captures the temporal correlation between consecutive frames, avoiding the loss of contextual information caused by processing individual frames in isolation. The processing involves calculating the attention weights of the query vector (Q), key vector (K), and value vector (V). After normalizing the attention weights using the Softmax function, they are weighted and summed with V to obtain the final visual feature vector. This 768-dimensional normalized vector contains both the visual detail features of a single frame and the temporal contextual information of consecutive frames, significantly improving the discriminative power and semantic relevance of the features.

[0033] For example, in the fintech field, this technical process can efficiently process various types of financial video data and extract core visual semantic information. For instance, for live-stream replay videos of financial products, the single-frame visual data obtained after parsing the RGB frame sequence includes product display images, yield charts, and the host's explanations. After being input into the visual encoder, the temporal attention enhancement mechanism can capture the temporal correlation of consecutive frames such as "host pointing to the yield chart → chart magnification → host interpreting the data," and the generated visual feature vector can accurately represent the core semantic of "product yield display." In financial regulatory scenarios, for online compliance training videos of financial institutions, the visual feature vector extracted through this process can be combined with temporal correlation to identify key stages such as "compliance clause display → lecturer emphasizing key points → case analysis demonstration," providing semantic support for training effectiveness evaluation and compliance verification. Furthermore, in stock market analysis videos, the temporal attention enhancement mechanism can capture the continuous frame logic of "market curve fluctuations → analyst pointing to fluctuation points → data annotation appearance," and the generated feature vector can accurately correspond to core scenarios such as "market turning point analysis," providing key visual semantic input for intelligent investment advisory systems and helping investors quickly locate effective information in the video.

[0034] In the healthcare field, this technological process provides core support for semantic understanding of medical videos, empowering scenarios such as clinical diagnosis and treatment, and medical education. For example, after parsing the RGB frame sequence of a surgical teaching video, a single frame of visual data contains images of surgical incisions, instrument operations, and tissue details. The visual encoder, through a temporal attention enhancement mechanism, can capture the temporal associations of continuous surgical steps such as "instrument entry into the incision → precise tissue separation → hemostasis." The generated visual feature vectors can clearly represent the semantic features of each surgical step, providing a foundation for the extraction of standardized surgical teaching segments. In telemedicine consultation videos, a single frame of visual data contains patient symptom displays, examination report images, and doctor consultation scenes. The temporal attention enhancement mechanism can capture the continuous frame associations of "doctor inquiring about symptoms → patient displaying lesions → doctor highlighting key points." The generated feature vectors can accurately correspond to core diagnostic and treatment steps such as "disease description and lesion display," helping consulting doctors quickly focus on key visual information. For dynamic medical imaging videos (such as ultrasound examination videos), the visual feature vectors generated by this process can be combined with temporal correlation to capture the dynamic changes in the lesion area, helping doctors to make more accurate judgments about the condition. Especially in primary healthcare institutions, this can improve the efficiency and accuracy of image analysis.

[0035] In one embodiment, step S300 includes: S3021, parsing the text input data to obtain text description data; S3022, inputting the text description data into the text encoder of the visual language embedding generator, quantizing the text description data through the text encoder, and outputting a text feature vector.

[0036] In this embodiment, the text input data serves as crucial semantic guidance information. It is flexible in form and highly adaptable, and can be either SRT subtitle files or manually annotated topic-related text descriptions. The core of parsing the text input data is to extract pure semantic content and remove irrelevant and redundant information. For SRT subtitle files, non-textual semantic elements such as timelines and formatting tags need to be stripped away, retaining only the original subtitle text. For manually annotated text, processing such as formatting and redundant character removal is required to ultimately obtain structured and semantically clear text description data, ensuring that subsequent encoding processes can focus on core semantic information.

[0037] The text encoder of the visual language embedding generator adopts the Transformer architecture and is the core component that transforms textual description data into numerical features. After the textual description data is input, it is first converted into word embedding vectors that the model can recognize. Then, through the self-attention mechanism in the Transformer architecture, the word dependencies in the text sequence are modeled, such as recognizing the semantic association between "risk warning" and "investment loss", and then integrating the semantic information of the entire text.

[0038] The quantization process is essentially a numerical transformation and normalization of semantic information. Through multi-layer network operations of the text encoder, the final output is a normalized text feature vector with a dimension of 768. This vector transforms the abstract semantics of the text into a computable numerical form, and the 768-dimensional design is consistent with the visual feature vector, providing a data foundation for subsequent cross-modal feature alignment and ensuring that visual and textual semantics can be correlated and calculated in the same feature space.

[0039] For example, in the fintech field, this technical process can provide precise textual semantic support for the semantic understanding of financial videos, empowering various business scenarios. For instance, in live-streaming financial product promotions, text input data can be the live stream's SRT subtitles (containing key information such as product yield, risk level, and investment period) or manually compiled core selling points of the product. After parsing, the resulting text description data is input into a text encoder, and the output text feature vectors can accurately represent core semantics such as "annualized yield of 4.5%" and "low-risk, stable." These text feature vectors can be aligned with the visual feature vectors of video frames, helping the system quickly locate key frames displaying product yield data and risk warning clauses, providing support for compliance review and extraction of product promotional materials. In stock market analysis videos, text description data can be summaries of market analysis reports or transcribed text of analysts' verbal interpretations. The feature vectors quantified by the text encoder can capture semantic information such as "the market index rose by 2%" and "the industry sector led the gains." Combined with visual features, this can accurately extract corresponding key frames from the video, such as data chart displays and market turning point analysis, providing core semantic input for intelligent investment advisory systems and assisting investors in efficiently obtaining decision-making information. Furthermore, in financial regulatory scenarios, for online meeting videos of financial institutions, the text description data can include meeting agendas and compliance clauses. The quantified text feature vectors can guide the system to identify key links in the video involving compliance statements and interpretation of transaction rules, thereby improving the targeting and efficiency of regulatory verification.

[0040] In the healthcare field, this technological process can deeply mine textual semantic information, synergistically empowering medical services and medical education in conjunction with video visual information. For example, in surgical teaching videos, text input data can use SRT surgical explanation subtitles (including surgical steps, instrument names, operational precautions, etc.) or text annotated by doctors highlighting core surgical steps. The parsed text description data is quantized by a text encoder, and the output text feature vector can accurately represent surgical semantics such as "incision location," "hemostasis," and "suture steps." After aligning these text features with the visual features of the surgical video frames, the system can quickly extract key frames of the corresponding surgical steps, forming standardized teaching segments for easy learning by medical staff. In telemedicine consultation videos, text description data can be manually annotated text such as patient case summaries and consultation discussion points. The quantified text feature vector can capture core medical semantics such as "history of hypertension" and "0.8cm diameter lung nodule." Combined with visual features such as patient symptom displays and examination report images in the video, this can help consulting doctors quickly focus on key diagnostic information and improve consultation efficiency. For medical science videos, the text input data can be the SRT subtitles or core knowledge point text of the science script. The encoded text feature vector can represent the science semantics such as "correct way to wear a mask" and "steps for blood glucose monitoring". After being aligned with the video visual features, key teaching frames in the science video can be extracted to generate concise and easy-to-understand science summaries, which helps to disseminate health knowledge efficiently.

[0041] In one embodiment, step S400 includes: S401, constructing several text prototypes based on the text feature vectors, and performing stability maintenance processing on the text prototypes through a momentum update mechanism; S402, obtaining unlabeled visual frames and their corresponding visual feature vectors from the RGB frame sequence; S403, calculating the similarity between the visual feature vectors of the unlabeled visual frames and the processed text prototypes, and obtaining a pseudo-label probability distribution based on the similarity; S404, comparing the pseudo-label probability distribution with a confidence threshold; S405, when the pseudo-label probability distribution is greater than the confidence threshold, taking the unlabeled visual frame corresponding to the pseudo-label probability as the target visual frame; S406, calculating an initial visual prototype based on the visual feature vectors of labeled keyframes; and S407, performing matching training on the visual encoder of the visual language embedding generator based on the target visual frame and the initial visual prototype using a cross-modal alignment loss function.

[0042] In this embodiment, the construction of text prototypes is based on text feature vectors. First, the number of keyframe categories K needs to be determined, and then K text prototypes are generated. To ensure the stability of the text prototypes, a momentum update mechanism is used for maintenance. This mechanism allows the text prototypes to be updated smoothly during the iteration process, avoiding prototype distortion caused by fluctuations in single batches of data.

[0043] Unlabeled visual frames are extracted directly from the RGB frame sequence without the need for manual labeling of category information, as their corresponding visual feature vectors have already been obtained in advance by the visual encoder of the visual language embedding generator. Next, the similarity between the visual feature vectors of the unlabeled visual frames and the text prototypes after momentum updates (usually using cosine similarity) is calculated, and the pseudo-label probability distribution is obtained based on this similarity.

[0044] Furthermore, the confidence threshold can be set to 0.8. The maximum value in the pseudo-label probability distribution is compared with this threshold. When the maximum value is greater than 0.8, it indicates that the unlabeled visual frame has a strong semantic correlation with the corresponding text prototype and has high annotation reliability. In this case, the unlabeled visual frame is determined as the target visual frame for subsequent model training. The initial visual prototype is calculated based on the visual feature vectors of the labeled keyframes. The initial visual prototype is obtained by averaging the visual feature vectors of labeled frames of the same type.

[0045] The Cross-Modal Alignment Loss Function is the core of visual encoder matching training. This loss function achieves accurate alignment between visual features and text semantics by minimizing the difference between the pseudo-label probability distribution of the target visual frame and the predicted distribution of the visual prototype, thereby optimizing the parameters of the visual encoder and improving its semantic discrimination ability.

[0046] For example, in the fintech field, this technical process can fully utilize a small amount of labeled data and a large amount of unlabeled financial video data to improve the accuracy and efficiency of keyframe extraction. For instance, in processing financial product compliance training videos, the text prototype can be constructed based on K types of keyframe themes such as "interpretation of compliance terms," ​​"risk warning explanation," and "case analysis demonstration." A momentum update mechanism ensures that the text prototype stably represents the semantics of various themes. A large number of unlabeled visual frames are extracted from the RGB frame sequence of the training video, and their similarity to the text prototype is calculated to obtain a pseudo-label probability distribution. Target visual frames with a confidence level higher than 0.8 (such as frames clearly displaying compliance clause text or frames where the instructor emphasizes risk points) are selected. An initial visual prototype is calculated by combining a small number of labeled keyframes (such as manually labeled "core compliance clause display" frames). A visual encoder is trained using a cross-modal alignment loss function, enabling the model to accurately identify similar keyframes. This process can be applied to financial regulatory scenarios, quickly extracting compliance-related key frames from massive amounts of online meeting videos and customer service videos from financial institutions, providing efficient support for regulatory verification; it can also be used for compliance review of financial product promotional videos, automatically identifying key frames such as illegal advertising and insufficient risk disclosure, thereby reducing compliance risks.

[0047] In the healthcare field, this technological process can effectively leverage the characteristics of medical video data to empower scenarios such as medical education and clinical diagnosis and treatment. Taking surgical teaching videos as an example, the text prototype can be constructed around K key surgical steps such as "incision location," "instrument operation," "hemostasis steps," and "suture procedure." After maintenance by a momentum update mechanism, it ensures that the prototype can stably reflect the semantic features of each step. Unlabeled visual frames are extracted from the RGB frame sequence of the surgical video, and their similarity with the text prototype is calculated to generate a pseudo-label probability distribution. Target visual frames with a confidence score of over 0.8 (such as frames that clearly show the incision location or frames showing precise instrument operation) are selected. An initial visual prototype is calculated based on a small number of manually labeled key surgical frames. The visual encoder is trained through a cross-modal alignment loss function, enabling the model to accurately capture the visual features of key steps in similar surgeries. This process can be used to generate standardized surgical teaching materials, automatically extracting core operation frames from different surgical videos to form unified teaching segments; it can also be applied to telemedicine consultations, extracting key frames related to the patient's condition from patient examination videos (such as endoscopic videos and ultrasound dynamic videos), and combining them with text descriptions (such as case summaries) to assist doctors in making rapid diagnoses. Especially in primary healthcare institutions, it can effectively improve the efficiency of medical resource utilization and diagnostic accuracy.

[0048] In one embodiment, step S500 includes: S501, obtaining labeled keyframes based on the RGB frame sequence; S502, encoding the labeled keyframes using a visual encoder of a trained visual language embedding generator to generate labeled visual feature vectors; S503, calculating a visual prototype based on the labeled visual feature vectors; S504, linearly fusing the visual prototype with a text prototype to obtain a hybrid prototype; and S505, processing the splicing features of the hybrid prototype using a two-layer perceptron to obtain an adaptive dynamic threshold.

[0049] In this embodiment, the labeled keyframes are representative frames with clear semantic labels selected from the RGB frame sequence arranged in chronological order. Only 5-10 frames are needed to meet the requirements of subsequent processes. The labels of these frames must be consistent with the core semantics of the text input data, such as corresponding to preset categories like "core information presentation" and "key operation steps". They serve as basic samples for model learning, providing accurate semantic references for the subsequent construction of visual prototypes.

[0050] The visual encoder of the trained visual language embedding generator employs a ViT-B / 32 architecture optimized through cross-modal alignment training. After annotated keyframes are input into this encoder, a temporal attention enhancement mechanism is used to capture the temporal correlations between frames. Simultaneously, gradient decoupling training avoids gradient conflicts between modalities, ultimately outputting a 768-dimensional normalized annotated visual feature vector. This vector contains both the core visual details of the annotated frame itself and incorporates feature information aligned with the text semantics, ensuring the accuracy and effectiveness of semantic representation.

[0051] The calculation of the visual prototype revolves around the labeled visual feature vectors of the same category, and is obtained by averaging the visual feature vectors of all labeled keyframes within the same category. This calculation method can fully aggregate the common visual semantics of labeled frames in that category, forming a visual prototype representation specific to that category, allowing the model to clearly grasp the common visual features of keyframes in different categories.

[0052] The hybrid prototype is obtained by linearly fusing a visual prototype and a textual prototype. During the fusion process, an adjustable modal weight is set to balance the visual detail representation capability of the visual prototype and the semantic guidance capability of the textual prototype. In this way, the hybrid prototype can retain the essential visual features of the video frame itself while accurately matching the core semantics of the text description, achieving deep fusion of visual and textual cross-modal information and providing more comprehensive semantic support for subsequent similarity calculations.

[0053] The adaptive dynamic threshold acquisition process first involves concatenating features from the visual and textual prototypes to form a joint feature vector containing information from both modalities. This joint feature vector is then input into a two-layer perceptron (MLP) for feature transformation. The transformed feature output is then normalized to the 0-1 range using a sigmoid activation function. Finally, it is multiplied by a preset baseline threshold to obtain the final adaptive dynamic threshold. This process allows the threshold to be flexibly adjusted based on the feature distribution of different categories of cross-modal prototypes, effectively improving the adaptability and accuracy of keyframe selection.

[0054] For example, in the fintech field, this technological process can accurately capture core semantic information in financial videos, enabling efficient operation across various business scenarios. For instance, in processing financial product promotion videos, keyframe annotation can select 5-10 frames with clear semantics, such as "product return chart display," "risk warning clause presentation," and "investment period explanation." A trained visual encoder generates annotated visual feature vectors, which are then combined with a text prototype constructed from corresponding text descriptions (e.g., "annualized return of 5%" and "risk level R2") to obtain a hybrid prototype. An adaptive dynamic threshold can automatically adjust the keyframe selection criteria based on the feature distribution of different product videos, accurately identifying frames highly relevant to core semantics such as "return display" and "risk warning," providing precise support for product promotion material editing and compliance review. In financial regulatory scenarios, for online trading training videos of financial institutions, keyframe annotation can correspond to categories such as "trading process demonstration" and "compliant operation specifications." Through hybrid prototypes and adaptive dynamic thresholds, key teaching frames and compliant operation demonstration frames can be quickly extracted from massive training videos, helping regulatory agencies efficiently verify the completeness and compliance of training content. Furthermore, in stock market analysis videos, this process can combine labeled keyframes (such as "market index turning point" and "industry sector leading data") with text descriptions (such as "Shanghai Composite Index rose 3%" and "new energy sector led the gains") to generate hybrid prototypes and dynamic thresholds that are adapted to market characteristics. It can accurately extract keyframes such as core market data display and analyst key viewpoint interpretation, providing high-quality information input for intelligent investment advisory systems.

[0055] In the healthcare field, this technology can be deeply adapted to the professional characteristics of medical videos, providing efficient support for clinical diagnosis and treatment, medical education, and other scenarios. Taking surgical teaching videos as an example, keyframe annotation can select frames from 5-10 key stages, such as "surgical incision location," "core instrument operation," and "postoperative suturing." After encoding and generating annotated visual feature vectors, these vectors are fused with the corresponding textual descriptions (such as "abdominal transverse incision" and "laparoscopic tissue separation") to form a hybrid prototype. Adaptive dynamic thresholds can automatically adjust the selection criteria based on the differences in video features of different surgical types (such as surgical procedures and minimally invasive procedures), accurately extracting key operation frames for each surgical stage to form standardized teaching segments, facilitating medical staff to quickly master core surgical skills. In telemedicine consultations, keyframe annotation can correspond to categories such as "patient lesion display," "examination report images," and "doctor's diagnosis explanation." The hybrid prototype can integrate the visual features of lesions with the semantic descriptions of the patient's condition in the case text, while the dynamic thresholds can adapt to the characteristics of different patients' medical videos, quickly filtering out keyframes relevant to the diagnosis. This helps consulting doctors focus on core diagnostic information, improving consultation efficiency and accuracy. For dynamic medical imaging videos (such as ultrasound examination videos), this process can combine labeled keyframes (such as "close-up of lesion area" and "abnormal organ structure") with text descriptions (such as "liver space-occupying lesion" and "kidney stone image") to generate targeted hybrid prototypes and dynamic thresholds, accurately extract key diagnostic frames in the images, and assist doctors in analyzing the condition more efficiently, especially providing strong technical support for primary healthcare institutions.

[0056] In one embodiment, step S600 includes: S601, obtaining a query frame based on the RGB frame sequence, and encoding the query frame using a visual encoder of a trained visual language embedding generator to obtain a query visual feature vector; S602, calculating the similarity between the query visual feature vector and the hybrid prototype; S603, comparing the similarity with an adaptive dynamic threshold, and filtering key candidate frames based on the comparison result; S604, performing redundant frame removal processing on the key candidate frames using a temporal nonmaximum suppression algorithm to obtain video key frames.

[0057] In this embodiment, the query frame is a frame to be judged selected from the complete RGB frame sequence, covering all frames in the video except for a small number of labeled keyframes. These frames are the core objects of keyframe screening, and whether they can be accurately identified as keyframes directly determines the completeness and representativeness of the final extraction results. The visual encoder of the trained visual language embedding generator is a ViT-B / 32 architecture that has been trained through cross-modal alignment and optimized by gradient decoupling. After the query frame is input into the encoder, the temporal correlation between frames is captured through the temporal attention enhancement mechanism, and a 768-dimensional normalized query visual feature vector is output. This vector not only retains the unique visual details of the query frame, but also incorporates feature information aligned with the semantics of the text, providing a high-quality feature foundation for subsequent similarity calculation.

[0058] The core of similarity calculation is to measure the semantic association between the query visual feature vector and the hybrid prototype. Cosine similarity is typically used as the calculation metric, representing the similarity by calculating the cosine of the angle between two vectors. The value ranges from -1 to 1, with a value closer to 1 indicating a stronger semantic association. The hybrid prototype is a linear fusion of the visual and textual prototypes, integrating detailed representations of visual features with semantic guidance from the text, allowing similarity calculation to simultaneously consider visual matching and semantic consistency.

[0059] The adaptive dynamic threshold is a threshold calculated separately for each keyframe category k. MLP and the sigmoid activation function can adaptively adjust the threshold size based on the feature distribution of the mixed prototype, making the threshold more closely match the semantic feature differences between different categories. The similarity of the query frame is compared with the adaptive dynamic threshold of the corresponding category. When the similarity is greater than the threshold, it indicates that the query frame highly matches the core semantics of the corresponding category, and it is identified as a key candidate frame.

[0060] Temporal Non-Maximum Suppression (NMS) is a crucial step in removing redundant key candidate frames. Its core logic is to select the most representative frames along the time dimension, avoiding key frame redundancy caused by semantic similarity between consecutive frames. This algorithm sets a time threshold of 0.5 seconds. For multiple key candidate frames with time intervals less than 0.5 seconds, only the frame with the highest similarity is retained as the final key frame, while the remaining frames are deemed redundant and discarded. The final output set of key frames not only fully covers the core semantics of the video but also ensures reasonable time intervals between frames, achieving efficient key frame selection.

[0061] For example, in the fintech field, this technology can accurately and efficiently extract keyframes from various financial videos, providing strong support for business operations. For instance, in a live stream replay of a financial product, the query frames encompass all frames from the entire live stream. A trained visual encoder encodes the query's visual feature vector, which is then compared to a hybrid prototype of categories such as "product benefit display" and "risk warning explanation." Candidate frames are selected using an adaptive dynamic threshold, and redundant frames within 0.5 seconds are removed using a temporal nonmaximum suppression algorithm, ultimately yielding keyframes that clearly present the product's core information. These keyframes can be used to generate product promotion summaries and compliance review evidence, helping financial institutions quickly verify whether live stream content contains illegal promotions or insufficient risk warnings. In financial regulatory scenarios, for online meeting videos and customer service videos from financial institutions, this process can quickly extract keyframes involving compliance statements, interpretations of transaction rules, and notifications of customer rights, providing regulatory agencies with accurate materials for efficiently verifying the compliance of financial institutions' business operations. Furthermore, in stock market analysis videos, this process can extract key frames such as "market index inflection point display," "industry sector leading data presentation," and "analyst core viewpoint interpretation," providing high-quality information input for intelligent investment advisory systems and assisting investors in quickly obtaining effective decision-making information.

[0062] In the healthcare field, this technological workflow can be deeply adapted to the professional characteristics of medical videos, empowering scenarios such as clinical diagnosis and treatment, and medical education. Taking surgical teaching videos as an example, the query frame contains all operation frames of the entire surgical procedure. After encoding, the similarity is calculated with mixed prototypes of categories such as "incision location," "instrument operation," "hemostasis steps," and "suture process." Candidate frames are filtered out through adaptive dynamic thresholding, and redundant frames with too close an interval are removed using a temporal nonmaximum suppression algorithm. Finally, a keyframe sequence that can completely present the key operations of the surgery is obtained. These keyframes can be used to create standardized surgical teaching segments, facilitating medical staff to quickly learn core surgical skills, especially providing efficient training resources for medical staff in primary healthcare institutions. In telemedicine consultations, for patient examination videos (such as endoscopic videos and dynamic ultrasound videos), this workflow can extract keyframes such as "close-up of the lesion area" and "display of organ structural abnormalities." Combined with the semantic information of the case text description, it helps consulting doctors quickly focus on core diagnostic and treatment information, improving consultation efficiency and accuracy. For medical science videos, this process can extract key frames such as "explanation of health knowledge points" and "key steps of operation demonstration" to generate concise and easy-to-understand science summaries, which can help disseminate health knowledge efficiently and help the public quickly master core health skills.

[0063] In one embodiment, a keyframe extraction device based on a prototype network is provided, which corresponds one-to-one with the keyframe extraction method based on a prototype network in the above embodiments. Referring to Figure 3, Figure 3 is a functional block diagram of a preferred embodiment of the keyframe extraction device based on a prototype network of the present invention. The module includes a data processing module 10, a model building module 20, a feature generation module 30, a text prototype module 40, a hybrid prototype module 50, and a keyframe module 60. The functional modules are described in detail below: Data processing module 10 is used to acquire raw input data and extract frames from the raw input data to obtain an RGB frame sequence; Model building module 20 is used to build a visual language embedding generator from a visual encoder and a text encoder; Feature generation module 30 is used to input the RGB frame sequence and text input data into the visual language embedding generator for parsing, and output visual feature vectors and text feature vectors respectively; Text prototype module 40 is used to build a text prototype based on the text feature vectors, generate a target visual frame by combining the visual feature vectors of the RGB frame sequence, and train the visual encoder of the visual language embedding generator using the target visual frame; Hybrid prototype module 50 is used to calculate a visual prototype based on the RGB frame sequence using the trained visual encoder of the visual language embedding generator, fuse the visual prototype with the text prototype to construct a hybrid prototype, and calculate an adaptive dynamic threshold based on the hybrid prototype; Keyframe module 60 is used to calculate the similarity between the RGB frame sequence and the hybrid prototype using the trained visual encoder of the visual encoder of the visual language embedding generator, and filter video keyframes based on the similarity and the adaptive dynamic threshold.

[0064] In one embodiment, the data processing module 10 includes: acquiring video input data and text input data, and preprocessing the video input data and text input data; extracting frames from the preprocessed video input data using a video decoder to obtain frame sampling data; and combining the frame sampling data in chronological order to obtain an RGB frame sequence.

[0065] In one embodiment, the feature generation module 30 includes: parsing the RGB frame sequence to obtain single-frame visual data; inputting the single-frame visual data into the visual encoder of the visual language embedding generator; and processing the single-frame visual data through the temporal attention enhancement mechanism of the visual encoder to obtain a visual feature vector.

[0066] In one embodiment, the feature generation module 30 includes: parsing the text input data to obtain text description data; inputting the text description data into the text encoder of the visual language embedding generator; quantizing the text description data through the text encoder; and outputting a text feature vector.

[0067] In one embodiment, the text prototype module 40 includes: constructing several text prototypes based on the text feature vectors, and performing stability maintenance processing on the text prototypes through a momentum update mechanism; obtaining unlabeled visual frames and their corresponding visual feature vectors from the RGB frame sequence; calculating the similarity between the visual feature vectors of the unlabeled visual frames and the processed text prototypes, and obtaining a pseudo-label probability distribution based on the similarity; comparing the pseudo-label probability distribution with a confidence threshold; when the pseudo-label probability distribution is greater than the confidence threshold, taking the unlabeled visual frame corresponding to the pseudo-label probability as the target visual frame; calculating an initial visual prototype based on the visual feature vectors of labeled keyframes; and performing matching training on the visual encoder of the visual language embedding generator based on the target visual frame and the initial visual prototype using a cross-modal alignment loss function.

[0068] In one embodiment, the hybrid prototype module 50 includes: obtaining labeled keyframes based on the RGB frame sequence; encoding the labeled keyframes using a visual encoder of a trained visual language embedding generator to generate labeled visual feature vectors; calculating a visual prototype based on the labeled visual feature vectors; linearly fusing the visual prototype with a text prototype to obtain a hybrid prototype; and processing the splicing features of the hybrid prototype using a two-layer perceptron to obtain an adaptive dynamic threshold.

[0069] In one embodiment, the keyframe module 60 includes: obtaining a query frame based on the RGB frame sequence, and encoding the query frame using a visual encoder of a trained visual language embedding generator to obtain a query visual feature vector; calculating the similarity between the query visual feature vector and the hybrid prototype; comparing the similarity with an adaptive dynamic threshold, and filtering key candidate frames based on the comparison result; and performing redundant frame removal processing on the key candidate frames using a temporal nonmaximum suppression algorithm to obtain video keyframes.

[0070] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram is shown in Figure 4. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external user terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a keyframe extraction method based on a prototype network on the server side.

[0071] In one embodiment, a computer device is provided, which can be a user terminal, and its internal structure diagram is shown in Figure 5. The computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used for communication with an external server via a network connection. When executed by a processor, the computer program implements the user-side functions or steps of a keyframe extraction method based on a prototype network. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: acquiring raw input data and extracting frames from the raw input data to obtain an RGB frame sequence; constructing a visual language embedding generator using a visual encoder and a text encoder; inputting the RGB frame sequence and text input data into the visual language embedding generator for parsing, and outputting visual feature vectors and text feature vectors respectively; constructing a text prototype based on the text feature vectors, generating a target visual frame by combining the visual feature vectors of the RGB frame sequence, and training the visual encoder of the visual language embedding generator using the target visual frame; calculating a visual prototype based on the RGB frame sequence using the trained visual encoder of the visual encoder of the visual language embedding generator, fusing the visual prototype with the text prototype to construct a hybrid prototype, and calculating an adaptive dynamic threshold based on the hybrid prototype; calculating the similarity between the RGB frame sequence and the hybrid prototype using the trained visual encoder of the visual encoder of the visual language embedding generator, and filtering video keyframes based on the similarity and the adaptive dynamic threshold.

[0072] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program performs the following steps: acquiring raw input data and extracting frames from the raw input data to obtain an RGB frame sequence; constructing a visual language embedding generator using a visual encoder and a text encoder; inputting the RGB frame sequence and text input data into the visual language embedding generator for parsing, and outputting visual feature vectors and text feature vectors respectively; constructing a text prototype based on the text feature vectors, generating a target visual frame by combining the visual feature vectors of the RGB frame sequence, and training the visual encoder of the visual language embedding generator using the target visual frame; calculating a visual prototype based on the RGB frame sequence using the trained visual encoder of the visual encoder of the visual language embedding generator, fusing the visual prototype with the text prototype to construct a hybrid prototype, and calculating an adaptive dynamic threshold based on the hybrid prototype; calculating the similarity between the RGB frame sequence and the hybrid prototype using the trained visual encoder of the visual encoder of the visual language embedding generator, and filtering video keyframes based on the similarity and the adaptive dynamic threshold.

[0073] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0074] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0075] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0076] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A keyframe extraction method based on a prototype network, characterized in that, Includes the following steps: The process involves: acquiring raw input data and extracting frames from it to obtain an RGB frame sequence; constructing a visual language embedding generator using a visual encoder and a text encoder; parsing the RGB frame sequence and text input data into the visual language embedding generator to output visual feature vectors and text feature vectors, respectively; constructing a text prototype based on the text feature vectors; generating a target visual frame by combining the visual feature vectors from the RGB frame sequence; training the visual encoder of the visual language embedding generator using the target visual frame; calculating the visual prototype based on the RGB frame sequence using the trained visual encoder of the visual language embedding generator; fusing the visual prototype with the text prototype to construct a hybrid prototype; calculating an adaptive dynamic threshold based on the hybrid prototype; and calculating the similarity between the RGB frame sequence and the hybrid prototype using the trained visual encoder of the visual encoder of the visual language embedding generator. Finally, selecting video keyframes based on the similarity and the adaptive dynamic threshold.

2. The keyframe extraction method based on prototype networks as described in claim 1, characterized in that, The step of acquiring raw input data and extracting frames from the raw input data to obtain an RGB frame sequence includes: acquiring video input data and text input data, and preprocessing the video input data and text input data; extracting frames from the preprocessed video input data using a video decoder to obtain frame sampling data; and combining the frame sampling data in chronological order to obtain an RGB frame sequence.

3. The keyframe extraction method based on prototype networks as described in claim 1, characterized in that, The step of inputting the RGB frame sequence into the visual language embedding generator for parsing and outputting a visual feature vector includes: parsing the RGB frame sequence to obtain single-frame visual data; inputting the single-frame visual data into the visual encoder of the visual language embedding generator; and processing the single-frame visual data through the temporal attention enhancement mechanism of the visual encoder to obtain a visual feature vector.

4. The keyframe extraction method based on prototype networks as described in claim 1, characterized in that, The step of inputting the text input data into the visual language embedding generator for parsing and outputting a text feature vector includes: parsing the text input data to obtain text description data; inputting the text description data into the text encoder of the visual language embedding generator; quantizing the text description data through the text encoder; and outputting a text feature vector.

5. The keyframe extraction method based on prototype networks as described in claim 1, characterized in that, The process of constructing text prototypes based on the text feature vectors, generating target visual frames by combining visual feature vectors from the RGB frame sequence, and training the visual encoder of the visual language embedding generator using the target visual frames includes: constructing several text prototypes based on the text feature vectors and maintaining the stability of the text prototypes through a momentum update mechanism; obtaining unlabeled visual frames and their corresponding visual feature vectors from the RGB frame sequence; calculating the similarity between the visual feature vectors of the unlabeled visual frames and the processed text prototypes, and obtaining a pseudo-label probability distribution based on the similarity; comparing the pseudo-label probability distribution with a confidence threshold; when the pseudo-label probability distribution is greater than the confidence threshold, using the unlabeled visual frame corresponding to the pseudo-label probability as the target visual frame; calculating an initial visual prototype based on the visual feature vectors of labeled keyframes; and training the visual encoder of the visual language embedding generator by matching the target visual frames and the initial visual prototypes using a cross-modal alignment loss function.

6. The keyframe extraction method based on prototype networks as described in claim 1, characterized in that, The visual encoder, trained by the visual language embedding generator, calculates a visual prototype based on an RGB frame sequence, fuses the visual prototype with a text prototype to construct a hybrid prototype, and calculates an adaptive dynamic threshold based on the hybrid prototype. This process includes: obtaining labeled keyframes based on the RGB frame sequence; encoding the labeled keyframes using the trained visual language embedding generator's visual encoder to generate labeled visual feature vectors; calculating a visual prototype based on the labeled visual feature vectors; linearly fusing the visual prototype with the text prototype to obtain a hybrid prototype; and processing the spliced ​​features of the hybrid prototype using a two-layer perceptron to obtain an adaptive dynamic threshold.

7. The keyframe extraction method based on prototype networks as described in claim 1, characterized in that, The step of using a visual encoder trained by a visual language embedding generator to calculate the similarity between the RGB frame sequence and the hybrid prototype, and filtering video keyframes based on the similarity and an adaptive dynamic threshold, includes: obtaining a query frame from the RGB frame sequence, encoding the query frame using the visual encoder trained by the visual language embedding generator to obtain a query visual feature vector; calculating the similarity between the query visual feature vector and the hybrid prototype; comparing the similarity with the adaptive dynamic threshold, and filtering key candidate frames based on the comparison result; and using a temporal nonmaximum suppression algorithm to remove redundant frames from the key candidate frames to obtain video keyframes.

8. A keyframe extraction device based on a prototype network, characterized in that, The keyframe extraction device based on the prototype network includes: a data processing module for acquiring raw input data and extracting frames from the raw input data to obtain an RGB frame sequence; a model building module for constructing a visual language embedding generator using a visual encoder and a text encoder; a feature generation module for inputting the RGB frame sequence and text input data into the visual language embedding generator for parsing, and outputting visual feature vectors and text feature vectors respectively; a text prototype module for constructing a text prototype based on the text feature vectors, generating a target visual frame by combining the visual feature vectors of the RGB frame sequence, and training the visual encoder of the visual language embedding generator using the target visual frame; a hybrid prototype module for calculating a visual prototype based on the RGB frame sequence using the trained visual encoder of the visual language embedding generator, fusing the visual prototype with the text prototype to construct a hybrid prototype, and calculating an adaptive dynamic threshold based on the hybrid prototype; and a keyframe module for calculating the similarity between the RGB frame sequence and the hybrid prototype using the trained visual encoder of the visual encoder of the visual language embedding generator, and filtering video keyframes based on the similarity and the adaptive dynamic threshold.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a prototype network-based keyframe extraction program stored in the memory and executable on the processor. When executed by the processor, the prototype network-based keyframe extraction program implements the steps of the prototype network-based keyframe extraction method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a keyframe extraction program based on a prototype network, which, when executed by a processor, implements the steps of the keyframe extraction method based on a prototype network as described in any one of claims 1-7.

Images