Key frame extraction method and device, equipment and medium
By standardizing multimodal data and extracting features using Bayesian neural networks, quantifying and dynamically fusing uncertainties, and selecting keyframes, the problem of insufficient accuracy in multimodal data is solved, and high-accuracy keyframe extraction in dynamic environments is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2026-01-13
- Publication Date
- 2026-05-01
AI Technical Summary
In the fields of healthcare and fintech, the dynamic volatility and uncertainty of multimodal data in existing technologies lead to insufficient accuracy in keyframe extraction. This makes it difficult to adaptively enhance the contribution of audio or text modalities when visual failure occurs, resulting in missed detections or false alarms of key pathological events or fraudulent activities.
By standardizing multimodal data, Bayesian features of visual, audio and text modalities are extracted using a Bayesian neural network, and uncertainty is quantified. An inverse variance weighting strategy is used to dynamically fuse features, and key frames are selected by combining spatiotemporal saliency. A refined temporal saliency score is then performed to select the optimal key frame.
It achieves adaptive perception and fusion of multimodal reliability fluctuations, significantly improving the accuracy and robustness of keyframe extraction, and can accurately capture key information in dynamic noise environments.
Smart Images
Figure CN121963039A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of machine learning technology, and in particular to a keyframe extraction method, apparatus, device, and medium. Background Technology
[0002] In the healthcare field, automated analysis and key information extraction from multimodal medical data streams (such as surgical videos, vital sign monitoring audio, and electronic medical record text) has become an important research direction. Traditional methods typically use predefined rules or static weighting to fuse multimodal information. For example, in endoscopic surgical video analysis, key steps are detected solely by visual frame difference methods, or event localization is simply achieved by combining audio alarms and text reports. However, medical scenarios are highly dynamic and uncertain, and existing technologies lack a quantification mechanism for the dynamic fluctuations in multimodal reliability. This makes it difficult to adaptively enhance the contribution of audio or text modalities when visual failures occur, leading to missed detections or false alarms of key pathological events (such as instrument operation errors or abnormal physiological parameters).
[0003] In the fintech sector, the integration of multimodal data from audiovisual and text sources to identify suspicious transactions, fraudulent activities, or compliance risks has become an industry necessity. Existing systems typically process video surveillance streams, customer service call recordings, and transaction log text independently, or employ simple decision-level fusion (such as majority voting) for event alerts. For example, in anti-fraud scenarios, systems often rely on facial recognition technology to analyze surveillance videos, combine this with voice emotion recognition to process call recordings, and then match the transaction text with rule patterns. However, significant modal uncertainty and heterogeneity exist in real-world financial scenarios: changes in lighting in the lobby or facial occlusion can reduce the confidence level of visual recognition; voice emotion analysis in noisy backgrounds is easily interfered with; and unstructured text (such as customer service records) contains ambiguities. Traditional static fusion methods cannot quantify these uncertainties, and it is even more difficult to adjust the fusion strategy based on the real-time reliability of each modality during the dynamic process of fraudulent activities. Summary of the Invention
[0004] This invention provides a keyframe extraction method, apparatus, computer equipment, and medium to solve the problem of low accuracy in existing keyframe extraction solutions on the market.
[0005] Firstly, a keyframe extraction method is provided, including: The pre-acquired multimodal data stream containing video, audio, and text is subjected to data standardization processing to generate standardized multimodal data; Bayesian features of different modalities are extracted from the standardized multimodal data to obtain video modal Bayesian features, audio modal Bayesian features, and text modal Bayesian features; The video modal Bayesian features, the audio modal Bayesian features, and the text modal Bayesian features are dynamically fused to generate a multimodal feature sequence; Key frames in the multimodal feature sequence are selected based on spatiotemporal saliency to obtain an initial key frame sequence; Temporal saliency scoring is performed on each keyframe in the initial keyframe sequence to obtain saliency scoring results; Based on the saliency score results, a preset number of keyframes are selected from the initial keyframe sequence in descending order of score to form the final keyframe sequence.
[0006] Secondly, a keyframe extraction device is provided, comprising: The data processing module is used to perform data standardization processing on the pre-acquired multimodal data stream containing video, audio and text to generate standardized multimodal data; The feature extraction module is used to extract Bayesian features of different modalities in the standardized multimodal data to obtain video modal Bayesian features, audio modal Bayesian features and text modal Bayesian features; The feature fusion module is used to dynamically fuse the video modal Bayesian features, the audio modal Bayesian features, and the text modal Bayesian features to generate a multimodal feature sequence; The keyframe scoring module is used to select keyframes in the multimodal feature sequence based on spatiotemporal saliency to obtain an initial keyframe sequence, and to perform temporal saliency scoring on each keyframe in the initial keyframe sequence to obtain a saliency scoring result. The keyframe selection module is used to select a preset number of keyframes from the initial keyframe sequence according to the saliency score results in descending order of score, thereby forming the final keyframe sequence. Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described keyframe extraction method.
[0007] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described keyframe extraction method.
[0008] The aforementioned keyframe extraction method, apparatus, computer equipment, and storage medium achieve the following: Structured multimodal input is formed by standardizing the original video, audio, and text data. Bayesian features for the visual, audio, and text modalities are extracted using three independent Bayesian neural networks. These networks not only output the mean vector of features representing semantic content but also generate corresponding variance vectors, thus quantifying the uncertainty of each modality's features. Next, the system dynamically calculates fusion weights based on the uncertainty of each modality, employing an inverse variance-based weighting strategy to assign higher weights to modalities with lower uncertainty, and generates a unified multimodal fusion feature sequence through weighted summation. Based on this, a comprehensive score is obtained by calculating the feature differences between adjacent frames, local contrast, and global saliency, and candidate keyframes are selected in segments to form an initial sequence. Subsequently, the initial keyframes undergo refined temporal saliency re-scoring, and the scores are sorted from high to low, selecting a preset number of optimal keyframes as the final output. This invention achieves adaptive perception and fusion of multimodal reliability fluctuations, significantly improving the accuracy of keyframe extraction. Attached Figure Description To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0009] Figure 1 This is a schematic diagram of an application environment for a keyframe extraction method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a keyframe extraction method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a keyframe extraction device in one embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation
[0010] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0011] The keyframe extraction method provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can standardize the raw video, audio, and text data from the client to form a structured multimodal input. Three independent Bayesian neural networks are then used to extract Bayesian features for the visual, audio, and text modalities, respectively. These networks not only output the mean vector of features representing semantic content but also generate corresponding variance vectors, thus quantifying the uncertainty of each modality's features. Next, the system dynamically calculates the fusion weights based on the uncertainty of each modality, employing an inverse variance-based weighting strategy to assign higher weights to modalities with lower uncertainty. A unified multimodal fusion feature sequence is then generated through weighted summation. Based on this, a comprehensive score is obtained by calculating the feature differences between adjacent frames, local contrast, and global saliency. Candidate keyframes are then selected in segments to form the initial sequence. Subsequently, the initial keyframes undergo refined temporal saliency re-scoring, and the scores are sorted from high to low. A predetermined number of optimal keyframes are selected as the final output. This achieves adaptive perception and fusion of multimodal reliability fluctuations, significantly improving the accuracy of keyframe extraction in dynamic noise environments. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.
[0012] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the keyframe extraction method provided in this embodiment of the invention includes the following steps: S1. Perform data standardization processing on the pre-acquired multimodal data stream containing video, audio and text to generate standardized multimodal data.
[0013] In the healthcare field, multimodal data streams typically manifest as real-time or time-series data sets that integrate various physiological and clinical information. For example, visual modalities may include endoscopic surgery videos, medical image sequences (such as ultrasound and dynamic CT scans), and patient monitoring videos (such as rehabilitation motion capture); audio modalities may encompass heart and lung sound auscultation records, doctor-patient consultation audio, and medical device alarm sounds; and textual modalities involve electronic medical record texts, real-time physiological parameter logs, diagnostic reports, and scientific research literature data. These multimodal data streams collectively characterize a patient's comprehensive health status, and joint analysis can support intelligent medical applications such as surgical navigation, early disease warning, and remote diagnosis and treatment. Bayesian uncertainty modeling is particularly important, as it can quantify uncertainties in real-world medical scenarios, such as medical image noise, sensor errors, and textual ambiguity.
[0014] In the fintech field, multimodal data streams primarily refer to dynamic data that integrates multi-source transaction and behavioral information. Visual modalities may include ATM or counter monitoring videos, remote face-to-face identity authentication videos, and investor emotional expression analysis videos; audio modalities commonly include customer service call recordings, trading floor ambient sounds, and conference call audio; text modalities encompass real-time news streams, financial report texts, social media sentiment, transaction order logs, contracts, and compliance documents. These multimodal data streams can be used for tasks such as fraud detection, high-risk transaction monitoring, customer service quality and efficiency analysis, and market sentiment prediction. Issues such as dynamically changing market environments, voice emotion spoofing, and text information delays and ambiguities can all be robustly modeled through uncertainty-aware multimodal fusion, improving the real-time performance and reliability of financial decisions.
[0015] In this embodiment of the invention, the step of performing data standardization processing on the pre-acquired multimodal data stream containing video, audio, and text to generate standardized multimodal data includes: The video in the multimodal data stream is sampled at preset time intervals to obtain a video frame sequence; The video frame sequence is subjected to RGB normalization to form a normalized video frame sequence; The audio in the multimodal data stream is subjected to frame-by-frame windowing processing to form frame-by-frame windowed audio data; Extract the Mel spectrogram of the framed and windowed audio data; The text in the multimodal data stream is segmented to obtain a segmented word sequence; The segmented word sequence is mapped to a vector to obtain a word vector sequence; The normalized video frame sequence, the Mel spectrogram, and the word vector sequence are combined to generate standardized multimodal data.
[0016] In detail, the process of sampling video frames in the multimodal data stream at preset time intervals to obtain a video frame sequence aims to extract representative image frames from the continuous video stream, thereby reducing data redundancy and controlling the computational scale.
[0017] In detail, the RGB normalization processing of the video frame sequence to form a normalized video frame sequence involves scaling the pixel values from the original range (e.g., 0–255) to [0,1] or standardizing them (mean = 0, variance = 1). Normalized frame data better meets the requirements of neural network processing, which is beneficial for improving the stability and generalization ability of model training.
[0018] In detail, the step of performing frame-segmentation and windowing processing on the audio in the multimodal data stream to form frame-segmented and windowed audio data involves dividing the audio stream into short time segments (e.g., 25ms per frame) and applying windowing functions such as Hamming windows to each frame to reduce spectral leakage. The frame-segmented and windowed audio data retains short-time stationary characteristics, laying the foundation for subsequent frequency domain feature extraction.
[0019] In detail, the extraction of the Mel spectrogram of the framed and windowed audio data is achieved by performing a short-time Fourier transform on the framed and windowed audio data to obtain the spectrum, then filtering it through a Mel filter bank and taking the logarithm to generate the Mel spectrogram.
[0020] In detail, the step of performing word segmentation on the text in the multimodal data stream to obtain a segmented word sequence involves using word segmentation tools (such as rule-based or statistical models) to divide the original text into word or sub-word sequences, eliminate format noise, and extract basic semantic units to prepare for subsequent vectorized representation.
[0021] In detail, the step of mapping the segmented word sequence into vectors to obtain a word vector sequence is to use a pre-trained word embedding model to map the segmented word sequence into dense vectors of fixed dimensions.
[0022] S2. Extract Bayesian features from different modalities in the standardized multimodal data to obtain video modal Bayesian features, audio modal Bayesian features, and text modal Bayesian features.
[0023] In this embodiment of the invention, the step of extracting Bayesian features from different modalities in the standardized multimodal data to obtain video modal Bayesian features, audio modal Bayesian features, and text modal Bayesian features includes: Extract the normalized video frame sequence, Mel spectrogram, and word vector sequence from the standardized multimodal data; Visual Bayesian features are obtained by performing forward propagation calculations on the normalized video frame sequence using a preset visual Bayesian neural network. The audio Bayesian features are obtained by forward propagation calculation of the Mel spectrogram using a preset audio Bayesian neural network. The word vector sequence is forward-propagated using a pre-defined text Bayesian neural network to obtain text Bayesian features; Uncertainty calculations are performed on the visual Bayesian features, the audio Bayesian features, and the text Bayesian features to obtain uncertainty score data; The visual Bayesian features, the audio modality Bayesian features, and the text Bayesian features are aligned by feature dimensions to form aligned visual Bayesian features, aligned audio Bayesian features, and aligned text Bayesian features. Based on the uncertainty scoring data, feature calibration is performed on the aligned visual Bayesian features, the aligned audio Bayesian features, and the aligned text Bayesian features to obtain video modality Bayesian features, audio modality Bayesian features, and text modality Bayesian features.
[0024] In detail, the step of using a preset visual Bayesian neural network to perform forward propagation calculation on the normalized video frame sequence to obtain visual Bayesian features employs Bayesian modeling techniques such as rank-1 parameterization. While extracting visual semantic features, it outputs the probability distribution parameters (mean and variance) of each feature dimension, forming visual Bayesian features with uncertainty measurement, providing a reliability assessment basis for subsequent fusion.
[0025] In detail, the rank-1 parameterization is a technique that parameterizes the weight matrix by restricting it to a rank-1 subspace. It is mainly used in Bayesian neural networks (BNNs) to improve parameter efficiency and scalability.
[0026] In detail, the step of using a pre-defined audio Bayesian neural network to perform forward propagation calculations on the Mel spectrogram to obtain audio Bayesian features involves using the audio Bayesian neural network to extract a probabilistic representation of time-frequency features from the audio signal through a Bayesian inference mechanism, outputting a feature mean vector and a variance vector. This process not only captures the semantic information of the audio content but also quantifies the uncertainty level of the audio mode in noisy environments or weak signal regions.
[0027] In detail, the step of using a pre-defined text Bayesian neural network to perform forward propagation calculations on the word vector sequence to obtain text Bayesian features involves inputting the word vector sequence into the text Bayesian neural network and obtaining the Bayesian representation of the text features through forward propagation. This network considers text-specific noise sources such as lexical ambiguity and recognition errors, generating a probability distribution for each semantic feature. This ensures that the text features not only contain semantic content but also include a quantitative estimate of the recognition confidence level.
[0028] In detail, the uncertainty calculation is performed on the visual Bayesian features, the audio Bayesian features, and the text Bayesian features to obtain uncertainty score data. Based on the variance information in the Bayesian features of each modality, the uncertainty score of each modality at each time step is calculated. This score is usually obtained by aggregating the feature variance vectors (such as using the L1 norm), quantifying the reliability of the modality at the current time step, and providing a numerical basis for subsequent feature calibration and fusion weight allocation.
[0029] In detail, the step of performing feature calibration on the aligned visual Bayesian features, aligned audio Bayesian features, and aligned text Bayesian features based on the uncertainty score data to obtain video modality Bayesian features, audio modality Bayesian features, and text modality Bayesian features involves smoothing or confidence weighting for modality features with high uncertainty, while retaining more original information for modality features with low uncertainty.
[0030] S3. Dynamically fuse the video modal Bayesian features, the audio modal Bayesian features, and the text modal Bayesian features to generate a multimodal feature sequence.
[0031] In this embodiment of the invention, the dynamic fusion of the video modality Bayesian features, the audio modality Bayesian features, and the text modality Bayesian features to generate a multimodal feature sequence includes: Uncertainty scalars are extracted from the video modality Bayesian features, the audio modality Bayesian features, and the text modality Bayesian features to form a set of uncertainty scalar values; The dynamic weights of the video modal Bayesian features, the audio modal Bayesian features, and the text modal Bayesian features are calculated based on the set of uncertainty scalar values. The video modal Bayesian features, the audio modal Bayesian features, and the text modal Bayesian features are weighted and fused according to the dynamic weights to generate a multimodal fusion vector; Uncertainty propagation calculation is performed on the multimodal fusion vector to obtain the uncertainty value; Based on the uncertainty value, feature normalization and time-series smoothing are performed on the multimodal fusion vector to generate a multimodal feature sequence.
[0032] In detail, the step of extracting the uncertainty scalars of the video modality Bayesian features, the audio modality Bayesian features, and the text modality Bayesian features to form a set of uncertainty scalar values is achieved by aggregating and calculating the feature variance vectors of the video, audio, and text modalities (such as taking the L1 norm or average value), thereby compressing the multidimensional uncertainty information into a single scalar.
[0033] In detail, the uncertainty propagation calculation is performed on the multimodal fusion vector to obtain the uncertainty value, and the overall uncertainty of the fusion feature is approximately estimated by weighted combination of the squares of each modality weight and the original variance.
[0034] In detail, the step of performing feature normalization and temporal smoothing on the multimodal fusion vector based on the uncertainty value to generate a multimodal feature sequence involves normalizing and adjusting the feature vector based on the uncertainty value to suppress the excessive influence of high uncertainty features; subsequently, temporal smoothing techniques (such as moving average) are applied to reduce feature jitter between adjacent frames and enhance temporal continuity.
[0035] In this embodiment of the invention, calculating the dynamic weights of the video modality Bayesian features, the audio modality Bayesian features, and the text modality Bayesian features based on the set of uncertainty scalar values includes: The uncertain scalar values in the set of uncertain scalar values are smoothed and small values are added to obtain a preprocessed set of uncertain scalar values; The penalty score for the set of preprocessed uncertainty scalar values is calculated based on a preset temperature coefficient. The penalty score is normalized using the Softmax function to generate dynamic weights for the video modality Bayesian features, the audio modality Bayesian features, and the text modality Bayesian features.
[0036] In detail, the smoothing and addition of a small value to the uncertain scalar values in the set of uncertain scalar values to obtain a preprocessed set of uncertain scalar values is achieved by performing a moving average of the uncertain values at adjacent time steps. This effectively reduces the impact of noise and abnormal fluctuations, enhancing the stability of weight allocation. Subsequently, a small positive value is added to each scalar value. This operation prevents numerical anomalies such as division by zero or logarithmically zero in subsequent calculations while maintaining the relative magnitude of the original uncertainties, providing a numerically stable input basis for weight calculation.
[0037] In detail, the penalty score for calculating the set of preprocessed uncertainty scalar values based on the preset temperature coefficient is to multiply the temperature coefficient by the uncertainty value and take the negative value, thus mapping the uncertainty to a negative score, so that the higher the uncertainty, the lower the score of the mode.
[0038] S4. Select key frames from the multimodal feature sequence based on spatiotemporal saliency to obtain an initial key frame sequence.
[0039] In this embodiment of the invention, the step of selecting keyframes from the multimodal feature sequence based on spatiotemporal saliency to obtain an initial keyframe sequence includes: Calculate the feature differences between adjacent features in the multimodal feature sequence to generate a difference score sequence; The multimodal feature sequence is sampled using preset window parameters to obtain a sampled feature sequence; Calculate the contrast of adjacent features in the sampled feature sequence to obtain a contrast sequence; Calculate the global significance of each feature in the sampled feature sequence; The comprehensive significance score of each feature in the multimodal feature sequence is calculated based on the difference score sequence, the contrast sequence, and the global significance. The multimodal feature sequence is segmented, and key frames of each segment are selected based on the comprehensive saliency score to obtain an initial key frame sequence.
[0040] In detail, the calculation of the feature differences between adjacent features in the multimodal feature sequence and the generation of the difference score sequence use Euclidean distance or cosine distance to measure the similarity of features between adjacent frames. The greater the difference, the more significant the scene change.
[0041] In detail, calculating the contrast of adjacent features in the sampled feature sequence to obtain a contrast sequence involves quantifying the richness and frequency of change of content within a local time period by measuring the difference distribution of feature vectors within the window. High-contrast windows typically contain more visual changes or semantic transitions, and the corresponding frames have higher local saliency.
[0042] In detail, calculating the global significance of each feature in the sampled feature sequence involves calculating the global significance of each feature within a sampling window relative to the entire video sequence. By comparing the differences between the features within the window and the global feature distribution (such as the mean of features across all frames), the uniqueness and representativeness of the window content within the entire video are evaluated.
[0043] In detail, the comprehensive significance score of each feature in the multimodal feature sequence is calculated based on the difference score sequence, the contrast sequence, and the global significance, and the three indicators of difference score sequence, local contrast sequence, and global significance are weighted and fused.
[0044] S5. Perform temporal saliency scoring on each keyframe in the initial keyframe sequence to obtain the saliency score result.
[0045] In this embodiment of the invention, the step of performing temporal saliency scoring on each keyframe in the initial keyframe sequence to obtain a saliency score result includes: Extract the feature vector of the timestamp corresponding to each keyframe in the initial keyframe sequence to obtain the feature vector sequence; Extract the uncertainty vector of the timestamp corresponding to each key frame in the initial key frame sequence to generate a key frame uncertainty set; Calculate the feature differences between adjacent keyframes in the initial keyframe sequence based on the feature vector sequence to obtain a feature difference sequence; Based on the uncertainty set of the keyframes, calculate the uncertainty penalty term of the initial keyframe sequence to generate a penalty term set; The feature difference sequence is normalized to obtain a normalized sequence; The temporal saliency score of each keyframe is calculated by multiplying the normalized sequence by the set of penalty terms in sequence, and the saliency score result is obtained.
[0046] In detail, the step of extracting the feature vector corresponding to the timestamp of each key frame in the initial key frame sequence to obtain the feature vector sequence is to extract the fused feature vector at the corresponding time based on the timestamp of each key frame in the initial key frame sequence.
[0047] In detail, the step of extracting the uncertainty vector corresponding to the timestamp of each keyframe in the initial keyframe sequence to generate a keyframe uncertainty set involves retrieving the uncertainty vector from the historical output of the Bayesian feature extraction module for each keyframe's corresponding timestamp. This set contains estimates of the feature variance or standard deviation of each modality (visual, audio, and text) at the corresponding time, quantifying the cognitive uncertainty and data noise present in the feature extraction process, and providing a probabilistic basis for evaluating the reliability of keyframes.
[0048] In detail, the step of calculating the feature differences between adjacent keyframes in the initial keyframe sequence based on the feature vector sequence to obtain the feature difference sequence is achieved by calculating the Euclidean distance or cosine distance between the corresponding feature vectors of adjacent keyframes in the initial keyframe sequence.
[0049] In detail, the step of calculating the uncertainty penalty term for the initial keyframe sequence based on the keyframe uncertainty set and generating a penalty term set is achieved by weighted averaging, transforming the multi-dimensional uncertainty vector into a scalar penalty term. This penalty term is negatively correlated with the uncertainty level; that is, the higher the uncertainty, the smaller the penalty term value.
[0050] S6. Based on the saliency score results, select a preset number of keyframes from the initial keyframe sequence in descending order of score to form the final keyframe sequence.
[0051] In this embodiment of the invention, the preset number of keyframes can be 5 frames, 10 frames, or 20 frames.
[0052] In the healthcare field, this solution can be applied to surgical video analysis and remote medical monitoring scenarios. By processing endoscopic surgical videos, operating room voice commands and equipment alarms in real time, as well as real-time summaries of electronic medical records, the system can dynamically quantify the reliability differences of multimodal data: for example, automatically reducing visual weights when bleeding causes visual blurring, and suppressing audio modal interference in noisy environments. Keyframe extraction based on Bayesian uncertainty can automatically locate key surgical steps and postoperative abnormal signs, and generate condensed visual reports to assist doctors in surgical debriefing, medical training, and remote consultations, while providing reliable labeled training data for medical AI models.
[0053] In the fintech field, this solution can serve intelligent risk control and compliance audit scenarios. By integrating counter monitoring video, customer service call recordings, and real-time transaction transaction text, the system can dynamically assess the credibility of multimodal information: for example, reducing reliance on facial recognition in low light conditions and enhancing text transaction record analysis in noisy backgrounds. Keyframe extraction based on uncertainty awareness can automatically capture time points of abnormal behavior, generate high-risk event snapshot sequences, assist anti-fraud models in locating key evidence, improve the efficiency of suspicious transaction reporting, and provide interpretable audiovisual text correlation analysis capabilities for regulatory technology.
[0054] As can be seen, in the above scheme, the original video, audio, and text data are standardized to form a structured multimodal input. Three independent Bayesian neural networks are then used to extract Bayesian features for the visual, audio, and text modalities, respectively. These networks not only output the mean vector of features representing semantic content but also generate corresponding variance vectors, thus quantifying the uncertainty of each modality's features. Next, the system dynamically calculates the fusion weights based on the uncertainty of each modality, employing an inverse variance-based weighting strategy to assign higher weights to modalities with lower uncertainty, and generates a unified multimodal fusion feature sequence through weighted summation. Based on this, a comprehensive score is obtained by calculating the feature differences between adjacent frames, local contrast, and global saliency, and candidate keyframes are selected in segments to form the initial sequence. Subsequently, the initial keyframes undergo refined temporal saliency re-scoring, and are sorted from high to low scores, selecting a predetermined number of optimal keyframes as the final output. This achieves adaptive perception and fusion of multimodal reliability fluctuations, significantly improving the robustness and accuracy of keyframe extraction in dynamic noise environments.
[0055] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0056] In one embodiment, a keyframe extraction device is provided, which corresponds one-to-one with the keyframe extraction methods described in the above embodiments. For example... Figure 3 As shown, the keyframe extraction device includes a data processing module 101, a feature extraction module 102, a feature fusion module 103, a keyframe scoring module 104, and a keyframe selection module 105. Detailed descriptions of each functional module are as follows: Data processing module 101 is used to perform data standardization processing on the pre-acquired multimodal data stream containing video, audio and text to generate standardized multimodal data; Feature extraction module 102 is used to extract Bayesian features of different modal data in the standardized multimodal data to obtain video modal Bayesian features, audio modal Bayesian features and text modal Bayesian features; The feature fusion module 103 is used to dynamically fuse the video modal Bayesian features, the audio modal Bayesian features, and the text modal Bayesian features to generate a multimodal feature sequence; The keyframe scoring module 104 is used to select keyframes in the multimodal feature sequence based on spatiotemporal saliency to obtain an initial keyframe sequence, and to perform temporal saliency scoring on each keyframe in the initial keyframe sequence to obtain a saliency scoring result. The keyframe selection module 105 is used to select a preset number of keyframes from the initial keyframe sequence according to the saliency score results in descending order of score, so as to form the final keyframe sequence.
[0057] In one embodiment, when performing the data standardization processing on the pre-acquired multimodal data stream containing video, audio, and text to generate standardized multimodal data, the data processing module 101 is specifically used for: The video in the multimodal data stream is sampled at preset time intervals to obtain a video frame sequence; The video frame sequence is subjected to RGB normalization to form a normalized video frame sequence; The audio in the multimodal data stream is subjected to frame-by-frame windowing processing to form frame-by-frame windowed audio data; Extract the Mel spectrogram of the framed and windowed audio data; The text in the multimodal data stream is segmented to obtain a segmented word sequence; The segmented word sequence is mapped to a vector to obtain a word vector sequence; The normalized video frame sequence, the Mel spectrogram, and the word vector sequence are combined to generate standardized multimodal data.
[0058] In one embodiment, the feature extraction module 102, when performing the extraction of Bayesian features from different modalities in the standardized multimodal data to obtain video modal Bayesian features, audio modal Bayesian features, and text modal Bayesian features, is specifically used for: Extract the normalized video frame sequence, Mel spectrogram, and word vector sequence from the standardized multimodal data; Visual Bayesian features are obtained by performing forward propagation calculations on the normalized video frame sequence using a preset visual Bayesian neural network. The audio Bayesian features are obtained by forward propagation calculation of the Mel spectrogram using a preset audio Bayesian neural network. The word vector sequence is forward-propagated using a pre-defined text Bayesian neural network to obtain text Bayesian features; Uncertainty calculations are performed on the visual Bayesian features, the audio Bayesian features, and the text Bayesian features to obtain uncertainty score data; The visual Bayesian features, the audio modality Bayesian features, and the text Bayesian features are aligned by feature dimensions to form aligned visual Bayesian features, aligned audio Bayesian features, and aligned text Bayesian features. Based on the uncertainty scoring data, feature calibration is performed on the aligned visual Bayesian features, the aligned audio Bayesian features, and the aligned text Bayesian features to obtain video modality Bayesian features, audio modality Bayesian features, and text modality Bayesian features.
[0059] In one embodiment, the feature fusion module 103, when performing the dynamic fusion of the video modal Bayesian features, the audio modal Bayesian features, and the text modal Bayesian features to generate a multimodal feature sequence, is specifically used for: Uncertainty scalars are extracted from the video modality Bayesian features, the audio modality Bayesian features, and the text modality Bayesian features to form a set of uncertainty scalar values; The dynamic weights of the video modal Bayesian features, the audio modal Bayesian features, and the text modal Bayesian features are calculated based on the set of uncertainty scalar values. The video modal Bayesian features, the audio modal Bayesian features, and the text modal Bayesian features are weighted and fused according to the dynamic weights to generate a multimodal fusion vector; Uncertainty propagation calculation is performed on the multimodal fusion vector to obtain the uncertainty value; Based on the uncertainty value, feature normalization and time-series smoothing are performed on the multimodal fusion vector to generate a multimodal feature sequence.
[0060] In one embodiment, the feature fusion module 103, when performing the dynamic weighting of the video modality Bayesian features, the audio modality Bayesian features, and the text modality Bayesian features based on the set of uncertainty scalar values, is specifically used for: The uncertain scalar values in the set of uncertain scalar values are smoothed and small values are added to obtain a preprocessed set of uncertain scalar values; The penalty score for the set of preprocessed uncertainty scalar values is calculated based on a preset temperature coefficient. The penalty score is normalized using the Softmax function to generate dynamic weights for the video modality Bayesian features, the audio modality Bayesian features, and the text modality Bayesian features.
[0061] In one embodiment, the keyframe scoring module 104, when performing the selection of keyframes from the multimodal feature sequence based on spatiotemporal saliency to obtain an initial keyframe sequence, is specifically used for: Calculate the feature differences between adjacent features in the multimodal feature sequence to generate a difference score sequence; The multimodal feature sequence is sampled using preset window parameters to obtain a sampled feature sequence; Calculate the contrast of adjacent features in the sampled feature sequence to obtain a contrast sequence; Calculate the global significance of each feature in the sampled feature sequence; The comprehensive significance score of each feature in the multimodal feature sequence is calculated based on the difference score sequence, the contrast sequence, and the global significance. The multimodal feature sequence is segmented, and key frames of each segment are selected based on the comprehensive saliency score to obtain an initial key frame sequence.
[0062] In one embodiment, the keyframe scoring module 104, when performing temporal saliency scoring on each keyframe in the initial keyframe sequence to obtain a saliency scoring result, is specifically used for: Extract the feature vector of the timestamp corresponding to each keyframe in the initial keyframe sequence to obtain the feature vector sequence; Extract the uncertainty vector of the timestamp corresponding to each key frame in the initial key frame sequence to generate a key frame uncertainty set; Calculate the feature differences between adjacent keyframes in the initial keyframe sequence based on the feature vector sequence to obtain a feature difference sequence; Based on the uncertainty set of the keyframes, calculate the uncertainty penalty term of the initial keyframe sequence to generate a penalty term set; The feature difference sequence is normalized to obtain a normalized sequence; The temporal saliency score of each keyframe is calculated by multiplying the normalized sequence by the set of penalty terms in sequence, and the saliency score result is obtained.
[0063] This invention provides a keyframe extraction device that performs dual status information verification on a user's bank card and ID card, ensuring accurate acquisition of the user's bank card information and ID card data. It generates front and back images of the ID card based on the ID card data, then verifies the user's input key by downloading and decrypting a pre-set key, ensuring the reliability of the user's ID card verification. Finally, it accesses services based on the bank card information and the front and back images of the ID card, improving the security of financial transactions.
[0064] Specific limitations regarding the keyframe extraction device can be found in the limitations of the keyframe extraction method described above, and will not be repeated here. Each module in the aforementioned keyframe extraction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the corresponding operations of each module.
[0065] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a keyframe extraction method on the server side.
[0066] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the client-side functions or steps of a keyframe extraction method.
[0067] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: The pre-acquired multimodal data stream containing video, audio, and text is subjected to data standardization processing to generate standardized multimodal data; Bayesian features of different modalities are extracted from the standardized multimodal data to obtain video modal Bayesian features, audio modal Bayesian features, and text modal Bayesian features; The video modal Bayesian features, the audio modal Bayesian features, and the text modal Bayesian features are dynamically fused to generate a multimodal feature sequence; Key frames in the multimodal feature sequence are selected based on spatiotemporal saliency to obtain an initial key frame sequence; Temporal saliency scoring is performed on each keyframe in the initial keyframe sequence to obtain saliency scoring results; Based on the saliency score results, a preset number of keyframes are selected from the initial keyframe sequence in descending order of score to form the final keyframe sequence.
[0068] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: The pre-acquired multimodal data stream containing video, audio, and text is subjected to data standardization processing to generate standardized multimodal data; Bayesian features of different modalities are extracted from the standardized multimodal data to obtain video modal Bayesian features, audio modal Bayesian features, and text modal Bayesian features; The video modal Bayesian features, the audio modal Bayesian features, and the text modal Bayesian features are dynamically fused to generate a multimodal feature sequence; Key frames in the multimodal feature sequence are selected based on spatiotemporal saliency to obtain an initial key frame sequence; Temporal saliency scoring is performed on each keyframe in the initial keyframe sequence to obtain saliency scoring results; Based on the saliency score results, a preset number of keyframes are selected from the initial keyframe sequence in descending order of score to form the final keyframe sequence.
[0069] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.
[0070] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0071] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0072] Finally, it should be noted that if any software tools or components not belonging to this company appear in the embodiments of the application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A keyframe extraction method, characterized in that, include: The pre-acquired multimodal data stream containing video, audio, and text is subjected to data standardization processing to generate standardized multimodal data; Bayesian features of different modalities are extracted from the standardized multimodal data to obtain video modal Bayesian features, audio modal Bayesian features, and text modal Bayesian features; The video modal Bayesian features, the audio modal Bayesian features, and the text modal Bayesian features are dynamically fused to generate a multimodal feature sequence; Key frames in the multimodal feature sequence are selected based on spatiotemporal saliency to obtain an initial key frame sequence; Temporal saliency scoring is performed on each keyframe in the initial keyframe sequence to obtain saliency scoring results; Based on the saliency score results, a preset number of keyframes are selected from the initial keyframe sequence in descending order of score to form the final keyframe sequence.
2. The keyframe extraction method as described in claim 1, characterized in that, The step of standardizing the pre-acquired multimodal data stream containing video, audio, and text to generate standardized multimodal data includes: The video in the multimodal data stream is sampled at preset time intervals to obtain a video frame sequence; The video frame sequence is subjected to RGB normalization to form a normalized video frame sequence; The audio in the multimodal data stream is subjected to frame-by-frame windowing processing to form frame-by-frame windowed audio data; Extract the Mel spectrogram of the framed and windowed audio data; The text in the multimodal data stream is segmented to obtain a segmented word sequence; The segmented word sequence is mapped to a vector to obtain a word vector sequence; The normalized video frame sequence, the Mel spectrogram, and the word vector sequence are combined to generate standardized multimodal data.
3. The keyframe extraction method as described in claim 1, characterized in that, The step of extracting Bayesian features from different modalities in the standardized multimodal data to obtain video modal Bayesian features, audio modal Bayesian features, and text modal Bayesian features includes: Extract the normalized video frame sequence, Mel spectrogram, and word vector sequence from the standardized multimodal data; Visual Bayesian features are obtained by performing forward propagation calculations on the normalized video frame sequence using a preset visual Bayesian neural network. The audio Bayesian features are obtained by forward propagation calculation of the Mel spectrogram using a preset audio Bayesian neural network. The word vector sequence is forward-propagated using a pre-defined text Bayesian neural network to obtain text Bayesian features; Uncertainty calculations are performed on the visual Bayesian features, the audio Bayesian features, and the text Bayesian features to obtain uncertainty score data; The visual Bayesian features, the audio modality Bayesian features, and the text Bayesian features are aligned by feature dimensions to form aligned visual Bayesian features, aligned audio Bayesian features, and aligned text Bayesian features. Based on the uncertainty scoring data, feature calibration is performed on the aligned visual Bayesian features, the aligned audio Bayesian features, and the aligned text Bayesian features to obtain video modality Bayesian features, audio modality Bayesian features, and text modality Bayesian features.
4. The keyframe extraction method as described in claim 1, characterized in that, The dynamic fusion of the video modal Bayesian features, the audio modal Bayesian features, and the text modal Bayesian features to generate a multimodal feature sequence includes: Uncertainty scalars are extracted from the video modality Bayesian features, the audio modality Bayesian features, and the text modality Bayesian features to form a set of uncertainty scalar values; The dynamic weights of the video modal Bayesian features, the audio modal Bayesian features, and the text modal Bayesian features are calculated based on the set of uncertainty scalar values. The video modal Bayesian features, the audio modal Bayesian features, and the text modal Bayesian features are weighted and fused according to the dynamic weights to generate a multimodal fusion vector; Uncertainty propagation calculation is performed on the multimodal fusion vector to obtain the uncertainty value; Based on the uncertainty value, feature normalization and time-series smoothing are performed on the multimodal fusion vector to generate a multimodal feature sequence.
5. The keyframe extraction method as described in claim 4, characterized in that, The step of calculating the dynamic weights of the video modality Bayesian features, the audio modality Bayesian features, and the text modality Bayesian features based on the set of uncertainty scalar values includes: The uncertain scalar values in the set of uncertain scalar values are smoothed and small values are added to obtain a preprocessed set of uncertain scalar values; The penalty score for the set of preprocessed uncertainty scalar values is calculated based on a preset temperature coefficient. The penalty score is normalized using the Softmax function to generate dynamic weights for the video modality Bayesian features, the audio modality Bayesian features, and the text modality Bayesian features.
6. The keyframe extraction method as described in claim 1, characterized in that, The initial keyframe sequence is obtained by selecting keyframes from the multimodal feature sequence based on spatiotemporal saliency, including: Calculate the feature differences between adjacent features in the multimodal feature sequence to generate a difference score sequence; The multimodal feature sequence is sampled using preset window parameters to obtain a sampled feature sequence; Calculate the contrast of adjacent features in the sampled feature sequence to obtain a contrast sequence; Calculate the global significance of each feature in the sampled feature sequence; The comprehensive significance score of each feature in the multimodal feature sequence is calculated based on the difference score sequence, the contrast sequence, and the global significance. The multimodal feature sequence is segmented, and key frames of each segment are selected based on the comprehensive saliency score to obtain an initial key frame sequence.
7. The keyframe extraction method as described in claim 1, characterized in that, The step of performing temporal saliency scoring on each keyframe in the initial keyframe sequence to obtain saliency scoring results includes: Extract the feature vector of the timestamp corresponding to each keyframe in the initial keyframe sequence to obtain the feature vector sequence; Extract the uncertainty vector of the timestamp corresponding to each key frame in the initial key frame sequence to generate a key frame uncertainty set; Calculate the feature differences between adjacent keyframes in the initial keyframe sequence based on the feature vector sequence to obtain a feature difference sequence; Based on the uncertainty set of the keyframes, calculate the uncertainty penalty term of the initial keyframe sequence to generate a penalty term set; The feature difference sequence is normalized to obtain a normalized sequence; The temporal saliency score of each keyframe is calculated by multiplying the normalized sequence by the set of penalty terms in sequence, and the saliency score result is obtained.
8. A keyframe extraction device, characterized in that, include: The data processing module is used to perform data standardization processing on the pre-acquired multimodal data stream containing video, audio and text to generate standardized multimodal data; The feature extraction module is used to extract Bayesian features of different modalities in the standardized multimodal data to obtain video modal Bayesian features, audio modal Bayesian features and text modal Bayesian features; The feature fusion module is used to dynamically fuse the video modal Bayesian features, the audio modal Bayesian features, and the text modal Bayesian features to generate a multimodal feature sequence; The keyframe scoring module is used to select keyframes in the multimodal feature sequence based on spatiotemporal saliency to obtain an initial keyframe sequence, and to perform temporal saliency scoring on each keyframe in the initial keyframe sequence to obtain a saliency scoring result. The keyframe selection module is used to select a preset number of keyframes from the initial keyframe sequence according to the saliency score results in descending order of score, to form the final keyframe sequence.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the keyframe extraction method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the keyframe extraction method as described in any one of claims 1 to 7.