Multi-modal key frame extraction method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, provides a multi-modal key frame extraction method, device and equipment and a medium, is applied to financial and medical health care service scenes, and can perform multi-modal feature extraction by utilizing each modal encoder so as to extract high-quality spatial-temporal features. Dynamic feature fusion is carried out based on the cross-modal attention network, so that the problem of feature drift caused by semantic deviation between modals can be solved, and the semantic consistency of fused features is enhanced; a differentiable enhancement network is utilized to enhance target video features and target audio features, cross-modal enhancement samples with consistent semantics are generated, the semantic disjunction problem caused by isolated enhancement of each modal is avoided, and the diversity and robustness of data are improved; the inter-frame saliency score is obtained based on the bidirectional long-short-term memory network so as to dynamically capture saliency changes under multi-mode cooperation, and therefore key frames with core semantics in a video can be recognized more accurately.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for extracting multimodal keyframes. Background Technology

[0002] In recent years, multimodal data processing technology has become one of the core research directions in the field of multimedia analytics. With the explosive growth of multimodal data such as video, audio, and text, effectively integrating information from different modalities and extracting key content has become a research hotspot. For example, by extracting interaction footage between bank tellers and customers, the cause of conflicts can be accurately located; by extracting key steps from surgical videos, teaching can be aided.

[0003] However, traditional keyframe extraction methods primarily rely on single-modal (e.g., visual features of video frames) for static or dynamic analysis, neglecting cross-modal semantic relationships. This results in insufficient robustness in complex scenes (e.g., varying lighting conditions, background noise). Furthermore, while existing multimodal fusion techniques have made some progress, they still suffer from the following limitations: (1) Intermodal feature drift problem: Most methods (such as early fusion or late fusion) fail to fully model the dynamic interaction between modalities, resulting in semantic bias in the fused features; (2) The isolation of data augmentation: Existing augmentation strategies (such as spatiotemporal transformation) are usually applied independently to each modality, lacking cross-modal consistency constraints, which weakens the effectiveness of self-supervised learning; (3) Static nature of keyframe extraction: Traditional methods are difficult to dynamically capture significant changes under multimodal collaboration. Summary of the Invention

[0004] In view of the above, it is necessary to provide a method, apparatus, device and medium for extracting multimodal keyframes, in order to solve the problems of inaccurate extraction of multimodal keyframes due to cross-modal feature drift and poor robustness of the extraction method.

[0005] A method for extracting multimodal keyframes, the method comprising: In response to the keyframe extraction instruction for the target multimodal data, the target multimodal data is preprocessed to obtain a target video frame sequence, a target audio signal, and a target text sequence; Multimodal feature extraction is performed on the target video frame sequence, the target audio signal, and the target text sequence using each modal encoder to obtain target spatiotemporal features corresponding to the target video frame sequence, target time-frequency features corresponding to the target audio signal, and target semantic features corresponding to the target text sequence. Based on a cross-modal attention network, the target spatiotemporal features, the target time-frequency features, and the target semantic features are dynamically fused to obtain feature-aligned target video features, target audio features, and target text features; The target video features and the target audio features are enhanced using a differentiable enhancement network to obtain target enhanced video and target enhanced audio; The target text features, the target enhanced video, and the target enhanced audio are processed using a bidirectional long short-term memory network to obtain the saliency score of each frame in the target multimodal data; Frames ranked first in the target multimodal data are selected as target keyframes according to their saliency scores from high to low.

[0006] A multimodal keyframe extraction device, the multimodal keyframe extraction device comprising: The preprocessing unit is used to preprocess the target multimodal data in response to the keyframe extraction instruction for the target multimodal data to obtain the target video frame sequence, the target audio signal and the target text sequence; The extraction unit is used to perform multimodal feature extraction on the target video frame sequence, the target audio signal, and the target text sequence using each modal encoder, to obtain the target spatiotemporal features corresponding to the target video frame sequence, the target time-frequency features corresponding to the target audio signal, and the target semantic features corresponding to the target text sequence. The fusion unit is used to dynamically fuse the target spatiotemporal features, the target time-frequency features, and the target semantic features based on a cross-modal attention network to obtain feature-aligned target video features, target audio features, and target text features; An enhancement unit is used to enhance the target video features and the target audio features using a differentiable enhancement network to obtain target enhanced video and target enhanced audio; The processing unit is used to process the target text features, the target enhanced video, and the target enhanced audio based on a bidirectional long short-term memory network to obtain the saliency score of each frame in the target multimodal data; The selection unit is used to select the frame ranked first in the target multimodal data as the target key frame in descending order of the salience score.

[0007] A computer device, the computer device comprising: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the multimodal keyframe extraction method.

[0008] A computer-readable storage medium storing at least one instruction, which is executed by a processor in a computer device to implement the multimodal keyframe extraction method.

[0009] As can be seen from the above technical solutions, this invention can utilize each modal encoder to extract multimodal features from the target video frame sequence, target audio signal, and target text sequence to extract high-quality spatiotemporal features; based on a cross-modal attention network, it can dynamically fuse the target spatiotemporal features, target time-frequency features, and target semantic features, which can solve the feature drift problem caused by semantic deviation between modalities and enhance the semantic consistency of the fused features; it can use a differentiable enhancement network to enhance the target video features and target audio features, generating semantically consistent cross-modal enhancement samples, avoiding the semantic disconnect problem caused by isolated enhancement of each modality, and improving the diversity and robustness of the data; based on a bidirectional long short-term memory network, it obtains inter-frame saliency scores to dynamically capture saliency changes under multimodal collaboration, thereby more accurately identifying key frames with core semantics in the video. Attached Figure Description

[0010] Figure 1 This is a flowchart of a preferred embodiment of the multimodal keyframe extraction method of the present invention.

[0011] Figure 2 This is a functional block diagram of a preferred embodiment of the multimodal keyframe extraction device of the present invention.

[0012] Figure 3 This is a schematic diagram of the structure of a computer device that implements the multimodal keyframe extraction method of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the multimodal keyframe extraction method of the present invention. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.

[0015] The multimodal keyframe extraction method is applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0016] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0017] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0018] The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0019] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0020] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0021] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0022] S10, in response to the key frame extraction instruction for the target multimodal data, preprocess the target multimodal data to obtain the target video frame sequence, the target audio signal and the target text sequence.

[0023] In this embodiment, the target multimodal data can be surveillance videos of bank branches, or promotional videos of funds, insurance and other products produced by financial institutions; the target multimodal data can also be medical surgery videos, or dynamic video images of medical examinations such as ultrasound dynamic videos and gastroscopy dynamic videos.

[0024] In this embodiment, the keyframe extraction instruction can be automatically triggered when the target multimodal data is detected to be uploaded to a specified interface or system.

[0025] In this embodiment, the preprocessing of the target multimodal data to obtain the target video frame sequence, target audio signal, and target text sequence includes: Video data is obtained from the target multimodal data, and the video data is normalized to obtain the target video frame sequence; Audio data is obtained from the target multimodal data, and the audio data is resampled to obtain the target audio signal; Text data is obtained from the target multimodal data, and word segmentation is performed on the text data to obtain the target text sequence.

[0026] By normalizing the video data, pixel values ​​can be adjusted to a uniform range, eliminating differences in brightness and contrast caused by different shooting devices and environments.

[0027] By resampling the audio data, the sampling rate can be standardized, ensuring the consistency of time-frequency feature extraction.

[0028] By performing word segmentation on the text data, continuous text can be broken down into discrete lexical units, preparing for semantic extraction by the text encoder.

[0029] The above embodiments can reduce the interference of data heterogeneity on subsequent feature extraction, improve the standardization of data of each modality, and provide high-quality preprocessed data for subsequent cross-modal feature alignment.

[0030] S11, using each modal encoder to perform multimodal feature extraction on the target video frame sequence, the target audio signal, and the target text sequence, respectively, to obtain the target spatiotemporal features corresponding to the target video frame sequence, the target time-frequency features corresponding to the target audio signal, and the target semantic features corresponding to the target text sequence.

[0031] In this embodiment, each modal encoder may include encoders for specifically extracting video features, audio features, and text features.

[0032] In this embodiment, the step of using each modal encoder to perform multimodal feature extraction on the target video frame sequence, the target audio signal, and the target text sequence to obtain target spatiotemporal features corresponding to the target video frame sequence, target time-frequency features corresponding to the target audio signal, and target semantic features corresponding to the target text sequence includes: Using a 3D Residual Next-101 encoder, the spatial and temporal features of the target video frame sequence are captured simultaneously through a three-dimensional convolutional kernel to obtain the spatiotemporal features of the target. The target audio signal is converted from time-domain audio to frequency-domain representation using a short-time Fourier transform algorithm, and the frequency-domain representation is input into a convolution-enhanced Transformer encoder for processing to obtain the target time-frequency features. The deep semantic information of the target text sequence is mined using a robust optimized BERT pre-trained encoder (RoBERTa encoder) to obtain context embedding features, and the context embedding features are determined as the target semantic features.

[0033] The spatial features may include object shapes, scene layouts, etc., and the temporal features may include changes in action, camera transitions, etc.

[0034] The 3D Residual Next-101 encoder can effectively alleviate the gradient vanishing problem in deep network training through the residual connection structure. The 3D Residual Next-101 encoder efficiently extracts multi-dimensional spatiotemporal features of videos. Compared with the traditional 2D convolution method that can only capture spatial features, it can more comprehensively reflect the dynamic information of videos.

[0035] The Conformer encoder combines the local perceptual capabilities of convolutional neural networks (such as capturing local time-frequency details) with the global modeling capabilities of Transformers (such as capturing long-term time-frequency dependencies) to output time-frequency features. The Conformer encoder can accurately capture the time-frequency characteristics of audio, extracting distinctive features from both short-duration voice commands and long-duration background sound effects.

[0036] The RoBERTa encoder mines deep semantic information from text through pre-training tasks such as masked language modeling, thereby outputting context embeddings. The RoBERTa encoder effectively extracts semantic features from text, capturing core semantic information even when the text contains missing keywords or non-standard expressions.

[0037] S12, dynamically fuse the target spatiotemporal features, the target time-frequency features, and the target semantic features based on a cross-modal attention network to obtain feature-aligned target video features, target audio features, and target text features.

[0038] In this embodiment, the dynamic fusion of the target spatiotemporal features, the target time-frequency features, and the target semantic features based on a cross-modal attention network to obtain feature-aligned target video features, target audio features, and target text features includes: Based on the target spatiotemporal features, the target time-frequency features, and the target semantic features, calculate the cross-attention weights between modalities to obtain the video-audio fusion weights, video-text fusion weights, and audio-text fusion weights. The first fusion feature of the video modality relative to the audio modality and the second fusion feature of the video modality relative to the text modality are calculated based on the video-audio fusion weight and the video-text fusion weight, respectively. The first fusion feature and the second fusion feature are then concatenated to obtain the target video feature. The third fusion feature of the audio modality relative to the video modality and the fourth fusion feature of the audio modality relative to the text modality are calculated based on the video-audio fusion weight and the audio-text fusion weight, respectively. The third fusion feature and the fourth fusion feature are then concatenated to obtain the target audio feature. The fifth fusion feature of the text modality relative to the video modality and the sixth fusion feature of the text modality relative to the audio modality are calculated based on the video-text fusion weight and the audio-text fusion weight, respectively. The fifth fusion feature and the sixth fusion feature are then concatenated to obtain the target text feature.

[0039] The cross-modal attention network is based on a cross-attention mechanism, which maps features of different modalities to a shared space through a learnable projection matrix and dynamically adjusts the contribution weights of each modality.

[0040] The above embodiments solve the feature drift problem caused by semantic deviation between modalities in traditional fusion methods, realize the dynamic alignment of multimodal features, and enhance the semantic consistency of fused features.

[0041] S13, the target video features and the target audio features are enhanced using a differentiable enhancement network to obtain target enhanced video and target enhanced audio.

[0042] In this embodiment, the differentiable enhancement network can be used to enhance video and audio data.

[0043] In this embodiment, the step of using a differentiable enhancement network to enhance the target video features and the target audio features to obtain enhanced target video and enhanced target audio includes: The learnable temporal deformation field is applied to the target video features using the differentiable enhancement network, and the target video features are differentiated by bilinear interpolation to obtain the target enhanced video; The target audio features are frequency-domain masked using the differentiable enhancement network to obtain the enhanced target audio.

[0044] Specifically, by differentiating the target video features through bilinear interpolation, the time delay between the video and other modalities (such as audio-visual asynchrony) can be corrected.

[0045] Specifically, using the differentiable enhancement network to perform frequency domain masking on the target audio features can ensure synchronization with the temporal perturbation of video enhancement.

[0046] The above embodiments can generate semantically consistent cross-modal augmented samples, effectively avoiding the semantic disconnect problem caused by isolated augmentation of each modality in traditional data augmentation, and improving the diversity and robustness of the data.

[0047] S14, The target text features, the target enhanced video, and the target enhanced audio are processed based on a bidirectional long short-term memory (LSTM) network to obtain the saliency score of each frame in the target multimodal data.

[0048] In this embodiment, the bidirectional long short-term memory network is used to model temporal dependencies.

[0049] In this embodiment, the processing of the target text features, the target enhanced video, and the target enhanced audio based on a bidirectional long short-term memory network to obtain the saliency score of each frame in the target multimodal data includes: The target-enhanced video, the target-enhanced audio, and the target text features are concatenated to obtain the target concatenated features; The target splicing features are input into the bidirectional long short-term memory network to model the temporal dependency relationship, thereby obtaining the forward hidden state and the reverse hidden state; The sigmoid function is used to process the forward hidden state and the reverse hidden state to obtain the saliency score for each frame.

[0050] The forward hidden state and the reverse hidden state can be processed using the following formula: ; in, Represents the frame-level significance score at time t; This represents the learnable weight matrix, which is a parameter that needs to be optimized during model training and is used to perform linear transformations on the bidirectional hidden states. This represents the forward hidden state at time t; This represents the reverse hidden state at time t; This represents the bias term, which is a parameter that needs to be optimized during model training and is used to adjust the result of the linear transformation. This represents the sigmoid function, which maps the calculation result to the interval [0,1] and is used to represent the saliency score of a frame.

[0051] The above embodiments enable the dynamic capture of significant changes under multimodal collaboration, and compared with traditional static scoring methods, it can more accurately identify key frames with core semantics in videos.

[0052] S15, select the frame ranked first in the target multimodal data as the target key frame according to the saliency score from high to low.

[0053] The preset position can be configured according to the accuracy requirements of the actual scenario.

[0054] In this embodiment, after selecting the frame ranked first in the target multimodal data as the target keyframe according to the saliency score from high to low, the target keyframe can be output to the downstream application system, thereby providing a keyframe sequence with clear structure and complete semantics to meet the needs of extracting core video content in different scenarios.

[0055] In this embodiment, in order to improve the system robustness and the accuracy of keyframe extraction, the network involved needs to be trained in advance before performing the above steps.

[0056] Specifically, the method further includes: Construct a training sample set; For each sample in the training sample set, the video features corresponding to the sample output by the cross-modal attention network and the enhanced video corresponding to the sample output by the differentiable enhancement network are considered positive samples; the audio features corresponding to the sample output by the cross-modal attention network and the enhanced audio corresponding to the sample output by the differentiable enhancement network are considered positive samples; the video features corresponding to the sample output by the cross-modal attention network and the enhanced videos corresponding to other samples output by the differentiable enhancement network are considered negative samples; the audio features corresponding to the sample output by the cross-modal attention network and the enhanced audio corresponding to other samples output by the differentiable enhancement network are considered negative samples. The feature vector of each sample is normalized to obtain the normalized embedding of each sample; The normalized temperature-scaled cross-entropy loss (NT-Xent loss) is calculated based on the positive and negative samples and the normalized embedding of each sample. The training sample set and the normalized temperature-scaled cross-entropy loss are used to jointly train each modal encoder, the cross-modal attention network, the differentiable enhancement network, and the bidirectional long short-term memory network.

[0057] For example, when the training sample set includes three original videos, namely video A, video B, and video C, and video A is taken as the core sample, then video B and video C are the other samples.

[0058] The normalized temperature-scaled cross-entropy loss can be calculated using the following formula based on the divided positive samples, negative samples, and the normalized embedding of each sample: ; ; in, This represents the total normalized temperature-scaled cross-entropy loss; Let represent the NT-Xent loss value for sample pair (i, j); This represents the transpose of the normalized embedding of the i-th sample; This represents the normalized embedding of the j-th sample; This represents the normalized embedding of the k-th sample; This represents the temperature coefficient, used to control the sensitivity of the loss to similarity. A smaller value, such as 0.07, is usually chosen to avoid excessively large differences in similarity. This indicates the batch size, which is the total number of samples contained in the current training batch. This indicates an indicator function, which takes the value 1 when k=i, and 0 otherwise (used to exclude sample i itself). This represents the dot product of the normalized embeddings of sample i and sample j, used to measure their similarity (the larger the dot product, the higher the similarity).

[0059] Where j is a positive sample of i (e.g., the original video sample i and its own augmented sample j): the above formula will make (Similarity) should be as high as possible, so that (Loss) should be as small as possible; when j is a negative sample of i (e.g., the original video sample i and the augmented sample j of other videos): the above formula will make (Similarity) should be kept as low as possible to avoid (Losses) increased.

[0060] The normalized temperature-scaled cross-entropy loss can measure whether the similarity of sample pairs in the feature space meets expectations. Through backpropagation of the loss, the model learns the feature representation that "positive sample pairs are more similar and negative sample pairs are more distant".

[0061] In the above embodiments, comparative learning can force the enhanced samples to maintain consistency with the original samples in the feature space, further improving the robustness and discriminative power of cross-modal features and enhancing the model's resistance to noise and interference.

[0062] This embodiment can be applied to keyframe extraction from multimodal videos in various scenarios.

[0063] For example, for news videos, the input data includes a 10-second, 25fps anchor narration segment, synchronized audio (with a 0.5-second delay), and automatically generated erroneous subtitles (missing keywords). This embodiment sequentially performs the following steps: encoder extracts features from each modality, time warp corrects video delay and spectral masking synchronizes audio, cross-modal attention aligns text and audiovisual content, and contrast loss constraint enhances sample consistency, ultimately outputting three keyframes: anchor opening, core content narration, and closing remarks. This embodiment can still accurately extract semantic keyframes even in cases of subtitle errors and audio-visual asynchrony.

[0064] For example, in the financial sector, for bank branch surveillance videos, input data includes counter interaction segments (30 seconds, 20fps), ambient audio (including crowd noise), and transaction record text (some information is abbreviated). This embodiment performs the following steps in sequence: encoder extracts features of each modality, time warp corrects for slight audio-visual delays and spectral mask filters background noise, cross-modal attention aligns transaction records with surveillance footage or audio, and contrast loss constraint enhances sample consistency, ultimately outputting four keyframes: customer submitting a large deposit certificate, teller operating a safe, customer signing confirmation, and suspected person loitering at the counter. This embodiment can still accurately extract risk-related keyframes even in noisy backgrounds and with simplified transaction records.

[0065] For example, in the medical field, for laparoscopic cholecystectomy surgery videos, the input data includes surgical operation segments (15 minutes, 15fps), surgeon's voice commands (including equipment noise), and surgical record text (real-time handwritten abbreviations). This embodiment sequentially performs the following steps: encoder extracts features of each modality, time warp corrects the delay between surgical footage and voice commands, spectral masking suppresses equipment noise, cross-modal attention aligns the surgical record with the surgical footage and voice, and contrast loss constraint enhances sample consistency, ultimately outputting 5 keyframes: incision establishment, gallbladder exposure, vascular clamping, cholecystectomy, and incision closure. This embodiment can still accurately extract keyframes of the core surgical steps even with surgical field occlusion and simplified recording.

[0066] As can be seen from the above technical solutions, this invention can utilize each modal encoder to extract multimodal features from the target video frame sequence, target audio signal, and target text sequence to extract high-quality spatiotemporal features; based on a cross-modal attention network, it can dynamically fuse the target spatiotemporal features, target time-frequency features, and target semantic features, which can solve the feature drift problem caused by semantic deviation between modalities and enhance the semantic consistency of the fused features; it can use a differentiable enhancement network to enhance the target video features and target audio features, generating semantically consistent cross-modal enhancement samples, avoiding the semantic disconnect problem caused by isolated enhancement of each modality, and improving the diversity and robustness of the data; based on a bidirectional long short-term memory network, it obtains inter-frame saliency scores to dynamically capture saliency changes under multimodal collaboration, thereby more accurately identifying key frames with core semantics in the video.

[0067] like Figure 2 The diagram shown is a functional block diagram of a preferred embodiment of the multimodal keyframe extraction device of the present invention. The multimodal keyframe extraction device 11 includes a preprocessing unit 110, an extraction unit 111, a fusion unit 112, an enhancement unit 113, a processing unit 114, and a selection unit 115. The module / unit referred to in this invention refers to a series of computer program segments that can be executed by a processor and perform a fixed function, and are stored in memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0068] The preprocessing unit 110 is used to preprocess the target multimodal data in response to the key frame extraction instruction for the target multimodal data to obtain the target video frame sequence, the target audio signal and the target text sequence.

[0069] In this embodiment, the target multimodal data can be surveillance videos of bank branches, or promotional videos of funds, insurance and other products produced by financial institutions; the target multimodal data can also be medical surgery videos, or dynamic video images of medical examinations such as ultrasound dynamic videos and gastroscopy dynamic videos.

[0070] In this embodiment, the keyframe extraction instruction can be automatically triggered when the target multimodal data is detected to be uploaded to a specified interface or system.

[0071] In this embodiment, the preprocessing unit 110 preprocesses the target multimodal data to obtain the target video frame sequence, target audio signal, and target text sequence, including: Video data is obtained from the target multimodal data, and the video data is normalized to obtain the target video frame sequence; Audio data is obtained from the target multimodal data, and the audio data is resampled to obtain the target audio signal; Text data is obtained from the target multimodal data, and word segmentation is performed on the text data to obtain the target text sequence.

[0072] By normalizing the video data, pixel values ​​can be adjusted to a uniform range, eliminating differences in brightness and contrast caused by different shooting devices and environments.

[0073] By resampling the audio data, the sampling rate can be standardized, ensuring the consistency of time-frequency feature extraction.

[0074] By performing word segmentation on the text data, continuous text can be broken down into discrete lexical units, preparing for semantic extraction by the text encoder.

[0075] The above embodiments can reduce the interference of data heterogeneity on subsequent feature extraction, improve the standardization of data of each modality, and provide high-quality preprocessed data for subsequent cross-modal feature alignment.

[0076] The extraction unit 111 is used to perform multimodal feature extraction on the target video frame sequence, the target audio signal and the target text sequence using each modal encoder, to obtain the target spatiotemporal features corresponding to the target video frame sequence, the target time-frequency features corresponding to the target audio signal and the target semantic features corresponding to the target text sequence.

[0077] In this embodiment, each modal encoder may include encoders for specifically extracting video features, audio features, and text features.

[0078] In this embodiment, the extraction unit 111 uses each modal encoder to perform multimodal feature extraction on the target video frame sequence, the target audio signal, and the target text sequence, respectively, to obtain target spatiotemporal features corresponding to the target video frame sequence, target time-frequency features corresponding to the target audio signal, and target semantic features corresponding to the target text sequence, including: Using a 3D Residual Next-101 encoder, the spatial and temporal features of the target video frame sequence are captured simultaneously through a three-dimensional convolutional kernel to obtain the spatiotemporal features of the target. The target audio signal is converted from time-domain audio to frequency-domain representation using a short-time Fourier transform algorithm, and the frequency-domain representation is input into a convolution-enhanced Transformer encoder for processing to obtain the target time-frequency features. The deep semantic information of the target text sequence is mined using a robust optimized BERT pre-trained encoder (RoBERTa encoder) to obtain context embedding features, and the context embedding features are determined as the target semantic features.

[0079] The spatial features may include object shapes, scene layouts, etc., and the temporal features may include changes in action, camera transitions, etc.

[0080] The 3D Residual Next-101 encoder can effectively alleviate the gradient vanishing problem in deep network training through the residual connection structure. The 3D Residual Next-101 encoder efficiently extracts multi-dimensional spatiotemporal features of videos. Compared with the traditional 2D convolution method that can only capture spatial features, it can more comprehensively reflect the dynamic information of videos.

[0081] The Conformer encoder combines the local perceptual capabilities of convolutional neural networks (such as capturing local time-frequency details) with the global modeling capabilities of Transformers (such as capturing long-term time-frequency dependencies) to output time-frequency features. The Conformer encoder can accurately capture the time-frequency characteristics of audio, extracting distinctive features from both short-duration voice commands and long-duration background sound effects.

[0082] The RoBERTa encoder mines deep semantic information from text through pre-training tasks such as masked language modeling, thereby outputting context embeddings. The RoBERTa encoder effectively extracts semantic features from text, capturing core semantic information even when the text contains missing keywords or non-standard expressions.

[0083] The fusion unit 112 is used to dynamically fuse the target spatiotemporal features, the target time-frequency features and the target semantic features based on a cross-modal attention network to obtain feature-aligned target video features, target audio features and target text features.

[0084] In this embodiment, the fusion unit 112 dynamically fuses the target spatiotemporal features, the target time-frequency features, and the target semantic features based on a cross-modal attention network to obtain feature-aligned target video features, target audio features, and target text features, including: Based on the target spatiotemporal features, the target time-frequency features, and the target semantic features, calculate the cross-attention weights between modalities to obtain the video-audio fusion weights, video-text fusion weights, and audio-text fusion weights. The first fusion feature of the video modality relative to the audio modality and the second fusion feature of the video modality relative to the text modality are calculated based on the video-audio fusion weight and the video-text fusion weight, respectively. The first fusion feature and the second fusion feature are then concatenated to obtain the target video feature. The third fusion feature of the audio modality relative to the video modality and the fourth fusion feature of the audio modality relative to the text modality are calculated based on the video-audio fusion weight and the audio-text fusion weight, respectively. The third fusion feature and the fourth fusion feature are then concatenated to obtain the target audio feature. The fifth fusion feature of the text modality relative to the video modality and the sixth fusion feature of the text modality relative to the audio modality are calculated based on the video-text fusion weight and the audio-text fusion weight, respectively. The fifth fusion feature and the sixth fusion feature are then concatenated to obtain the target text feature.

[0085] The cross-modal attention network is based on a cross-attention mechanism, which maps features of different modalities to a shared space through a learnable projection matrix and dynamically adjusts the contribution weights of each modality.

[0086] The above embodiments solve the feature drift problem caused by semantic deviation between modalities in traditional fusion methods, realize the dynamic alignment of multimodal features, and enhance the semantic consistency of fused features.

[0087] The enhancement unit 113 is used to enhance the target video features and the target audio features using a differentiable enhancement network to obtain target enhanced video and target enhanced audio.

[0088] In this embodiment, the differentiable enhancement network can be used to enhance video and audio data.

[0089] In this embodiment, the enhancement unit 113 uses a differentiable enhancement network to enhance the target video features and the target audio features, resulting in enhanced target video and enhanced target audio, including: The learnable temporal deformation field is applied to the target video features using the differentiable enhancement network, and the target video features are differentiated by bilinear interpolation to obtain the target enhanced video; The target audio features are frequency-domain masked using the differentiable enhancement network to obtain the enhanced target audio.

[0090] Specifically, by differentiating the target video features through bilinear interpolation, the time delay between the video and other modalities (such as audio-visual asynchrony) can be corrected.

[0091] Specifically, using the differentiable enhancement network to perform frequency domain masking on the target audio features can ensure synchronization with the temporal perturbation of video enhancement.

[0092] The above embodiments can generate semantically consistent cross-modal augmented samples, effectively avoiding the semantic disconnect problem caused by isolated augmentation of each modality in traditional data augmentation, and improving the diversity and robustness of the data.

[0093] The processing unit 114 is used to process the target text features, the target enhanced video, and the target enhanced audio based on a bidirectional long short-term memory (LSTM) network to obtain the saliency score of each frame in the target multimodal data.

[0094] In this embodiment, the bidirectional long short-term memory network is used to model temporal dependencies.

[0095] In this embodiment, the processing unit 114 processes the target text features, the target enhanced video, and the target enhanced audio based on a bidirectional long short-term memory network to obtain the saliency score of each frame in the target multimodal data, including: The target-enhanced video, the target-enhanced audio, and the target text features are concatenated to obtain the target concatenated features; The target splicing features are input into the bidirectional long short-term memory network to model the temporal dependency relationship, thereby obtaining the forward hidden state and the reverse hidden state; The sigmoid function is used to process the forward hidden state and the reverse hidden state to obtain the saliency score for each frame.

[0096] The forward hidden state and the reverse hidden state can be processed using the following formula: ; in, Represents the frame-level significance score at time t; This represents the learnable weight matrix, which is a parameter that needs to be optimized during model training and is used to perform linear transformations on the bidirectional hidden states. This represents the forward hidden state at time t; This represents the reverse hidden state at time t; This represents the bias term, which is a parameter that needs to be optimized during model training and is used to adjust the result of the linear transformation. This represents the sigmoid function, which maps the calculation result to the interval [0,1] and is used to represent the saliency score of a frame.

[0097] The above embodiments enable the dynamic capture of significant changes under multimodal collaboration, and compared with traditional static scoring methods, it can more accurately identify key frames with core semantics in videos.

[0098] The selection unit 115 is used to select the frame ranked first in the target multimodal data as the target key frame according to the order of the salience score from high to low.

[0099] The preset position can be configured according to the accuracy requirements of the actual scenario.

[0100] In this embodiment, after selecting the frame ranked first in the target multimodal data as the target keyframe according to the saliency score from high to low, the target keyframe can be output to the downstream application system, thereby providing a keyframe sequence with clear structure and complete semantics to meet the needs of extracting core video content in different scenarios.

[0101] In this embodiment, in order to improve the system robustness and the accuracy of keyframe extraction, the network involved needs to be trained in advance before performing the above steps.

[0102] Specifically, construct a training sample set; For each sample in the training sample set, the video features corresponding to the sample output by the cross-modal attention network and the enhanced video corresponding to the sample output by the differentiable enhancement network are considered positive samples; the audio features corresponding to the sample output by the cross-modal attention network and the enhanced audio corresponding to the sample output by the differentiable enhancement network are considered positive samples; the video features corresponding to the sample output by the cross-modal attention network and the enhanced videos corresponding to other samples output by the differentiable enhancement network are considered negative samples; the audio features corresponding to the sample output by the cross-modal attention network and the enhanced audio corresponding to other samples output by the differentiable enhancement network are considered negative samples. The feature vector of each sample is normalized to obtain the normalized embedding of each sample; The normalized temperature-scaled cross-entropy loss (NT-Xent loss) is calculated based on the positive and negative samples and the normalized embedding of each sample. The training sample set and the normalized temperature-scaled cross-entropy loss are used to jointly train each modal encoder, the cross-modal attention network, the differentiable enhancement network, and the bidirectional long short-term memory network.

[0103] For example, when the training sample set includes three original videos, namely video A, video B, and video C, and video A is taken as the core sample, then video B and video C are the other samples.

[0104] The normalized temperature-scaled cross-entropy loss can be calculated using the following formula based on the divided positive samples, negative samples, and the normalized embedding of each sample: ; ; in, This represents the total normalized temperature-scaled cross-entropy loss; Let represent the NT-Xent loss value for sample pair (i, j); This represents the transpose of the normalized embedding of the i-th sample; This represents the normalized embedding of the j-th sample; This represents the normalized embedding of the k-th sample; This represents the temperature coefficient, used to control the sensitivity of the loss to similarity. A smaller value, such as 0.07, is usually chosen to avoid excessively large differences in similarity. This indicates the batch size, which is the total number of samples contained in the current training batch. This indicates an indicator function, which takes the value 1 when k=i, and 0 otherwise (used to exclude sample i itself). This represents the dot product of the normalized embeddings of sample i and sample j, used to measure their similarity (the larger the dot product, the higher the similarity).

[0105] Where j is a positive sample of i (e.g., the original video sample i and its own augmented sample j): the above formula will make (Similarity) should be as high as possible, so that (Loss) should be as small as possible; when j is a negative sample of i (e.g., the original video sample i and the augmented sample j of other videos): the above formula will make (Similarity) should be kept as low as possible to avoid (Losses) increased.

[0106] The normalized temperature-scaled cross-entropy loss can measure whether the similarity of sample pairs in the feature space meets expectations. Through backpropagation of the loss, the model learns the feature representation that "positive sample pairs are more similar and negative sample pairs are more distant".

[0107] In the above embodiments, comparative learning can force the enhanced samples to maintain consistency with the original samples in the feature space, further improving the robustness and discriminative power of cross-modal features and enhancing the model's resistance to noise and interference.

[0108] This embodiment can be applied to keyframe extraction from multimodal videos in various scenarios.

[0109] For example, for news videos, the input data includes a 10-second, 25fps anchor narration segment, synchronized audio (with a 0.5-second delay), and automatically generated erroneous subtitles (missing keywords). This embodiment sequentially performs the following steps: encoder extracts features from each modality, time warp corrects video delay and spectral masking synchronizes audio, cross-modal attention aligns text and audiovisual content, and contrast loss constraint enhances sample consistency, ultimately outputting three keyframes: anchor opening, core content narration, and closing remarks. This embodiment can still accurately extract semantic keyframes even in cases of subtitle errors and audio-visual asynchrony.

[0110] For example, in the financial sector, for bank branch surveillance videos, input data includes counter interaction segments (30 seconds, 20fps), ambient audio (including crowd noise), and transaction record text (some information is abbreviated). This embodiment performs the following steps in sequence: encoder extracts features of each modality, time warp corrects for slight audio-visual delays and spectral mask filters background noise, cross-modal attention aligns transaction records with surveillance footage or audio, and contrast loss constraint enhances sample consistency, ultimately outputting four keyframes: customer submitting a large deposit certificate, teller operating a safe, customer signing confirmation, and suspected person loitering at the counter. This embodiment can still accurately extract risk-related keyframes even in noisy backgrounds and with simplified transaction records.

[0111] For example, in the medical field, for laparoscopic cholecystectomy surgery videos, the input data includes surgical operation segments (15 minutes, 15fps), surgeon's voice commands (including equipment noise), and surgical record text (real-time handwritten abbreviations). This embodiment sequentially performs the following steps: encoder extracts features of each modality, time warp corrects the delay between surgical footage and voice commands, spectral masking suppresses equipment noise, cross-modal attention aligns the surgical record with the surgical footage and voice, and contrast loss constraint enhances sample consistency, ultimately outputting 5 keyframes: incision establishment, gallbladder exposure, vascular clamping, cholecystectomy, and incision closure. This embodiment can still accurately extract keyframes of the core surgical steps even with surgical field occlusion and simplified recording.

[0112] As can be seen from the above technical solutions, this invention can utilize each modal encoder to extract multimodal features from the target video frame sequence, target audio signal, and target text sequence to extract high-quality spatiotemporal features; based on a cross-modal attention network, it can dynamically fuse the target spatiotemporal features, target time-frequency features, and target semantic features, which can solve the feature drift problem caused by semantic deviation between modalities and enhance the semantic consistency of the fused features; it can use a differentiable enhancement network to enhance the target video features and target audio features, generating semantically consistent cross-modal enhancement samples, avoiding the semantic disconnect problem caused by isolated enhancement of each modality, and improving the diversity and robustness of the data; based on a bidirectional long short-term memory network, it obtains inter-frame saliency scores to dynamically capture saliency changes under multimodal collaboration, thereby more accurately identifying key frames with core semantics in the video.

[0113] like Figure 3 The diagram shown is a schematic representation of the structure of a computer device that implements the multimodal keyframe extraction method of the present invention.

[0114] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a multimodal keyframe extraction program.

[0115] Those skilled in the art will understand that the schematic diagram is merely an example of computer device 1 and does not constitute a limitation on computer device 1. Computer device 1 can be either a bus topology or a star topology. Computer device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, computer device 1 may also include input / output devices, network access devices, etc.

[0116] It should be noted that the computer device 1 described is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0117] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a portable hard drive of the computer device 1. In other embodiments, the memory 12 can be an external storage device of the computer device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device 1. Furthermore, the memory 12 can include both internal and external storage units of the computer device 1. The memory 12 can be used not only to store application software and various types of data installed on the computer device 1, such as the code of a multimodal keyframe extraction program, but also to temporarily store data that has been output or will be output.

[0118] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the computer device 1, connecting various components of the computer device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing a multimodal keyframe extraction program) and calls data stored in the memory 12 to perform various functions of the computer device 1 and process data.

[0119] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the various embodiments of the multimodal keyframe extraction method described above, for example... Figure 1 The steps are shown.

[0120] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, which describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into a preprocessing unit 110, an extraction unit 111, a fusion unit 112, an enhancement unit 113, a processing unit 114, and a selection unit 115.

[0121] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the multimodal keyframe extraction method described in the various embodiments of the present invention.

[0122] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0123] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0124] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0125] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0126] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The bus is represented by only one straight line, but this does not mean that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0127] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0128] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish a communication connection between the computer device 1 and other computer devices.

[0129] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the computer device 1 and to display a visual user interface.

[0130] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0131] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0132] Combination Figure 1 The memory 12 in the computer device 1 stores multiple instructions to implement a multimodal keyframe extraction method, and the processor 13 can execute the multiple instructions to achieve the following: In response to the keyframe extraction instruction for the target multimodal data, the target multimodal data is preprocessed to obtain a target video frame sequence, a target audio signal, and a target text sequence; Multimodal feature extraction is performed on the target video frame sequence, the target audio signal, and the target text sequence using each modal encoder to obtain target spatiotemporal features corresponding to the target video frame sequence, target time-frequency features corresponding to the target audio signal, and target semantic features corresponding to the target text sequence. Based on a cross-modal attention network, the target spatiotemporal features, the target time-frequency features, and the target semantic features are dynamically fused to obtain feature-aligned target video features, target audio features, and target text features; The target video features and the target audio features are enhanced using a differentiable enhancement network to obtain target enhanced video and target enhanced audio; The target text features, the target enhanced video, and the target enhanced audio are processed using a bidirectional long short-term memory network to obtain the saliency score of each frame in the target multimodal data; Frames ranked first in the target multimodal data are selected as target keyframes according to their saliency scores from high to low.

[0133] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0134] It should be noted that all data involved in this case was legally obtained. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0135] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0136] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0137] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0138] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0139] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0140] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0141] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this invention can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for extracting multimodal keyframes, characterized in that, The multimodal keyframe extraction method includes: In response to the keyframe extraction instruction for the target multimodal data, the target multimodal data is preprocessed to obtain a target video frame sequence, a target audio signal, and a target text sequence; Multimodal feature extraction is performed on the target video frame sequence, the target audio signal, and the target text sequence using each modal encoder to obtain target spatiotemporal features corresponding to the target video frame sequence, target time-frequency features corresponding to the target audio signal, and target semantic features corresponding to the target text sequence. Based on a cross-modal attention network, the target spatiotemporal features, the target time-frequency features, and the target semantic features are dynamically fused to obtain feature-aligned target video features, target audio features, and target text features; The target video features and the target audio features are enhanced using a differentiable enhancement network to obtain target enhanced video and target enhanced audio; The target text features, the target enhanced video, and the target enhanced audio are processed using a bidirectional long short-term memory network to obtain the saliency score of each frame in the target multimodal data; Frames ranked first in the target multimodal data are selected as target keyframes according to their saliency scores from high to low.

2. The multimodal keyframe extraction method as described in claim 1, characterized in that, The preprocessing of the target multimodal data to obtain the target video frame sequence, target audio signal, and target text sequence includes: Video data is obtained from the target multimodal data, and the video data is normalized to obtain the target video frame sequence; Audio data is obtained from the target multimodal data, and the audio data is resampled to obtain the target audio signal; Text data is obtained from the target multimodal data, and word segmentation is performed on the text data to obtain the target text sequence.

3. The multimodal keyframe extraction method as described in claim 1, characterized in that, The step of using each modal encoder to perform multimodal feature extraction on the target video frame sequence, the target audio signal, and the target text sequence to obtain target spatiotemporal features corresponding to the target video frame sequence, target time-frequency features corresponding to the target audio signal, and target semantic features corresponding to the target text sequence includes: Using a 3D Residual Next-101 encoder, the spatial and temporal features of the target video frame sequence are captured simultaneously through a three-dimensional convolutional kernel to obtain the spatiotemporal features of the target. The target audio signal is converted from time-domain audio to frequency-domain representation using a short-time Fourier transform algorithm, and the frequency-domain representation is input into a convolution-enhanced Transformer encoder for processing to obtain the target time-frequency features. The encoder utilizes a robustly optimized BERT pre-training method to mine deep semantic information of the target text sequence to obtain context embedding features, and then determines the context embedding features as the target semantic features.

4. The multimodal keyframe extraction method as described in claim 1, characterized in that, The dynamic fusion of the target spatiotemporal features, target time-frequency features, and target semantic features based on a cross-modal attention network to obtain feature-aligned target video features, target audio features, and target text features includes: Based on the target spatiotemporal features, the target time-frequency features, and the target semantic features, calculate the cross-attention weights between modalities to obtain the video-audio fusion weights, video-text fusion weights, and audio-text fusion weights. The first fusion feature of the video modality relative to the audio modality and the second fusion feature of the video modality relative to the text modality are calculated based on the video-audio fusion weight and the video-text fusion weight, respectively. The first fusion feature and the second fusion feature are then concatenated to obtain the target video feature. The third fusion feature of the audio modality relative to the video modality and the fourth fusion feature of the audio modality relative to the text modality are calculated based on the video-audio fusion weight and the audio-text fusion weight, respectively. The third fusion feature and the fourth fusion feature are then concatenated to obtain the target audio feature. The fifth fusion feature of the text modality relative to the video modality and the sixth fusion feature of the text modality relative to the audio modality are calculated based on the video-text fusion weight and the audio-text fusion weight, respectively. The fifth fusion feature and the sixth fusion feature are then concatenated to obtain the target text feature.

5. The multimodal keyframe extraction method as described in claim 1, characterized in that, The step of enhancing the target video features and the target audio features using a differentiable enhancement network to obtain target enhanced video and target enhanced audio includes: The learnable temporal deformation field is applied to the target video features using the differentiable enhancement network, and the target video features are differentiated by bilinear interpolation to obtain the target enhanced video; The target audio features are frequency-domain masked using the differentiable enhancement network to obtain the enhanced target audio.

6. The multimodal keyframe extraction method as described in claim 1, characterized in that, The process of processing the target text features, the target enhanced video, and the target enhanced audio using a bidirectional long short-term memory network to obtain the saliency score for each frame in the target multimodal data includes: The target-enhanced video, the target-enhanced audio, and the target text features are concatenated to obtain the target concatenated features; The target splicing features are input into the bidirectional long short-term memory network to model the temporal dependency relationship, thereby obtaining the forward hidden state and the reverse hidden state; The sigmoid function is used to process the forward hidden state and the reverse hidden state to obtain the saliency score for each frame.

7. The multimodal keyframe extraction method as described in claim 1, characterized in that, The method further includes: Construct a training sample set; For each sample in the training sample set, the video features corresponding to the sample output by the cross-modal attention network and the enhanced video corresponding to the sample output by the differentiable enhancement network are considered positive samples; the audio features corresponding to the sample output by the cross-modal attention network and the enhanced audio corresponding to the sample output by the differentiable enhancement network are considered positive samples; the video features corresponding to the sample output by the cross-modal attention network and the enhanced videos corresponding to other samples output by the differentiable enhancement network are considered negative samples; the audio features corresponding to the sample output by the cross-modal attention network and the enhanced audio corresponding to other samples output by the differentiable enhancement network are considered negative samples. The feature vector of each sample is normalized to obtain the normalized embedding of each sample; The normalized temperature-scaled cross-entropy loss is calculated based on the positive and negative samples and the normalized embedding of each sample. The training sample set and the normalized temperature-scaled cross-entropy loss are used to jointly train each modal encoder, the cross-modal attention network, the differentiable enhancement network, and the bidirectional long short-term memory network.

8. A multimodal keyframe extraction device, characterized in that, The multimodal keyframe extraction device includes: The preprocessing unit is used to preprocess the target multimodal data in response to the keyframe extraction instruction for the target multimodal data to obtain the target video frame sequence, the target audio signal and the target text sequence; The extraction unit is used to perform multimodal feature extraction on the target video frame sequence, the target audio signal, and the target text sequence using each modal encoder, to obtain the target spatiotemporal features corresponding to the target video frame sequence, the target time-frequency features corresponding to the target audio signal, and the target semantic features corresponding to the target text sequence. The fusion unit is used to dynamically fuse the target spatiotemporal features, the target time-frequency features, and the target semantic features based on a cross-modal attention network to obtain feature-aligned target video features, target audio features, and target text features; An enhancement unit is used to enhance the target video features and the target audio features using a differentiable enhancement network to obtain target enhanced video and target enhanced audio. The processing unit is used to process the target text features, the target enhanced video, and the target enhanced audio based on a bidirectional long short-term memory network to obtain the saliency score of each frame in the target multimodal data; The selection unit is used to select the frame ranked first in the target multimodal data as the target key frame in descending order of the salience score.

9. A computer device, characterized in that, The computer device includes: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the multimodal keyframe extraction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in a computer device to implement the multimodal keyframe extraction method as described in any one of claims 1 to 7.

Images