Key frame feature extraction method based on cross-modal attention and related equipment

Abstract

The invention relates to the technical field of video analysis and processing, can be applied to the field of science and technology finance / digital medical treatment, and discloses a cross-modal attention-based key frame feature extraction method and related equipment. The method comprises the following steps: performing multi-modal data decomposition on an input video to obtain a video frame sequence, an audio signal and a motion optical flow sequence; respectively extracting visual, audio and motion features by adopting a multi-mode encoder; mapping each modal feature to a unified feature space, executing cross-modal attention fusion in the space, calculating an association relationship between modals and generating a fusion feature; dynamically adjusting the weight coefficient of each mode through a gating adaptive weight mechanism to obtain weighted key frame characterization features; and constructing a meta-learning task based on representation, forming a loss function through similarity calculation of a support set and a query set, and optimizing parameters in a cross-modal attention fusion process. According to the method, the accuracy and stability of key frame feature extraction are improved based on cross-modal attention.

Description

Technical Field

[0001] This application relates to the field of video analysis and processing technology, and can be applied to the fields of fintech / digital healthcare, particularly to a keyframe feature extraction method and related equipment based on cross-modal attention. Background Technology

[0002] Keyframe extraction from videos is a crucial step in video understanding, retrieval, and regulatory analysis. Existing technologies mainly fall into three categories: The first category relies on handcrafted features, such as optical flow, color histograms, SIFT (Scale-Invariant Feature Transform) / ORB (Oriented Fast and Rotated BRIEF), and then filters keyframes through thresholding or clustering. However, this approach is susceptible to occlusion, blurring, and scale variations, resulting in insufficient robustness. The second category uses static deep neural networks, such as CNN (Convolutional Neural Network) or ViT (Vision Transformer), combined with global pooling or attention scoring to independently extract frame features and filter keyframes. However, the network structure and computational path are fixed, making it impossible to dynamically adjust the receptive field and computational resources based on video content, leading to insufficient discrimination capabilities in complex scenes or with significant scale differences. The third category incorporates temporal modeling, such as LSTM (Long Short-Term Memory), Transformer, and TSN (Temporal Segment Network). While existing technologies like Network-based or Soft-DTW (Differentiable Dynamic Time Warping) are used for inter-frame relationship modeling or temporal alignment, they generally separate frame feature extraction from temporal alignment, are highly dependent on large-scale annotation, and have weak generalization ability under cross-scene or limited sample conditions. Furthermore, some MoE (Mixture of Experts) / dynamic routing schemes, although supporting conditional computation, only adjust routes at the parameter or expert level, and the route selection is not deeply coupled with task semantics, causing keyframe determination to deviate from the task's focus. Therefore, current technologies cannot solve the problems of computational paths not being dynamically adjusted based on video content and task, the disconnect between frame-level feature and temporal consistency modeling, insufficient multi-scale spatiotemporal semantic expression, and the lack of coordination between keyframe selection and task semantics. Summary of the Invention

[0003] The main technical problems addressed by the implementation of this application are the fixed computational path for existing keyframe feature extraction, insufficient spatiotemporal semantic expression, and weak task-related feature enhancement capabilities.

[0004] To address the aforementioned technical problems, the first technical solution adopted in this application is: providing a keyframe feature extraction method based on cross-modal attention, comprising: performing multimodal data decomposition on a received input video to obtain a video frame sequence, an audio signal, and a motion optical flow sequence; using a multimodal encoder to extract features from the video frame sequence, the audio signal, and the motion optical flow sequence respectively to obtain corresponding visual features, audio features, and motion features; mapping the visual features, audio features, and motion features to a unified feature space, and performing cross-modal attention fusion within the unified feature space to calculate the correlation between modalities, and generating fused features based on the correlation; dynamically adjusting the modal weight coefficients of the fused features through a gating adaptive weighting mechanism to obtain adaptively weighted keyframe representation features; constructing a meta-learning task based on the keyframe representation features, calculating a construction loss through the similarity between the support set and the query set, and updating the parameters in the cross-modal attention fusion process according to the construction loss.

[0005] Optionally, the step of mapping the visual features, audio features, and motion features to a unified feature space and performing cross-modal attention fusion within the unified feature space includes: performing linear transformation and projection processing on the visual features, audio features, and motion features respectively, so that the visual features, audio features, and motion features are mapped to a feature space of the same dimension, obtaining a unified-dimensional modal representation; processing the unified-dimensional modal representation through a multi-head attention mechanism, calculating the query vector, key vector, and value vector respectively; constructing an attention weight matrix based on the query vector and key vector, and performing a weighted summation of the attention weight matrix and the value vector to obtain the output features of each attention head; concatenating and linearly transforming the output features of each attention head to obtain a preliminary fused cross-modal feature representation; and performing normalization processing on the cross-modal feature representation to limit the cross-modal feature representation to a preset feature distribution space, obtaining the fused attention representation.

[0006] Optionally, the step of dynamically adjusting the modal weight coefficients of the fused features through a gating adaptive weighting mechanism to obtain adaptively weighted keyframe representation features includes: initializing the weight parameters of the fused features to generate initial weight coefficients for each modality of visual features, audio features, and motion features; inputting the initial weight coefficients to a preset gating function for activation operation to obtain modal weight control coefficients; performing element-wise weighting processing on the fused features according to the modal weight control coefficients to obtain weighted modal fused features; and performing normalization scaling processing on the weighted modal fused features to obtain the keyframe representation features.

[0007] Optionally, the step of constructing a meta-learning task based on the keyframe representation features, calculating a loss by comparing the similarity between the support set and the query set, and updating the parameters in the cross-modal attention fusion process based on the constructed loss includes: performing set partitioning processing on the keyframe representation features according to a preset sample partitioning strategy to obtain a support set and a query set for meta-learning training; performing feature aggregation operations on the keyframe representation features in the support set to generate an aggregated representation vector corresponding to the support set; calculating the similarity between the aggregated representation vector of the support set and the keyframe representation features of the query set to obtain a similarity calculation result between the features of the support set and the query set; inputting the similarity calculation result into a pre-constructed loss function to generate a loss value for model parameter optimization; and updating the parameters in the cross-modal attention fusion process based on the loss value.

[0008] Optionally, the step of performing multimodal data decomposition on the received input video to obtain a video frame sequence, an audio signal, and a motion optical flow sequence includes: decoding and extracting frames from the input video according to a preset frame sampling rule to obtain a continuous video frame sequence; separating and resampling the audio signal of the input video to obtain an audio signal sequence aligned with the time span of the video frame sequence; performing inter-frame displacement calculation based on pixel changes in adjacent frames of the video frame sequence to generate a motion optical flow sequence for characterizing motion changes; and performing timestamp alignment or sampling index alignment on the video frame sequence, the audio signal sequence, and the motion optical flow sequence to obtain a time-series corresponding multimodal data sequence.

[0009] Optionally, the step of using a multimodal encoder to extract features from the video frame sequence, the audio signal, and the motion optical flow sequence to obtain corresponding visual features, audio features, and motion features includes: inputting the video frame sequence into a preset visual coding network and performing spatiotemporal feature coding operations to obtain visual features corresponding to the video frame sequence; inputting the audio signal sequence into a preset audio coding network and performing time-domain or frequency-domain feature coding operations to obtain audio features corresponding to the audio signal sequence; inputting the motion optical flow sequence into a preset motion coding network and performing inter-frame motion mode coding operations to obtain motion features corresponding to the motion optical flow sequence; and performing feature tensor dimension alignment processing on the visual features, the audio features, and the motion features to obtain the dimension-consistent visual features, audio features, and motion features.

[0010] Optionally, the step of calculating the correlation between modalities and generating fused features based on the correlation includes: extracting statistical parameters describing the input distribution state of visual features, audio features, and motion features based on visual features, audio features, and motion features in a unified feature space; generating a correlation control signal for adjusting the interaction relationship of visual features, audio features, and motion features through the statistical parameters; setting the correlation control signal as a correction parameter during the calculation of the correlation relationship of visual features, audio features, and motion features, and updating the calculation of the correlation relationship through the correction parameter; and performing a fusion operation according to the corrected correlation relationship to generate fused features with dynamic correlation adjustment.

[0011] To address the aforementioned technical problems, the second technical solution adopted in this application is: providing a keyframe feature extraction device based on cross-modal attention, comprising: a multimodal data decomposition module for performing multimodal data decomposition on received input video to obtain video frame sequences, audio signals, and motion optical flow sequences; a multimodal encoder module for extracting features from the video frame sequences, audio signals, and motion optical flow sequences using a multimodal encoder to obtain corresponding visual features, audio features, and motion features respectively; a fusion feature generation module for mapping the visual features, audio features, and motion features to a unified feature space, performing cross-modal attention fusion within the unified feature space, calculating the correlation between modalities, and generating fused features based on the correlation; a keyframe representation feature module for dynamically adjusting the modal weight coefficients of the fused features through a gating adaptive weighting mechanism to obtain adaptively weighted keyframe representation features; and a meta-learning optimization module for constructing a meta-learning task based on the keyframe representation features, calculating a construction loss based on the similarity between the support set and the query set, and updating the parameters in the cross-modal attention fusion process according to the construction loss.

[0012] To solve the above-mentioned technical problems, the third technical solution adopted in the embodiments of this application is: to provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the keyframe feature extraction method based on cross-modal attention as described above.

[0013] To solve the above-mentioned technical problems, the fourth technical solution adopted in the embodiments of this application is: to provide a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by an electronic device, the electronic device performs the keyframe feature extraction method based on cross-modal attention as described above.

[0014] Unlike related technologies, this application achieves cross-modal correlation modeling and adaptive fusion of visual, audio, and motion features by performing multimodal data decomposition and unified feature space mapping on the input video. In the feature extraction stage, this scheme employs independent multimodal encoder networks to ensure the semantic integrity of each modality. In the fusion stage, a cross-modal attention mechanism is introduced to calculate the correlation between modalities and generate fused features, enabling the model to dynamically capture the complementarity and consistency of multi-source information, thereby improving the discriminative ability and temporal stability of keyframe representations. The overall method of this application structurally achieves unified expression and adaptive fusion of multimodal features, solving the problems of insufficient modal interaction and coarse fusion leading to low keyframe extraction accuracy in existing technologies. Attached Figure Description

[0015] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0016] Figure 1 This is a schematic diagram of the operating environment of the keyframe feature extraction method based on cross-modal attention provided in the embodiments of this application.

[0017] Figure 2 This is a schematic diagram of the execution flow of the keyframe feature extraction method based on cross-modal attention provided in the embodiments of this application.

[0018] Figure 3 This is a schematic diagram of the execution flow of cross-modal attention fusion in the keyframe feature extraction method based on cross-modal attention provided in the embodiments of this application.

[0019] Figure 4 This is a schematic diagram of the execution flow for generating fused features in the keyframe feature extraction method based on cross-modal attention provided in the embodiments of this application.

[0020] Figure 5 This is an example diagram of a data processing flow in the keyframe feature extraction method based on cross-modal attention provided in the embodiments of this application.

[0021] Figure 6 This is a schematic diagram of the system structure of the keyframe feature extraction device based on cross-modal attention provided in the embodiments of this application.

[0022] Figure 7 This is a schematic diagram of the hardware structure of an electronic device that performs a keyframe feature extraction method based on cross-modal attention, as provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Software tools, components, or servers not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0024] It should be noted that, unless otherwise specified, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device schematic diagram or the order in the flowchart.

[0025] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0026] To facilitate understanding of this embodiment, a keyframe feature extraction method based on cross-modal attention disclosed in this application will first be described in detail. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the operating environment of the keyframe feature extraction method based on cross-modal attention provided in the embodiments of this application, such as... Figure 1 As shown, the execution subject of the keyframe feature extraction method based on cross-modal attention provided in this application embodiment is generally an electronic device with a certain computing power, such as a computer device. In some possible implementations, this keyframe feature extraction method based on cross-modal attention can be implemented by the processor calling computer-readable instructions stored in memory. Figure 1 The computer equipment mentioned can be a server. A server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This can be understood as... Figure 1 The number of computer devices shown is merely illustrative and can be expanded in any number according to actual needs.

[0027] Please continue reading. Figure 2 , Figure 2 This is a schematic diagram of the execution flow of the keyframe feature extraction method based on cross-modal attention provided in the embodiments of this application, as shown below. Figure 2 As shown, it includes the following steps: S1. Perform multimodal data decomposition on the received input video to obtain the video frame sequence, audio signal and motion optical flow sequence.

[0028] Step S1 involves multimodal data decomposition: simultaneously extracting three complementary signals from the input video stream—frame-by-frame visual sequences, time-aligned audio waveforms, and optical flow estimated from the pixel displacements of adjacent frames. These three data streams provide precise temporally aligned input for subsequent "visual / acoustic / motion" three-branch coding, ensuring that cross-modal attention can establish a correlation on a unified time axis. For example, in smart counter quality inspection or remote face-to-face signing scenarios in fintech, visual frames capture the clarity and authenticity of the customer's held documents, audio signals verify the consistency of readings or spoken information, and optical flow detects micro-expressions and suspicious actions (such as rapid removal / obstruction of documents). In digital healthcare remote consultations or intraoperative recording scenarios, visual frames convey the state of instruments and the surgical field, the audio track records medical orders and key points spoken by the surgeon, and optical flow quantifies the trajectory and operational stability of surgical instruments, thus providing multi-source evidence for keyframe extraction and event localization.

[0029] As an optional implementation, step S1 may specifically include steps S11 to S14.

[0030] S11. Decode and extract frames from the input video according to the preset frame sampling rules to obtain a continuous video frame sequence.

[0031] Step S11 involves frame-level decomposition and standardized sampling of the video signal. Through preset frame rate decoding, frame extraction, and keyframe extraction algorithms, the continuous video stream is transformed into a temporally consistent and texture-preserving frame sequence. This process ensures the uniformity of input data in spatial resolution and time intervals during subsequent motion estimation and spatiotemporal coding stages, thus providing a foundation for stable extraction of visual features and cross-modal alignment.

[0032] S12. The audio signal of the input video is separated and resampled to obtain an audio signal sequence that is aligned with the time span of the video frame sequence.

[0033] In step S12, the audio signal is independently extracted and resampled in the temporal domain, ensuring that the audio sequence is strictly aligned with the video frame sequence in terms of time span. Through processing such as unified sampling rate, denoising, and amplitude normalization, the differences between video and audio in terms of sampling period, phase reference, etc., are eliminated, enabling subsequent intermodal fusion calculations to be based on a comparable temporal index system.

[0034] S13. Perform inter-frame displacement calculation based on pixel changes in adjacent frames in the video frame sequence to generate a motion optical flow sequence to characterize motion changes.

[0035] In step S13, based on pixel or feature changes in adjacent frames, local motion vectors are calculated to generate an optical flow sequence describing the dynamic changes of objects or scenes. The optical flow data characterizes the displacement direction and velocity amplitude of local regions in the video, providing the motion feature encoder with input that directly reflects temporal dynamics. This enables the system to capture spatiotemporal motion patterns beyond visual features, achieving fine-grained modeling of video dynamic behavior.

[0036] S14. Perform timestamp alignment or sampling index alignment on the video frame sequence, audio signal sequence and motion optical flow sequence to obtain the time-corresponding multimodal data sequence.

[0037] In step S14, multimodal temporal synchronization is performed, mapping video frames, audio signals, and optical flow data to a unified time index. Through timestamp correction, sampling index interpolation, or padding operations, it is ensured that each modality corresponds to a consistent data segment at the same time point. This step eliminates the time drift problem between modalities, ensuring that the cross-modal attention mechanism executes under a unified time reference, thereby improving the accuracy and reliability of multimodal feature fusion.

[0038] Through steps S11 to S14, temporal consistency preprocessing of the input video across visual, audio, and motion modalities is achieved, ensuring strict alignment of multi-source heterogeneous data on the time axis and maintaining sampling consistency. This process effectively eliminates sampling bias and temporal drift issues between different modalities, guaranteeing that features from each modality participate in computation under the same time reference during subsequent feature extraction and cross-modal attention fusion, thereby improving the accuracy of inter-modal correlation and fusion stability. Furthermore, by introducing optical flow sequences, the system can extract motion information from dynamic changes between frames, complementing the temporal features of static visual features and audio signals. This enhances the model's ability to perceive video spatiotemporal semantics and event transitions, providing a reliable data foundation for keyframe determination and high-precision feature representation.

[0039] S2. A multimodal encoder is used to extract features from the video frame sequence, audio signal and motion optical flow sequence respectively, and the corresponding visual features, audio features and motion features are obtained respectively.

[0040] In step S2, a multimodal encoder is used to extract features in parallel from video frame sequences, audio signals, and motion optical flow sequences, encoding the original visual, auditory, and motion information into structured high-dimensional feature representations. The visual branch is responsible for modeling spatial layout and object appearance, the audio branch models acoustic energy distribution and semantic features, and the motion branch captures dynamic changes and displacement patterns between frames, thereby achieving deep representation of multi-source information and providing a foundation for cross-modal attention fusion. In the fintech field, this mechanism can be used for the comprehensive recognition of multimodal signals in smart teller machine monitoring or remote face-to-face video: visual features represent customer identity and document authenticity, audio features analyze voice commands or reading content, and motion features detect abnormal actions (such as covering or quickly swapping documents), achieving higher-credibility identity verification and business compliance monitoring. In digital healthcare scenarios, multimodal encoders can simultaneously extract visual textures (surgical instruments, surgical field status), audio signals (doctor's commands, equipment alarm sounds), and motion optical flow (trajector trajectories and operating speeds) from surgical videos, thereby enabling spatiotemporal modeling and risk identification of surgical behavior and providing multi-source support for intraoperative decision support or postoperative analysis.

[0041] As an optional implementation, step S2 may specifically include steps S21 to S24.

[0042] S21. Input the video frame sequence into the preset visual coding network, perform spatiotemporal feature coding operation, and obtain the visual features corresponding to the video frame sequence.

[0043] In step S21, a visual coding network is used to perform spatiotemporal feature modeling on the video frame sequence, extracting visual features that characterize the spatial structure of the image, the motion trajectory of the target, and the semantics of the scene. The network can adopt a convolutional neural network or a temporal convolutional network structure to capture local texture, object contours, and temporal change trends, thereby achieving feature encoding of video information in both spatial and temporal dimensions.

[0044] S22. Input the audio signal sequence into the preset audio coding network, perform time-domain or frequency-domain feature coding operations, and obtain the audio features corresponding to the audio signal sequence.

[0045] In step S22, the audio signal is modeled in the time or frequency domain using an audio coding network to extract acoustic elements such as energy distribution, frequency band resonance, and speech rhythm. The encoded audio features can describe the speech changes or environmental sound information accompanying the events in the video, providing acoustic contextual clues for cross-modal matching.

[0046] S23. Input the motion optical flow sequence into the preset motion coding network, perform inter-frame motion mode coding operation, and obtain the motion features corresponding to the motion optical flow sequence.

[0047] In step S23, the optical flow sequence is modeled using a motion coding network to learn dynamic attributes such as local displacement, acceleration, and motion direction, thereby forming an abstract description of the motion of the target or shot in the video. This motion feature compensates for the shortcomings of static visual features in the temporal dimension, enhancing the system's ability to represent actions, behaviors, and motion patterns.

[0048] S24. Perform feature tensor dimension alignment processing on visual features, audio features and motion features to obtain visual features, audio features and motion features with consistent dimensions.

[0049] In step S24, dimension alignment and scale unification are performed on the feature tensors of visual, audio, and motion types to ensure that the features of different modalities are consistent in terms of channel number, time step, and feature dimension. Through linear transformation or tensor resampling operations, it is ensured that the features of each modality can be matched and interact in the same feature space during subsequent cross-modal attention fusion, thereby improving the computability and accuracy of multimodal fusion.

[0050] As an example, in the multimodal feature encoding stage described above, feature representations of each modality can be extracted from the raw video data using a multimodal feature encoder. The visual branch uses Vision Transformer (ViT) to process video frames and generate image patch-based embedding vectors; the audio branch uses the Wav2Vec 2.0 model to extract acoustic features; and the motion branch analyzes the optical flow field using 3D-CNN to capture temporal motion patterns. For the visual modality, given an input frame... ViT divides it into non-overlapping equal-sized segments. The visual feature matrix is ​​obtained by linear projection of image patches: , The visual embedding dimension is represented by a row in the matrix corresponding to the features of each image patch. Audio feature extraction uses the Wav2Vec 2.0 model to process the original waveform. ( (Number of audio sample points) This is expressed as the number of compressed time steps. This represents the dimension of the audio features. Output temporal features: Motion features are processed using 3D-CNN to analyze the optical flow field. ( (length of optical flow sequence) , This represents the dimension of motion features, where features at each time step capture local motion patterns.

[0051] S3. Map visual features, audio features, and motion features to a unified feature space, perform cross-modal attention fusion within the unified feature space, calculate the correlation between modalities, and generate fused features based on the correlation.

[0052] Step S3 maps visual, audio, and motion features to a shared feature space and performs cross-modal attention fusion within this space, generating fused features by calculating the correlations between different modalities. The unified feature space ensures the comparability of features across modalities in semantic dimension and scale, while the cross-modal attention mechanism, guided by dynamic weights, strengthens the complementarity and information correlation between different modalities, enabling the model to adaptively focus on key moments and core modal signals. In fintech scenarios, this mechanism can be used for intelligent risk control video analysis or remote credit verification: simultaneously monitoring changes in the customer's visual expressions, voice tone, and body movements within the unified feature space to comprehensively determine the authenticity and consistency of the face-to-face verification behavior. For example, when audio features show abnormal speech rate, visual features detect gaze drift, and motion features record unnatural hand movements, the system can trigger a risk warning. In digital healthcare scenarios, cross-modal attention fusion can comprehensively analyze intraoperative video, audio, and motion signals. For example, in surgical monitoring, the visual modality reflects the relationship between instruments and the surgical field, the audio modality provides physician instructions and team collaboration information, and the motion modality characterizes the force and stability of the operation. The system can focus on key moments (such as cutting and suturing) through attention mechanisms to achieve multimodal understanding and quality assessment of the surgical process.

[0053] As an optional implementation, please refer to Figure 3 , Figure 3 This is a schematic diagram of the execution flow of cross-modal attention fusion in the keyframe feature extraction method based on cross-modal attention provided in the embodiments of this application, as shown below. Figure 3 As shown, it can specifically include the following steps S31 to S35.

[0054] S31. Perform linear transformation and projection processing on the visual features, audio features and motion features respectively, so that the visual features, audio features and motion features are mapped to the feature space of the same dimension, so as to obtain a modal representation of the same dimension.

[0055] In step S31, visual, audio, and motion features are mapped to a feature space of the same dimension through linear transformation and projection operations, completing the initial semantic alignment between modalities. This process ensures that different modalities are comparable in dimensionality and distribution, establishing a unified input representation foundation for subsequent attention weight calculation and cross-modal fusion.

[0056] S32. Process the modal representation of the same dimension through a multi-head attention mechanism to calculate the query vector, key vector and value vector respectively.

[0057] In step S32, a multi-head attention mechanism is used to reorganize the modal representations within a unified feature space. The system generates query vectors, key vectors, and value vectors respectively, and computes the inter-modal dependencies in parallel on multiple attention heads, enabling the model to capture cross-modal semantic interactions and contextual relationships from multiple attention perspectives.

[0058] S33. Construct an attention weight matrix based on the query vector and key vector, and sum the attention weight matrix with the value vector to obtain the output features of each attention head.

[0059] In step S33, an attention weight matrix is ​​constructed based on the correlation between the query vector and the key vector, and the weights are applied to the value vector to obtain the multi-head attention output features. This process essentially realizes the weighted flow of information between different modal features, which strengthens the modal information with strong correlation and suppresses the irrelevant parts, thereby improving the discriminativeness of feature fusion.

[0060] S34. The output features of each attention head are concatenated and linearly transformed to obtain a preliminary fused cross-modal feature representation.

[0061] In step S34, the features output by multi-head attention are concatenated and linearly transformed to form a holistic cross-modal fusion representation. The concatenation operation preserves the independent attention results of multi-head attention in different semantic dimensions, and the linear mapping further compresses and reorganizes the feature dimensions, laying the foundation for generating a high-level unified cross-modal representation.

[0062] S35. Perform normalization processing on the cross-modal feature representation to limit the cross-modal feature representation to a preset feature distribution space, thereby obtaining the fused attention representation.

[0063] In step S34, the fused features are normalized and scaled to ensure that the distribution of each modality after fusion is stable and matches the expected feature space. The normalization operation reduces the distribution shift caused by modal differences, enabling the system to have higher fusion consistency and generalization ability in subsequent tasks (such as keyframe extraction or similarity calculation).

[0064] As an example, after multimodal feature extraction, the system enters the dynamic fusion Transformer stage. This stage employs a three-level fusion strategy, including an early fusion layer, a deep fusion layer, and a late fusion layer, to achieve dynamic cross-modal attention fusion of visual, audio, and motion features within a unified feature space. The early fusion layer concatenates the multimodal features and projects them into a shared space. , For learnable weight matrix, This indicates a concatenation operation along the feature dimension. Indicates the hidden layer dimension. This represents the bias term. The deep fusion layer employs a multi-head cross-attention mechanism (based on the number of heads). For example). For the first Size: Query matrix Key matrix Sum matrix Each by Obtained through linear transformation, The dimension is the subspace dimension. The result after concatenating the outputs from multiple sources is: , For N independent output projection matrices, the features of the multi-head attention output are input into different... In the middle, then all of them The projection matrix outputs are summed (feature enhancement can be performed for different tasks during model training). Initialize to random values, but n has a 20% probability of being a zero-value matrix, while ensuring that at least one n is not a zero-value matrix. Finally, LayerNorm implements residual joins. Post-normalization. Late-stage gating fusion is dynamically adjusted via modal weights: Gating vector via the sigmoid function activation, This represents element-wise multiplication, enabling adaptive weighting of modal features. This is represented as a bias term, and the final output is the feature. .

[0065] Through steps S31 to S35, a unified mapping and deep fusion of visual, audio, and motion features are achieved, enabling dynamic interaction and semantic compensation of multimodal features within a unified feature space. The multi-head attention mechanism adaptively allocates attention weights across different modalities, highlighting the most discriminative modal information according to task requirements and effectively eliminating information redundancy and scale differences between modalities. Through this cross-modal attention fusion process, the system establishes a connection between visual content, acoustic information, and motion behavior at a unified semantic level, enabling the fused features to more accurately represent the true semantics and spatiotemporal relationships of events in the video. This mechanism significantly improves the accuracy and robustness of keyframe feature extraction, providing a highly consistent and separable input feature foundation for subsequent adaptive weighting and meta-learning optimization.

[0066] As another alternative implementation, please refer to Figure 4 , Figure 4 This is a schematic diagram of the execution flow for generating fused features in the keyframe feature extraction method based on cross-modal attention provided in this application embodiment, as shown below. Figure 4As shown, it can specifically include the following steps S36 to S39.

[0067] S36. Based on visual features, audio features, and motion features in a unified feature space, extract statistical parameters to describe the input distribution state of visual features, audio features, and motion features.

[0068] In step S36, statistical measures are extracted from the input distributions of visual, audio, and motion features within a unified feature space to describe the state characteristics of each modality at the current moment. These statistical parameters may include mean, variance, skewness, kurtosis, or distribution density estimation, reflecting the dynamic stability and information fluctuation of the modal data. Through these parameters, the system can quantify the activity level of each modality and its contribution changes in the current scene, providing a quantitative basis for the subsequent generation of cross-modal modulation signals.

[0069] S37. Generate correlation modulation signals for adjusting the interaction between visual features, audio features and motion features through statistical parameters.

[0070] In step S37, the extracted statistical parameters are used to generate a cross-modal correlation modulation signal to reflect the interdependence between visual, audio, and motion modalities. This modulation signal can be regarded as a kind of "global attention bias term". By describing the degree of cooperation and uncertainty distribution between modalities, it enables subsequent fusion calculations to not only rely on static weights, but also dynamically adjust the intensity of feature interactions based on modal states.

[0071] S38. In the process of calculating the correlation between visual features, audio features and motion features, the correlation control signal is set as the correction parameter, and the calculation of the correlation is updated by the correction parameter.

[0072] In step S38, during the calculation of intermodal correlations, a control signal is introduced as a correction parameter to dynamically adjust the attention weights or correlation matrix. The addition of the correction parameter enables the system to adjust the intensity of feature interactions between different modalities in real time, thereby suppressing anomalous modalities (such as noisy audio or blurred images) and enhancing the influence of high-confidence modalities, making the fusion calculation more robust and adaptive.

[0073] S39. Perform a fusion operation based on the corrected association relationship to generate a fusion feature with dynamic association adjustment.

[0074] In step S39, a fusion operation is performed based on the corrected modal correlation to generate a fusion feature representation with dynamic adjustment capability. This fusion result integrates the statistical state and dynamic interaction information between modalities, and can adaptively reflect the collaborative change trend of multimodal data in different scenarios, providing a robust and high-resolution feature foundation for subsequent keyframe feature extraction and model optimization.

[0075] Through steps S36 to S39, a statistically driven dynamic control mechanism is introduced into the cross-modal fusion process, enabling the system to adaptively adjust the correlation between modalities based on the current distribution of visual, audio, and motion modalities. By characterizing modal activity and confidence levels using statistical parameters and generating corresponding control signals, the model dynamically balances the contribution ratios of different modalities in the fusion process, effectively suppressing the interference of noisy modalities on the overall features. This mechanism transforms the calculation of cross-modal attention from static weight allocation to a dynamic update process based on modal state feedback, allowing the fused features to reflect the changing trends of multimodal information in real time. The resulting fused features show significant improvements in temporal stability, modal synergy, and feature discriminative power, providing higher-level feature support for the accurate generation of keyframe representations and robust optimization of downstream learning tasks. S4. The modal weight coefficients of the fused features are dynamically adjusted through a gating adaptive weighting mechanism to obtain adaptively weighted keyframe representation features.

[0076] Step S4 employs a gated adaptive weighting mechanism to dynamically adjust the modal weight coefficients in the fused features, enabling the model to adaptively allocate modal weights based on changes in feature contribution. The gating mechanism uses a non-linear activation function to modulate the weights, thereby highlighting the most critical modal features and suppressing redundant or noisy information in different scenarios, ultimately yielding adaptively weighted keyframe representation features. In fintech scenarios, this mechanism can be used in intelligent monitoring or risk assessment systems, automatically increasing the weight of motion modalities under unstable lighting or strong audio interference to enhance abnormal behavior recognition capabilities. In digital healthcare scenarios, the system can dynamically adjust the visual and audio modal weights based on the real-time feature distribution during surgery, maintaining high-precision intraoperative state understanding even during periods of instrument occlusion or dense voice commands.

[0077] As an optional implementation, step S4 may specifically include steps S41 to S44.

[0078] S41. Initialize the weight parameters of the fused features to generate the initial weight coefficients for each modality of the corresponding visual features, audio features, and motion features.

[0079] Step S41 is the weight initialization stage, which establishes the initial proportional relationship between visual, audio, and motion modalities in the weighted calculation by setting the initial weight parameters for each modality in the fused features. This process provides a benchmark weight for the gating mechanism, giving subsequent dynamic adjustments a balanced reference and avoiding feature imbalance caused by weight bias in the early stages of training.

[0080] S42. Input the initial weight coefficients into the preset gating function to perform activation calculations and obtain the modal weight control coefficients.

[0081] In step S42, the initial weight parameters are input into a gating function for nonlinear activation to generate modal weight control coefficients. The gating function (e.g., Sigmoid or Tanh) is used to compress the weight range and achieve continuously differentiable dynamic adjustment, enabling the system to smoothly respond to modal state changes and providing learnable adaptive control capabilities for feature fusion.

[0082] S43. The fusion features are weighted element by element according to the modal weight control coefficient to obtain the weighted modal fusion features.

[0083] In step S43, based on the weight control coefficients of the gating output, element-wise weighting is performed on the fused features to achieve dynamic weighting at the modality level. Through the weighting process, the influence of different modal features is redistributed according to their current effectiveness, thereby strengthening key modalities and suppressing noisy modalities, making the fused features more consistent with the true semantic distribution of video events.

[0084] S44. Normalize and scale the weighted modal fusion features to obtain keyframe representation features.

[0085] In step S44, based on the weight control coefficients of the gating output, an element-wise weighting operation is performed on the fused features to achieve dynamic weighting at the modality level. Through the weighting process, the influence of different modal features is redistributed according to their current effectiveness, thereby strengthening key modalities and suppressing noisy modalities, making the fused features more consistent with the true semantic distribution of video events.

[0086] Through steps S41 to S44, adaptive weighting and dynamic balancing of the fused features are achieved, enabling the system to automatically adjust weight allocation based on the contribution of different modal features to the current task, thereby enhancing the discriminative power and robustness of the overall feature representation. Through the nonlinear control mechanism of the gating function, the model can continuously perceive changes in the state of visual, audio, and motion modalities during training and inference, and correct the weight coefficients in real time, giving the weighting process self-learning and adaptive capabilities. This mechanism effectively avoids the modal imbalance problem caused by fixed weights, ensuring that key modal features are fully strengthened during the fusion process, while suppressing the interference of invalid or noisy modalities on the fusion results. The normalized keyframe representation features show significant improvements in temporal stability, modal consistency, and semantic aggregation, providing a high-quality, dynamically optimized input feature foundation for downstream similarity calculation and meta-learning tasks.

[0087] S5. Construct a meta-learning task based on keyframe representation features, calculate the construction loss by the similarity between the support set and the query set, and update the parameters in the cross-modal attention fusion process based on the construction loss.

[0088] Step S5 constructs a meta-learning task based on keyframe representation features, establishes a task-level loss function by calculating the similarity between the support set and the query set, and optimizes the parameters in the cross-modal attention fusion process using loss feedback. The meta-learning structure enables the model to quickly adapt to new scenarios from a small number of samples, achieving self-evolutionary learning in cross-modal tasks. In fintech scenarios, this mechanism can be used for tasks such as abnormal transaction identification or remote identity verification: the system learns typical transaction patterns or standard face-to-face features through the support set, and quickly distinguishes between query samples (e.g., fragments of new customer behavior) through similarity measurement. In digital healthcare scenarios, the meta-learning structure can be used for personalized surgical procedure analysis or disease image-assisted discrimination; the model can transfer learning across different patients and scenarios, automatically adjusting fusion parameters to maintain high diagnostic consistency.

[0089] As an optional implementation, step S5 may specifically include steps S51 to S55.

[0090] S51. The keyframe representation features are divided into sets according to the preset sample division strategy to obtain the support set and query set for meta-learning training.

[0091] In step S51, based on a preset sample partitioning strategy, the keyframe representation features are divided into a support set and a query set for meta-learning training. The support set provides a reference representation of the category or pattern, while the query set is used to verify the model's generalization performance on new samples, thus forming a task-level data structure input.

[0092] S52. Perform feature aggregation operation on the keyframe representation features in the support set to generate the aggregated representation vector corresponding to the support set.

[0093] In step S52, feature aggregation operations are performed on the keyframe representation features in the support set to generate an aggregated representation vector for the support set. Aggregation operations are typically based on mean pooling or weighted fusion to form a centralized representation of similar samples in the feature space, facilitating efficient subsequent similarity calculations.

[0094] S53. Calculate the similarity between the aggregated representation vector of the support set and the keyframe representation features of the query set to obtain the similarity calculation results between the features of the support set and the query set.

[0095] Step S53 calculates the similarity between the query set samples and the aggregated feature vectors of the support set to measure the distance relationship between different samples in the feature space. The similarity measure can be cosine similarity, Euclidean distance, or attention-based weighted relevance index to achieve a unified measure between cross-modal features.

[0096] S54. Input the similarity calculation result into the pre-built loss function to generate a loss value for model parameter optimization.

[0097] In step S54, the similarity calculation result is input into a predefined loss function to generate the loss value required for model optimization. The loss function reflects the model's discrimination error under the current task, provides gradient feedback for the fusion network and attention mechanism, and guides parameter updates.

[0098] S55. Update the parameters in the cross-modal attention fusion process based on the loss value.

[0099] In step S55, the parameters of the cross-modal attention fusion module are updated based on the loss feedback results, enabling the model to adaptively adjust feature associations and modal weight distributions in subsequent tasks. Through continuous iteration, the system learns the optimal combination relationship between cross-modal features from task feedback, improving generalization performance and stability.

[0100] As an example, after completing the dynamic fusion phase, the model enters the meta-learning optimization phase. This phase builds a meta-learning optimizer based on the Prototypical Networks framework to perform contrastive learning and temporal consistency constraints on the fused features under few-shot conditions, achieving adaptive optimization of the model parameters. Specifically, in the meta-training phase, each task contains a support set. and query set For the keyframe classification task, the prototype network computes the category prototype: ,in, Represented as a DFT encoder, Indicate category Number of supported samples (keyframes / non-keyframes) Let represent the input feature vector of the sample, and y represent the artificial label corresponding to the input sample. The prediction of the query sample is obtained by Euclidean distance. ( )calculate: Employing contrastive loss to align cross-modal features: in Represents cosine similarity. Represents the temperature parameter. Temporal consistency loss constrains continuous frame prediction, where t is time t: The total loss function is: . These represent manually set weights, which allow the model to focus on learning specific tasks.

[0101] Through steps S51 to S55, an adaptive optimization mechanism based on a meta-learning structure is implemented, enabling the system to dynamically update and continuously optimize parameters during cross-modal attention fusion based on the similarity calculation results between the support set and the query set. This mechanism allows the model to quickly learn the correlation patterns between modalities from a small number of representative samples, rather than relying on large-scale sample training, thus achieving cross-task transfer and rapid convergence. Through similarity-driven loss feedback updates, the model continuously strengthens the representation ability of highly correlated modalities in the feature space and weakens the influence of redundant or noisy modalities, thereby effectively improving the discriminative power and generalization performance of the fused features. This scheme enables the system to have self-learning and self-correction capabilities, flexibly adapting to changes in data distribution under different application scenarios, and providing a reliable optimization mechanism for improving the accuracy of keyframe feature extraction and the long-term stability of cross-modal fusion.

[0102] As an example, please refer to Figure 5 , Figure 5 This is an example diagram of a data processing flow in the keyframe feature extraction method based on cross-modal attention provided in the embodiments of this application, such as... Figure 5As shown, firstly, a multimodal feature encoder extracts visual, audio, and motion modal features from the input video. Specifically, visual features are encoded using a VisionTransformer network for spatiotemporal features, audio features are extracted using a Wav2Vec 2.0 network for acoustic features, and motion features are modeled using a 3D convolutional neural network (3D CNN) for optical flow patterns, thus obtaining visual, audio, and motion feature representations respectively. Then, these three types of modal features are input into a dynamic cross-modal attention mechanism module, performing cross-modal feature fusion within a unified feature space. This module progressively achieves information interaction and association enhancement between modalities through a multi-level fusion structure, specifically including an early fusion layer, a deep fusion layer, and a late fusion layer. Early fusion is used for preliminary association modeling of low-level modal features, deep fusion is used to capture complex interaction relationships between modalities, and late fusion is used to synthesize the results of multiple attention layers to form the final fused feature representation. Finally, the fused keyframe representation features are input into prototypical networks to construct a meta-learning task for adaptive model optimization. This stage constructs support and query sets, calculates the similarity between features, and updates the parameters of the cross-modal attention module based on loss feedback, thereby achieving self-learning and rapid generalization of the feature space. Through the aforementioned process, the method in this embodiment can establish highly correlated dynamic interaction relationships between multimodal signals, obtaining semantically consistent and highly discriminative keyframe features, providing a high-quality feature foundation for subsequent video content analysis and intelligent recognition.

[0103] As another example, taking the analysis of teaching videos for the elderly as an example, the input video content includes visual actions such as doctors conducting physical examinations and viewing films, accompanied by audio statements such as "step-by-step explanation" (e.g., "Now listen to the heart") and motion optical flow features generated by hand operations. Traditional single-modal video analysis methods, because they rely only on visual changes, often miss moments with small visual changes but key semantics, such as auscultation. The embodiments of this application can enhance the importance of such moments through audio features, enabling the model to identify the semantic key frame of "auscultation". The key frame sequence output by the Dynamic Fusion Transformer (DFT) module in this example includes: (1) the examination action stage, corresponding to the activation of high motion features; (2) the auscultation moment, where the audio modality triggers gating weight adjustment; and (3) the finished product display stage, where visual saliency and audio summary statements jointly enhance the weight. This example verifies the significant improvement of multimodal fusion in the completeness of key frame extraction, especially when the single modality signal is weak, other modalities can provide effective complementary information, thereby ensuring the coherence and accuracy of temporal recognition.

[0104] The keyframe feature extraction method based on cross-modal attention provided in this application achieves deep correlation modeling and adaptive weight adjustment of visual, audio, and motion information through a collaborative design of multimodal feature encoding, dynamic fusion Transformer, and meta-learning optimization. This method can dynamically capture complementary relationships between modalities within a unified feature space, significantly improving the discriminativeness and temporal consistency of keyframe representations. It achieves adaptive balancing of modal contributions through a gating mechanism, effectively avoiding information bias caused by the failure of a single modality. Combined with a prototype network-based meta-learning optimization framework, the model possesses rapid generalization capabilities in tasks with few samples and across different scenarios. The overall solution demonstrates higher robustness and generalization performance in multimodal scenarios such as video understanding, financial risk identification, and medical image analysis.

[0105] Please continue reading. Figure 6 , Figure 6 This is a schematic diagram of the system structure of the keyframe feature extraction device based on cross-modal attention provided in the embodiments of this application, as shown below. Figure 6 As shown, the keyframe feature extraction device 60 based on cross-modal attention includes: a multimodal data decomposition module 61, a multimodal encoder module 62, a fusion feature generation module 63, a keyframe representation feature module 64, and a meta-learning optimization module 65.

[0106] The multimodal data decomposition module 61 is specifically used to perform multimodal data decomposition on the received input video to obtain video frame sequences, audio signals, and motion optical flow sequences.

[0107] The multimodal encoder module 62 is specifically used to extract features from the video frame sequence, the audio signal, and the motion optical flow sequence using a multimodal encoder, respectively, to obtain corresponding visual features, audio features, and motion features.

[0108] The fusion feature generation module 63 is specifically used to map the visual features, audio features and motion features to a unified feature space, perform cross-modal attention fusion in the unified feature space, calculate the correlation between modalities, and generate fusion features based on the correlation. The keyframe representation feature module 64 is specifically used to dynamically adjust the modal weight coefficients of the fused features through a gated adaptive weighting mechanism to obtain adaptively weighted keyframe representation features. The meta-learning optimization module 65 is specifically used to construct a meta-learning task based on the keyframe representation features, calculate the construction loss by the similarity between the support set and the query set, and update the parameters in the cross-modal attention fusion process according to the construction loss.

[0109] As an optional implementation, the fusion feature generation module 63 is further specifically used to perform linear transformation and projection processing on the visual features, audio features, and motion features respectively, so that the visual features, audio features, and motion features are mapped to a feature space of the same dimension to obtain a unified-dimensional modal representation; to process the unified-dimensional modal representation through a multi-head attention mechanism, and to calculate the query vector, key vector, and value vector respectively; to construct an attention weight matrix based on the query vector and key vector, and to perform a weighted summation of the attention weight matrix and the value vector to obtain the output features of each attention head; to perform concatenation and linear transformation processing on the output features of each attention head to obtain a preliminary fused cross-modal feature representation; and to perform normalization processing on the cross-modal feature representation to limit the cross-modal feature representation to a preset feature distribution space to obtain the fused attention representation.

[0110] As an optional implementation, the keyframe representation feature module 64 is further specifically used to initialize the weight parameters of the fused features to generate initial weight coefficients for each modality of visual features, audio features, and motion features; input the initial weight coefficients to a preset gating function for activation operation to obtain modality weight control coefficients; perform element-wise weighting processing on the fused features according to the modality weight control coefficients to obtain weighted modality fused features; and perform normalization scaling processing on the weighted modality fused features to obtain the keyframe representation features.

[0111] As an optional implementation, the meta-learning optimization module 65 is further configured to perform set partitioning processing on the keyframe representation features according to a preset sample partitioning strategy to obtain a support set and a query set for meta-learning training; perform feature aggregation operation on the keyframe representation features in the support set to generate an aggregated representation vector corresponding to the support set; calculate the similarity between the aggregated representation vector of the support set and the keyframe representation features of the query set to obtain a similarity calculation result between the features of the support set and the query set; input the similarity calculation result into a pre-constructed loss function to generate a loss value for model parameter optimization; and update the parameters in the cross-modal attention fusion process according to the loss value.

[0112] As an optional implementation, the multimodal data decomposition module 61 is further specifically used to decode and extract frames from the input video according to a preset frame sampling rule to obtain a continuous video frame sequence; to separate and resample the audio signal of the input video to obtain an audio signal sequence aligned with the time span of the video frame sequence; to perform inter-frame displacement calculation based on the pixel changes of adjacent frames in the video frame sequence to generate a motion optical flow sequence for characterizing motion changes; and to perform timestamp alignment or sampling index alignment processing on the video frame sequence, the audio signal sequence, and the motion optical flow sequence to obtain a time-corresponding multimodal data sequence.

[0113] As an optional implementation, the multimodal encoder module 62 is further configured to input the video frame sequence into a preset visual coding network, perform spatiotemporal feature coding operations to obtain visual features corresponding to the video frame sequence; input the audio signal sequence into a preset audio coding network, perform time-domain or frequency-domain feature coding operations to obtain audio features corresponding to the audio signal sequence; input the motion optical flow sequence into a preset motion coding network, perform inter-frame motion mode coding operations to obtain motion features corresponding to the motion optical flow sequence; and perform feature tensor dimension alignment processing on the visual features, the audio features, and the motion features to obtain the dimension-consistent visual features, audio features, and motion features.

[0114] As an optional implementation, the fusion feature generation module 63 is further specifically used to extract statistical parameters describing the input distribution state of visual features, audio features, and motion features based on visual features, audio features, and motion features in a unified feature space; generate an association control signal for adjusting the interaction relationship of visual features, audio features, and motion features through the statistical parameters; set the association control signal as a correction parameter during the calculation of the association relationship of visual features, audio features, and motion features, and update the calculation of the association relationship through the correction parameter; perform a fusion operation according to the corrected association relationship to generate a fusion feature with dynamic association adjustment.

[0115] It should be noted that the aforementioned keyframe feature extraction device based on cross-modal attention can execute the keyframe feature extraction method based on cross-modal attention provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in the embodiments of the keyframe feature extraction device based on cross-modal attention can be found in the keyframe feature extraction method based on cross-modal attention provided in the embodiments of this application.

[0116] Figure 7 This is a schematic diagram of the hardware structure of an electronic device that performs a keyframe feature extraction method based on cross-modal attention, as provided in an embodiment of this application. Figure 7 As shown, the electronic device 700 includes: One or more processors 710 and memory 720, Figure 7 Take the 710 processor as an example.

[0117] The processor 710 and memory 720 can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0118] The memory 720, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the keyframe feature extraction method based on cross-modal attention in the embodiments of this application. The processor 710 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 720, thereby implementing the keyframe feature extraction method based on cross-modal attention in the above-described method embodiments.

[0119] The memory 720 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the cross-modal attention-based keyframe feature extraction device, etc. Furthermore, the memory 720 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 720 may optionally include memory remotely located relative to the processor 710, and these remote memories may be connected to the cross-modal attention-based keyframe feature extraction device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0120] The one or more modules are stored in the memory 720. When executed by the one or more processors 710, they perform the keyframe feature extraction method based on cross-modal attention in any of the above method embodiments. For example, they perform the methods described above. Figure 2 Method steps S1 to S5, Figure 3 Method steps S31 to S35, Figure 4 Steps S36 to S39 in the method are implemented. Figure 6 The functions of modules 61-65 in the document.

[0121] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0122] This application provides a non-volatile computer-readable storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 7 One of the processors 710 can enable the one or more processors to execute the keyframe feature extraction method based on cross-modal attention in any of the above method embodiments, for example, to perform the above-described... Figure 2 Method steps S1 to S5, Figure 3 Method steps S31 to S35, Figure 4 Steps S36 to S39 in the method are implemented. Figure 6 The functions of modules 61-65 in the document.

[0123] This application provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions that, when executed by an electronic device, enable the electronic device to perform the keyframe feature extraction method based on cross-modal attention in any of the above method embodiments, for example, to perform the above-described method. Figure 2 Method steps S1 to S5, Figure 3 Method steps S31 to S35, Figure 4 Steps S36 to S39 in the method are implemented. Figure 6 The functions of modules 61-65 in the document.

[0124] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A keyframe feature extraction method based on cross-modal attention, characterized in that, include: The received input video is decomposed into multimodal data to obtain video frame sequences, audio signals, and motion optical flow sequences; A multimodal encoder is used to extract features from the video frame sequence, the audio signal, and the motion optical flow sequence, respectively, to obtain corresponding visual features, audio features, and motion features. The visual features, audio features, and motion features are mapped to a unified feature space, and cross-modal attention fusion is performed within the unified feature space. The correlation between modalities is calculated, and fused features are generated based on the correlation. The modal weight coefficients of the fused features are dynamically adjusted by a gated adaptive weighting mechanism to obtain adaptively weighted keyframe representation features. A meta-learning task is constructed based on the keyframe representation features. The construction loss is calculated by the similarity between the support set and the query set. The parameters in the cross-modal attention fusion process are updated according to the construction loss.

2. The keyframe feature extraction method based on cross-modal attention according to claim 1, characterized in that, The step of mapping the visual features, audio features, and motion features to a unified feature space, and performing cross-modal attention fusion within the unified feature space, includes: The visual features, audio features, and motion features are subjected to linear transformation and projection processing respectively, so that the visual features, audio features, and motion features are mapped to the feature space of the same dimension, thereby obtaining a modal representation of the same dimension. The modal representation of the unified dimension is processed by a multi-head attention mechanism to calculate the query vector, key vector, and value vector respectively. An attention weight matrix is ​​constructed based on the query vector and the key vector. The attention weight matrix is ​​then weighted and summed with the value vector to obtain the output features of each attention head. The output features of each attention head are concatenated and linearly transformed to obtain a preliminary fused cross-modal feature representation; Normalization is performed on the cross-modal feature representation to confine it to a preset feature distribution space, thereby obtaining the fused attention representation.

3. The keyframe feature extraction method based on cross-modal attention according to claim 1, characterized in that, The step of dynamically adjusting the modal weight coefficients of the fused features through a gated adaptive weighting mechanism to obtain adaptively weighted keyframe representation features includes: Initialize the weight parameters of the fused features to generate initial weight coefficients for each modality of visual features, audio features, and motion features; The initial weight coefficients are input into a preset gating function for activation calculation to obtain the modal weight control coefficients; The fusion features are weighted element-wise according to the modal weight control coefficients to obtain the weighted modal fusion features; The weighted modality fusion features are normalized and scaled to obtain the keyframe representation features.

4. The keyframe feature extraction method based on cross-modal attention according to claim 1, characterized in that, The steps of constructing a meta-learning task based on the keyframe representation features, calculating the construction loss through the similarity between the support set and the query set, and updating the parameters in the cross-modal attention fusion process based on the construction loss include: The keyframe representation features are partitioned according to a preset sample partitioning strategy to obtain a support set and a query set for meta-learning training. Perform feature aggregation operations on the keyframe representation features in the support set to generate an aggregated representation vector corresponding to the support set; The similarity between the aggregated representation vector of the support set and the keyframe representation features of the query set is calculated to obtain the similarity calculation result between the features of the support set and the query set. Input the similarity calculation result into the pre-built loss function to generate a loss value for model parameter optimization; The parameters in the cross-modal attention fusion process are updated based on the loss value.

5. The keyframe feature extraction method based on cross-modal attention according to claim 1, characterized in that, The step of performing multimodal data decomposition on the received input video to obtain a video frame sequence, audio signal, and motion optical flow sequence includes: The input video is decoded and frame-sampling is performed according to a preset frame sampling rule to obtain a continuous video frame sequence; The audio signal of the input video is separated and resampled to obtain an audio signal sequence aligned with the time span of the video frame sequence; Based on the pixel changes of adjacent frames in the video frame sequence, inter-frame displacement calculation is performed to generate a motion optical flow sequence to characterize motion changes; The video frame sequence, the audio signal sequence, and the motion optical flow sequence are time-stamp aligned or sample index aligned to obtain a time-series corresponding multimodal data sequence.

6. The keyframe feature extraction method based on cross-modal attention according to claim 1, characterized in that, The step of using a multimodal encoder to extract features from the video frame sequence, the audio signal, and the motion optical flow sequence to obtain corresponding visual features, audio features, and motion features includes: The video frame sequence is input into a preset visual coding network, and spatiotemporal feature coding operation is performed to obtain the visual features corresponding to the video frame sequence. The audio signal sequence is input into a preset audio coding network, and time-domain or frequency-domain feature coding operations are performed to obtain the audio features corresponding to the audio signal sequence. The motion optical flow sequence is input into a preset motion coding network, and inter-frame motion mode coding is performed to obtain the motion features corresponding to the motion optical flow sequence. The visual features, audio features, and motion features are subjected to feature tensor dimension alignment processing to obtain the visual features, audio features, and motion features after dimension consistency processing.

7. The keyframe feature extraction method based on cross-modal attention according to claim 1, characterized in that, The step of calculating the correlation between modalities and generating fused features based on the correlation includes: Based on visual features, audio features, and motion features in a unified feature space, statistical parameters are extracted to describe the input distribution state of the visual features, audio features, and motion features. The statistical parameters are used to generate correlation control signals for adjusting the interaction relationship between the visual features, audio features, and motion features; In the process of calculating the correlation between the visual features, audio features and motion features, the correlation control signal is set as a correction parameter, and the calculation of the correlation is updated through the correction parameter; Perform fusion operations based on the corrected association relationships to generate fusion features with dynamic association adjustment.

8. A keyframe feature extraction device based on cross-modal attention, characterized in that, include: The multimodal data decomposition module is used to perform multimodal data decomposition on the received input video to obtain video frame sequences, audio signals, and motion optical flow sequences. The multimodal encoder module is used to extract features from the video frame sequence, the audio signal, and the motion optical flow sequence using a multimodal encoder, respectively, to obtain corresponding visual features, audio features, and motion features; The fusion feature generation module is used to map the visual features, audio features and motion features to a unified feature space, perform cross-modal attention fusion in the unified feature space, calculate the correlation between modalities, and generate fusion features based on the correlation. The keyframe representation feature module is used to dynamically adjust the modal weight coefficients of the fused features through a gated adaptive weighting mechanism to obtain adaptively weighted keyframe representation features. The meta-learning optimization module is used to construct a meta-learning task based on the keyframe representation features, calculate the construction loss by the similarity between the support set and the query set, and update the parameters in the cross-modal attention fusion process according to the construction loss.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the keyframe feature extraction method based on cross-modal attention as described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by an electronic device, cause the electronic device to perform the keyframe feature extraction method based on cross-modal attention as described in any one of claims 1-7.

Images