Key frame feature extraction method and device for multi-modal data, equipment and medium

Abstract

The invention relates to the technical field of data analysis, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses a key frame feature extraction method, device, equipment and medium for multi-modal data, and the method comprises the following steps: carrying out differentiable neural architecture search on a plurality of multi-modal original data streams to obtain multi-modal undetermined features; carrying out importance analysis on the multi-modal undetermined features to obtain importance scores, screening out an effective modal feature set from the multi-modal undetermined features, carrying out dynamic sparse connection on the effective modal feature set to obtain fused modal features, carrying out time sequence attention distillation on the fused modal features to obtain frame-level attention weights, and carrying out time sequence attention distillation on the fused modal features to obtain frame-level attention weights; and carrying out frame-level key analysis on the multi-modal original data stream to obtain a key frame of the multi-modal original data stream. According to the method, the unified modeling capability of the multi-modal time sequence is improved, dynamic adjustment of the modal weight and redundant information compression are realized, and the overall processing efficiency is remarkably improved while the calculation burden is reduced.

Description

Methods, apparatus, equipment and media for keyframe feature extraction from multimodal data Technical Field

[0001] This invention relates to the field of data analysis technology, and in particular to a method, apparatus, device, and medium for extracting keyframe features from multimodal data. Background Technology

[0002] Multimodal perception technology has been widely used in the field of intelligent analysis, but the differences in structure, distribution, and temporal characteristics of different modalities make unified modeling complex. Existing methods generally rely on fixed feature extraction and fusion structures, making it difficult to dynamically adjust the modality contribution based on changes in input content. Furthermore, multiple feature paths need to be retained simultaneously during the inference stage, resulting in a heavy computational burden. At the same time, multimodal temporal data generally contain a large number of redundant frames, and keyframe selection and feature learning are independent of each other, lacking an end-to-end correlation optimization mechanism, leading to insufficient utilization of temporal information and limited model efficiency.

[0003] In the healthcare field, when faced with heterogeneous data such as images, monitoring signals, and electronic medical records, the modal contribution often cannot be dynamically adjusted according to changes in the patient's condition due to the fixed fusion method and the lack of adaptive structure. At the same time, a large amount of redundant time-series data cannot be effectively filtered out, which limits the computational load, response speed, and anomaly identification accuracy of real-time monitoring and clinical decision-making systems.

[0004] In the fintech business, when dealing with multi-source data such as transaction time series, market indicators and user behavior, it is difficult to adapt to the characteristics of frequent data fluctuations and rapid changes in modal importance. Fixed-structure feature pipelines often require a large amount of parallel computing and are difficult to effectively remove low-value time series segments, resulting in excessive resource consumption and increased response latency in risk control models under high concurrency environments, and easy loss of key dynamic signals in complex behavior recognition.

[0005] Therefore, existing multimodal time series processing methods have limited capabilities in unified modeling, dynamic adjustment of modal weights, and compression of redundant information, resulting in insufficient utilization of key information, high computational burden, and limited overall efficiency. Summary of the Invention

[0006] This invention provides a method, apparatus, device, and medium for key frame feature extraction of multimodal data. Its main purpose is to solve the problems that existing multimodal time series processing methods have limited capabilities in unified modeling, dynamic adjustment of modality weights, and compression of redundant information, resulting in insufficient utilization of key information, high computational burden, and limited overall efficiency.

[0007] In a first aspect, to achieve the above objective, the present invention provides a method for keyframe feature extraction from multimodal data, comprising: acquiring several multimodal raw data streams; performing a differentiable neural architecture search on the multimodal raw data streams to obtain multimodal undetermined features; performing importance analysis on the multimodal undetermined features to obtain importance scores, and selecting an effective modal feature set from the multimodal undetermined features based on the importance scores; performing dynamic sparse connections on the effective modal feature set to obtain fused modal features; performing temporal attention distillation on the fused modal features to obtain frame-level attention weights; and performing frame-level keyframe analysis on the multimodal raw data streams based on the frame-level attention weights to obtain keyframes of the multimodal raw data streams.

[0008] Secondly, the present invention also provides a keyframe feature extraction device for multimodal data, comprising: a feature search module for acquiring several multimodal raw data streams, performing a differentiable neural architecture search on the multimodal raw data streams to obtain multimodal undetermined features; an importance analysis module for performing importance analysis on the multimodal undetermined features to obtain importance scores, and filtering out a set of effective modal features from the multimodal undetermined features based on the importance scores; a feature connection module for performing dynamic sparse connection on the set of effective modal features to obtain fused modal features; an attention distillation module for performing temporal attention distillation on the fused modal features to obtain frame-level attention weights; and a keyframe analysis module for performing frame-level keyframe analysis on the multimodal raw data streams based on the frame-level attention weights to obtain keyframes of the multimodal raw data streams.

[0009] Thirdly, the present invention also provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the aforementioned method for extracting keyframe features from multimodal data.

[0010] Fourthly, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the above-described method for keyframe feature extraction of multimodal data.

[0011] This invention acquires several multimodal raw data streams, performs a differentiable neural architecture search on these streams to obtain undetermined multimodal features, and adaptively learns the network structure and feature extraction path best suited to the characteristics of different modalities without human intervention. It then performs importance analysis on these undetermined features to obtain importance scores, and selects a set of effective modal features based on these scores. This effectively removes redundant and noisy features while retaining key information, highlighting modal features that contribute significantly to the target task. Finally, it performs dynamic sparse connections on the set of effective modal features to obtain fused modal features, fully utilizing complementary multimodal information while avoiding redundant computation and noise interference caused by fully connected fusion. The perturbation process focuses the feature interaction relationships more on the key connection paths that substantially contribute to the target task. Temporal attention distillation is performed on the fused modal features to obtain frame-level attention weights. While maintaining key temporal discrimination information, the ability to model complex temporal sequences is transferred from a high-complexity model to a lightweight model, allowing for more efficient and stable focus on key time frames that have a decisive impact on the task results. Based on the frame-level attention weights, frame-level criticality analysis is performed on the multimodal raw data stream to quickly focus on the time segments with the most informational value to the target task in massive continuous data, obtaining the key frames of the multimodal raw data stream. By improving the unified modeling capability of multimodal temporal sequences, dynamic adjustment of modality weights and compression of redundant information are achieved, significantly improving overall processing efficiency while reducing computational burden. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 is a schematic diagram of an application environment for a keyframe feature extraction method for multimodal data according to an embodiment of the present invention; Figure 2 is a schematic flowchart of a keyframe feature extraction method for multimodal data according to an embodiment of the present invention; Figure 3 is a schematic flowchart of the feature importance analysis process in a keyframe feature extraction method for multimodal data according to an embodiment of the present invention; Figure 4 is a schematic diagram of a module of a keyframe feature extraction device for multimodal data according to an embodiment of the present invention; Figure 5 is a schematic diagram of the structure of an electronic device for implementing a keyframe feature extraction method for multimodal data according to an embodiment of the present invention; Figure 6 is another schematic diagram of the structure of an electronic device for implementing a keyframe feature extraction method for multimodal data according to an embodiment of the present invention.

[0014] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0015] To enable those skilled in the art to better understand the technical solutions of this disclosure, and to fully understand and implement the process of how this disclosure applies technical means to solve technical problems and achieve corresponding technical effects, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. The embodiments of this disclosure and the various features within them can be combined with each other without conflict, and the resulting technical solutions are all within the protection scope of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort should fall within the protection scope of this disclosure.

[0016] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but includes other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0017] This application provides a method for extracting keyframe features from multimodal data. The execution subject of this method includes, but is not limited to, at least one electronic device that can be configured to execute the device provided in this application, such as a server or a terminal. In other words, the method for extracting keyframe features from multimodal data can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0018] This invention provides a keyframe feature extraction method for multimodal data, applicable to the application environment shown in Figure 1. The client communicates with the server via a network. The server can obtain several multimodal raw data streams from the client, perform a differentiable neural architecture search on these streams to obtain undetermined multimodal features. This method adaptively learns the network structure and feature extraction path best suited to the characteristics of different modal data without human intervention. Importance analysis is performed on the undetermined multimodal features to obtain importance scores. Based on these scores, a set of effective modal features is selected, effectively removing redundant and noisy features while retaining key information, highlighting modal features that contribute significantly to the target task. Dynamic sparse connections are then performed on the effective modal feature set to obtain fused modal features. This fully utilizes complementary multimodal information while avoiding redundant computation and noise interference from fully connected fusion, making feature interaction relationships more robust. The system focuses on key connection paths that substantially contribute to the target task. Temporal attention distillation is performed on the fused modal features to obtain frame-level attention weights. While maintaining key temporal discrimination information, the system transfers complex temporal modeling capabilities from high-complexity models to lightweight models, enabling more efficient and stable focus on key time frames that decisively influence task results. Based on the frame-level attention weights, frame-level criticality analysis is performed on the multimodal raw data stream. This quickly focuses on the time segments with the most informational value to the target task from massive continuous data, obtaining the key frames of the multimodal raw data stream. By improving the unified modeling capability of multimodal temporal sequences, dynamic adjustment of modality weights and compression of redundant information are achieved, significantly improving overall processing efficiency while reducing computational burden. Finally, the key frames of the multimodal raw data stream are output and fed back to the user client. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0019] The following explanation of this invention relates to the present invention, which adaptively learns the optimal feature extraction structure from multimodal raw data, effectively filters and sparsely fuses features from different modalities, and further combines a temporal attention distillation mechanism to accurately characterize the importance of key time frames. This reduces feature redundancy and computational complexity while fully mining key information in multimodal time series data, achieving efficient and accurate extraction of key frames and improving the efficiency, robustness, and interpretability of multimodal time series analysis.

[0020] Referring to Figure 2, a flowchart illustrating a keyframe feature extraction method for multimodal data according to an embodiment of the present invention is shown. In this embodiment, the keyframe feature extraction method for multimodal data includes: S1, acquiring several multimodal raw data streams, performing a differentiable neural architecture search on the multimodal raw data streams to obtain multimodal features to be determined.

[0021] In this embodiment of the invention, the multimodal raw data stream is a collection of raw data synchronously acquired by multiple different types of sensors or data sources and organized in chronological order at the same time or in the same task scenario, such as images, videos, voice, radar signals or other sensor information.

[0022] Differentiable neural architecture search is a gradient optimization-based method that parameterizes the neural network structure and introduces it into a continuously differentiable search space, automatically learning the optimal network structure through backpropagation. Multimodal undetermined features are candidate feature representations obtained after performing differentiable neural architecture search on the original multimodal data stream, but which have not yet undergone final screening.

[0023] In specific healthcare scenarios, it can be applied to the intelligent analysis and assisted diagnosis of multi-source medical data. For example, in intensive care or chronic disease management scenarios, it can simultaneously acquire multimodal raw data streams such as patients' medical image frames, vital sign time-series data, and biochemical test results. By searching for network structures that automatically adapt to different modalities of data through a differentiable neural architecture, it can uncover potential correlations between image evolution features and changes in physiological indicators, forming multimodal undetermined features for disease risk assessment, disease progression prediction, or treatment effect monitoring, thereby improving the accuracy and personalization of diagnostic decisions.

[0024] In specific fintech scenarios, it can be applied to transaction behavior analysis and risk control. For example, in anti-fraud or credit assessment scenarios, it can acquire multimodal raw data streams such as user transaction sequences, device fingerprints, behavioral trajectories, and account operation logs. Through differentiable neural architecture search, it can automatically construct feature extraction structures adapted to different data characteristics, extract multimodal undetermined features such as abnormal transaction rhythms, behavioral pattern deviations, and device correlation characteristics, and use them to identify potential fraudulent behavior, assess user credit risk, or achieve real-time risk control early warning, thereby improving the security and stability of the financial system.

[0025] In this embodiment of the invention, the step of performing a differentiable neural architecture search on the multimodal raw data stream to obtain multimodal undetermined features includes: normalizing the video frames of the multimodal raw data stream to obtain a normalized data stream; generating a set of differentiable candidate operators based on the normalized data stream; constructing an architecture search space based on the set of differentiable candidate operators; searching the set of differentiable candidate operators using the architecture search space to obtain architecture selection parameters for each candidate operator in the set of differentiable candidate operators; sorting each candidate operator by weight based on the architecture selection parameters, and selecting target operator combinations based on the sorting results; performing forward propagation on the normalized data stream of each modality using the target operator combinations to obtain a set of undetermined feature vectors for each modality in the normalized data stream; and summarizing the sets of undetermined feature vectors for all modalities into multimodal undetermined features.

[0026] In this embodiment of the invention, the video frames of the multimodal raw data stream are subjected to size alignment and resolution unification processing to eliminate spatial scale differences caused by different acquisition devices or encoding methods; the pixel intensity of each video frame is numerically normalized to map the original pixel values ​​of the video frame to a preset standard range, and the timestamp is aligned and compensated simultaneously to finally form a normalized data stream that is consistent in spatial scale, numerical distribution and time dimension.

[0027] In this embodiment of the invention, various differentiable feature extraction operators are constructed for the data characteristics of different modalities in multimodal data streams, including different convolution scales, temporal modeling operators or cross-modal interaction operators. The parameterization method ensures that each differentiable feature extraction operator can participate in gradient update during training, thereby forming a set of differentiable candidate operators covering multiple feature expression capabilities.

[0028] Furthermore, the candidate operators in the set of differentiable candidate operators are organized according to preset connection relationships, hierarchical structures and combination rules to form a continuous searchable space containing multiple operator types, connection paths and combination methods, namely the architecture search space.

[0029] In this embodiment of the invention, a set of differentiable candidate operators is embedded into the architecture search space. By introducing continuously differentiable architecture selection parameters, the selection probability of different candidate operators is modeled. During the modeling process, the architecture parameters and network weight parameters are jointly optimized. Driven by the backpropagation mechanism, the contribution of each candidate operator is updated by gradient, thereby gradually adjusting the architecture selection parameters of each candidate operator. Finally, the architecture selection parameters corresponding to each candidate operator in the set of differentiable candidate operators are obtained, which are used to characterize their priority and importance in the target network structure.

[0030] In this embodiment of the invention, the weight of each candidate operator is quantitatively evaluated based on the architecture selection parameters, and the candidate operators are sorted by weight according to a preset sorting rule. Based on the sorting results, combined with screening strategies such as threshold constraints, quantity constraints, or cumulative contribution constraints, candidate operators with higher weights and significant contributions to task performance are retained first, while operators with lower weights or redundancy are removed. Finally, the selected and retained candidate operators are combined to form a target operator combination that meets the performance and complexity requirements.

[0031] In this embodiment of the invention, normalized data streams are sequentially input into a network structure composed of target operators. Forward propagation yields the response results of each modality at different levels. Multidimensional feature representations reflecting temporal variations, spatial structure, and modal characteristics are extracted, generating a corresponding set of undetermined feature vectors. Further, the undetermined feature vectors output by each modality are uniformly integrated according to preset concatenation, weighting, or alignment rules to obtain multimodal undetermined features containing cross-modal complementary information.

[0032] In this embodiment of the invention, for each mode of the normalized data stream, a structure is constructed containing... The architecture search space for candidate operators is optimized using gradient descent.

[0033] in, Indicates the first Time of the first The undetermined eigenvectors of each modality This represents the total number of candidate operators in the set of differentiable candidate operators. It is the first A number of candidate operators Indicates the first The th mode Architecture selection parameters for candidate operators Indicates the temperature coefficient. Indicates the first The th mode Architecture selection parameters for candidate operators Indicates the first One modality, Indicates the first time, Indicates the first In the architecture search space of the modality, the first A number of candidate operators Indicates the first Time of the first Normalized data stream of each modality, This indicates exponentiation.

[0034] This invention searches for differentiable neural architectures from several frames of multimodal raw data streams. It can adaptively learn the network structure and feature extraction path that best suits the characteristics of different modal data without human intervention, thereby fully exploring the complementary information and temporal correlation between the modalities.

[0035] S2. Perform importance analysis on the multimodal features to be determined, obtain importance scores, and select a set of effective modal features from the multimodal features to be determined based on the importance scores.

[0036] In this embodiment of the invention, the importance score is a numerical index assigned to the undetermined features of each modality using an importance analysis method, reflecting the relative contribution of the undetermined features. The effective modal feature set is a subset of features selected from the undetermined features of multimodalities based on the importance score, which has a high contribution to the target task and low noise and redundancy.

[0037] In specific healthcare scenarios, it can be used for feature optimization and decision support based on multi-source clinical data. For example, in early disease screening or prognostic assessment, importance analysis can be performed on multimodal features from medical imaging, vital sign monitoring, and laboratory indicators to quantify the contribution of different features to disease identification or risk prediction. Based on this, a set of effective modal features that significantly impact diagnostic results can be selected, thereby reducing redundant information interference and highlighting key physiological indicators and imaging features.

[0038] In specific fintech scenarios, it can be used for risk identification and refined risk control modeling. For example, in the process of anti-fraud or credit assessment, importance analysis can be performed on multimodal undetermined features such as transaction behavior characteristics, account operation characteristics, and device and environmental characteristics to identify key behavioral patterns and related features that contribute significantly to risk assessment. Based on this, an effective set of modal features can be constructed for real-time risk scoring and strategy triggering, thereby reducing model complexity while ensuring identification accuracy.

[0039] Figure 3 is a flowchart illustrating the feature importance analysis process in a keyframe feature extraction method for multimodal data according to an embodiment of the present invention.

[0040] In this embodiment of the invention, the step of performing importance analysis on the multimodal undetermined features to obtain an importance score includes: constructing a lightweight gating network based on the multimodal undetermined features; using the lightweight gating network to perform modal response analysis on the set of undetermined feature vectors for each modality in the multimodal undetermined features to obtain the original importance response value for each modality; and normalizing the original importance response value to obtain an importance score.

[0041] In this embodiment of the invention, dimension alignment and feature compression are performed on the undetermined features of each modality to reduce computational complexity and eliminate scale differences. Subsequently, a lightweight gating network consisting of a small number of fully connected layers or one-dimensional convolutional layers is constructed to jointly model the undetermined features of multiple modalities and learn the dynamic weights of each modality feature in the current task scenario. The activation intensity of different modal features is adaptively adjusted through a gating function, and finally a lightweight gating network with a compact structure, controlled parameter scale, and modality selection capability is formed to achieve efficient fusion and control of multimodal features.

[0042] Furthermore, the undetermined feature vectors of each modality are input into a lightweight gating network, and the gating output is obtained through forward propagation. This quantifies the activation intensity and contribution of different modal features in the current task, thereby generating the corresponding original importance response value for each modality.

[0043] In this embodiment of the invention, the original importance response value corresponding to each mode is scaled and adjusted according to a preset normalization rule, so that the original importance response value is mapped to a unified numerical range, and the importance results of different modes are comparable, and finally an importance score reflecting the relative contribution of each mode is formed.

[0044] In this embodiment of the invention, the step of selecting a set of effective modal features from the multimodal undetermined features based on the importance scores includes: randomly perturbing the importance scores to obtain a set of perturbed scores; smoothing the distribution of the set of perturbed scores using a preset temperature parameter to form a set of smoothed scores; sorting each modality of the multimodal undetermined features in descending order of importance based on the set of smoothed scores to obtain sorted undetermined features; and selecting a preset number of effective modal features from the sorted undetermined features.

[0045] In this embodiment of the invention, while keeping the relative structure of the original importance scores unchanged, a random perturbation term with controlled amplitude is introduced to randomly adjust the importance score corresponding to each mode, so as to simulate uncertainty and enhance the robustness of the sorting process to local fluctuations, thereby forming a set of perturbation scores containing multiple possible changes.

[0046] Furthermore, by introducing a temperature parameter to adjust the numerical distribution of the perturbation score, the sharpness of extreme scores is reduced, making the overall distribution of the importance score of each mode smoother and more stable, thereby avoiding a single mode from occupying too high a weight due to accidental fluctuations.

[0047] In this embodiment of the invention, the smoothed importance score is used as the evaluation criterion to compare and rank the undetermined features corresponding to different modes, and prioritize the mode features with higher importance, thereby forming a ranked undetermined feature sequence that reflects the relative contribution of each mode.

[0048] Furthermore, based on the pre-set feature quantity screening rules, a preset number of modal features with high contribution to the task are selected from the top-ranked undetermined features, and redundant features with low importance are eliminated, ultimately constructing a set of effective modal features with a compact structure and high information density.

[0049] In this embodiment of the invention, the Gumbel-TopK mechanism is used to filter out a preset number of effective modal feature sets:

[0050]

[0051] in, Indicates the first Always Activated Effective modal features under modal constraints This indicates selecting from the features to be ranked. A mechanism for selecting effective modal features for each modality. Represents the normalization function. Indicates the first Always Activated Importance scores under modal constraints Indicates the first Always Activated A random noise vector under modal constraints This indicates that the random noise follows a Gumbel distribution with a mean of 0 and a scale parameter of 1. This represents a multilayer perceptron network. Indicates the first Time of the first Average pooling of the undetermined eigenvectors of each modality.

[0052] This invention analyzes the importance of multimodal features and selects a set of effective modal features based on their importance scores. This allows for the effective removal of redundant and noisy features while retaining key information, highlighting modal features that contribute significantly to the target task.

[0053] S3. Perform dynamic sparse connection on the effective modal feature set to obtain fused modal features.

[0054] In this embodiment of the invention, dynamic sparse connectivity involves dynamically selecting partial feature connections based on the correlation between effective modal features or task requirements during feature fusion, and establishing a sparse connection mechanism for mapping between effective modal features. The fused modal features are joint feature representations formed by associating, interacting with, and integrating effective modal features from different modalities through dynamic sparse connectivity, capable of simultaneously representing multimodal information and their complementary relationships.

[0055] In specific healthcare scenarios, it can be applied to the deep fusion and comprehensive evaluation of multi-source clinical information. For example, in disease diagnosis or efficacy evaluation scenarios, selected medical imaging features, physiological monitoring features, and laboratory indicator features are used to form an effective modal feature set. Through dynamic sparse connections, only feature interaction relationships that are significantly related to the current condition are established, reducing interference between irrelevant features, forming a fused modal feature set to characterize the patient's overall health status.

[0056] In specific fintech scenarios, this can be applied to the joint modeling of complex risk characteristics. For example, in anti-fraud or comprehensive risk control scenarios, effective modal features such as user transaction behavior characteristics, account attribute characteristics, and device environment characteristics can be dynamically and sparsely connected. Based on the real-time business scenario, the association paths between key features can be adaptively selected to construct fused modal features for identifying potential abnormal patterns or risky behaviors.

[0057] In this embodiment of the invention, the step of dynamically sparsely connecting the effective modal feature set to obtain fused modal features includes: analyzing the connection relationships of the effective modal feature set to obtain multiple adjustable connection channels; generating a fusion weight for each modality based on the effective modal feature set; filtering the adjustable connection channels based on the fusion weight to obtain retained connection channels; constructing a sparse connection structure based on the retained connection channels; and fusing the effective modal feature set according to the sparse connection structure to obtain fused modal features.

[0058] In this embodiment of the invention, based on the correlation, complementarity and task requirements between modal features, the potential interaction paths of effective modal features of different modalities are analyzed, and the possible information transmission relationships between effective modal features are abstracted into multiple independently adjustable connection channels to represent the information fusion paths under different feature combination methods.

[0059] Furthermore, the discrimination ability and contribution of each modality's effective modal features are analyzed through attention mechanisms, gating functions, or lightweight mapping networks. Based on the response intensity and correlation of each modality feature in the target task, corresponding fusion weights are generated. Then, through normalization constraints, the weights of each effective modality feature satisfy the preset allocation rules, thereby obtaining the fusion weights of each modality.

[0060] In this embodiment of the invention, the importance of each adjustable connection channel is evaluated by using the fusion weight as the screening criterion. Connection channels that are related to the effective modal features with high weights and have a high contribution to the task are retained first, while connection channels with low contribution are suppressed or eliminated, thereby reducing connection redundancy.

[0061] Furthermore, the reserved connection channels are used as effective information transmission paths; with the reserved connection channels as constraints, the original fully connected or dense connection relationships are pruned, redundant connections that are not retained are removed, and explicit connections are established only between the reserved connection channels; according to the preset topology rules and parameter sharing strategy, each connection node and its mapping relationship are reconstructed to form a sparse connection structure with controlled number of connections and clear computational paths.

[0062] In this embodiment of the invention, selective connection relationships are established for each modal feature in the effective modal feature set based on the sparse connection structure, and feature interaction is performed only on the reserved connection channels; according to the sparse topological constraints, the effective modal features are weighted and information is aggregated to suppress interference between irrelevant or redundant modes; through layer-by-layer fusion and feature update, the effective modal features from different modes are integrated on the sparse connection path, and finally a fused modal feature with compact structure and strong information complementarity is obtained.

[0063] This invention generates fused modal features by dynamically sparsely connecting the effective modal feature set. This can make full use of multimodal complementary information while avoiding redundant computation and noise interference caused by full-connection fusion, and make the feature interaction relationship more focused on the key connection path that makes a substantial contribution to the target task.

[0064] S4. Perform temporal attention distillation on the fused modal features to obtain frame-level attention weights.

[0065] In this embodiment of the invention, temporal attention distillation is a process of introducing an attention mechanism in the time dimension and compressing and transferring key information from complex temporal features to a simpler representation through distillation. Frame-level attention weights are numerical weights assigned to each time frame during the temporal attention distillation process, representing the relative importance of that frame in the overall temporal feature analysis.

[0066] In specific healthcare scenarios, this technology can be applied to identify critical moments in continuous monitoring data. For example, in patient vital sign monitoring or medical imaging follow-up, temporal attention distillation can be performed on fusion modal features that integrate multimodal information such as imaging, electrocardiogram, and blood oxygenation to automatically highlight key time frames that are indicative of changes in the patient's condition. Corresponding frame-level attention weights are then generated to identify critical moments of abnormality or early signals of disease deterioration, thereby assisting doctors in timely intervention and accurate diagnosis.

[0067] In specific fintech scenarios, it can be applied to time-sensitive analysis of transaction behavior. For example, in high-frequency trading monitoring or anti-fraud scenarios, temporal attention distillation can be performed on fusion modal features that integrate transaction sequences, account behavior, and device characteristics to identify key transaction moments that have a significant impact on risk assessment. Frame-level attention weights are then generated to highlight time periods in which abnormal transactions occur, thereby improving the timeliness and accuracy of risk identification and supporting real-time risk control decisions.

[0068] In this embodiment of the invention, the step of performing temporal attention distillation on the fused modal features to obtain frame-level attention weights includes: obtaining the feature temporal order of the fused modal features; converting the fused modal features into a fused feature sequence according to the feature temporal order; performing temporal dependency modeling on the fused feature sequence in a preset teacher network to generate a temporal attention modeling structure for the teacher network; calculating the attention response value of each time frame in the fused feature sequence based on the temporal attention modeling structure; generating a temporal attention distribution for the teacher network based on the attention response value; using the temporal attention distribution as a distillation target, and performing temporal attention distillation optimization on a preset student network based on the distillation target to obtain an optimized student network; and using the optimized student network to perform forward propagation on the fused feature sequence to obtain frame-level attention weights.

[0069] In this embodiment of the invention, based on the timestamp or frame index information of the fusion modal features during the original data acquisition process, the features in the fusion modal features are time-aligned and sorted so that the discrete fusion modal features are arranged in the actual order of occurrence, forming a fusion feature sequence that can reflect the temporal evolution law.

[0070] In this embodiment of the invention, the formula for calculating the fusion modal features of the teacher network is as follows:

[0071] in, Indicating teacher network in Timing of fusion modal features Indicates the first Always Activated Effective modal features under modal constraints Indicates the first Time of the first Layer normalization operation for undetermined eigenvectors of each modality.

[0072] Furthermore, the fused feature sequence is input into a pre-defined teacher network and organized chronologically. By introducing a temporal modeling module (including recurrent neural networks, temporal convolution, or self-attention mechanisms), the dependencies between fused features at different time steps are correlated, characterizing long-term and short-term temporal relationships. During the training of the teacher network, the high model capacity and expressive power of the teacher network are utilized to adaptively learn the contribution of features at each time step to the overall task, forming a temporal attention modeling structure that can dynamically allocate time weights to represent the key temporal patterns and evolutionary relationships of the fused feature sequence.

[0073] In this embodiment of the invention, the fused feature sequence is input into the temporal attention modeling structure of the trained teacher network, and the corresponding temporal attention weight parameters are extracted; the fused features of each time frame are used as queries, keys and values, and the correlation between features at different times is measured according to the attention calculation mechanism; the correlation is normalized to obtain the attention weights that reflect the degree of contribution of each time frame to the overall sequence modeling, and the attention weights are mapped to the corresponding attention response values.

[0074] Furthermore, the attention response values ​​of each time frame are processed and normalized to achieve a uniform scale, so that the attention response values ​​form a comparable weight distribution in the time dimension, thereby obtaining a temporal attention distribution that reflects the teacher network's attention to different time frames.

[0075] In this embodiment of the invention, according to the distillation objective, a temporal attention distillation mechanism is introduced into the preset student network during the training process. By calculating the difference between the fused features of the student network and the teacher network frame by frame in the time dimension, and using the frame-level attention weights adaptively generated by the bidirectional gated recurrent unit to weight and constrain the difference, the student network focuses on aligning with the feature representation of the teacher network in key time frames during the optimization process. This effectively inherits the temporal modeling ability and multimodal feature expression ability of the teacher network while maintaining the compactness of the model structure, and finally obtains the student network optimized by temporal attention distillation.

[0076] Furthermore, the calculation formula for optimizing the student network through attention distillation loss is as follows:

[0077] in, Indicates distillation loss, Indicates the first Frame-level attention weights at each moment Indicates the total number of time frames. Indicates the student network in the first The fusion characteristics of the output at any time Indicates learnable weights, The teacher network is in the first The fusion characteristics of the output at any time This represents the Sigmoid activation function. Indicates a forward-gated loop unit. This indicates a backward-gated loop unit, and the arrows indicate the forward and backward propagation directions.

[0078] Furthermore, the fused feature sequence is input into the student network after structural pruning and parameter optimization, and the fused feature sequence is forward propagated in chronological order. The student network performs layer-by-layer mapping and feature transformation of the fused features of each time frame through the built-in lightweight temporal modeling and attention calculation module. In the forward inference process without introducing back updates, the student network estimates the contribution of the fused features at different times based on the learned parameters and outputs the corresponding frame-level attention weights, thereby obtaining the frame-level attention weights that represent the relative importance of each time frame.

[0079] This invention performs temporal attention distillation on fused modal features and generates frame-level attention weights. This allows for the transfer of complex temporal modeling capabilities from high-complexity models to lightweight models while maintaining key temporal discrimination information. It enables more efficient and stable focus on key time frames that have a decisive impact on task results, thereby reducing interference from irrelevant frames and improving the accuracy and interpretability of temporal feature representation.

[0080] S5. Perform frame-level criticality analysis on the multimodal raw data stream according to the frame-level attention weights to obtain the key frames of the multimodal raw data stream.

[0081] In this embodiment of the invention, frame-level criticality analysis is an analysis process that compares each time frame based on frame-level attention weights to determine the degree of contribution of different time frames to the target task. Key frames are time frames that are representative and contain high information for task discrimination or event understanding, selected from the multimodal raw data stream after frame-level criticality analysis.

[0082] In specific healthcare scenarios, it can be applied to continuous medical monitoring and diagnostic assistance. For example, during intensive care or remote follow-up, frame-level criticality analysis can be performed on multimodal raw data streams, including medical images, vital sign waveforms, and monitoring videos, based on frame-level attention weights. This automatically filters out key frames that reflect sudden changes in the condition, abnormal signs, or critical diagnostic and treatment nodes, helping medical staff quickly locate high-risk moments, reducing the workload of manual data retrieval, and improving the timeliness and accuracy of clinical decision-making.

[0083] In specific fintech scenarios, it can be applied to transaction process monitoring and risk event tracing. For example, in anti-fraud or transaction compliance review scenarios, frame-level criticality analysis can be performed on multimodal raw data streams containing transaction sequences, user operation behaviors, and device status information based on frame-level attention weights. This automatically identifies key frames highly correlated with abnormal transactions or risky behaviors, enabling rapid identification of critical time points where problematic transactions occur. This improves risk identification efficiency and supports post-event auditing and real-time risk control decisions.

[0084] In this embodiment of the invention, the step of performing frame-level criticality analysis on the multimodal raw data stream based on the frame-level attention weight to obtain key frames of the multimodal raw data stream includes: selecting time frames corresponding to frame-level attention weights greater than a preset weight threshold as candidate frames; performing temporal redundancy detection on adjacent candidate frames and selecting target frames from the candidate frames based on the detection results; obtaining the target frame position of the target frame in the fused feature sequence, and locating the key frame backtracking in the multimodal raw data stream based on the target frame position.

[0085] In this embodiment of the invention, the frame-level attention weights corresponding to each time frame are compared one by one according to a pre-set weight threshold, and candidate frames with frame-level attention weights higher than the weight threshold are selected. It is believed that the frame-level attention weights have high importance in the overall time series analysis.

[0086] Furthermore, candidate frames that are temporally adjacent or closely spaced are compared to analyze their similarity in temporal position and feature representation. If there is information duplication or high similarity, the more representative candidate frames are retained and redundant frames are removed, thereby obtaining target frames with concentrated and reasonably distributed information.

[0087] In this embodiment of the invention, the index or time position of the target frame in the fused feature sequence is determined, and the correspondence between the index or time position and the original multimodal data stream is used to accurately map the target frame back into the original multimodal data stream, thereby tracing back to locate the corresponding key frame.

[0088] This invention performs frame-level criticality analysis on multimodal raw data streams based on frame-level attention weights and extracts key frames. This enables the rapid focusing of the most informational time segments for the target task in massive continuous data, reducing the interference of irrelevant or redundant data on the analysis results.

[0089] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0090] Figure 4 shows a functional block diagram of a keyframe feature extraction device for multimodal data provided in an embodiment of the present invention.

[0091] This disclosure provides a keyframe feature extraction device for multimodal data, which corresponds one-to-one with the keyframe feature extraction method for multimodal data described in the previous embodiment. As shown in Figure 4, the keyframe feature extraction device 100 for multimodal data can be installed in an electronic device. Depending on its functions, the keyframe feature extraction device 100 includes a feature search module 101, an importance analysis module 102, a feature connection module 103, an attention distillation module 104, and a keyframe analysis module 105. The functional modules are described in detail below: Feature search module 101 is used to acquire several multimodal raw data streams, perform differentiable neural architecture search on the multimodal raw data streams, and obtain multimodal features to be determined; Importance analysis module 102 is used to perform importance analysis on the multimodal features to be determined, obtain importance scores, and select an effective modal feature set from the multimodal features to be determined based on the importance scores; Feature connection module 103 is used to perform dynamic sparse connection on the effective modal feature set to obtain fused modal features; Attention distillation module 104 is used to perform temporal attention distillation on the fused modal features to obtain frame-level attention weights; Keyness analysis module 105 is used to perform frame-level keyness analysis on the multimodal raw data streams based on the frame-level attention weights to obtain key frames of the multimodal raw data streams.

[0092] In one embodiment, the feature search module 101 performs a differentiable neural architecture search on the multimodal raw data stream to obtain multimodal undetermined features, including: normalizing the video frames of the multimodal raw data stream to obtain a normalized data stream; generating a set of differentiable candidate operators based on the normalized data stream; constructing an architecture search space based on the set of differentiable candidate operators; searching the set of differentiable candidate operators using the architecture search space to obtain architecture selection parameters for each candidate operator in the set of differentiable candidate operators; ranking each candidate operator by weight based on the architecture selection parameters, and selecting target operator combinations based on the ranking results; performing forward propagation on the normalized data stream of each modality using the target operator combinations to obtain a set of undetermined feature vectors for each modality in the normalized data stream; and summing the sets of undetermined feature vectors for all modalities into multimodal undetermined features.

[0093] In one embodiment, the importance analysis module 102 performs importance analysis on the multimodal undetermined features to obtain an importance score, including: constructing a lightweight gating network based on the multimodal undetermined features; using the lightweight gating network to perform modal response analysis on the set of undetermined feature vectors for each modality in the multimodal undetermined features to obtain the original importance response value for each modality; and normalizing the original importance response value to obtain an importance score.

[0094] In one embodiment, the importance analysis module 102 performs the following steps when selecting a set of effective modal features from the multimodal undetermined features based on the importance scores: randomly perturbing the importance scores to obtain a perturbed score set; smoothing the distribution of the perturbed score set using a preset temperature parameter to form a smoothed score set; sorting each mode of the multimodal undetermined features in descending order of importance based on the smoothed score set to obtain sorted undetermined features; and selecting a preset number of effective modal features from the sorted undetermined features.

[0095] In one embodiment, the feature connection module 103 performs dynamic sparse connection on the effective modal feature set to obtain fused modal features, including: analyzing the connection relationships of the effective modal feature set to obtain multiple adjustable connection channels; generating a fusion weight for each modality based on the effective modal feature set; filtering the adjustable connection channels based on the fusion weight to obtain retained connection channels; constructing a sparse connection structure based on the retained connection channels; and fusing the effective modal feature set according to the sparse connection structure to obtain fused modal features.

[0096] In one embodiment, the attention distillation module 104 performs temporal attention distillation on the fused modal features to obtain frame-level attention weights, including: obtaining the feature temporal order of the fused modal features; converting the fused modal features into a fused feature sequence according to the feature temporal order; performing temporal dependency modeling on the fused feature sequence in a preset teacher network to generate a temporal attention modeling structure for the teacher network; calculating the attention response value for each time frame in the fused feature sequence based on the temporal attention modeling structure; generating a temporal attention distribution for the teacher network based on the attention response value; using the temporal attention distribution as the distillation target, and optimizing a preset student network using temporal attention distillation based on the distillation target to obtain an optimized student network; and using the optimized student network to perform forward propagation on the fused feature sequence to obtain frame-level attention weights.

[0097] In one embodiment, the criticality analysis module 105 performs frame-level criticality analysis on the multimodal raw data stream based on the frame-level attention weights to obtain key frames of the multimodal raw data stream, including: selecting time frames corresponding to the frame-level attention weights that are greater than a preset weight threshold as candidate frames; performing temporal redundancy detection on adjacent candidate frames and filtering out target frames from the candidate frames based on the detection results; obtaining the target frame position of the target frame in the fused feature sequence, and locating the key frame backtracking in the multimodal raw data stream based on the target frame position.

[0098] In this invention, the specific limitations of the keyframe feature extraction device for multimodal data can be found in the above-described limitations of the keyframe feature extraction method for multimodal data, and will not be repeated here. Each module in the aforementioned keyframe feature extraction device for multimodal data can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0099] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram is shown in Figure 5. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a keyframe feature extraction method for multimodal data on the server side.

[0100] In one embodiment, a computer device is provided, which can be a client, and its internal structure diagram is shown in Figure 6. The computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a keyframe feature extraction method for multimodal data on the client side.

[0101] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: acquiring a plurality of multimodal raw data streams; performing a differentiable neural architecture search on the multimodal raw data streams to obtain multimodal features to be determined; performing importance analysis on the multimodal features to be determined to obtain importance scores, and filtering out a set of effective modal features from the multimodal features to be determined based on the importance scores; performing dynamic sparse connections on the set of effective modal features to obtain fused modal features; performing temporal attention distillation on the fused modal features to obtain frame-level attention weights; and performing frame-level criticality analysis on the multimodal raw data streams based on the frame-level attention weights to obtain keyframes of the multimodal raw data streams.

[0102] In the several embodiments provided by this invention, it should be understood that the disclosed devices and apparatuses can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0103] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0104] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0105] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0106] In some embodiments of this example, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the steps of the method described in the above embodiments.

[0107] The readable storage medium of the present invention stores a computer program, which, when executed by a processor of an electronic device, can perform the following: acquire several multimodal raw data streams; perform a differentiable neural architecture search on the multimodal raw data streams to obtain multimodal undetermined features; perform importance analysis on the multimodal undetermined features to obtain importance scores, and select a set of effective modal features from the multimodal undetermined features based on the importance scores; perform dynamic sparse connections on the set of effective modal features to obtain fused modal features; perform temporal attention distillation on the fused modal features to obtain frame-level attention weights; and perform frame-level criticality analysis on the multimodal raw data streams based on the frame-level attention weights to obtain keyframes of the multimodal raw data streams.

[0108] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0109] Computer-readable storage media may also store at least one computer-executable program / instruction, such as computer-readable instructions. Computer-readable storage media include, but are not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Computer-readable storage media may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions stored on the computer-readable storage medium, the various methods described above can be performed.

[0110] In addition, the computer device may include (but is not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (e.g., keyboard, mouse, speakers, etc.).

[0111] The processor can communicate with external devices via the I / O bus through wired or wireless networks.

[0112] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product / computer program product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0113] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0114] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0115] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0116] It should be noted that, in this disclosure, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element limited by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0117] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

[0118] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely for illustrative purposes and do not represent actual use.

Claims

1. A method for keyframe feature extraction from multimodal data, characterized in that, The method includes: acquiring several multimodal raw data streams; performing a differentiable neural architecture search on the multimodal raw data streams to obtain multimodal undetermined features; performing importance analysis on the multimodal undetermined features to obtain importance scores, and selecting an effective modal feature set from the multimodal undetermined features based on the importance scores; performing dynamic sparse connections on the effective modal feature set to obtain fused modal features; performing temporal attention distillation on the fused modal features to obtain frame-level attention weights; and performing frame-level keyness analysis on the multimodal raw data streams based on the frame-level attention weights to obtain keyframes of the multimodal raw data streams.

2. The keyframe feature extraction method for multimodal data as described in claim 1, characterized in that, The step of performing a differentiable neural architecture search on the multimodal raw data stream to obtain multimodal undetermined features includes: normalizing the video frames of the multimodal raw data stream to obtain a normalized data stream; generating a set of differentiable candidate operators based on the normalized data stream; constructing an architecture search space based on the set of differentiable candidate operators; searching the set of differentiable candidate operators using the architecture search space to obtain architecture selection parameters for each candidate operator in the set of differentiable candidate operators; ranking each candidate operator by weight based on the architecture selection parameters, and selecting target operator combinations based on the ranking results; performing forward propagation on the normalized data stream of each modality using the target operator combinations to obtain a set of undetermined feature vectors for each modality in the normalized data stream; and summing the sets of undetermined feature vectors for all modalities into multimodal undetermined features.

3. The keyframe feature extraction method for multimodal data as described in claim 2, characterized in that, The step of performing importance analysis on the multimodal undetermined features to obtain importance scores includes: constructing a lightweight gating network based on the multimodal undetermined features; using the lightweight gating network to perform modal response analysis on the set of undetermined feature vectors for each modality in the multimodal undetermined features to obtain the original importance response value for each modality; and normalizing the original importance response value to obtain the importance score.

4. The keyframe feature extraction method for multimodal data as described in claim 1, characterized in that, The step of selecting a set of effective modal features from the multimodal undetermined features based on the importance scores includes: randomly perturbing the importance scores to obtain a set of perturbed scores; smoothing the distribution of the set of perturbed scores using a preset temperature parameter to form a set of smoothed scores; sorting each modality of the multimodal undetermined features in descending order of importance based on the set of smoothed scores to obtain sorted undetermined features; and selecting a preset number of effective modal features from the sorted undetermined features.

5. The keyframe feature extraction method for multimodal data as described in claim 1, characterized in that, The step of dynamically sparsely connecting the effective modal feature set to obtain fused modal features includes: analyzing the connection relationships of the effective modal feature set to obtain multiple adjustable connection channels; generating a fusion weight for each modality based on the effective modal feature set; filtering the adjustable connection channels based on the fusion weight to obtain retained connection channels; constructing a sparse connection structure based on the retained connection channels; and fusing the effective modal feature set according to the sparse connection structure to obtain fused modal features.

6. The keyframe feature extraction method for multimodal data as described in claim 1, characterized in that, The step of performing temporal attention distillation on the fused modal features to obtain frame-level attention weights includes: obtaining the feature temporal order of the fused modal features; converting the fused modal features into a fused feature sequence according to the feature temporal order; performing temporal dependency modeling on the fused feature sequence in a preset teacher network to generate a temporal attention modeling structure for the teacher network; calculating the attention response value for each time frame in the fused feature sequence based on the temporal attention modeling structure; generating a temporal attention distribution for the teacher network based on the attention response value; using the temporal attention distribution as the distillation target, and performing temporal attention distillation optimization on a preset student network based on the distillation target to obtain an optimized student network; and using the optimized student network to perform forward propagation on the fused feature sequence to obtain frame-level attention weights.

7. The keyframe feature extraction method for multimodal data as described in claim 1, characterized in that, The step of performing frame-level criticality analysis on the multimodal raw data stream based on the frame-level attention weights to obtain key frames of the multimodal raw data stream includes: selecting time frames corresponding to frame-level attention weights greater than a preset weight threshold as candidate frames; performing temporal redundancy detection on adjacent candidate frames and selecting target frames from the candidate frames based on the detection results; obtaining the target frame position of the target frame in the fused feature sequence, and locating the key frame backtracking in the multimodal raw data stream based on the target frame position.

8. A keyframe feature extraction device for multimodal data, characterized in that, The device includes: a feature search module for acquiring several multimodal raw data streams, performing differentiable neural architecture search on the multimodal raw data streams to obtain multimodal undetermined features; an importance analysis module for performing importance analysis on the multimodal undetermined features to obtain importance scores, and filtering out a set of effective modal features from the multimodal undetermined features based on the importance scores; a feature connection module for performing dynamic sparse connections on the set of effective modal features to obtain fused modal features; an attention distillation module for performing temporal attention distillation on the fused modal features to obtain frame-level attention weights; and a key analysis module for performing frame-level key analysis on the multimodal raw data streams based on the frame-level attention weights to obtain key frames of the multimodal raw data streams.

9. An electronic device, characterized in that, The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the keyframe feature extraction method for multimodal data as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the keyframe feature extraction method for multimodal data as described in any one of claims 1 to 7.

Images