Key frame extraction method and device based on time sequence gating, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, provides a key frame extraction method and device based on time sequence gating, equipment and a medium, is applied to financial and medical health old-age service scenes, and can calculate the contribution degree score of each modal based on a low-dimensional long and short-term memory network and a modal special feature library. A time sequence change rule of each modal feature is accurately captured based on a historical feature state, and time sequence coherence is kept; according to the method, active modal features are generated based on the gating network and the optimal calculation graph, the active modal with high contribution degree at the current moment can be reserved, the calculation cost is reduced, meanwhile, dynamic allocation and release of calculation resources are achieved on the hardware level based on the optimal calculation graph, and the real-time performance is improved; the active modal features are fused based on a query-key value attention network, and meanwhile, current feature similarity and historical context information are considered, so that self-adaptive precise fusion of the active modal features is realized; and a target key frame is selected from the fusion features by adopting a clustering algorithm, so that time sequence faults are further avoided.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and medium for extracting keyframes based on time-gated control. Background Technology

[0002] In the fields of video analytics and multimodal data processing, keyframe extraction technology significantly reduces the computational complexity of subsequent processing by selecting the most representative frames from a continuous frame sequence. For example, by extracting the interaction between a bank teller and a customer, the cause of a conflict can be quickly located; by extracting key steps from surgical videos, medical teaching videos can be generated efficiently.

[0003] Existing technologies typically employ static feature fusion strategies, such as early fusion or late fusion. However, these methods struggle to adapt to dynamically changing modal correlations and incur high computational costs. Furthermore, existing dynamic feature fusion strategies also suffer from the following limitations: (1) Static modality processing: It is impossible to adjust the modality participation in real time, resulting in redundant calculations. For example, when the video scene changes abruptly, some modalities (such as audio) may temporarily fail, but static fusion will still force the calculation of all modal features.

[0004] (2) Lack of temporal continuity: Although a dynamic gating mechanism was proposed, it did not consider the preservation of historical modal states, resulting in temporal discontinuity when selecting key frames.

[0005] (3) Insufficient hardware efficiency: Although the existing dynamic compilation technology has optimized the single-modal network, it has not been specifically accelerated for the multimodal gating-fusion pipeline, making it difficult to meet the real-time requirements. Summary of the Invention

[0006] In view of the above, it is necessary to provide a keyframe extraction method, apparatus, device and medium based on time-gated keyframe extraction, which aims to solve the problems of high computational cost, time discontinuity and inability to meet real-time requirements when performing multimodal keyframe extraction.

[0007] A keyframe extraction method based on temporal gating, the keyframe extraction method based on temporal gating includes: In response to the keyframe extraction instruction for the target multimodal data, the target multimodal data is preprocessed to obtain the feature vector of each modality; The feature vectors of each mode are processed using a low-dimensional long short-term memory network and a mode-specific feature library to obtain the contribution score of each mode. In a gated network, a gate mask is generated based on the contribution score of each mode, and the optimal computation graph is obtained by reconstructing the topology of the computation graph in real time based on the gate mask using dynamic compilation technology. Active modal features are generated by filtering the feature vectors of each modality based on the gating mask and the optimal computation graph. The active modality features are fused using a query-key-value attention network to obtain fused features; A clustering algorithm is used to select target keyframes from the fused features.

[0008] A keyframe extraction device based on time-gated control, the keyframe extraction device based on time-gated control includes: The preprocessing unit is used to preprocess the target multimodal data in response to the keyframe extraction instruction for the target multimodal data to obtain the feature vector of each modality; The processing unit is used to process the feature vector of each mode based on a low-dimensional long short-term memory network and a mode-specific feature library to obtain the contribution score of each mode. The generation unit is used to generate a gating mask in the gating network based on the contribution score of each mode, and to reconstruct the topology of the computation graph in real time based on the gating mask using dynamic compilation technology to obtain the optimal computation graph. The generation unit is further configured to filter and generate active modal features from the feature vectors of each modality based on the gating mask and the optimal computation graph; The fusion unit is used to fuse the active modality features based on a query-key-value attention network to obtain fused features; The selection unit is used to select target keyframes from the fused features using a clustering algorithm.

[0009] A computer device, the computer device comprising: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the timing-gated keyframe extraction method.

[0010] A computer-readable storage medium storing at least one instruction, which is executed by a processor in a computer device to implement the timing-gated keyframe extraction method.

[0011] As can be seen from the above technical solutions, this invention can calculate the contribution score of each modality based on a low-dimensional long short-term memory network and a modality-specific feature library, so as to accurately capture the temporal change pattern of each modality feature based on historical feature states and maintain temporal continuity; it generates active modality features based on a gating network and an optimal computation graph, which can retain active modalities with high contribution at the current moment, reduce computational costs, and realize dynamic allocation and release of computing resources at the hardware level based on the optimal computation graph, thereby improving real-time performance; it fuses active modality features based on a query-key-value attention network, and considers the similarity of current features and historical context information to achieve adaptive and accurate fusion of active modality features; and it uses a clustering algorithm to select target keyframes from the fused features to further avoid temporal discontinuities. Attached Figure Description

[0012] Figure 1 This is a flowchart of a preferred embodiment of the keyframe extraction method based on time-gating of the present invention.

[0013] Figure 2 This is a functional block diagram of a preferred embodiment of the keyframe extraction device based on time-gated control of the present invention.

[0014] Figure 3 This is a schematic diagram of the structure of a computer device that implements a preferred embodiment of the keyframe extraction method based on time-gating according to the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the keyframe extraction method based on time-gating according to the present invention. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.

[0017] The keyframe extraction method based on time-gating is applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0018] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0019] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0020] The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0021] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0022] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0023] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0024] S10, in response to the keyframe extraction instruction for the target multimodal data, preprocess the target multimodal data to obtain the feature vector of each modality.

[0025] In this embodiment, the target multimodal data may include bank branch videos in the financial field or operating room videos in the medical field.

[0026] In this embodiment, the keyframe extraction command can be automatically triggered when the target multimodal data is detected to be uploaded to a designated platform.

[0027] In this embodiment, different preprocessing methods can be used for different types of modal data. For example, spatial features of video frames can be extracted using a lightweight convolutional neural network (CNN); audio can be converted into spectral features using a short-time fourier transform (STFT); financial time-series data can be processed using standardization algorithms (such as Z-score standardization), and statistical features (mean, variance, volatility) can be extracted using sliding window sampling; effective features of medical image data can be obtained using grayscale normalization, noise removal (such as Gaussian filtering), and region of interest extraction algorithms; and standardized features of physiological signals can be obtained through filtering and denoising (such as Butterworth filtering) and feature point extraction (such as R-wave detection of electrocardiogram signals).

[0028] Furthermore, the dimensions of the features extracted from each modality can be adjusted to ensure the compatibility of subsequent fusion processing, such as by using pooling operations and feature mapping to transform modal features of different dimensions into the same dimensional space.

[0029] The above embodiments can remove noise interference from the original data, extract the core effective features of each modality, reduce data redundancy, provide high-quality input data for subsequent dynamic feature extraction, and improve the efficiency and accuracy of subsequent processing.

[0030] S11, based on a low-dimensional long short-term memory (LSTM) network and a modality-specific feature library, processes the feature vector of each modality to obtain the contribution score of each modality.

[0031] In this embodiment, the low-dimensional long short-term memory network can be 32-64 dimensional, with only 1 / 5 of the number of parameters of the traditional time series model, making it suitable for embedded deployment.

[0032] In this embodiment, the process of processing the feature vector of each modality based on a low-dimensional long short-term memory network and a modality-specific feature library to obtain the contribution score of each modality includes: Calculate the feature difference between adjacent frames based on the feature vector of each modality; The most recently written features are read from the modality-specific feature library as the initial hidden state of the low-dimensional long short-term memory network; Starting from the initial hidden state, the contribution score of each modality is calculated based on the feature difference between adjacent frames; The modality-specific feature library includes a fixed-capacity first-in-first-out (FIFO) queue maintained for each modality. When an active modality is detected, the current features of that modality are written to the corresponding FIFO queue in real time, and old features are deleted according to the feature writing time when the corresponding FIFO queue is full. When a modality is detected to change from an active state to an inactive state, the writing of features to the corresponding FIFO queue for that modality is stopped, and the written features in the corresponding FIFO queue for that modality are retained. When a modality is detected to change from an inactive state to an active state, the most recently written features are read from the corresponding FIFO queue for that modality as the initial hidden state of the low-dimensional long short-term memory network. When a modality is detected to change from an inactive state to an active state, and the corresponding FIFO queue for that modality is empty, zero-vector filling is used as the initial hidden state of the low-dimensional long short-term memory network.

[0033] The feature difference between adjacent frames can be represented as follows: ; in, The feature difference between adjacent frames is used to characterize the intensity of feature change of the m-th mode at time t; This represents the eigenvector of the m-th mode at time t, representing a specified dimension. This represents the eigenvector of the m-th mode at time t-1 with a specified dimension.

[0034] The contribution score for each mode is represented as follows: ; in, This represents the contribution score of the m-th mode at time t; This represents the Sigmoid function, used to compress scores to the interval [0,1]. Represents the learnable parameter matrix; Indicates the bias term; This represents the hidden state of the LSTM, initially consisting of the most recently written features extracted from the modality-specific feature library, and subsequently the output vector of the LSTM hidden layer.

[0035] Through the above embodiments, the temporal variation patterns of each modality feature can be accurately captured, the importance of each modality at the current moment can be quantified, and a scientific basis can be provided for subsequent modality selection. At the same time, the lightweight design reduces the computational overhead and ensures real-time processing capabilities.

[0036] In this embodiment, the capacity of each first-in-first-out queue can be 5.

[0037] As the model is trained, the active features of each modality are constantly updated, causing the gating value to change continuously. This in turn affects whether the features of each modality are added to the queue, and the features in the queue will serve as the initial state of the LSTM, affecting subsequent calculations, thus forming a dynamic loop.

[0038] Furthermore, the modality-specific feature library can effectively preserve the historical feature states of each modality, ensuring that key contextual information is not lost during modality switching, guaranteeing temporal continuity, and providing support for event evolution analysis of long sequence data.

[0039] S12, In the gated network, a gate mask is generated based on the contribution score of each mode, and the optimal computation graph is obtained by reconstructing the topology of the computation graph in real time based on the gate mask using dynamic compilation technology.

[0040] In this embodiment, generating the gating mask based on the contribution score of each modality includes: The contribution scores of each mode are sorted in descending order to obtain an ordered sequence; Starting from the first contribution score in the ordered sequence, the scores are accumulated sequentially until the accumulated value reaches a preset threshold. The number of modes currently participating in the accumulation is then determined as the number of active modes. The gate mask is generated based on the number of active modes; Specifically, after generating the gating mask, CUDA (Compute Unified Device Architecture) stream synchronization is triggered.

[0041] The preset threshold can be configured according to actual needs; for example, the preset threshold can be configured to 0.8.

[0042] Of course, the number of active modes can also be configured to a fixed value, such as 2.

[0043] The gate mask can be represented as follows: ; in, The gate mask is represented by k; the number of active modes is represented by k; and the total number of modes is represented by M. This indicates that the corresponding mode is the active mode; This indicates that the corresponding mode is an inactive mode and needs to be pruned.

[0044] The immediate triggering of unified computing device architecture flow synchronization after the generation of the gating mask effectively avoids subsequent waste of computing resources.

[0045] Through the above embodiments, dynamic sparsity selection of modes can be achieved, retaining only the active modes with high contribution at the current moment, greatly reducing redundant calculations and improving system operating efficiency.

[0046] In this embodiment, the step of obtaining the optimal computation graph by real-time reconstruction of the computation graph topology based on the gating mask using dynamic compilation technology includes: Obtain the complete computation graph of the entire multimodal model, and divide the complete computation graph into multiple independent subgraph units according to the modality type; wherein each subgraph unit includes the full-process computation logic of the corresponding single modality; The optimal computation graph is formed by selecting active subgraph cells from the subgraph cells according to the gating mask. The optimal computation graph is used to guide the fine-grained optimization of the feature vectors of the selected active modalities based on the low-dimensional long short-term memory network to obtain the active modal features.

[0047] For example, a visual subgraph can include the computation process of a convolutional neural network (CNN) + LSTM, an audio subgraph can include the computation process of short-time Fourier transform + LSTM, a financial time series subgraph can include the computation process of standardization + statistical feature extraction + LSTM, and a medical image subgraph can include the computation process of preprocessing + region of interest extraction + LSTM.

[0048] Subgraphs are dynamically scheduled based on gating masks, and only subgraphs corresponding to active modes are computed, while inactive mode subgraphs are skipped, thus avoiding invalid computations.

[0049] This allows LSTM and attention computation to be combined into a single CUDA core, reducing memory transfer overhead and improving computational efficiency.

[0050] Furthermore, it can pre-allocate the video memory space for each mode, avoiding the latency caused by dynamic memory allocation at runtime, thereby ensuring real-time processing performance.

[0051] Tests on graphics processors show that this optimization reduces multimodal inference latency by 12%.

[0052] Through the above embodiments, dynamic allocation and release of computing resources are realized at the hardware level, optimizing the execution efficiency of the multimodal gating-fusion pipeline and effectively improving inference speed compared to traditional static networks.

[0053] S13, based on the gating mask and the optimal computation graph, select and generate active modal features from the feature vectors of each modality.

[0054] In this embodiment, after filtering out the features in the active state according to the gating mask, the selected features can be further refined and optimized by controlling LSTM based on the optimal computation graph (such as more accurate historical state encoding, long-term dependency capture, etc.) to obtain the active modality features.

[0055] S14, The active modality features are fused based on a query-key attention network to obtain fused features.

[0056] In this embodiment, the fusion of the active modality features based on the query-key-value attention network to obtain the fused features includes: Calculate the product of the active modality features and the key projection matrix to obtain the key vector for each modality; Calculate the product of the active modality features and the value projection matrix to obtain the value vector for each modality; Historical fusion features are obtained and processed using a multi-layer perceptron (MLP) to obtain a query vector; The fusion weights are calculated based on the query vector and the key vector using a query-key-value attention mechanism. The fusion feature is obtained by weighted fusion based on the fusion weights and the value vectors.

[0057] The key projection matrix and the value projection matrix can be initialized using a random Gaussian distribution and can be optimized through training.

[0058] The fusion weights can be represented as follows: ; in, This represents the fusion weights at time t; Indicates the transpose of the query vector; This represents the bond vector of mode m at time t; This represents the feature dimension scaling factor, which is the dimension of the key vector.

[0059] The fusion features are represented as follows: ; in, This represents the fusion characteristics at time t; This represents the value vector of mode m at time t; , representing the active modality index set.

[0060] Through the above embodiments, it is possible to simultaneously consider the current feature similarity and historical context information, achieve adaptive and accurate fusion of active modality features, highlight the contribution of key modalities, and improve the representational ability of fused features.

[0061] S15, a clustering algorithm is used to select target keyframes from the fused features.

[0062] In this embodiment, the step of using a clustering algorithm to select the target keyframe from the fused features includes: Obtain the target scene to which the target multimodal data belongs, and configure the current sliding window according to the target scene; The fused feature is divided into continuous sub-features according to the current sliding window; wherein each sub-feature includes a fused feature at a continuous time step. Clustering is performed within each segment feature, and the feature closest to the corresponding cluster center is selected from each segment feature as each candidate keyframe; The keyframe selection network is used to select the candidate keyframe with the highest probability from each candidate keyframe as the target keyframe. The keyframe selection network comprises two sub-layers, each consisting of a multilayer perceptron and a sigmoid layer.

[0063] The entire keyframe selection process is executed cyclically using a time-series sliding window. The size of the sliding window can be flexibly configured according to the application scenario (e.g., a 5-minute window can be used in financial scenarios, and a 10-second window can be used in medical scenarios).

[0064] In the above embodiments, candidate keyframes are initially selected based on clustering algorithms to reduce computational load. Then, the target keyframe is further selected from the candidate keyframes through a keyframe selection network. This can accurately filter out the most representative keyframes, fully capture the event evolution process, avoid temporal discontinuities, and significantly improve the accuracy of keyframe extraction compared to traditional fixed fusion weight methods. At the same time, the loop execution mode can also adapt to the needs of long sequence data processing.

[0065] In this embodiment, the method further includes: Noise is sampled based on the Gumbel distribution to obtain Gumbel noise; The contribution score of each mode is fused with the Gumbel noise to obtain the fusion result; The fusion result is processed by the Softmax function to obtain a differentiable gated mask; Forward computation is performed using the differentiable gating mask, and backward propagation is performed based on the differentiable gating mask to update the network parameters of the low-dimensional long short-term memory network, the gating network, and the query-key-value attention network.

[0066] Through the above embodiments, the Gumbel-Softmax technique can be used to make the gating decision differentiable, supporting end-to-end training. The differentiable design ensures the integrity of model training.

[0067] This embodiment can be applied to multiple fields, such as video analysis, monitoring systems, autonomous driving, and medical image analysis. It aims to improve the accuracy and efficiency of keyframe extraction while reducing computing resource consumption. It is particularly suitable for real-time multimodal analysis in resource-constrained environments, such as abnormal event detection in intelligent monitoring, multi-sensor fusion in autonomous driving, and synchronous analysis of physiological signals in telemedicine.

[0068] For example: surveillance videos in bank lobbies or operating rooms: 1. Input the following data collected synchronously: Visual modality: 1920×1080 resolution video (10fps); Audio modality: 16kHz sampling rate mono audio; Motion mode: Optical flow characteristics (calculated using the Farneback dense optical flow algorithm).

[0069] 2. The key event was: a pedestrian suddenly ran in the surveillance footage (t=32 frames) accompanied by shouting.

[0070] 3. The output of the time series modeling includes: , , ; in, This represents the contribution score of the visual modality at time t=32; This represents the contribution score of the audio modality at time t=32; This represents the contribution score of the motion mode at time t=32.

[0071] Top-2 gating selects audio and motion modalities (such as visual modalities being temporarily pruned).

[0072] 4. Feature fusion: Motion mode at time t Attention fusion weights: (Due to significant abrupt changes in optical flow, the weighting is higher); Audio modality at time t Attention fusion weights: .

[0073] 5. Function of the feature library: At frame t=33, the pedestrian leaves the frame, and the motion mode becomes invalid (the gating mask of the motion mode at t=33). ); When the visual modality is reactivated at frame t=34, it is taken from the first-in-first-out queue of the visual modality. Recover the features of frame t=31 to maintain coherence.

[0074] 6. The output results include: accurately marking frames t=32-34 as keyframe sequences, fully capturing the event evolution process of "stillness → running → disappearance", while traditional methods may miss audio clues or produce temporal breaks due to fixed fusion weights.

[0075] As can be seen from the above technical solutions, this invention can calculate the contribution score of each modality based on a low-dimensional long short-term memory network and a modality-specific feature library, so as to accurately capture the temporal change pattern of each modality feature based on historical feature states and maintain temporal continuity; it generates active modality features based on a gating network and an optimal computation graph, which can retain active modalities with high contribution at the current moment, reduce computational costs, and realize dynamic allocation and release of computing resources at the hardware level based on the optimal computation graph, thereby improving real-time performance; it fuses active modality features based on a query-key-value attention network, and considers the similarity of current features and historical context information to achieve adaptive and accurate fusion of active modality features; and it uses a clustering algorithm to select target keyframes from the fused features to further avoid temporal discontinuities.

[0076] like Figure 2 The diagram shown is a functional block diagram of a preferred embodiment of the keyframe extraction device based on time-gated control according to the present invention. The keyframe extraction device 11 based on time-gated control includes a preprocessing unit 110, a processing unit 111, a generation unit 112, a fusion unit 113, and a selection unit 114. The module / unit referred to in this invention refers to a series of computer program segments that can be executed by a processor and perform a fixed function, and are stored in memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0077] The preprocessing unit 110 is used to preprocess the target multimodal data in response to the keyframe extraction instruction for the target multimodal data to obtain the feature vector of each modality.

[0078] In this embodiment, the target multimodal data may include bank branch videos in the financial field or operating room videos in the medical field.

[0079] In this embodiment, the keyframe extraction command can be automatically triggered when the target multimodal data is detected to be uploaded to a designated platform.

[0080] In this embodiment, different preprocessing methods can be used for different types of modal data. For example, spatial features of video frames can be extracted using a lightweight convolutional neural network (CNN); audio can be converted into spectral features using a short-time fourier transform (STFT); financial time-series data can be processed using standardization algorithms (such as Z-score standardization), and statistical features (mean, variance, volatility) can be extracted using sliding window sampling; effective features of medical image data can be obtained using grayscale normalization, noise removal (such as Gaussian filtering), and region of interest extraction algorithms; and standardized features of physiological signals can be obtained through filtering and denoising (such as Butterworth filtering) and feature point extraction (such as R-wave detection of electrocardiogram signals).

[0081] Furthermore, the dimensions of the features extracted from each modality can be adjusted to ensure the compatibility of subsequent fusion processing, such as by using pooling operations and feature mapping to transform modal features of different dimensions into the same dimensional space.

[0082] The above embodiments can remove noise interference from the original data, extract the core effective features of each modality, reduce data redundancy, provide high-quality input data for subsequent dynamic feature extraction, and improve the efficiency and accuracy of subsequent processing.

[0083] The processing unit 111 is used to process the feature vector of each modality based on a low-dimensional long short-term memory network (LSTM) and a modality-specific feature library to obtain the contribution score of each modality.

[0084] In this embodiment, the low-dimensional long short-term memory network can be 32-64 dimensional, with only 1 / 5 of the number of parameters of the traditional time series model, making it suitable for embedded deployment.

[0085] In this embodiment, the processing unit 111 processes the feature vector of each modality based on a low-dimensional long short-term memory network and a modality-specific feature library to obtain the contribution score of each modality, including: Calculate the feature difference between adjacent frames based on the feature vector of each modality; The most recently written features are read from the modality-specific feature library as the initial hidden state of the low-dimensional long short-term memory network; Starting from the initial hidden state, the contribution score of each modality is calculated based on the feature difference between adjacent frames; The modality-specific feature library includes a fixed-capacity first-in-first-out (FIFO) queue maintained for each modality. When an active modality is detected, the current features of that modality are written to the corresponding FIFO queue in real time, and old features are deleted according to the feature writing time when the corresponding FIFO queue is full. When a modality is detected to change from an active state to an inactive state, the writing of features to the corresponding FIFO queue for that modality is stopped, and the written features in the corresponding FIFO queue for that modality are retained. When a modality is detected to change from an inactive state to an active state, the most recently written features are read from the corresponding FIFO queue for that modality as the initial hidden state of the low-dimensional long short-term memory network. When a modality is detected to change from an inactive state to an active state, and the corresponding FIFO queue for that modality is empty, zero-vector filling is used as the initial hidden state of the low-dimensional long short-term memory network.

[0086] The feature difference between adjacent frames can be represented as follows: ; in, The feature difference between adjacent frames is used to characterize the intensity of feature change of the m-th mode at time t; This represents the eigenvector of the m-th mode at time t, representing a specified dimension. This represents the eigenvector of the m-th mode at time t-1 with a specified dimension.

[0087] The contribution score for each mode is represented as follows: ; in, This represents the contribution score of the m-th mode at time t; This represents the Sigmoid function, used to compress scores to the interval [0,1]. Represents the learnable parameter matrix; Indicates the bias term; This represents the hidden state of the LSTM, initially consisting of the most recently written features extracted from the modality-specific feature library, and subsequently the output vector of the LSTM hidden layer.

[0088] Through the above embodiments, the temporal variation patterns of each modality feature can be accurately captured, the importance of each modality at the current moment can be quantified, and a scientific basis can be provided for subsequent modality selection. At the same time, the lightweight design reduces the computational overhead and ensures real-time processing capabilities.

[0089] In this embodiment, the capacity of each first-in-first-out queue can be 5.

[0090] As the model is trained, the active features of each modality are constantly updated, causing the gating value to change continuously. This in turn affects whether the features of each modality are added to the queue, and the features in the queue will serve as the initial state of the LSTM, affecting subsequent calculations, thus forming a dynamic loop.

[0091] Furthermore, the modality-specific feature library can effectively preserve the historical feature states of each modality, ensuring that key contextual information is not lost during modality switching, guaranteeing temporal continuity, and providing support for event evolution analysis of long sequence data.

[0092] The generation unit 112 is used to generate a gating mask in the gating network based on the contribution score of each mode, and to reconstruct the topology of the computation graph in real time based on the gating mask using dynamic compilation technology to obtain the optimal computation graph.

[0093] In this embodiment, the generation unit 112 generates a gating mask based on the contribution score of each modality, including: The contribution scores of each mode are sorted in descending order to obtain an ordered sequence; Starting from the first contribution score in the ordered sequence, the scores are accumulated sequentially until the accumulated value reaches a preset threshold. The number of modes currently participating in the accumulation is then determined as the number of active modes. The gate mask is generated based on the number of active modes; Specifically, after generating the gating mask, CUDA (Compute Unified Device Architecture) stream synchronization is triggered.

[0094] The preset threshold can be configured according to actual needs; for example, the preset threshold can be configured to 0.8.

[0095] Of course, the number of active modes can also be configured to a fixed value, such as 2.

[0096] The gate mask can be represented as follows: ; in, The gate mask is represented by k; the number of active modes is represented by k; and the total number of modes is represented by M. This indicates that the corresponding mode is the active mode; This indicates that the corresponding mode is an inactive mode and needs to be pruned.

[0097] The immediate triggering of unified computing device architecture flow synchronization after the generation of the gating mask effectively avoids subsequent waste of computing resources.

[0098] Through the above embodiments, dynamic sparsity selection of modes can be achieved, retaining only the active modes with high contribution at the current moment, greatly reducing redundant calculations and improving system operating efficiency.

[0099] In this embodiment, the generation unit 112 obtains the optimal computation graph by reconstructing the computation graph topology in real time based on the gating mask using dynamic compilation technology, including: Obtain the complete computation graph of the entire multimodal model, and divide the complete computation graph into multiple independent subgraph units according to the modality type; wherein each subgraph unit includes the full-process computation logic of the corresponding single modality; The optimal computation graph is formed by selecting active subgraph cells from the subgraph cells according to the gating mask. The optimal computation graph is used to guide the fine-grained optimization of the feature vectors of the selected active modalities based on the low-dimensional long short-term memory network to obtain the active modal features.

[0100] For example, a visual subgraph can include the computation process of a convolutional neural network (CNN) + LSTM, an audio subgraph can include the computation process of short-time Fourier transform + LSTM, a financial time series subgraph can include the computation process of standardization + statistical feature extraction + LSTM, and a medical image subgraph can include the computation process of preprocessing + region of interest extraction + LSTM.

[0101] Subgraphs are dynamically scheduled based on gating masks, and only subgraphs corresponding to active modes are computed, while inactive mode subgraphs are skipped, thus avoiding invalid computations.

[0102] This allows LSTM and attention computation to be combined into a single CUDA core, reducing memory transfer overhead and improving computational efficiency.

[0103] Furthermore, it can pre-allocate the video memory space for each mode, avoiding the latency caused by dynamic memory allocation at runtime, thereby ensuring real-time processing performance.

[0104] Tests on graphics processors show that this optimization reduces multimodal inference latency by 12%.

[0105] Through the above embodiments, dynamic allocation and release of computing resources are realized at the hardware level, optimizing the execution efficiency of the multimodal gating-fusion pipeline and effectively improving inference speed compared to traditional static networks.

[0106] The generation unit 112 is further configured to filter and generate active modal features from the feature vectors of each modality based on the gating mask and the optimal computation graph.

[0107] In this embodiment, after filtering out the features in the active state according to the gating mask, the selected features can be further refined and optimized by controlling LSTM based on the optimal computation graph (such as more accurate historical state encoding, long-term dependency capture, etc.) to obtain the active modality features.

[0108] The fusion unit 113 is used to fuse the active modality features based on a query-key attention network to obtain fused features.

[0109] In this embodiment, the fusion unit 113 fuses the active modality features based on a query-key-value attention network to obtain fused features including: Calculate the product of the active modality features and the key projection matrix to obtain the key vector for each modality; Calculate the product of the active modality features and the value projection matrix to obtain the value vector for each modality; Historical fusion features are obtained and processed using a multi-layer perceptron (MLP) to obtain a query vector; The fusion weights are calculated based on the query vector and the key vector using a query-key-value attention mechanism. The fusion feature is obtained by weighted fusion based on the fusion weights and the value vectors.

[0110] The key projection matrix and the value projection matrix can be initialized using a random Gaussian distribution and can be optimized through training.

[0111] The fusion weights can be represented as follows: ; in, This represents the fusion weights at time t; Indicates the transpose of the query vector; This represents the bond vector of mode m at time t; This represents the feature dimension scaling factor, which is the dimension of the key vector.

[0112] The fusion features are represented as follows: ; in, This represents the fusion characteristics at time t; This represents the value vector of mode m at time t; , representing the active modality index set.

[0113] Through the above embodiments, it is possible to simultaneously consider the current feature similarity and historical context information, achieve adaptive and accurate fusion of active modality features, highlight the contribution of key modalities, and improve the representational ability of fused features.

[0114] The selection unit 114 is used to select target keyframes from the fusion features using a clustering algorithm.

[0115] In this embodiment, the selection unit 114 employs a clustering algorithm to select target keyframes from the fused features, including: Obtain the target scene to which the target multimodal data belongs, and configure the current sliding window according to the target scene; The fused feature is divided into continuous sub-features according to the current sliding window; wherein each sub-feature includes a fused feature at a continuous time step. Clustering is performed within each segment feature, and the feature closest to the corresponding cluster center is selected from each segment feature as each candidate keyframe; The keyframe selection network is used to select the candidate keyframe with the highest probability from each candidate keyframe as the target keyframe. The keyframe selection network comprises two sub-layers, each consisting of a multilayer perceptron and a sigmoid layer.

[0116] The entire keyframe selection process is executed cyclically using a time-series sliding window. The size of the sliding window can be flexibly configured according to the application scenario (e.g., a 5-minute window can be used in financial scenarios, and a 10-second window can be used in medical scenarios).

[0117] In the above embodiments, candidate keyframes are initially selected based on clustering algorithms to reduce computational load. Then, the target keyframe is further selected from the candidate keyframes through a keyframe selection network. This can accurately filter out the most representative keyframes, fully capture the event evolution process, avoid temporal discontinuities, and significantly improve the accuracy of keyframe extraction compared to traditional fixed fusion weight methods. At the same time, the loop execution mode can also adapt to the needs of long sequence data processing.

[0118] In this embodiment, noise is sampled based on the Gumbel distribution to obtain Gumbel noise; The contribution score of each mode is fused with the Gumbel noise to obtain the fusion result; The fusion result is processed by the Softmax function to obtain a differentiable gated mask; Forward computation is performed using the differentiable gating mask, and backward propagation is performed based on the differentiable gating mask to update the network parameters of the low-dimensional long short-term memory network, the gating network, and the query-key-value attention network.

[0119] Through the above embodiments, the Gumbel-Softmax technique can be used to make the gating decision differentiable, supporting end-to-end training. The differentiable design ensures the integrity of model training.

[0120] This embodiment can be applied to multiple fields, such as video analysis, monitoring systems, autonomous driving, and medical image analysis. It aims to improve the accuracy and efficiency of keyframe extraction while reducing computing resource consumption. It is particularly suitable for real-time multimodal analysis in resource-constrained environments, such as abnormal event detection in intelligent monitoring, multi-sensor fusion in autonomous driving, and synchronous analysis of physiological signals in telemedicine.

[0121] For example: surveillance videos in bank lobbies or operating rooms: 1. Input the following data collected synchronously: Visual modality: 1920×1080 resolution video (10fps); Audio modality: 16kHz sampling rate mono audio; Motion mode: Optical flow characteristics (calculated using the Farneback dense optical flow algorithm).

[0122] 2. The key event was: a pedestrian suddenly ran in the surveillance footage (t=32 frames) accompanied by shouting.

[0123] 3. The output of the time series modeling includes: , , ; in, This represents the contribution score of the visual modality at time t=32; This represents the contribution score of the audio modality at time t=32; This represents the contribution score of the motion mode at time t=32.

[0124] Top-2 gating selects audio and motion modalities (such as visual modalities being temporarily pruned).

[0125] 4. Feature fusion: Motion mode at time t Attention fusion weights: (Due to significant abrupt changes in optical flow, the weighting is higher); Audio modality at time t Attention fusion weights: .

[0126] 5. Function of the feature library: At frame t=33, the pedestrian leaves the frame, and the motion mode becomes invalid (the gating mask of the motion mode at t=33). ); When the visual modality is reactivated at frame t=34, it is taken from the first-in-first-out queue of the visual modality. Recover the features of frame t=31 to maintain coherence.

[0127] 6. The output results include: accurately marking frames t=32-34 as keyframe sequences, fully capturing the event evolution process of "stillness → running → disappearance", while traditional methods may miss audio clues or produce temporal breaks due to fixed fusion weights.

[0128] As can be seen from the above technical solutions, this invention can calculate the contribution score of each modality based on a low-dimensional long short-term memory network and a modality-specific feature library, so as to accurately capture the temporal change pattern of each modality feature based on historical feature states and maintain temporal continuity; it generates active modality features based on a gating network and an optimal computation graph, which can retain active modalities with high contribution at the current moment, reduce computational costs, and realize dynamic allocation and release of computing resources at the hardware level based on the optimal computation graph, thereby improving real-time performance; it fuses active modality features based on a query-key-value attention network, and considers the similarity of current features and historical context information to achieve adaptive and accurate fusion of active modality features; and it uses a clustering algorithm to select target keyframes from the fused features to further avoid temporal discontinuities.

[0129] like Figure 3 The diagram shown is a schematic representation of the structure of a computer device that implements a preferred embodiment of the keyframe extraction method based on time-gating according to the present invention.

[0130] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a keyframe extraction program based on timing gating.

[0131] Those skilled in the art will understand that the schematic diagram is merely an example of computer device 1 and does not constitute a limitation on computer device 1. Computer device 1 can be either a bus topology or a star topology. Computer device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, computer device 1 may also include input / output devices, network access devices, etc.

[0132] It should be noted that the computer device 1 described is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0133] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a portable hard drive of the computer device 1. In other embodiments, the memory 12 can be an external storage device of the computer device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device 1. Furthermore, the memory 12 can include both internal storage units and external storage devices of the computer device 1. The memory 12 can be used not only to store application software and various types of data installed on the computer device 1, such as the code of a keyframe extraction program based on timing gating, but also to temporarily store data that has been output or will be output.

[0134] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the computer device 1, connecting various components of the computer device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing a keyframe extraction program based on timing gating) and calls data stored in the memory 12 to perform various functions of the computer device 1 and process data.

[0135] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the various embodiments of the time-gating-based keyframe extraction method described above, for example... Figure 1 The steps are shown.

[0136] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into a preprocessing unit 110, a processing unit 111, a generation unit 112, a fusion unit 113, and a selection unit 114.

[0137] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the time-gated keyframe extraction method described in the various embodiments of this invention.

[0138] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0139] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0140] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0141] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0142] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The bus is represented by only one straight line, but this does not mean that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0143] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0144] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the computer device 1 and other computer devices.

[0145] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the computer device 1 and to display a visual user interface.

[0146] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0147] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0148] Combination Figure 1 The memory 12 in the computer device 1 stores multiple instructions to implement a keyframe extraction method based on time-gating, and the processor 13 can execute the multiple instructions to achieve the following: In response to the keyframe extraction instruction for the target multimodal data, the target multimodal data is preprocessed to obtain the feature vector of each modality; The feature vectors of each mode are processed using a low-dimensional long short-term memory network and a mode-specific feature library to obtain the contribution score of each mode. In a gated network, a gate mask is generated based on the contribution score of each mode, and the optimal computation graph is obtained by reconstructing the topology of the computation graph in real time based on the gate mask using dynamic compilation technology. Active modal features are generated by filtering the feature vectors of each modality based on the gating mask and the optimal computation graph. The active modality features are fused using a query-key-value attention network to obtain fused features; A clustering algorithm is used to select target keyframes from the fused features.

[0149] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0150] It should be noted that all data involved in this case was legally obtained. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0151] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0152] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0153] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0155] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0156] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0157] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this invention can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A keyframe extraction method based on temporal gating, characterized in that, The keyframe extraction method based on temporal gating includes: In response to the keyframe extraction instruction for the target multimodal data, the target multimodal data is preprocessed to obtain the feature vector of each modality; The feature vectors of each mode are processed using a low-dimensional long short-term memory network and a mode-specific feature library to obtain the contribution score of each mode. In a gated network, a gate mask is generated based on the contribution score of each mode, and the optimal computation graph is obtained by reconstructing the topology of the computation graph in real time based on the gate mask using dynamic compilation technology. Active modal features are generated by filtering the feature vectors of each modality based on the gating mask and the optimal computation graph. The active modality features are fused using a query-key-value attention network to obtain fused features; A clustering algorithm is used to select target keyframes from the fused features.

2. The keyframe extraction method based on temporal gating as described in claim 1, characterized in that, The feature vector of each modality is processed based on a low-dimensional long short-term memory network and a modality-specific feature library to obtain the contribution score of each modality, including: Calculate the feature difference between adjacent frames based on the feature vector of each modality; The most recently written features are read from the modality-specific feature library as the initial hidden state of the low-dimensional long short-term memory network; Starting from the initial hidden state, the contribution score of each modality is calculated based on the feature difference between adjacent frames; The modality-specific feature library includes a fixed-capacity first-in-first-out (FIFO) queue maintained for each modality. When an active modality is detected, the current features of that modality are written to the corresponding FIFO queue in real time, and old features are deleted according to the feature writing time when the corresponding FIFO queue is full. When a modality is detected to change from an active state to an inactive state, the writing of features to the corresponding FIFO queue for that modality is stopped, and the written features in the corresponding FIFO queue for that modality are retained. When a modality is detected to change from an inactive state to an active state, the most recently written features are read from the corresponding FIFO queue for that modality as the initial hidden state of the low-dimensional long short-term memory network. When a modality is detected to change from an inactive state to an active state, and the corresponding FIFO queue for that modality is empty, zero-vector filling is used as the initial hidden state of the low-dimensional long short-term memory network.

3. The keyframe extraction method based on temporal gating as described in claim 1, characterized in that, The generation of the gate mask based on the contribution score of each modality includes: The contribution scores of each mode are sorted in descending order to obtain an ordered sequence; Starting from the first contribution score in the ordered sequence, the scores are accumulated sequentially until the accumulated value reaches a preset threshold. The number of modes currently participating in the accumulation is then determined as the number of active modes. The gate mask is generated based on the number of active modes; Specifically, after generating the gating mask, the unified computing device architecture stream synchronization is triggered.

4. The keyframe extraction method based on time-gated control as described in claim 1, characterized in that, The process of obtaining the optimal computation graph by real-time reconstruction of the computation graph topology based on the gating mask using dynamic compilation technology includes: Obtain the complete computation graph of the entire multimodal model, and divide the complete computation graph into multiple independent subgraph units according to the modality type; wherein, each subgraph unit includes the full-process computation logic of the corresponding single modality; The optimal computation graph is formed by selecting active subgraph cells from the subgraph cells according to the gating mask. The optimal computation graph is used to guide the fine-grained optimization of the feature vectors of the selected active modalities based on the low-dimensional long short-term memory network to obtain the active modal features.

5. The keyframe extraction method based on temporal gating as described in claim 1, characterized in that, The query-key-value attention network is used to fuse the active modality features to obtain the fused features, which include: Calculate the product of the active modality features and the key projection matrix to obtain the key vector for each modality; Calculate the product of the active modality features and the value projection matrix to obtain the value vector for each modality; Historical fusion features are obtained, and the historical fusion features are processed using a multilayer perceptron to obtain a query vector; The fusion weights are calculated based on the query vector and the key vector using a query-key-value attention mechanism. The fusion feature is obtained by weighted fusion based on the fusion weights and the value vectors.

6. The keyframe extraction method based on temporal gating as described in claim 1, characterized in that, The step of using a clustering algorithm to select target keyframes from the fused features includes: Obtain the target scene to which the target multimodal data belongs, and configure the current sliding window according to the target scene; The fused feature is divided into continuous sub-features according to the current sliding window; wherein each sub-feature includes a fused feature at a continuous time step. Clustering is performed within each segment feature, and the feature closest to the corresponding cluster center is selected from each segment feature as each candidate keyframe; The keyframe selection network is used to select the candidate keyframe with the highest probability from each candidate keyframe as the target keyframe. The keyframe selection network comprises two sub-layers, each consisting of a multilayer perceptron and a sigmoid layer.

7. The keyframe extraction method based on temporal gating as described in claim 1, characterized in that, The method further includes: Noise is sampled based on the Günber distribution to obtain Günber noise; The contribution score of each mode is fused with the Gumbel noise to obtain the fusion result; The fusion result is processed by the Softmax function to obtain a differentiable gated mask; Forward computation is performed using the differentiable gating mask, and backward propagation is performed based on the differentiable gating mask to update the network parameters of the low-dimensional long short-term memory network, the gating network, and the query-key-value attention network.

8. A keyframe extraction device based on time-gated control, characterized in that, The keyframe extraction device based on time-gated control includes: The preprocessing unit is used to preprocess the target multimodal data in response to the keyframe extraction instruction for the target multimodal data to obtain the feature vector of each modality; The processing unit is used to process the feature vector of each mode based on a low-dimensional long short-term memory network and a mode-specific feature library to obtain the contribution score of each mode. The generation unit is used to generate a gating mask in the gating network based on the contribution score of each mode, and to reconstruct the topology of the computation graph in real time based on the gating mask using dynamic compilation technology to obtain the optimal computation graph. The generation unit is further configured to filter and generate active modal features from the feature vectors of each modality based on the gating mask and the optimal computation graph; The fusion unit is used to fuse the active modality features based on a query-key-value attention network to obtain fused features; The selection unit is used to select target keyframes from the fused features using a clustering algorithm.

9. A computer device, characterized in that, The computer device includes: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the keyframe extraction method based on timing gating as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in a computer device to implement the keyframe extraction method based on time-gated control as described in any one of claims 1 to 7.

Images