A key frame extraction method, device, equipment and medium

Abstract

The application relates to the technical field of artificial intelligence, in particular to a key frame extraction method and device, equipment and medium. Applied to a medical scene, in the application, the advantages of multi-modal information such as images, audio and text in a video are fully combined, the contribution degrees of various modes to key frame prediction are dynamically calculated, and adaptive weighted fusion of features is realized. The fusion mode based on the contribution degree avoids the problems of information redundancy or weakening of key features that may be caused by traditional fixed weight fusion, so that the feature vector after fusion can more accurately reflect the importance of the video frame. Compared with a single mode key frame extraction method, the method comprehensively considers the rich information of the video content in the visual, auditory and semantic levels, thereby effectively improving the accuracy and robustness of the key frame extraction result, and better meeting the demand of video retrieval, content summary, intelligent analysis and other practical application scenarios for high-quality key frames.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a keyframe extraction method, apparatus, device, and medium. Background Technology

[0002] In the medical field, keyframe extraction technology is commonly used for the analysis and processing of medical image sequences (such as echocardiography, endoscopic videos, and dynamic CT / MRI sequences). Traditional methods mainly rely on handcrafted features (such as SIFT or HOG) of a single modality (e.g., visual information) for keyframe selection. However, these methods struggle to capture complex spatiotemporal relationships and have limited generalization ability in multimodal scenarios. In recent years, deep learning technology, through the combined use of convolutional neural networks (CNNs) and long short-term memory networks (LSTMs), has significantly improved the ability to extract spatiotemporal features. However, existing methods typically employ simple feature concatenation or weighted averaging in multimodal fusion, lacking dynamic modeling of intermodal interactions. This results in low information fusion efficiency and consequently, low accuracy in keyframe extraction. Therefore, improving the accuracy of keyframes during keyframe extraction has become a pressing issue. Summary of the Invention

[0003] In view of this, embodiments of this application provide a keyframe extraction method, apparatus, device, and medium to solve the problem of low accuracy in keyframe extraction.

[0004] In a first aspect, embodiments of this application provide a keyframe extraction method, the keyframe extraction method comprising: Acquire the video to be extracted, which includes multiple video frames, each video frame including image, audio signal and text data; For any video frame, feature extraction is performed on the image to obtain an image feature vector; feature extraction is performed on the audio signal to obtain an audio feature vector; and feature extraction is performed on the text data to obtain a text feature vector. Based on the image feature vector, the audio feature vector, and the text feature vector, calculate the contribution of each modality in the image, the audio signal, and the text data to the keyframe prediction; Based on the contribution of each modality to the keyframe prediction, the image feature vector, the audio feature vector, and the text feature vector are fused to obtain the fused feature vector. Based on the fused feature vectors of all video frames, keyframes are extracted to obtain keyframe extraction results.

[0005] Secondly, embodiments of this application provide a keyframe extraction device, the keyframe extraction device comprising: The acquisition module is used to acquire the video to be extracted, which includes multiple video frames, and each video frame includes image, audio signal and text data; The extraction module is used to extract features from the image for any video frame to obtain an image feature vector, extract features from the audio signal to obtain an audio feature vector, and extract features from the text data to obtain a text feature vector. The calculation module is used to calculate the contribution of each modality in the image, audio signal and text data to keyframe prediction based on the image feature vector, the audio feature vector and the text feature vector; The fusion module is used to fuse the image feature vector, the audio feature vector, and the text feature vector according to the contribution of each modality to the keyframe prediction, so as to obtain the fused feature vector. The module is used to extract keyframes based on the fused feature vectors of all video frames, and obtain keyframe extraction results.

[0006] Thirdly, embodiments of this application provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the keyframe extraction method as described above.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the keyframe extraction method as described above.

[0008] The advantages of this application compared to the prior art are: This application fully leverages the advantages of multimodal information from images, audio, and text in videos, achieving adaptive weighted fusion of features by dynamically calculating the contribution of each modality to keyframe prediction. This contribution-based fusion method avoids the information redundancy or weakening of key features that may occur with traditional fixed-weight fusion, enabling the fused feature vector to more accurately reflect the importance of video frames. Compared to single-modal keyframe extraction methods, this method comprehensively considers the rich information of video content at the visual, auditory, and semantic levels, thereby effectively improving the accuracy and robustness of keyframe extraction results and better meeting the needs of high-quality keyframes in practical application scenarios such as video retrieval, content summarization, and intelligent analysis. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic diagram of the application environment of a keyframe extraction method provided in an embodiment of this application; Figure 2 This is a schematic flowchart of a keyframe extraction method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a keyframe extraction device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0013] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0014] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0015] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0016] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0017] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0018] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0019] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0020] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0021] To illustrate the technical solution of this application, specific embodiments are described below.

[0022] One embodiment of this application provides a keyframe extraction method that can be applied to, for example... Figure 1 In this application environment, the client communicates with the server. The client includes, but is not limited to, handheld computers, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, and personal digital assistants (PDAs). The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0023] To illustrate the technical solution of this application, specific embodiments are described below.

[0024] See Figure 2 This is a flowchart illustrating a keyframe extraction method provided in an embodiment of this application, as shown below. Figure 2 As shown, the keyframe extraction method may include the following steps.

[0025] S201: Obtain the video to be extracted. The video to be extracted includes multiple video frames, and each video frame includes image, audio signal and text data.

[0026] In step S201, the video to be extracted is an audio-visual video of a certain period of time, including multiple video frames, each video frame including images, audio signals and text data.

[0027] In this embodiment, in the modern medical system, video has become one of the core carriers for clinical diagnosis, surgical guidance, medical teaching, and case review. From real-time images of laparoscopic surgery and dynamic monitoring of ultrasound examinations to video interaction in remote consultations and process recording of medical experiments, massive amounts of medical video data contain crucial information related to disease assessment and medical quality. However, medical videos often have characteristics such as long duration, uneven information density, and fragmented key content. If medical staff browse frame by frame, it will not only consume a lot of time and energy, but may also miss core diagnostic and treatment details due to visual fatigue. Against this background, video keyframe extraction technology based on medical scenarios has emerged. It can accurately capture key frames in videos that carry core medical information, enabling efficient screening, focusing, and value extraction of medical video resources, providing strong technical support for diverse medical scenarios.

[0028] In this embodiment, keyframe extraction involves acquiring the video to be extracted. This video can be obtained through methods such as accessing a video storage database, a real-time streaming interface, or uploading a local file from the user. The video storage database can pre-store a large amount of historical video data, allowing the system to quickly locate and retrieve the target video based on video identifier information via a query interface. The real-time streaming interface is suitable for processing dynamic content such as live videos, enabling real-time reception and parsing of video stream data. The user's local file upload method allows users to directly submit video files from their personal devices to the system, meeting personalized video processing needs. During the acquisition process, the system automatically detects and converts the video format to ensure that subsequent processing modules are compatible with video data of different encoding formats. It also verifies the video integrity; if data is missing or corrupted, a re-acquisition mechanism will be triggered or an error message will be sent to the user.

[0029] S202: For any video frame, perform feature extraction on the image to obtain an image feature vector, perform feature extraction on the audio signal to obtain an audio feature vector, and perform feature extraction on the text data to obtain a text feature vector.

[0030] In step S202, the image feature vector represents the spatial features of the image, the audio feature vector represents the audio features in the video frame, and the text feature vector represents the text features in the video frame.

[0031] In this embodiment, for any video frame, feature extraction is performed on the image to obtain an image feature vector, feature extraction is performed on the audio signal to obtain an audio feature vector, and feature extraction is performed on the text data to obtain a text feature vector. Image feature extraction can employ a convolutional neural network model, using multi-layer convolutional operations to capture low-level features such as edges, textures, and colors, as well as high-level semantic features such as object contours and spatial layout. Audio feature extraction can utilize Mel-frequency cepstral coefficients (MFCC) or Mel-frequency spectrograms, combined with recurrent neural networks to model the temporal and spectral features of the audio. Text feature extraction can leverage pre-trained language models, such as the BERT model, to convert text data into vector representations containing semantic information, making feature vectors from different modalities comparable in dimensionality and semantic space, laying the foundation for subsequent multimodal feature fusion.

[0032] It should be noted that after acquiring the video to be extracted, preprocessing is performed. This preprocessing includes size normalization and noise removal of the image portions of the video frames to unify image resolution and reduce interference. Simultaneously, the audio signal undergoes sampling rate conversion and filtering to ensure audio data consistency. For text data, format standardization is performed, converting different text formats into a unified character encoding format to provide high-quality foundational data for subsequent keyframe extraction.

[0033] Optionally, feature extraction is performed on the image to obtain an image feature vector, and feature extraction is performed on the audio signal to obtain an audio feature vector, including: Spatial feature extraction is performed on the image to obtain a spatial feature map. Temporal feature extraction is then performed on the spatial feature map based on a pre-defined bidirectional recurrent neural network to obtain an image feature vector. The audio signal is processed to obtain the Mel spectrogram of the audio signal. Temporal features are extracted from the Mel spectrogram to obtain the audio feature vector.

[0034] In this embodiment, spatial feature extraction can employ a pre-trained MobileNetV3 model. Through layer-by-layer convolution and pooling operations, spatial features at different levels are extracted from the image to form multi-scale spatial feature maps, thereby preserving the positional relationships and detailed information of targets in the image. Bidirectional recurrent neural networks (such as BiLSTM) can simultaneously utilize the historical and future temporal dependencies of spatial feature maps. Through a gating mechanism, they can selectively memorize and forget key temporal information, thus transforming static spatial features into image feature vectors with temporal dynamic characteristics.

[0035] For audio signal processing, the original audio is first segmented, windowed, and subjected to Fourier transform to convert the time-domain signal into a spectrogram. Then, it is mapped to a Mel spectrogram through a Mel filter bank to simulate the sensitivity of the human ear to different frequencies. Next, a convolutional neural network can be used to extract the local spectral features of the Mel spectrogram, and then a long short-term memory network (LSTM) can be combined to model the dynamic characteristics of the spectrogram over time. Finally, an audio feature vector containing comprehensive information in both the time and frequency domains of the audio is output.

[0036] S203: Calculate the contribution of each modality in the image, audio signal, and text data to keyframe prediction based on the image feature vector, audio feature vector, and text feature vector.

[0037] In step S203, the contribution is the degree of importance of the corresponding mode in the keyframe prediction process.

[0038] In this embodiment, the contribution of each modality in the image, audio signal, and text data to keyframe prediction is calculated based on the image feature vector, audio feature vector, and text feature vector.

[0039] The formula for calculating the contribution of each modality to keyframe prediction is as follows: in, S represents the contribution of the video frame at time t in the k-th modality, where K is the total number of modalities (3 in this embodiment), k is the k-th modality, and S does not contain any subset of the k-th modality (e.g., when K=3, S can be {1}, {2}, {1,2}, etc.). The final prediction function of a multimodal model, After adding the k-th mode to the modality subset S, the predicted value is obtained through feature fusion and model inference. The predicted value is obtained by reasoning using only the features of a subset of modalities S.

[0040] In this embodiment, the contribution is represented by the corresponding SHAP (SHapley Additive exPlanations) value. The SHAP value, through a game-theoretic-based explanatory model, decomposes the keyframe prediction result into the sum of the marginal contributions of each modality's features, quantifying the positive or negative impact of different modalities in the prediction process. Specifically, for the image modality, its SHAP value reflects the proportion of contribution of dynamic temporal information in the spatial feature map to the saliency of the keyframe event. The SHAP value for the audio modality reflects the correlation strength between the temporal variation characteristics of the spectrogram and the audio features of the keyframe. The SHAP value for the text modality measures the degree to which semantic information in the text data supports the description of the keyframe content.

[0041] S204: Based on the contribution of each modality to the keyframe prediction, perform feature fusion on the image feature vector, audio feature vector, and text feature vector to obtain the fused feature vector.

[0042] In step S204, the fused feature vector is obtained by weighted summation of the contribution of each modality and its corresponding feature vector. The fused feature vector characterizes the importance of different modalities in keyframe determination at different times, preserving the unique information of each modality while achieving organic integration of multimodal features, thus laying the foundation for subsequent keyframe prediction based on fused features.

[0043] In this embodiment, based on the contribution of each modality to keyframe prediction, the image feature vector, audio feature vector, and text feature vector are fused to obtain the fused feature vector. That is, the contribution of each modality to keyframe prediction is weighted and summed with the image feature vector, audio feature vector, and text feature vector to obtain the fused feature vector.

[0044] In this embodiment, the image feature vector, audio feature vector, and text feature vector are multiplied by their respective contribution coefficients, and then the three weighted vectors are summed element-wise to generate a fused feature vector containing multimodal key information. This fusion method can adaptively highlight the feature representation of high-contribution modalities while suppressing redundant information in low-contribution modalities, making the fused features more focused on the modal features that play a dominant role in keyframe prediction, thus laying the foundation for accurate identification of subsequent keyframes.

[0045] Optionally, based on the contribution of each modality to keyframe prediction, feature fusion is performed on the image feature vector, audio feature vector, and text feature vector to obtain a fused feature vector, including: Linear transformations are performed on the image feature vector, audio feature vector, and text feature vector to obtain the attention weight value for each modality; Based on the attention weight value of each modality, the image feature vector, audio feature vector, and text feature vector are weighted to obtain the image attention feature vector, audio attention feature vector, and text attention feature vector. Based on the contribution of each modality to keyframe prediction, the image attention feature vector, audio attention feature vector, and text attention feature vector are weighted and summed to obtain the fused feature vector.

[0046] In this embodiment, a linear transformation is performed on the image feature vector, audio feature vector, and text feature vector to obtain the attention weight value for each modality. The calculation formula for the attention weight value of each modality is as follows: in, Let be the attention weight value of the video frame at time t in the k-th modality. The function is one that transforms any real vector into a probability distribution. This is the weight matrix (implemented via 1×1 convolution) corresponding to the video frame at time t in the k-th modality. Its function is to capture the dependencies between feature channels. Let be the feature vector of the video frame at time t in the k-th modality. This is a bias term.

[0047] In this embodiment, these attention weights quantify the importance of different modal features in the keyframe prediction task, enabling dynamic allocation of multimodal data. For example, when there are significant visual changes in the video to be extracted, the attention weight of the image modality is increased accordingly, thus giving it higher priority in the subsequent feature fusion process. When the audio signal contains key dialogue or environmental sound effects, the weight of the audio modality is enhanced to highlight its impact on keyframe decision-making. The attention weight of the text modality is adjusted based on the semantic matching degree between the text description and the video content, ensuring that the text information can effectively assist in the accurate localization of keyframes.

[0048] Based on the attention weight value of each modality, the image feature vector, audio feature vector, and text feature vector are weighted to obtain the image attention feature vector, audio attention feature vector, and text attention feature vector. The calculation formula for the attention feature vector of each modality is as follows: in, Let be the attention feature vector of the video frame at time t in the k-th modality. , Let be the feature vector of the video frame at time t in the k-th mode.

[0049] Based on the contribution of each modality to keyframe prediction, the image attention feature vector, audio attention feature vector, and text attention feature vector are weighted and summed to obtain the fused feature vector.

[0050] In this embodiment, the image attention feature vector, audio attention feature vector, and text attention feature vector are weighted and summed according to the contribution of each modality to keyframe prediction, resulting in a fused feature vector. This approach fully considers the differentiated roles of different modal data in the keyframe prediction task, avoiding fusion bias caused by an excessively high or low proportion of a single modality feature. By dynamically assigning contribution weights related to keyframe determination for each modality, the visual saliency of the image, the key event cues in the audio, and the semantic guidance information in the text can work synergistically, effectively improving the ability of the fused features to represent key information in the video content and laying a more reliable data foundation for subsequent accurate keyframe extraction.

[0051] Optionally, based on the contribution of each modality to keyframe prediction, feature fusion is performed on the image feature vector, audio feature vector, and text feature vector to obtain a fused feature vector, which also includes: The target weight value of each modality is calculated based on the attention weight value of each modality and the contribution of each modality to the keyframe prediction. The image attention feature vector, audio attention feature vector, and text attention feature vector are weighted and summed according to the target weight value of each modality to obtain the fused feature vector.

[0052] In this embodiment, the attention weight value of each modality is added to the contribution of each modality to the keyframe prediction to obtain the target weight value of each modality. Based on the target weight value of each modality, the image attention feature vector, audio attention feature vector and text attention feature vector are weighted and summed to obtain the fused feature vector.

[0053] In this embodiment, the target weight value of each modality is calculated based on the attention weight value of each modality and the contribution of each modality to keyframe prediction. This comprehensively considers the dynamic focusing ability of different modal data under the attention mechanism and their inherent contribution to the keyframe prediction task, thus making the target weight value of each modality more in line with the actual prediction requirements. The target weight value calculated in this way retains the focusing information of important features within the modality reflected by the attention weight value, and also incorporates the proportion of influence between modalities on the overall task reflected by the contribution. This allows for a more accurate highlighting of key modal information and suppression of secondary modal interference when subsequently weighting and summing the image attention feature vector, audio attention feature vector, and text attention feature vector, ultimately improving the representational ability of the fused feature vector.

[0054] Optionally, based on the attention weight value of each modality and the contribution of each modality to keyframe prediction, the target weight value of each modality is calculated, including: For any modality, calculate the rate of increase of the attention weight value corresponding to the modality relative to the contribution. Based on the increase rate, the contribution is adjusted to obtain the target weight value for the corresponding mode.

[0055] In this embodiment, for any modality, the increase rate of the attention weight value corresponding to the modality relative to the contribution is calculated; that is, the ratio of the difference between the attention weight value and the contribution of the modality to the contribution value is used to obtain the increase rate. The product of the increase rate and the contribution value is used to obtain the corresponding increase amount. The sum of the increase amount and the contribution value is used to obtain the corresponding target weight value.

[0056] S205: Extract keyframes based on the fused feature vectors of all video frames to obtain keyframe extraction results.

[0057] In step S205, a key frame refers to a selected, iconic video frame in a continuous video frame sequence that can characterize core features or states.

[0058] In this embodiment, keyframes are extracted based on the fused feature vectors of all video frames to obtain keyframe extraction results. Specifically, the fused feature vectors are input into a preset keyframe classification model. This model comprehensively evaluates the fused feature vectors of each video frame and outputs a probability value indicating whether the video frame is a keyframe. The model learns the differences in fused features between keyframes and non-keyframes through training data. When the fused feature vectors are input, it can classify video frames based on the learned feature patterns, thereby selecting video frames with probability values ​​higher than a preset threshold as keyframes. During model training, a cross-entropy loss function is used to optimize model parameters to improve the accuracy and robustness of keyframe classification, ensuring that the extracted keyframes accurately reflect the core content of the video sequence.

[0059] In this embodiment, keyframes are extracted based on the fused feature vectors of all video frames to obtain keyframe extraction results. This fully integrates the multi-dimensional feature information of video frames and uses a preset keyframe classification model to accurately evaluate and classify the fused feature vectors, effectively improving the accuracy and reliability of keyframe extraction.

[0060] Optionally, keyframes are extracted based on the fused feature vectors of all video frames to obtain keyframe extraction results, including: The fused feature vector is processed using a pre-defined neural network to obtain the keyframe probability value for each video frame. The video frame whose keyframe probability value is greater than a preset threshold is identified as a keyframe.

[0061] In this embodiment, the preset neural network can adopt a deep learning architecture that includes an attention mechanism. By assigning dynamic weights to different dimensions of information in the fused feature vector, it focuses on feature components closely related to the core content of the video, thereby improving the accuracy of keyframe probability value calculation. The preset neural network processes the fused feature vector to obtain the keyframe probability value of each video frame. Video frames with keyframe probability values ​​greater than a preset threshold are identified as keyframes. The setting of the preset threshold needs to be combined with the actual distribution characteristics of the video data. It can be achieved by statistical analysis on a large-scale labeled dataset to select a value that achieves the best balance between keyframe extraction accuracy and recall, ensuring that important keyframes are not missed due to an excessively high threshold, nor are too many redundant frames introduced due to an excessively low threshold.

[0062] Optionally, based on the fused feature vectors of all video frames, keyframes are extracted. After obtaining the keyframe extraction results, the method further includes: For any target video frame in the keyframe extraction results, spatial interpretation processing is performed on the target video frame to obtain the gradient-weighted class activation heatmap of the target video frame. Based on the contribution of each modality in all target video frames to the keyframe prediction, and sorted chronologically, the contribution of each modality over time is obtained.

[0063] In this embodiment, for any target video frame in the keyframe extraction results, spatial interpretation processing is performed on the target video frame to obtain a gradient-weighted activation heatmap of the target video frame. The gradient-weighted activation heatmap is a visualization tool that can accurately locate the regions in the target video frame that play a decisive role in model decisions, providing quantitative and visual support for the analytical value of the keyframes.

[0064] Convolutional neural networks (CNNs) are used to extract features from target video frames. The last convolutional layer in the model is selected as the feature extraction layer—the feature map of this layer preserves both intra-frame spatial location information and high-level semantic information, representing a balance between spatial localization and semantic association. For example, ResNet-50-based models typically choose the "conv5_block3_out" layer as the key layer. The target video frame is input into the video analysis model, and forward propagation is performed to the key convolutional layer, saving the feature map tensor output by this layer. The target category is defined, and the gradient of the score corresponding to that category in the model output with respect to the key layer feature map is calculated. Global average pooling is performed on the gradient tensor of each feature channel, converting the single-channel gradient into a scalar value, and the "average importance" of the gradient for that channel is calculated. If the gradient pooling value of a channel is positive, it indicates that the feature of that channel has a positive contribution to the target category; if it is negative, it has a negative contribution. Through weighted fusion of weights and feature maps, high-level features are mapped back to the original pixel space to form a heatmap.

[0065] In this embodiment, based on the contribution of each modality to keyframe prediction in all target video frames, and sorted chronologically, the contribution of each modality over time is obtained. By analyzing the contribution curves of different modalities on the time axis, the dynamic changes of each modality's information during the video keyframe recognition process can be intuitively observed. For example, the contribution of some modalities increases significantly during specific time periods of the video, while remaining at a low level during other time periods. This variation pattern helps to deeply understand the impact mechanism of different modal features on the keyframe extraction task.

[0066] This application fully leverages the advantages of multimodal information from images, audio, and text in videos, achieving adaptive weighted fusion of features by dynamically calculating the contribution of each modality to keyframe prediction. This contribution-based fusion method avoids the information redundancy or weakening of key features that may occur with traditional fixed-weight fusion, enabling the fused feature vector to more accurately reflect the importance of video frames. Compared to single-modal keyframe extraction methods, this method comprehensively considers the rich information of video content at the visual, auditory, and semantic levels, thereby effectively improving the accuracy and robustness of keyframe extraction results and better meeting the needs of high-quality keyframes in practical application scenarios such as video retrieval, content summarization, and intelligent analysis.

[0067] Please see Figure 3 , Figure 3 This is a schematic diagram of a keyframe extraction device according to an embodiment of this application. This keyframe extraction device corresponds one-to-one with the keyframe extraction method in the above embodiments. Please refer to [link / reference] for details. Figure 2 as well as Figure 2 The relevant descriptions in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown. See also... Figure 3 The keyframe extraction device 30 includes: an acquisition module 31, an extraction module 32, a calculation module 33, a fusion module 34, and a result module 35.

[0068] The acquisition module 31 is used to acquire the video to be extracted. The video to be extracted includes multiple video frames, and each video frame includes image, audio signal and text data.

[0069] The extraction module 32 is used to extract features from an image for any video frame to obtain an image feature vector, extract features from an audio signal to obtain an audio feature vector, and extract features from text data to obtain a text feature vector.

[0070] The calculation module 33 is used to calculate the contribution of each modality in the image, audio signal and text data to the keyframe prediction based on the image feature vector, audio feature vector and text feature vector.

[0071] The fusion module 34 is used to fuse the image feature vector, audio feature vector and text feature vector according to the contribution of each modality to the keyframe prediction, so as to obtain the fused feature vector.

[0072] Module 35 is used to extract keyframes based on the fused feature vectors of all video frames, and obtain the keyframe extraction results.

[0073] Optionally, the extraction module 32 includes: The first extraction unit is used to extract spatial features from the image to obtain a spatial feature map. Based on a preset bidirectional recurrent neural network, temporal features are extracted from the spatial feature map to obtain an image feature vector.

[0074] The second extraction unit is used to process the audio signal to obtain the Mel spectrogram of the audio signal, and to extract the temporal features from the Mel spectrogram to obtain the audio feature vector.

[0075] Optionally, the fusion module 34 includes: The transformation unit is used to perform linear transformations on the image feature vector, audio feature vector, and text feature vector to obtain the attention weight value for each modality.

[0076] The weighting unit is used to weight the image feature vector, audio feature vector, and text feature vector according to the attention weight value of each modality, so as to obtain the image attention feature vector, audio attention feature vector, and text attention feature vector.

[0077] The first summation unit is used to perform weighted summation of the image attention feature vector, audio attention feature vector, and text attention feature vector based on the contribution of each modality to the keyframe prediction, so as to obtain the fused feature vector.

[0078] Optionally, the fusion module 34 further includes: The calculation unit is used to calculate the target weight value of each modality based on the attention weight value of each modality and the contribution of each modality to the keyframe prediction.

[0079] The second summation unit is used to perform weighted summation of the image attention feature vector, audio attention feature vector, and text attention feature vector based on the target weight value of each modality, so as to obtain the fused feature vector.

[0080] Optionally, the above-mentioned computing unit includes: The computational subunit is used to calculate the rate of increase of the attention weight value relative to the contribution for any given modality.

[0081] The correction unit is used to correct the contribution based on the increase rate to obtain the target weight value of the corresponding mode.

[0082] Optionally, the above-mentioned module 35 includes: The unit is used to process the fused feature vector using a preset neural network to obtain the keyframe probability value for each video frame.

[0083] The determination unit is used to determine the video frame whose key frame probability value is greater than a preset threshold as a key frame.

[0084] Optionally, the keyframe extraction device 30 further includes: The processing module is used to perform spatial interpretation processing on any target video frame in the keyframe extraction results to obtain the gradient-weighted class activation heatmap of the target video frame.

[0085] The sorting module is used to sort the contributions of each modality to keyframe prediction in all target video frames according to time order, and obtain the contribution of each modality as time changes.

[0086] It should be noted that the information interaction and execution process between the above-mentioned units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0087] Figure 4 This is a schematic diagram of the structure of a computer device provided in one embodiment of this application. For example... Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown in the diagram), a memory, and a computer program stored in the memory and capable of running on at least one processor, wherein the processor executes the computer program to implement the steps in any of the above-described keyframe extraction method embodiments.

[0088] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 4 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0089] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0090] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of the computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0091] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0092] The implementation of all or part of the processes in the methods of the above embodiments can also be accomplished by a computer program product. When the computer program product is run on a computer device, it enables the computer device to execute the steps in the above method embodiments.

[0093] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0094] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0095] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0096] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0097] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A keyframe extraction method, characterized in that, The keyframe extraction method includes: Acquire the video to be extracted, which includes multiple video frames, each video frame including image, audio signal and text data; For any video frame, feature extraction is performed on the image to obtain an image feature vector; feature extraction is performed on the audio signal to obtain an audio feature vector; and feature extraction is performed on the text data to obtain a text feature vector. Based on the image feature vector, the audio feature vector, and the text feature vector, calculate the contribution of each modality in the image, the audio signal, and the text data to the keyframe prediction; Based on the contribution of each modality to the keyframe prediction, the image feature vector, the audio feature vector, and the text feature vector are fused to obtain the fused feature vector. Based on the fused feature vectors of all video frames, keyframes are extracted to obtain keyframe extraction results.

2. The keyframe extraction method as described in claim 1, characterized in that, The step of fusing the image feature vector, the audio feature vector, and the text feature vector based on the contribution of each modality to keyframe prediction, to obtain a fused feature vector, includes: A linear transformation is performed on the image feature vector, the audio feature vector, and the text feature vector to obtain the attention weight value for each modality; Based on the attention weight value of each modality, the image feature vector, the audio feature vector, and the text feature vector are weighted to obtain the image attention feature vector, the audio attention feature vector, and the text attention feature vector; Based on the contribution of each modality to keyframe prediction, the image attention feature vector, the audio attention feature vector, and the text attention feature vector are weighted and summed to obtain the fused feature vector.

3. The keyframe extraction method as described in claim 2, characterized in that, The step of fusing the image feature vector, the audio feature vector, and the text feature vector based on the contribution of each modality to the keyframe prediction to obtain the fused feature vector further includes: The target weight value of each modality is calculated based on the attention weight value of each modality and the contribution of each modality to the keyframe prediction. The image attention feature vector, the audio attention feature vector, and the text attention feature vector are weighted and summed according to the target weight value of each modality to obtain the fused feature vector.

4. The keyframe extraction method as described in claim 3, characterized in that, The step of calculating the target weight value for each modality based on the attention weight value of each modality and the contribution of each modality to keyframe prediction includes: For any modality, calculate the rate of increase of the attention weight value corresponding to the modality relative to the contribution. Based on the increase rate, the contribution is adjusted to obtain the target weight value of the corresponding mode.

5. The keyframe extraction method as described in claim 1, characterized in that, The step of extracting keyframes based on the fused feature vectors of all video frames to obtain keyframe extraction results includes: The fused feature vector is processed using a preset neural network to obtain the keyframe probability value for each video frame. The video frame whose keyframe probability value is greater than a preset threshold is identified as a keyframe.

6. The keyframe extraction method as described in claim 1, characterized in that, After extracting keyframes based on the fused feature vectors of all video frames to obtain the keyframe extraction results, the process further includes: For any target video frame in the keyframe extraction results, spatial interpretation processing is performed on the target video frame to obtain the gradient-weighted class activation heatmap of the target video frame; Based on the contribution of each modality in all target video frames to the keyframe prediction, and sorted chronologically, the contribution of each modality over time is obtained.

7. The keyframe extraction method as described in claim 1, characterized in that, The step of extracting features from the image to obtain an image feature vector and extracting features from the audio signal to obtain an audio feature vector includes: Spatial feature extraction is performed on the image to obtain a spatial feature map. Temporal feature extraction is then performed on the spatial feature map based on a preset bidirectional recurrent neural network to obtain the image feature vector. The audio signal is processed to obtain a Mel spectrogram of the audio signal, and time-series features are extracted from the Mel spectrogram to obtain the audio feature vector.

8. A keyframe extraction device, characterized in that, The keyframe extraction device includes: The acquisition module is used to acquire the video to be extracted, which includes multiple video frames, and each video frame includes image, audio signal and text data; The extraction module is used to extract features from the image for any video frame to obtain an image feature vector, extract features from the audio signal to obtain an audio feature vector, and extract features from the text data to obtain a text feature vector. The calculation module is used to calculate the contribution of each modality in the image, audio signal and text data to keyframe prediction based on the image feature vector, the audio feature vector and the text feature vector; The fusion module is used to fuse the image feature vector, the audio feature vector, and the text feature vector according to the contribution of each modality to the keyframe prediction, so as to obtain the fused feature vector. The module is used to extract keyframes based on the fused feature vectors of all video frames, and obtain keyframe extraction results.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the keyframe extraction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the keyframe extraction method as described in any one of claims 1 to 7.

Images