Video key frame extraction method and device, electronic equipment and storage medium
By performing feature extraction, hash value grouping, and hierarchical clustering on video frame sequences, adjusting similar frame clusters, and selecting key frames, the problem of low accuracy in video key frame extraction in existing technologies is solved, achieving higher extraction accuracy and adaptability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2026-01-13
- Publication Date
- 2026-05-01
AI Technical Summary
Existing video keyframe extraction methods have relatively coarse measurement of similar frames, which affects the extraction accuracy of keyframes. They also have poor adaptability to complex scenes, high computational complexity, and are difficult to apply to long videos.
By acquiring video frame sequences, extracting features, grouping them according to hash values, and performing hierarchical clustering, the centroid similarity between the video frame features and the initial similar frame clusters is adjusted to select key frames within the target similar frame clusters.
It improves the accuracy and adaptability of video keyframe extraction, reduces computational complexity, and better preserves the semantic and temporal information of the video.
Smart Images
Figure CN121963040A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, applicable to the fields of fintech and healthcare, and particularly to a method, apparatus, electronic device, and storage medium for extracting keyframes from a video. Background Technology
[0002] Video keyframe extraction is a technique for selecting the most representative frames from a video frame sequence. In video analysis, it's often necessary to extract key frames so users can quickly understand the crucial content within lengthy videos. For example, in the fintech field of auto insurance, frames containing the damaged parts of a vehicle can be extracted from videos of damaged vehicles to assess damage. Similarly, in the healthcare field of medical video analysis, frames containing lesions can be extracted from ultrasound videos of patients for diagnostic analysis by medical personnel.
[0003] Currently, keyframe extraction in video primarily relies on algorithms such as deep learning. For example, deep learning algorithms can be used to group similar frames and then select keyframes based on these similarities. However, current keyframe extraction methods rely on relatively coarse measurements of similarity, which affects the accuracy of keyframe extraction. Summary of the Invention
[0004] The main objective of this application is to provide a video keyframe extraction method, apparatus, electronic device, and storage medium, aiming to solve the technical problem of coarse measurement of similar frames and improve the accuracy of video keyframe extraction.
[0005] To achieve the above objectives, a first aspect of this application proposes a video keyframe extraction method, the method comprising: Obtain a video frame sequence; wherein the video frame sequence includes at least two video frames; Feature extraction is performed on each video frame to obtain video frame features; The video frames are grouped according to the hash values corresponding to the features of each video frame to obtain similar frame groups; Hierarchical clustering is performed based on the similarity of video frame features between every two video frames in the similar frame group to obtain the initial similar frame cluster of the similar frame group; Based on the similarity between the video frame features of the video frame and the centroids of each initial similar frame cluster, the initial similar frame clusters are adjusted to obtain the target similar frame cluster to which the video frame belongs; Frames are selected based on the video frame features within the target similar frame cluster to obtain keyframes.
[0006] In some embodiments, the step of performing hierarchical clustering based on the similarity of video frame features between every two video frames in the similar frame group to obtain an initial cluster of similar frames includes: For the video frames within the similar frame group, the video frame features of the video frames are determined as nodes, and an edge is created between every two nodes to obtain a video frame similarity graph. Different nodes in the video frame similarity graph are identified as different first similar frame clusters, and the variance increment is determined based on every two first similar frame clusters having the edge. All nodes in the two first similar frame clusters with the smallest variance increment are merged into the second similar frame cluster, and the second similar frame cluster is determined as the initial similar frame cluster.
[0007] In some embodiments, determining the variance increment based on every two first similar frame clusters having the edge includes: For every two first similar frame clusters with the edge, the product of the number of nodes in the two first similar frame clusters and the sum of the number of nodes are divided to obtain the first variance median value. The centroid of each first similar frame cluster is obtained by calculating the mean of the nodes within each first similar frame cluster. The second variance median value is obtained by summing the squares of the differences between the centroids of the two first similar frame clusters. The variance increment is obtained by multiplying the first variance median value and the second variance median value.
[0008] In some embodiments, before determining the variance increment based on every two first similar frame clusters having the edge, the method further includes: The cosine similarity is calculated for every two nodes to obtain the cosine similarity, and the cosine similarity is determined as the weight of the edge between the two nodes. For every two nodes, if the weight of the edge between the two nodes is less than or equal to a preset similarity threshold, the edge between the two nodes is deleted.
[0009] In some embodiments, at least two video frames have different time steps, and the nodes within each of the initial similar frame clusters are the video frame features; The step of adjusting the initial similar frame clusters based on the similarity between the video frame features of the video frame and the centroids of each initial similar frame cluster to obtain the target similar frame cluster to which the video frame belongs includes: The centroid of each initial similar frame cluster is obtained by calculating the mean of the nodes within each initial similar frame cluster, and the normalized similarity is obtained by calculating the similarity between the video frame features at each time step and the centroid of each initial similar frame cluster. The average similarity is calculated by averaging the normalized similarity of the time steps within the preset frame window length. The similarity frame divergence for each time step is obtained by calculating the divergence based on the normalized similarity and the average similarity for each time step. The temporal consistency loss value is calculated by averaging the similar frame divergence at the time step within the frame window length, and the initial similar frame cluster is adjusted based on the temporal consistency loss value to obtain the target similar frame cluster to which the video frame belongs.
[0010] In some embodiments, the step of calculating a normalized similarity based on the video frame features at each time step and the centroids of each initial similar frame cluster includes: The ratio similarity is calculated by multiplying the transpose vector of the video frame feature at each time step with the centroid of each initial similar frame cluster and a preset temperature coefficient. For each time step, the ratio similarity is calculated exponentially to obtain an initial exponential similarity. The summation of the initial index similarities is calculated to obtain the total index similarity. The normalized similarity is obtained by calculating the ratio between each initial exponential similarity and the sum exponential similarity.
[0011] In some embodiments, the step of selecting keyframes based on the video frame features within the target similar frame cluster includes: Attention weights are calculated based on the features of every two video frames within the same cluster of similar frames to the target. For each video frame feature, the attention weights of the video frame feature and other video frame features within the same target similar frame cluster are summed to obtain the attention score of each video frame feature. The video frame with the highest attention score within the target similar frame cluster is determined as the keyframe.
[0012] To achieve the above objectives, a second aspect of this application provides a video keyframe extraction apparatus, the apparatus comprising: A frame acquisition module is used to acquire a video frame sequence; wherein the video frame sequence includes at least two video frames; The feature extraction module is used to extract features from each of the video frames to obtain video frame features; The grouping module is used to group the video frames according to the hash values corresponding to the features of each video frame, so as to obtain similar frame groups; The clustering module is used to perform hierarchical clustering based on the similarity of video frame features between every two video frames in the similar frame group to obtain the initial similar frame cluster of the similar frame group. The cluster adjustment module is used to adjust the initial similar frame clusters based on the similarity between the video frame features of the video frame and the centroids of each initial similar frame cluster, so as to obtain the target similar frame cluster to which the video frame belongs; The frame selection module is used to select frames based on the video frame features within the target similar frame cluster to obtain key frames.
[0013] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.
[0014] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.
[0015] The video keyframe extraction method, apparatus, electronic device, and storage medium proposed in this application acquire a video frame sequence and group it according to the hash values corresponding to the video frame features of each video frame. This allows video frames with similar visual features to be classified into the same group. Furthermore, hierarchical clustering is performed based on the similarity of the video frame features between every two video frames in the similar frame group. This allows for further hierarchical clustering on top of the initial grouping to obtain more refined clusters of similar frames. Then, the initial similar frame clusters are adjusted based on the similarity between the video frame features of the video frames and the centroids of each initial similar frame cluster to improve the accuracy of cluster partitioning, rather than coarsely partitioning similar frames, such as using only hash values. Finally, keyframes are selected within the target similar frame clusters. This method selects keyframes that represent the main information of the video from frames that are semantically and temporally similar, thereby improving the accuracy of video keyframe extraction. Attached Figure Description
[0016] Figure 1 This is a flowchart of the video keyframe extraction method provided in the embodiments of this application; Figure 2 yes Figure 1 The flowchart for step 104 in the document; Figure 3 yes Figure 2 The flowchart for step 202 in the document; Figure 4 This is a flowchart of a video keyframe extraction method provided in another embodiment of this application; Figure 5 yes Figure 1 The flowchart for step 105 in the document; Figure 6 yes Figure 5 The flowchart for step 501 in the document; Figure 7 yes Figure 1 The flowchart for step 106 in the document; Figure 8 This is a system architecture diagram of the video keyframe extraction process provided in the embodiments of this application; Figure 9 This is a system architecture diagram of the self-supervised learning module provided in the embodiments of this application; Figure 10 This is a schematic diagram of the video keyframe extraction device provided in the embodiments of this application; Figure 11 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0018] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0020] First, let's analyze some of the terms used in this application: Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can be a simulation of the information processes of human consciousness and thought. AI can also be the theory, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning. This application can acquire and process relevant data based on AI technology.
[0021] Video keyframe extraction is a core task of video analysis and summarization. Its goal is to select the most representative frames from a long video sequence in order to preserve the semantic content and temporal structure of the original video.
[0022] Locality-Sensitive Hashing (LSH) is a technique that maps high-dimensional data to low-dimensional codes. LSH makes it more likely that similar data will generate similar or identical hash values. LSH can be applied to information retrieval, data mining, and recommendation systems.
[0023] Hierarchical clustering is a clustering technique that creates a tree-like clustering structure by hierarchically dividing a dataset.
[0024] Traditional video keyframe extraction methods primarily rely on hand-designed features, such as SIFT (Scale Invariant Feature Transform) or HOG (Image Feature Detection) algorithms, combined with clustering algorithms (such as k-means) to extract keyframes. For example, video frames can be grouped into similar frames, and keyframes can be selected based on these similarities. However, current video keyframe extraction methods use relatively coarse measurements of similarity, affecting extraction accuracy. Furthermore, these methods are poorly adaptable to complex scenes, and their computational complexity increases dramatically with video size, making them unsuitable for long videos. Additionally, these methods typically focus only on content similarity, ignoring the temporal logical relationships between keyframes. In summary, the keyframe extraction accuracy of these methods is relatively low.
[0025] Based on this, embodiments of this application provide a video keyframe extraction method, apparatus, electronic device, and storage medium, aiming to improve the accuracy of video keyframe extraction.
[0026] The video keyframe extraction method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the video keyframe extraction method in this application is described.
[0027] The video keyframe extraction method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms; the software can be an application that implements the video keyframe extraction method, but is not limited to the above forms.
[0028] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0029] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.
[0030] Figure 1 This is an optional flowchart of the video keyframe extraction method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps 101 to 106.
[0031] Step 101: Obtain a video frame sequence; wherein the video frame sequence includes at least two video frames; Step 102: Extract features from each video frame to obtain video frame features; Step 103: Group the video frames according to the hash values corresponding to the features of each video frame to obtain similar frame groups; Step 104: Perform hierarchical clustering based on the similarity of video frame features between every two video frames in the similar frame group to obtain the initial similar frame clusters of the similar frame group. Step 105: Adjust the initial similar frame clusters based on the similarity between the video frame features of the video frame and the centroids of each initial similar frame cluster to obtain the target similar frame cluster to which the video frame belongs. Step 106: Select frames based on the video frame features within the target similar frame cluster to obtain key frames.
[0032] The beneficial effects of this application's embodiments include, but are not limited to: by acquiring video frame sequences and grouping them according to the hash values corresponding to the video frame features of each video frame, video frames with similar visual features can be classified into the same group. Furthermore, hierarchical clustering is performed based on the similarity of the video frame features of every two video frames in the similar frame group. This allows for further hierarchical clustering on the basis of the initial grouping to obtain more refined clusters of similar frames. Then, the initial similar frame clusters are adjusted according to the similarity between the video frame features of the video frames and the centroids of each initial similar frame cluster to improve the accuracy of cluster division, rather than coarsely dividing similar frames, such as dividing similar frames solely by hash values. Finally, keyframes are selected within the target similar frame clusters. This allows for the selection of keyframes that represent the main information of the video from frames that are semantically and temporally similar, thereby improving the accuracy of video keyframe extraction.
[0033] In step 101 of some embodiments, the video frame sequence consists of a series of static images (i.e., video frames) arranged in chronological order. The video frame sequence may include video frames arranged sequentially at different time steps. For example, the video frame sequence can be obtained by capturing video with a camera, or by other means, and is not limited to these. In some embodiments, such as in a car insurance scenario in the fintech field, a damaged vehicle can be photographed to obtain video frames containing the damaged parts of the vehicle. As another example, in a medical video analysis scenario in the healthcare field, a patient can be monitored for a timeout to obtain video frames containing ultrasound videos of the patient's lesions.
[0034] In step 102 of some embodiments, the video frame features are feature vectors of the video frames. For example, the video frame features can be obtained by extracting features from each video frame using an image encoder based on a deep learning model (such as the ViT model).
[0035] In step 103 of some embodiments, video frame features can be mapped to hash values using a hash function, such as a random projection hash function. Specifically, the hash value can be binary encoding or other formats, and is not limited thereto. Similar frame grouping is the grouping of video frames obtained based on the hash values. In some embodiments, it should be noted that all video frames within each similar frame group have the same hash value.
[0036] In some embodiments, video frame features are mapped to hash values using a random projection hash function, whereby the hash value is defined as follows: .in, The video frame features represent the i-th video frame (or simply the i-th frame); This represents the hash value obtained by mapping the i-th frame through the j-th hash function; Represents a random weight vector The transpose of the matrix, It follows a standard normal distribution; The sign function is defined, and its output is -1, 0, or 1. In some embodiments, a collision threshold of L is set. If two video frames collide on at least L hash functions (i.e., the mapped hash values are the same), then the two video frames are determined to be similar frames and assigned to the same video frame group. Specifically, the collision threshold L can be half the number of random projection hash functions K, i.e., L = K / 2.
[0037] In step 104 of some embodiments, the initial similar frame cluster is a cluster obtained by hierarchically clustering the video frame features of the video frames in each similar frame group. As for the specific process of hierarchical clustering, please refer to the detailed description of steps 201 to 203 below, which will not be repeated here.
[0038] In step 105 of some embodiments, the cluster to which the video frame belongs can be adjusted from the original cluster (i.e., the initial similar frame cluster) to another cluster (i.e., the target similar frame cluster), thereby improving the similarity between the video frame and other video frames in its cluster.
[0039] In step 106 of some embodiments, a frame can be selected from the video frame features within each target similarity frame cluster as a key frame. For example, a frame can be randomly selected from each target similarity frame cluster as a key frame, or the frame with the highest attention score can be selected from each target similarity frame cluster as a key frame, and this is not limited to these methods.
[0040] Please see Figure 2 In some embodiments, step 104 may include, but is not limited to, steps 201 to 203: Step 201: For video frames within a similar frame group, determine the video frame features of the video frames as nodes, and create an edge between every two nodes to obtain a video frame similarity graph. Step 202: Different nodes of the video frame similarity graph are identified as different first similar frame clusters, and the variance increment is determined based on every two first similar frame clusters with edges. Step 203: Merge all nodes in the two first similar frame clusters with the smallest variance increment into the second similar frame cluster, and determine the second similar frame cluster as the initial similar frame cluster.
[0041] The advantage of this embodiment lies in constructing a video frame similarity graph by mapping video frame features within similar frame groups to nodes and creating edge connections between nodes. This provides a structured representation of inter-frame similarity relationships, offering a graph data foundation for hierarchical clustering. By treating each node in the video frame similarity graph as a distinct first similar frame cluster, first similar frame clusters with connected edges and increasing variance are merged. This allows for the quantification of the feature distribution dispersion change when merging each pair of first similar frame clusters, selecting the cluster with the smallest variance increment for merging to generate a second similar frame cluster. This improves the similarity of features within the merged cluster, thereby enhancing the accuracy of video keyframe extraction.
[0042] In step 201 of some embodiments, the video frame similarity graph includes nodes and edges connecting two nodes. The nodes of the video frame similarity graph are video frame features. It should be noted that all nodes in each video frame similarity graph belong to the same similar frame group.
[0043] In step 202 of some embodiments, the existence of an edge between two first similar frame clusters means that at least two nodes belonging to different first similar frame clusters are connected by an edge. It should be noted that this embodiment only calculates the variance increment for two first similar frame clusters that have an edge. If there is no edge between two first similar frame clusters (i.e., no connection relationship), the variance increment is not calculated, and no subsequent cluster merging operation is performed.
[0044] In step 203 of some embodiments, the second similar frame cluster is a cluster obtained by merging all nodes in the two first similar frame clusters with the smallest variance increment.
[0045] Please see Figure 3 In some embodiments, step 202 may include, but is not limited to, steps 301 to 304: Step 301: For every two first similar frame clusters with edges, divide the product of the number of nodes in the two first similar frame clusters by the sum of the number of nodes to obtain the first variance median value. Step 302: Calculate the mean value of each node in the first similar frame cluster to obtain the centroid of each first similar frame cluster; Step 303: Calculate the sum of squares based on the difference between the centroids of the two first similar frame clusters to obtain the median value of the second variance; Step 304: Calculate the variance increment by multiplying the first variance median value and the second variance median value.
[0046] The advantage of this embodiment lies in that, for each pair of first similar frame clusters with edges, the ratio of the product of their node counts to the sum of their node counts is used as the first variance median. Then, the centroid is calculated by taking the mean of the nodes within each first similar frame cluster, and the second variance median is calculated by summing the squares of the differences between the two centroids. This accurately measures the centroid distance between the two clusters in the feature space, reflecting the degree of content difference between different clusters. Finally, the variance increment is obtained by multiplying the first and second variance medians. This comprehensively evaluates the incremental impact of the merging operation on the feature dispersion (i.e., variance) within the cluster. Therefore, during hierarchical clustering, the two clusters with the highest similarity of features within the new cluster are selected for merging each time, thereby improving the accuracy of video keyframe extraction.
[0047] In step 301 of some embodiments, as illustrated in the formula for step 304 below, the first variance median value is .
[0048] In step 302 of some embodiments, specifically, the centroid of the first similar frame cluster is the mean of the nodes within the first similar frame cluster.
[0049] In step 303 of some embodiments, as illustrated in the formula for step 304 below, the median value of the second variance is... .
[0050] In step 304 of some embodiments, the variance increment is defined as shown in the following formula: , In the formula, This represents the p-th first similar frame cluster; This represents the q-th first similar frame cluster; Cluster and cluster The variance increment; Cluster The total number of nodes in the cluster, also known as the cluster size; Cluster The total number of nodes in the middle; Cluster The center of mass; Cluster The center of mass; This represents the square of the Euclidean norm.
[0051] In some embodiments, the Ward criterion (i.e., the variance minimization criterion) can be applied for hierarchical clustering. Specifically, the two first similar frame clusters with the smallest variance increment can be selected for merging.
[0052] Please see Figure 4In some embodiments, prior to step 202, the video keyframe extraction method may further include, but is not limited to, steps 401 to 402: Step 401: Calculate the cosine similarity for every two nodes to obtain the cosine similarity, and determine the cosine similarity as the weight of the edge between the two nodes. Step 402: For every two nodes, if the weight of the edge between the two nodes is less than or equal to a preset similarity threshold, delete the edge between the two nodes.
[0053] The advantage of this embodiment lies in its calculation of the cosine similarity between every two nodes and using it as the weight of an edge. If the weight of an edge between two nodes is less than or equal to a preset similarity threshold, the edge is deleted. This removes edges between nodes with low similarity, merging only clusters with connections. Consequently, in subsequent hierarchical clustering, it avoids merging frames with excessively different visual content into the same cluster, improving the semantic consistency of features within clusters. This not only reduces unnecessary computation and improves cluster merging efficiency but also enhances the similarity of content within the initial similar frame clusters obtained after merging, thereby improving the accuracy of video keyframe extraction.
[0054] In step 401 of some embodiments, cosine similarity is defined as... ,in, This represents the cosine similarity between the i-th frame and the j-th frame; Represents the video frame features of the i-th frame. The transpose of ; This represents the video frame features of the j-th frame; Representing video frame features The modulus length; Representing video frame features The length of the module.
[0055] In step 402 of some embodiments, the similarity threshold can be any decimal between 0 and 1. For example, the similarity threshold can be 0.7. In another embodiment, the similarity threshold can also be other values, and is not limited thereto.
[0056] Please see Figure 5 In some embodiments, at least two video frames have different time steps, and the nodes within each initial cluster of similar frames are video frame features. Step 105 may include, but is not limited to, steps 501 through 504: Step 501: Calculate the mean value of the nodes in each initial similar frame cluster to obtain the centroid of each initial similar frame cluster, and calculate the similarity between the video frame features at each time step and the centroid of each initial similar frame cluster to obtain the normalized similarity. Step 502: Calculate the average similarity based on the normalized similarity of time steps within the preset frame window length. Step 503: Calculate the divergence based on the normalized similarity and average similarity at each time step to obtain the similar frame divergence at each time step. Step 504: Calculate the average value of the similar frame divergence within the time step of the frame window to obtain the temporal consistency loss value, and adjust the initial similar frame cluster according to the temporal consistency loss value to obtain the target similar frame cluster to which the video frame belongs.
[0057] The advantage of this embodiment lies in that it calculates the centroid by averaging the nodes within each initial similar frame cluster, and calculates normalized similarity based on the video frame features at each time step and the centroids of each initial similar frame cluster. This allows for the evaluation of the correlation between each frame and the centroids of each cluster. An average similarity is calculated based on the normalized similarity of time steps within a preset frame window length, and divergence is calculated based on the normalized similarity and average similarity at each time step to obtain the similar frame divergence at each time step. This quantifies the temporal deviation of the similarity between frames and clusters. Then, the temporal consistency loss value is calculated by averaging the similar frame divergence of time steps within the frame window length, and the initial similar frame clusters are adjusted based on this loss value. This ensures that the clustering results conform to the temporal patterns of the video content, rather than performing isolated and coarse inter-frame similarity measurements, thereby improving the accuracy of keyframe extraction.
[0058] In step 501 of some embodiments, the calculation process of normalized similarity can be referred to the specific description of step 604 below, and will not be repeated here.
[0059] In step 502 of some embodiments, it should be noted that if the frame window length is T, the average similarity is the average of the normalized similarities over T time steps.
[0060] For example, the definition of average similarity is... .in, Indicates average similarity; T represents frame window length; This represents the normalized similarity at time step t; This indicates a summation operation on the normalized similarity within the frame window length T.
[0061] In step 503 of some embodiments, the similarity frame divergence is defined as... .in, Indicates the divergence of similar frames; This indicates that KL divergence calculation is being performed; This represents the normalized similarity at time step t; This represents the average similarity.
[0062] In some embodiments, KL divergence can be calculated using the KL divergence function. It should be noted that KL (Kullback-Leibler divergence), also known as relative entropy or information divergence, is used to measure the difference between two probability distributions.
[0063] In step 504 of some embodiments, the cluster to which the video frame belongs may be adjusted, for example, by changing the cluster to which it belongs from the initial similar frame cluster to the target similar frame cluster, in order to minimize the temporal consistency loss value.
[0064] Please see Figure 6 In some embodiments, step 501 may include, but is not limited to, steps 601 to 604: Step 601: Calculate the ratio of the transpose vector of the video frame features at each time step to the centroid of each initial similar frame cluster, and the preset temperature coefficient to obtain the ratio similarity. Step 602: Calculate the exponential similarity of the ratio at each time step to obtain the initial exponential similarity. Step 603: Sum the initial index similarities to obtain the total index similarity. Step 604: Calculate the normalized similarity by performing a ratio between each initial index similarity and the sum index similarity.
[0065] The advantage of this embodiment lies in the fact that it calculates the ratio similarity by multiplying the transpose vector of the video frame features at each time step by the product of the centroid of each initial similar frame cluster and a preset temperature coefficient. The ratio similarity at each time step is then exponentially calculated to obtain the initial exponential similarity, thus mapping the similarity value to a positive range. The initial exponential similarities at each time step are summed to obtain the total exponential similarity. Then, the ratio of the initial exponential similarity at each time step to the total exponential similarity is calculated to obtain the normalized similarity at each time step. This transforms the similarity between each frame and each cluster into a probability distribution, reflecting the similarity between the frame and each cluster (i.e., the probability that the frame belongs to that cluster). This makes the clusters obtained subsequently based on temporal consistency loss values more accurate, thereby improving the accuracy of keyframe extraction.
[0066] In step 601 of some embodiments, as illustrated by the formula for step 604 below, the ratio similarity is... .
[0067] In step 602 of some embodiments, as illustrated in the formula for step 604 below, the initial exponential similarity is... .
[0068] In step 603 of some embodiments, as illustrated in the formula for step 604 below, the summation exponent similarity is... .
[0069] In step 604 of some embodiments, the normalized similarity can be a softmax similarity. For example, the normalized similarity is defined as shown in the following formula: , In the formula, This represents the normalized similarity at time step t; Represents an exponential function; Represents the video frame features of frame t. The transpose of ; This represents the centroid of the k-th initial cluster of similar frames; This represents the centroid of the h-th initial cluster of similar frames; Indicates the temperature coefficient; This represents the summation over all initial similar frame clusters, where K represents the total number of initial similar frame clusters.
[0070] In some embodiments, the temperature coefficient is 0.1. In another embodiment, the temperature coefficient may be other values, and is not limited thereto.
[0071] Please see Figure 7 In some embodiments, step 106 may include, but is not limited to, steps 701 to 703: Step 701: Calculate the attention weights based on the features of every two video frames within the same cluster of similar frames to obtain the attention weights. Step 702: For each video frame feature, sum the attention weights of the video frame feature and other video frame features within the same target similar frame cluster to obtain the attention score of each video frame feature. Step 703: The video frame with the highest attention score within the target similar frame cluster is identified as the keyframe.
[0072] The advantage of this embodiment lies in that it calculates attention weights based on the features of every two video frames within a cluster of similar frames with the same target, thus quantifying the correlation between any two frames within the same cluster. For each video frame feature, the attention weights of the video frame feature and other video frame features within the cluster are summed to obtain an attention score for each video frame feature. This allows for the identification of the central frame that best represents the semantic content of the entire cluster based on the attention score. The video frame with the highest attention score within the cluster is determined as the keyframe. This selects the most representative frame within the cluster as the keyframe, rather than randomly or simply selecting keyframes, thereby improving the accuracy of keyframe extraction.
[0073] In step 701 of some embodiments, the attention weight is defined as Among them, it means; This represents the normalized exponential function (softmax function). The video frame features represent the i-th video frame (or simply the i-th frame); This represents the video frame features of the j-th frame; and These represent the learnable weight parameters; Represents the video frame features of the j-th frame. and weight parameters The transpose of the product of .
[0074] In step 702 of some embodiments, the attention score is defined as follows: , In the formula, This represents the attention score for the i-th frame; This represents the attention weights for the i-th and j-th frames; Indicates characteristic functions; This indicates the cluster to which the j-th frame belongs; This indicates the cluster to which the i-th frame belongs.
[0075] It should be noted that if the cluster to which the j-th frame belongs is the same as the cluster to which the i-th frame belongs, that is, if the condition is met... ,but The value is 1. Conversely, if the cluster to which the j-th frame belongs is different from the cluster to which the i-th frame belongs, then... The value is 0.
[0076] In step 703 of some embodiments, the video frame with the highest attention score can be selected as the keyframe from each cluster of similar frames to the target.
[0077] Please see Figure 8In some embodiments, the overall architecture of the video keyframe extraction process may include an input module, a feature extraction module, a self-supervised learning module, and an output module. The input module takes the raw video stream data as input and performs preprocessing operations such as standardization and resizing on the video frames. The feature extraction module converts the video frames into 768-dimensional feature vectors based on a pre-trained deep learning model. The self-supervised learning module uses the Locality Sensitive Hash (LSH) algorithm to quickly group similar frames, uses a hierarchical clustering algorithm based on the Ward criterion (variance minimization criterion) to cluster the video frame features to obtain initial clusters of similar frames, calculates the temporal consistency loss function value to adjust the clusters, and calculates attention scores using a multi-head self-attention algorithm to select keyframes. The output module outputs the final selected keyframe sequence.
[0078] Please see Figure 9 In some embodiments, the self-supervised learning module can be used to implement the following steps: generating a random projective hash (LSH) function, assigning frames to corresponding buckets based on hash values to construct hash buckets, and achieving rapid grouping of similar frames. Then, an inter-frame similarity graph (i.e., the video frame similarity graph mentioned above) is constructed, the number of edges is reduced through threshold pruning, and the video frame features are clustered using a hierarchical clustering algorithm based on the variance minimization criterion, thus enabling fine-grained clustering within hash buckets. By comparing the clustering assignments of adjacent frames, the KL divergence loss is calculated to obtain the temporal consistency loss value, thereby optimizing the temporal coherence of video frame features within the cluster. By calculating multi-head self-attention weights and selecting keyframes based on attention scores, representative frames can be selected from the clustering results.
[0079] Please see Figure 10 This application also provides a video keyframe extraction device that can implement the above-described video keyframe extraction method. The device includes: The frame acquisition module 801 is used to acquire a video frame sequence; wherein the video frame sequence includes at least two video frames. The feature extraction module 802 is used to extract features from each video frame to obtain video frame features; The grouping module 803 is used to group video frames according to the hash values corresponding to the features of each video frame to obtain similar frame groups; Clustering module 804 is used to perform hierarchical clustering based on the similarity of video frame features between every two video frames in the similar frame group to obtain the initial similar frame clusters of the similar frame group. The cluster adjustment module 805 is used to adjust the initial similar frame clusters based on the similarity between the video frame features of the video frame and the centroids of each initial similar frame cluster, so as to obtain the target similar frame cluster to which the video frame belongs. The frame selection module 806 is used to select keyframes based on the video frame features within the target similar frame cluster.
[0080] In one embodiment, the video keyframe extraction device further includes a threshold edge pruning module, which is used to: calculate the cosine similarity for every two nodes, obtain the cosine similarity, and determine the cosine similarity as the weight of the edge between the two nodes; for every two nodes, if the weight of the edge between the two nodes is less than or equal to a preset similarity threshold, delete the edge between the two nodes.
[0081] The specific implementation of this video keyframe extraction device is basically the same as the specific implementation of the video keyframe extraction method described above, and will not be repeated here.
[0082] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described video keyframe extraction method. This electronic device can include any smart terminal such as a tablet computer or an in-vehicle computer.
[0083] Please see Figure 11 , Figure 11 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the video keyframe extraction method of the embodiments of this application. The 903 input / output interface is used to implement information input and output. The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.
[0084] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described video keyframe extraction method.
[0085] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0086] It should be noted that the software tools or components not belonging to our company that appear in the embodiments of this application are merely examples and do not represent actual use.
[0087] The embodiments described in this application are intended to more clearly illustrate the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0088] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.
[0089] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0090] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0091] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0092] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0093] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, or indirect coupling or communication connection between the apparatus or units, and may be electrical, mechanical, or other forms.
[0094] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0095] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0096] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0097] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.
Claims
1. A method for extracting keyframes from a video, characterized in that, The method includes: Obtain a video frame sequence; wherein the video frame sequence includes at least two video frames; Feature extraction is performed on each video frame to obtain video frame features; The video frames are grouped according to the hash values corresponding to the features of each video frame to obtain similar frame groups; Hierarchical clustering is performed based on the similarity of video frame features between every two video frames in the similar frame group to obtain the initial similar frame cluster of the similar frame group; Based on the similarity between the video frame features of the video frame and the centroids of each initial similar frame cluster, the initial similar frame clusters are adjusted to obtain the target similar frame cluster to which the video frame belongs; Frames are selected based on the video frame features within the target similar frame cluster to obtain keyframes.
2. The method according to claim 1, characterized in that, The step of performing hierarchical clustering based on the similarity of video frame features between every two video frames in the similar frame group to obtain the initial similar frame cluster includes: For the video frames within the similar frame group, the video frame features of the video frames are determined as nodes, and an edge is created between every two nodes to obtain a video frame similarity graph. Different nodes in the video frame similarity graph are identified as different first similar frame clusters, and the variance increment is determined based on every two first similar frame clusters having the edge. All nodes in the two first similar frame clusters with the smallest variance increment are merged into the second similar frame cluster, and the second similar frame cluster is determined as the initial similar frame cluster.
3. The method according to claim 2, characterized in that, The step of determining the variance increment based on every two first similar frame clusters having the edge includes: For every two first similar frame clusters with the edge, the product of the number of nodes in the two first similar frame clusters and the sum of the number of nodes are divided to obtain the first variance median value. The centroid of each first similar frame cluster is obtained by calculating the mean of the nodes within each first similar frame cluster. The second variance median value is obtained by summing the squares of the differences between the centroids of the two first similar frame clusters. The variance increment is obtained by multiplying the first variance median value and the second variance median value.
4. The method according to claim 2, characterized in that, Before determining the variance increment based on every two first similar frame clusters having the edge, the method further includes: The cosine similarity is calculated for every two nodes to obtain the cosine similarity, and the cosine similarity is determined as the weight of the edge between the two nodes. For every two nodes, if the weight of the edge between the two nodes is less than or equal to a preset similarity threshold, the edge between the two nodes is deleted.
5. The method according to any one of claims 1 to 4, characterized in that, At least two video frames have different time steps, and the nodes in each of the initial similar frame clusters are the features of the video frames; The step of adjusting the initial similar frame clusters based on the similarity between the video frame features of the video frame and the centroids of each initial similar frame cluster to obtain the target similar frame cluster to which the video frame belongs includes: The centroid of each initial similar frame cluster is obtained by calculating the mean of the nodes within each initial similar frame cluster, and the normalized similarity is obtained by calculating the similarity between the video frame features at each time step and the centroid of each initial similar frame cluster. The average similarity is calculated by averaging the normalized similarity of the time steps within the preset frame window length. The similarity frame divergence for each time step is obtained by calculating the divergence based on the normalized similarity and the average similarity for each time step. The temporal consistency loss value is calculated by averaging the similar frame divergence at the time step within the frame window length, and the initial similar frame cluster is adjusted based on the temporal consistency loss value to obtain the target similar frame cluster to which the video frame belongs.
6. The method according to claim 5, characterized in that, The step of calculating the normalized similarity based on the video frame features at each time step and the centroids of each initial similar frame cluster includes: The ratio similarity is calculated by multiplying the transpose vector of the video frame feature at each time step with the centroid of each initial similar frame cluster and a preset temperature coefficient. For each time step, the ratio similarity is calculated exponentially to obtain an initial exponential similarity. The summation of the initial index similarities is calculated to obtain the total index similarity. The normalized similarity is obtained by calculating the ratio between each initial exponential similarity and the sum exponential similarity.
7. The method according to any one of claims 1 to 4, characterized in that, The step of selecting keyframes based on the video frame features within the target similar frame cluster includes: Attention weights are calculated based on the features of every two video frames within the same cluster of similar frames to the target. For each video frame feature, the attention weights of the video frame feature and other video frame features within the same target similar frame cluster are summed to obtain the attention score of each video frame feature. The video frame with the highest attention score within the target similar frame cluster is determined as the keyframe.
8. A video keyframe extraction device, characterized in that, The device includes: A frame acquisition module is used to acquire a video frame sequence; wherein the video frame sequence includes at least two video frames; The feature extraction module is used to extract features from each of the video frames to obtain video frame features; The grouping module is used to group the video frames according to the hash values corresponding to the features of each video frame, so as to obtain similar frame groups; The clustering module is used to perform hierarchical clustering based on the similarity of video frame features between every two video frames in the similar frame group to obtain the initial similar frame cluster of the similar frame group. The cluster adjustment module is used to adjust the initial similar frame clusters based on the similarity between the video frame features of the video frame and the centroids of each initial similar frame cluster, so as to obtain the target similar frame cluster to which the video frame belongs; The frame selection module is used to select frames based on the video frame features within the target similar frame cluster to obtain key frames.
9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the video keyframe extraction method according to any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the video keyframe extraction method according to any one of claims 1 to 7.