Video key frame extraction method based on cover frame difference and related device

Abstract

The invention discloses a video key frame extraction method based on cover frame difference and a related device, and the method comprises the steps: obtaining cover frames of a to-be-processed video clip, calculating the frame feature value of each cover frame in a cover frame set, calculating the similarity between the cover frames based on the frame feature values, and obtaining the similarity between the cover frames; dividing video clips into a plurality of scene groups according to cover inter-frame similarity, on this basis, calculating inter-frame change degrees of the clips in the scene groups, adaptively determining the number of key frames in each group according to the inter-frame change degrees, and finally selecting key frames which can best represent content change in the scene groups. The calculation cost of the key frame extraction process is obviously reduced, and the frame extraction efficiency is improved.

Description

Technical Field

[0001] This invention pertains to video retrieval technology, and particularly relates to a method and related apparatus for extracting video keyframes based on cover frame differences. Background Technology

[0002] With the widespread application of artificial intelligence technology, video content analysis based on image recognition has become a crucial foundational capability in fields such as video surveillance, smart security, and intelligent transportation. Existing video retrieval systems typically improve video management and utilization efficiency by automating the analysis and indexing of video footage captured by cameras, enabling the rapid location of relevant segments from massive amounts of video data.

[0003] However, traditional video retrieval schemes mostly rely on storing the complete video stream and analyzing features frame by frame. While this approach can comprehensively reflect the video content, it leads to high system costs and performance bottlenecks in practical applications.

[0004] On the one hand, in the cloud storage stage, the system usually needs to upload and save complete videos or high-frequency extracted frame data, resulting in a huge amount of data and a significant increase in cloud storage space occupation and data transmission costs. On the other hand, in the video analysis and retrieval stage, if there is a lack of an adaptive processing mechanism for the degree of change in the picture, the terminal or edge device needs to extract and compare features from a large number of redundant frames, which not only increases the computational burden and energy consumption, but also limits the real-time retrieval performance.

[0005] In addition, existing keyframe selection methods are mostly based on fixed-interval extraction or simple similarity judgment between adjacent frames, lacking dynamic perception of the changing features of video content. This can easily lead to insufficient representativeness of keyframes, resulting in an excessive number of redundant keyframes in some static scenes, while there are insufficient keyframes in scenes with drastic changes, ultimately affecting the accuracy and efficiency of retrieval.

[0006] Therefore, there is an urgent need for a technical solution that can adaptively extract keyframes based on the changing characteristics of video content. This solution can effectively reduce the storage and computing costs of the system while ensuring the representativeness of keyframes and the accuracy of retrieval, thereby improving the overall performance and practical value of the video retrieval system. Summary of the Invention

[0007] Based on this, the present invention aims to propose a video keyframe extraction method and related apparatus based on cover frame differences. It uses the video cover frame as the basis for keyframe extraction, and extracts keyframes of each scene group through scene grouping and difference calculation, so as to overcome the inefficiency and high cost problems caused by global frame comparison.

[0008] In a first aspect, the present invention provides a method for extracting video keyframes based on cover frame differences, comprising:

[0009] Obtain the cover frames of several video clips and construct a cover frame set;

[0010] Feature calculations are performed on each cover frame in the cover frame set to obtain the frame feature value of each cover frame;

[0011] Several video segments are grouped into scene groups based on frame feature values;

[0012] Calculate the degree of inter-frame variation of video clips within a scene group, and allocate the number of keyframes to each scene group based on the degree of inter-frame variation.

[0013] Extract keyframes from video clips within a scene group based on the assigned number of keyframes.

[0014] Furthermore, scene grouping of several video segments based on frame feature values ​​includes:

[0015] Construct a cover frame sequence by arranging the cover frames in the cover frame set in order;

[0016] For the current cover frame in the cover frame sequence, calculate its inter-frame similarity with adjacent frames based on the frame feature value, and record it as the first inter-frame similarity.

[0017] When the similarity between the first frames reaches the preset grouping threshold, the video segments corresponding to the current cover frame and its adjacent frames are divided into the same scene group.

[0018] Furthermore, scene grouping of several video segments based on frame feature values ​​includes:

[0019] When the similarity between the first frames reaches the first grouping threshold, the video segments corresponding to the current cover frame and its adjacent frames are divided into the same scene group; otherwise, the following fuzzy judgment process is entered:

[0020] When the first inter-frame similarity reaches the second grouping threshold, the inter-frame similarity between the current cover frame and its nearest neighbor frames is calculated based on the feature value and recorded as the second inter-frame similarity. The nearest neighbor frame is the cover frame in the cover frame sequence that has a preset frame interval with the current cover frame.

[0021] The similarity between the first frame is recorded as the fuzzy current similarity. The fuzzy current similarity is compared with the similarity between the second frame to calculate the similarity difference.

[0022] When the similarity difference meets the preset fluctuation condition, the video segments corresponding to the current cover frame and its adjacent frames are divided into the same scene group.

[0023] Furthermore, the frame feature values ​​include at least one of the following: perceptual hash value, color histogram vector, and image structure index.

[0024] Furthermore, when the frame feature values ​​include at least two items, calculating the inter-frame similarity based on the frame feature values ​​includes:

[0025] Calculate the sub-feature similarity between the two cover frames corresponding to the feature values ​​of each frame;

[0026] The inter-frame similarity between the two cover frames is calculated by weighting the similarity of each sub-feature.

[0027] Furthermore, the degree of inter-frame variation of video clips within a scene group is calculated, and the number of keyframes allocated to each scene group is determined based on the degree of inter-frame variation, including:

[0028] Merge video clips from the same scene group into a single group of clips in sequence;

[0029] The group of segments is sampled at a preset sampling interval to form a sampling frame sequence;

[0030] Calculate the inter-frame similarity between adjacent frames in the sampled frame sequence, and denote it as the third inter-frame similarity;

[0031] The inter-frame change rate of the group of segments is calculated based on the third inter-frame similarity.

[0032] Based on the inter-frame change rate, the scene groups are classified according to the preset classification rules to obtain the scene classification results;

[0033] Based on the scene classification results, the number of keyframes for each scene group is determined according to the preset allocation rules.

[0034] Furthermore, extracting keyframes from video clips within a scene group based on the assigned number of keyframes includes:

[0035] The first frame of a scene group is the group cover frame, and the frames other than the group cover frame are candidate frames.

[0036] Calculate the inter-frame similarity between the candidate frame and the group cover frame respectively, and denot it as the fourth inter-frame similarity;

[0037] The candidate frames are sorted according to the fourth inter-frame similarity corresponding to each candidate frame to form a candidate frame sequence;

[0038] Extract candidate frames from the candidate frame sequence that correspond to the number of keyframes and use them as keyframes.

[0039] Furthermore, extracting candidate frames corresponding to the number of keyframes from the candidate frame sequence as keyframes includes:

[0040] Candidate frames in the candidate frame sequence are arranged in descending order of similarity between the fourth frame. Candidate frames corresponding to the number of key frames are extracted sequentially from the candidate frame sequence according to the preset frame distribution rules.

[0041] Secondly, the present invention provides a video keyframe extraction device based on cover frame differences, comprising:

[0042] The cover frame acquisition module is used to acquire cover frames from several video clips and construct a cover frame set.

[0043] The frame feature calculation module is used to perform feature calculations on each cover frame in the cover frame set to obtain the frame feature value of each cover frame.

[0044] The scene grouping module is used to group several video clips into scene groups based on frame feature values.

[0045] The keyframe allocation module is used to calculate the degree of inter-frame variation of video clips within a scene group and allocate the number of keyframes to each scene group based on the degree of inter-frame variation.

[0046] The keyframe extraction module is used to extract keyframes from video clips within a scene group based on the assigned number of keyframes.

[0047] Thirdly, the present invention provides an electronic device including a memory storing computer-executable instructions and a processor, wherein when the computer-executable instructions are executed by the processor, the device performs the steps of the video keyframe extraction method based on cover frame differences provided in the first aspect.

[0048] Fourthly, the present invention provides a readable storage medium storing a computer-executable program that, when executed, can implement the various steps of the video keyframe extraction method based on cover frame differences provided in the first aspect.

[0049] Compared with existing keyframe extraction methods, the present invention has the following advantages:

[0050] This invention proposes a video keyframe extraction method and related apparatus based on cover frame differences. By introducing scene grouping based on cover frame differences before keyframe extraction, the keyframe extraction is upgraded from local processing of a single video segment to global optimization of the entire group of homogeneous video content, achieving scene-level adaptive control of keyframe allocation and selection. Video scene clustering is achieved through cover frame feature value calculation and similarity determination, effectively isolating different scenes and aggregating video segments with consistent visual content within the same scene. This makes the statistics of inter-frame change rate and the allocation of keyframes more consistent with real dynamic change characteristics. A further embodiment uses multi-feature fusion to calculate inter-frame similarity. By combining a fuzzy grouping strategy based on the similarity of neighboring frames and fluctuation conditions, the robustness and accuracy of scene grouping can be significantly improved, avoiding misgrouping problems caused by single features or critical similarity. More preferably, by constructing group segments, calculating inter-frame change rates, and adaptively allocating the number of keyframes based on scene classification, the number of keyframes is matched with the scene complexity. Furthermore, by ranking candidate frames and group cover frames based on similarity and using a distributed extraction mechanism, it is ensured that the extracted keyframes can fully cover representative changes within the scene. This invention can effectively improve the accuracy and representativeness of keyframe extraction, reduce redundant frames, optimize downstream video management, retrieval, and analysis processes, and achieve more efficient video summary generation that better matches the content characteristics. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0052] Figure 1 A flowchart illustrating the implementation of the video keyframe extraction method based on cover frame differences provided in this embodiment of the invention.

[0053] Figure 2 This is a schematic diagram of the video keyframe extraction device based on cover frame differences provided in an embodiment of the present invention;

[0054] Figure 3 This is an electronic device architecture diagram provided for an embodiment of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] See Figure 1 An embodiment of the present invention provides a method for extracting video keyframes based on cover frame differences, comprising the following steps:

[0057] Step S110. Obtain the cover frames of several video clips and construct a cover frame set.

[0058] This step extracts the first frame or other specified frame from each video segment as the cover frame of that segment, thus forming a cover frame set. The cover frames are arranged in chronological order to form a cover frame sequence.

[0059] Specifically, video clips can originate from the temporal segmentation of complete video content. This involves dividing the original long video into several temporally continuous segments according to a preset strategy. The segmentation strategy can be based on a fixed duration or adaptively based on content changes, camera transitions, or optical flow changes, thus forming standardized segments with sequential order on the timeline. Video clips can also originate from multiple temporally discontinuous independent video resources, such as short videos uploaded by users, scattered footage from different recording devices, or discrete recording segments generated by the monitoring system based on event triggers. Although these clips lack temporal continuity, they are still included in the processing as part of the target video set.

[0060] In a further embodiment, if a video segment has multiple sources, after acquiring the video segment, it can be further processed to unify its frame rate, resolution, or encoding format as needed to eliminate structural differences between different sources and ensure that subsequent scene grouping and keyframe extraction processes are performed on a consistent data basis.

[0061] In some embodiments, if the front-end platform has already saved the cover frames of each segment, the cover frame sequence can be read directly without repeated decoding. In a more preferred embodiment, when video segment lengths are inconsistent or frames are dropped, frame compensation or selecting an intermediate frame of the segment as a replacement cover frame can be used to ensure representation stability.

[0062] Step S120. Perform feature calculation on each cover frame in the cover frame set to obtain the frame feature value of each cover frame.

[0063] This step provides quantifiable image content features for scene grouping, enabling cover frames from different video clips to be compared, measured, and clustered numerically. Specifically, as representative images of each video clip in terms of visual semantics, the cover frame contains various information such as color distribution, texture structure, and geometric layout. Directly comparing pixel matrices is not only computationally expensive but also fails to effectively reflect visual differences across different dimensions. Therefore, it is necessary to perform feature processing on the cover frame to extract frame feature values ​​that can describe the characteristics of the video content and are more computationally efficient.

[0064] Specifically, the frame feature values ​​calculated in this step are a set of feature vectors or codes that represent the frame content in numerical form. By comparing the distance or similarity between frame feature values, the degree of similarity between different video segments in terms of content can be quantitatively determined, thereby supporting the subsequent scene grouping process.

[0065] In a further embodiment, the calculation of frame feature values ​​can employ various indicator systems adapted to the image content features. For example, perceptual hash (pHash) features can be used to describe the perceptual similarity relationships of images at the overall structural level; color histogram vectors can also be used to characterize the statistical features of different color distributions in the frame; or structural similarity-related indicators (such as image structure index) can be used to reflect the texture, brightness, and contrast structure information of the image.

[0066] In a more preferred embodiment, the above-mentioned multiple features can also be used in combination to improve the stability and discriminativeness of the cover frame similarity measurement. In some implementations, a perceptual hash value, a color histogram vector, and an image structure index are calculated for each cover frame, and these values ​​are used as multiple sub-features of the frame features.

[0067] Step S130. Group several video segments into scene groups based on frame feature values ​​to obtain several scene groups.

[0068] This step aggregates video clips with similar content into the same group based on the frame feature values ​​of their cover frames. This allows subsequent keyframe extraction to be uniformly judged at the "scene" granularity, thereby improving the efficiency and accuracy of keyframe extraction. Since different video clips may come from the same shooting scene or adjacent consecutive events, their cover frames exhibit high consistency in visual features such as color style, scene layout, and texture structure. Conversely, even if video clips are of the same length, significant differences in their content will result in significant separation of cover frames in the feature space. Therefore, this step, by calculating the inter-frame similarity of cover frames and performing clustering, enables the structured organization of video content.

[0069] Specifically, a cover frame sequence is constructed by arranging cover frames in chronological or file order, ensuring that adjacent frames in the sequence have a certain natural proximity in terms of source. The inter-frame similarity between the current cover frame and its adjacent frames is calculated. This similarity is based on the frame feature values ​​obtained in step S120, such as through hash distance, color histogram cosine similarity, or structural index difference. When the similarity reaches a preset grouping threshold, the video segments corresponding to the current frame and its adjacent frames are considered to belong to the same scene in terms of content, and thus can be directly grouped into the same scene group. This method provides clear criteria for establishing scene groups and reduces computational complexity.

[0070] In some embodiments, to enhance the ability of inter-frame similarity calculation to comprehensively represent different visual characteristics, a weighted sub-feature similarity strategy is adopted when calculating similarity based on multiple frame feature values ​​of the cover frame. Specifically, since a single feature can often only depict one aspect of video content—for example, perceptual hash values ​​are better suited to expressing overall texture and structure, color histograms are good at reflecting the color distribution of the image, and image structure indexes can measure the preservation of local structure and brightness—directly relying on a single feature may lead to a bias in similarity judgment on a specific dimension, making it difficult to accurately reflect the degree of similarity between the actual content of two cover frames. Therefore, a weighted combination of multiple sub-feature similarities is adopted to make inter-frame similarity more robust and discriminative.

[0071] Specifically, firstly, the sub-feature similarity of each of the two cover frames in each feature dimension is calculated. For example, the hash similarity corresponding to the hash distance, the overlap of the color histogram distribution, and the preservation of the structure index are calculated so that each sub-feature similarity can quantify the closeness of the two frames in that feature dimension. After obtaining multiple sub-feature similarities, these similarities are weighted based on preset weights to obtain the final inter-frame similarity.

[0072] The weights can be configured based on the discriminative power, stability, or contribution to scene semantics of each sub-feature. For example, when a video scene is sensitive to color changes, the weight of color features can be increased; while for videos with stable ambient light but significant structural changes, the weight of the structure index can be appropriately increased, thus making the calculation results more consistent with actual visual differences.

[0073] In further embodiments, the weights of sub-feature similarity can be determined in various ways, including pre-setting fixed weights, allocating proportions based on experience or statistical results (e.g., giving higher weights to structural features to enhance sensitivity to differences in scene layout), or dynamically adjusting weights (e.g., automatically optimizing weights based on feature variance, trends, or sample distribution when processing different types of videos to make the similarity results more consistent with the actual content characteristics of the videos). Furthermore, a normalization strategy can be used to avoid an excessive influence of scale differences on the final similarity of a particular feature value; for example, mapping sub-feature similarities to a unified interval before weighted calculation to improve the stability of the fusion results.

[0074] In other implementations, an adaptive weighting mechanism can be employed, adjusting weights in real time by evaluating fluctuations in frame features within a scene group. For example, when a specific feature dimension varies significantly within a scene group, its participation in similarity calculation can be increased to enhance the sensitivity of scene grouping. Furthermore, optimal weights can be trained using machine learning methods, such as supervised learning with a large number of labeled samples, to make the weighted similarity more closely resemble the true similarity determined by human judgment.

[0075] In a further embodiment, to adapt to the effects of slight changes in cover frames or lighting fluctuations caused by different shooting conditions, a dual-threshold and fuzzy judgment strategy is introduced in the grouping process. When the similarity between the current cover frame and the first frame of its adjacent frames reaches a high first grouping threshold, they can be directly grouped. When the similarity between the first frame and the second frame is between the first grouping threshold and the second grouping threshold, a fuzzy judgment process is entered. The similarity between the current cover frame and its nearest neighbor (such as cover frames spaced several positions apart) is calculated to assess the consistency of the content over a larger span. The difference between the first and second frame similarities is further compared to determine whether it meets the preset fluctuation conditions, such as whether the difference falls within the allowable deviation range or whether it maintains a similar trend. When the fluctuation conditions are met, the current frame and its adjacent frames are considered to still belong to the same scene group, thereby enhancing the robustness of scene grouping to local noise and slight lighting changes.

[0076] Specifically, step S130 includes the following steps:

[0077] When the similarity between the first frames reaches the first grouping threshold, the video segments corresponding to the current cover frame and its adjacent frames are divided into the same scene group; otherwise, the following fuzzy judgment process is entered:

[0078] When the first inter-frame similarity reaches the second grouping threshold, the inter-frame similarity between the current cover frame and its nearest neighbor frames is calculated based on the feature value and recorded as the second inter-frame similarity. The nearest neighbor frame is the cover frame in the cover frame sequence that has a preset frame interval with the current cover frame.

[0079] The similarity between the first frame is recorded as the fuzzy current similarity. The fuzzy current similarity is compared with the similarity between the second frame to calculate the similarity difference.

[0080] When the similarity difference meets the preset fluctuation condition, the video segments corresponding to the current cover frame and its adjacent frames are divided into the same scene group.

[0081] In a more preferred embodiment, scene grouping may also consider the following optional mechanisms: First, the grouping threshold can be dynamically adjusted based on multiple feature dimensions. For example, when color features are stable but texture features differ significantly, the decision weight of color features can be increased to adapt to the content characteristics of different types of videos. Second, a sliding window-style multi-frame consistency judgment can be adopted, that is, not only comparing the current frame with neighboring frames, but also comparing several frames before and after it, making the grouping results smoother and more continuous. Third, for videos with rapidly changing scene content, an upper limit can be set to prevent too many frames from being forcibly merged. For videos with slowly changing content, the length of the scene group can be extended to reduce unnecessary segmentation. Fourth, structural features such as color principal component direction, local binary mode (LBP), and image gradient direction statistics can be further introduced as grouping references to enhance the content discrimination capability of the grouping.

[0082] For example, the following is an example of a cover frame sequence consisting of 300 cover frames.

[0083] For the current cover frame Calculate its relationship with adjacent frames similarity ,if If they match, they are considered to be in the same scene; otherwise, proceed to the following fuzzy scene judgment:

[0084] like Then the cover frame is considered , The corresponding video clips belong to different scene groups, if If the similarity between two adjacent frames is considered to be at a critical point, neither significantly similar nor significantly different, then further comparison between the current frame and its nearest neighbor frames is needed, for example... , The similarity is used to determine whether it is a gradual scene or a sudden scene change.

[0085] Let j=1, and denote To blur the current similarity, i.e. ,when At that time, loop counts for j and calculate the similarity between the current frame and its nearest neighbor frames. ,like This indicates a large fluctuation in similarity, suggesting a sudden change in the scene or cover frame. , The corresponding video clips belong to different scene groups; if If the inter-frame similarity fluctuations are small and show a gradually stable trend, it is determined to be a gradual change, and the current blurred similarity is then updated. And let j = j + 1 to continue calculating the similarity between the current frame and its nearest neighbor frames. If the fuzzy judgment process ends, the video segments corresponding to the current cover frame and its adjacent frames will be divided into the same scene group. All are preset thresholds. The maximum nearest neighbor frame interval is preset.

[0086] Step S140. Calculate the degree of inter-frame variation of video segments within a scene group, and allocate the number of keyframes to each scene group based on the degree of inter-frame variation.

[0087] This step involves quantitatively analyzing the changes in video clip content within each scene group, and dynamically allocating the number of keyframes to be extracted for each scene group based on the degree of change.

[0088] Since scene groups are constructed based on cover frame similarity, video clips within the same scene group exhibit high consistency in semantic content or shooting conditions. However, the content may still change to varying degrees over time, such as the movement of the subject, changes in lighting, camera shake, and electronic zoom. Grouping scenes solely based on cover frames is insufficient to accurately reflect the strength of content changes within a group. Therefore, it is necessary to further analyze the changes in frame sequences within scene groups to adaptively determine the number of keyframes. This step aims to allocate fewer keyframes to scenes with less change, reducing redundancy, and allocate more keyframes to scenes with significant changes, thereby improving the representativeness and coverage of the content and achieving a dynamic balance between storage resources and retrieval performance.

[0089] In some embodiments, for each scene group, the system first merges the frame sequences belonging to multiple video segments within the scene group in chronological order to form a continuous content sequence, denoted as a group segment, so as to evaluate content changes on a unified time axis. Subsequently, the system extracts a sampled frame sequence from the group segment according to a preset sampling period and calculates the similarity between adjacent sampled frames.

[0090] To comprehensively measure the actual changes in video content over a short timescale, the degree of change is usually measured by a fusion of multiple indicators, including pixel difference, structural difference, and color distribution difference. This allows the change to reflect not only changes in brightness or color, but also the magnitude of changes in visual information such as local structure, texture, and target contours.

[0091] In a further embodiment, multiple similarity indices are calculated for adjacent sampled frames and then summed with weights. For example, the pixel mean square error is calculated to describe the overall brightness change, the structural similarity index is calculated to measure the degree of preservation of local structure, and the color histogram difference is calculated to show the changes in color distribution. Assigning different weights to these similarity indices can emphasize the importance of different visual content. Subsequently, the similarity of all adjacent sampled frames within the scene group is statistically analyzed, such as calculating the average inter-frame change rate, median change rate, or the change intensity estimated by the depth model, to constitute the inter-frame change degree index of the scene group.

[0092] In a further embodiment, to more clearly map the rate of change to the number of keyframes, the system can classify or continuously map the rate of change. For example, the rate of change can be divided into three categories: low change, medium change, and high change, and assigned different numbers of keyframes to each; or the rate of change can be directly mapped to a range of keyframe numbers using a linear or nonlinear mapping function. The greater the change, the more keyframes; the smaller the change, the fewer keyframes. In some cases, additional factors such as the duration of the scene group, the number of segments, and the number of moving subjects can be combined for multi-factor weighting to make the allocation of keyframe numbers more consistent with the scene semantics.

[0093] In other implementations, adaptive strategies can be used to dynamically determine the threshold. For example, the rate of change for each scene group can be clustered to automatically generate a change level classification; or the threshold can be dynamically adjusted based on the statistical patterns of long-term videos, ensuring that the system maintains stable keyframe allocation performance under different monitoring points and shooting environments. Furthermore, the optimal number of keyframes can be predicted using machine learning models. For instance, a supervised learning model can be built using historical videos and retrieval results to automatically output the number of keyframes for each scene group. For groups with minimal scene changes, the number of keyframes can be further limited to a certain minimum value to prevent meaningless redundancy.

[0094] Step S150. Extract keyframes from video clips within a scene group based on the assigned number of keyframes.

[0095] This step identifies the frame sequence in each scene group that best represents the core content change trend of that scene. Compared to the traditional method of selecting frame by frame or sampling at equal intervals, this step uses the similarity with the cover frame of the scene group as a metric to select the frame with the greatest difference from the core of the scene from a large number of candidate frames as the key frame. This ensures that the final extracted key frame not only maintains the representativeness of the scene but also highlights the points of content change, thereby improving its effectiveness in subsequent retrieval and classification.

[0096] Specifically, firstly, multiple video clips within a scene group are merged along the timeline to form a group segment. The first frame of this group segment is set as the group cover frame, serving as the reference frame for the entire scene group and typically representing the static background, subject position, or initial state of the video group. All frames except the group cover frame are used as candidate frames in subsequent similarity calculations. Based on this, the inter-frame similarity between each candidate frame and the group cover frame is calculated. For example, a fourth inter-frame similarity is generated based on pixel structure, color distribution, or depth features to measure the degree of change of the candidate frame relative to the initial visual state of the scene. The greater the change, the lower the similarity, and the more likely the corresponding frame is to become a keyframe.

[0097] In a further embodiment, to facilitate sorting candidate frames according to the degree of change, the system sorts all candidate frames according to the similarity between their corresponding fourth frames, from largest to smallest or smallest to largest, forming a candidate frame sequence. Typically, a low-to-high similarity sort is used, meaning frames with greater change appear earlier, thus prioritizing frames with more significant differences and stronger representativeness as keyframes. After sorting, based on the number of keyframes allocated in step S140, a corresponding number of frames are sequentially extracted from the sorted candidate frame sequence to form the final keyframe set.

[0098] In a further embodiment, to avoid keyframes being concentrated in a specific local time segment of a video clip, the system can introduce frame distribution rules. For example, a segmented equal sampling method can be used in the sorting sequence, that is, the candidate frame sequence is divided into several range intervals according to the number of keyframes, and the top-ranked frame in each interval is selected as the keyframe to ensure the balanced coverage of keyframes on the timeline. In another approach, a minimum time interval or minimum sequence number difference can be set between two adjacent keyframes to prevent keyframes from being too dense and causing excessive content repetition.

[0099] In other alternative embodiments, visual perception models can be used to optimize keyframe selection. For example, high-level semantic features (such as object category, action state, scene type, etc.) of candidate frames can be extracted using deep neural networks, and secondary filtering can be performed in conjunction with similarity ranking results to jointly determine keyframes from both variability and semantic importance perspectives. In some cases, to further improve retrieval efficiency, candidate frame sequences can be pre-filtered, for example, redundant frames with excessive similarity to the cover frame can be removed, retaining only frames with potential information gain, thereby improving ranking accuracy while reducing computational burden.

[0100] Furthermore, in some more flexible implementations, keyframes can be dynamically updated. This involves re-evaluating the importance of remaining candidate frames based on their similarity to the selected ones during the keyframe selection process. This avoids excessive duplication of subsequently extracted frames with the selected ones, improving the information diversity of the keyframe set. When needed, priority rules can be set based on specific content (such as the appearance of moving objects, camera transitions, sudden changes in lighting, etc.) to assign weights to frames near specific events, ensuring that the extracted keyframes more fully reflect scene changes and key events.

[0101] The disclosed method can be implemented using various types of devices. Therefore, the present invention also discloses an apparatus corresponding to the above method, and specific embodiments are given below for detailed description.

[0102] like Figure 2 As shown, one embodiment of the present invention provides a video keyframe extraction device based on cover frame differences, comprising:

[0103] The cover frame acquisition module 202 is used to acquire cover frames of several video clips and construct a cover frame set;

[0104] The frame feature calculation module 204 is used to perform feature calculation on each cover frame in the cover frame set to obtain the frame feature value of each cover frame.

[0105] The scene grouping module 206 is used to group several video segments into several scene groups based on frame feature values.

[0106] The keyframe number allocation module 208 is used to calculate the degree of inter-frame variation of video segments within a scene group and allocate the number of keyframes to each scene group according to the degree of inter-frame variation.

[0107] The keyframe extraction module 210 is used to extract keyframes from video clips within a scene group based on the assigned number of keyframes.

[0108] The device provided in this application embodiment has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0109] The methods and related apparatuses mentioned in the above embodiments are described with reference to the method flowcharts and / or structural diagrams provided in the embodiments of this application. Specifically, each block of the method flowchart and / or structural diagram, as well as combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 A process or multiple processes and / or structures illustrate the steps of the functions specified in one or more boxes.

[0110] The following embodiments illustrate the application of this method to a computer device. It is understood that the computer device can be any device with computing and processing capabilities, including but not limited to servers or personal laptops. In one embodiment, the computer device can be an application server, which can be a server used to run the application under test.

[0111] See Figure 3 This document illustrates a hardware block diagram of an electronic device intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0112] like Figure 3As shown, the electronic device includes: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0113] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0114] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0115] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0116] The memory stores a program, which the processor can call. The program is used to implement the various processing steps of the aforementioned video keyframe extraction scheme based on cover frame differences.

[0117] This invention also provides a readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements various processing flows of the video keyframe extraction scheme based on cover frame differences provided in any possible implementation of the above embodiments and / or in combination with the embodiments.

[0118] The invention has been described in particular detail above with respect to possible scenarios, and those skilled in the art will recognize that the invention can be practiced through other embodiments. Specific naming of components, capitalization of terms, attributes, data structures, or any other programming or structural aspects are not mandatory or important, and the mechanisms or features of implementing the invention may have different names, forms, or procedures. The system can be implemented through a combination of hardware and software (as described), entirely through hardware elements, or entirely through software elements. The specific division of functions among the various system components described herein is merely exemplary and not mandatory; rather, the functions performed by a single system component can be performed by multiple components, or the functions performed by multiple components can be performed by a single component.

[0119] Those skilled in the art should understand that the various steps of the disclosed methods can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using device-executable program code, which can then be stored in a storage device for execution by the computing device. Alternatively, they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Therefore, the embodiments disclosed in this invention are not limited to any specific hardware and software combination.

[0120] The programs (also referred to as programs, software, software applications, or code) executable by these computing devices include machine instructions of a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0121] Certain aspects of this invention include the process steps and instructions described herein in algorithmic form. It should be noted that the process steps and instructions of this invention can be implemented in software, firmware, and / or hardware, and when implemented in software, they can be downloaded, stored on various operating systems and operated from said platforms.

[0122] Those skilled in the art will understand that the structures shown in the figures are merely block diagrams of some structures related to the present application and do not constitute a limitation on the terminal device to which the present application is applied. Specific terminal devices may include more or fewer components than those shown in the figures, or combine certain components, or have different component arrangements.

[0123] In the description of this specification, the use of terms such as "one embodiment," "some embodiments," "example," "specific example," or "possible design," etc., refers to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0124] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0125] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting video keyframes based on cover frame differences, characterized in that, include: Obtain the cover frames of several video clips and construct a cover frame set; Feature calculations are performed on each cover frame in the cover frame set to obtain the frame feature value of each cover frame; Based on the frame feature values, several video segments are grouped into scene groups to obtain several scene groups; Calculate the degree of inter-frame variation of video segments within the scene group, and allocate the number of keyframes to each scene group based on the degree of inter-frame variation; Keyframes are extracted from the video clips of the scene group based on the assigned number of keyframes.

2. The method according to claim 1, characterized in that, The step of grouping several video segments into scenes based on the frame feature values ​​includes: The cover frames in the set of cover frames are arranged in order to construct a cover frame sequence; For the current cover frame in the cover frame sequence, calculate its inter-frame similarity with adjacent frames based on the frame feature value, and record it as the first inter-frame similarity. When the similarity between the first frames reaches the preset grouping threshold, the video segments corresponding to the current cover frame and its adjacent frames are divided into the same scene group.

3. The method according to claim 2, characterized in that, The method further includes: When the similarity between the first frames reaches the first grouping threshold, the video segments corresponding to the current cover frame and its adjacent frames are divided into the same scene group; otherwise, the following fuzzy judgment process is entered: When the first inter-frame similarity reaches the second grouping threshold, the inter-frame similarity between the current cover frame and its nearest neighbor frames is calculated based on the feature value and recorded as the second inter-frame similarity. The nearest neighbor frame is the cover frame in the cover frame sequence that has a preset frame interval with the current cover frame. The first inter-frame similarity is recorded as the fuzzy current similarity. The fuzzy current similarity is compared with the second inter-frame similarity to calculate the similarity difference. When the similarity difference meets the preset fluctuation condition, the video segments corresponding to the current cover frame and its adjacent frames are divided into the same scene group.

4. The method according to claim 2, characterized in that, When the frame feature values ​​include at least two items, calculating the inter-frame similarity based on the frame feature values ​​includes: Calculate the sub-feature similarity between the two cover frames corresponding to the feature values ​​of each frame; The inter-frame similarity between the two cover frames is calculated by weighting the similarity of each sub-feature.

5. The method according to claim 1, characterized in that, The calculation of the inter-frame variation degree of video segments within the scene group, and the allocation of keyframes to each scene group based on the inter-frame variation degree, includes: Merge video clips from the same scene group into a single group of clips in sequence; The group of segments is sampled at a preset sampling interval to form a sampling frame sequence; Calculate the inter-frame similarity between adjacent frames in the sampled frame sequence, and denote it as the third inter-frame similarity; The inter-frame change rate of the group of segments is calculated based on the third inter-frame similarity. Based on the inter-frame change rate, the scene group is classified according to a preset classification rule to obtain the scene classification result; Based on the scene classification results, the number of keyframes for each scene group is determined according to the preset allocation rules.

6. The method according to claim 5, characterized in that, Extracting keyframes from video clips within a scene group based on the allocated number of keyframes includes: The first frame of the group segment is designated as the group cover frame of the scene group, and the frames other than the group cover frame are designated as candidate frames. Calculate the inter-frame similarity between the candidate frame and the group cover frame respectively, and record it as the fourth inter-frame similarity; The candidate frames are sorted according to the fourth inter-frame similarity corresponding to each candidate frame to form a candidate frame sequence; Candidate frames corresponding to the number of key frames are extracted from the candidate frame sequence and used as key frames.

7. The method according to claim 6, characterized in that, The step of extracting candidate frames corresponding to the number of keyframes from the candidate frame sequence as keyframes includes: The candidate frames in the candidate frame sequence are arranged in descending order of the fourth inter-frame similarity. According to the preset frame distribution rules, candidate frames corresponding to the number of key frames are extracted sequentially from the candidate frame sequence as key frames.

8. A video keyframe extraction device based on cover frame differences, characterized in that, include: The cover frame acquisition module is used to acquire cover frames from several video clips and construct a cover frame set. The frame feature calculation module is used to perform feature calculation on each cover frame in the cover frame set to obtain the frame feature value of each cover frame. The scene grouping module is used to group several video segments into scene groups based on the frame feature values ​​to obtain several scene groups. The keyframe number allocation module is used to calculate the degree of inter-frame variation of video segments within the scene group, and allocate the number of keyframes to each scene group according to the degree of inter-frame variation. The keyframe extraction module is used to extract keyframes from video clips within a scene group based on the assigned number of keyframes.

9. An electronic device, characterized in that, The device includes a memory storing computer-executable instructions and a processor, which, when executed by the processor, causes the device to perform the video keyframe extraction method based on cover frame differences as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, It contains a computer-executable program that, when executed, enables the video keyframe extraction method based on cover frame differences as described in any one of claims 1 to 7.

Images