A method for extracting key frames of compressed video

Abstract

The application provides an extraction method for key frames of compressed video, comprising the following steps: acquiring all video frames in a video and a preset number of required key frames; wherein the last frame of the video is a fixed key frame; extracting feature vectors of the video frames except the last frame in the video by using a trained embedding model; determining the number of clustering clusters according to the preset number, clustering the video frames according to the number of clustering clusters and the feature vectors of the video frames except the last frame, to obtain the clustering cluster to which each video frame belongs; extracting one non-fixed key frame for compressing the video from each clustering cluster, and taking the fixed key frame and the non-fixed key frame as key frames for compressing the video.

Description

Technical Field

[0001] This invention relates to the field of digital video coding, specifically to the field of video frame sampling within digital video coding, and particularly to a method for extracting keyframes for compressed video. Background Technology

[0002] With the development of machine vision, more and more visual information is being processed by computers. Compared to how the human eye processes visual information, computer processing is faster and more efficient, and avoids the problem of decreased processing accuracy due to eye fatigue. In fact, machine vision evaluation of videos focuses more on the semantic information expressed by the video itself, rather than subjective and objective evaluation indicators such as video sharpness, peak signal-to-noise ratio (PSNR), and structural similarity (SSIM) found in the Human Visual System (HVS). Therefore, temporal resampling of the original video and encoding the sampled video frames into a video stream has become an effective strategy for removing redundancy in machine vision encoding. At the decoding end, depth-temporal interpolation techniques can be used to use adjacent frames after decoding the video stream as reference frames to predict the pixel content and video frame content discarded during the resampling process, thereby reconstructing the original video.

[0003] However, in the temporal domain, the method of sampling video frames is uniform sampling at the same time interval. For example, using a fixed step size of x2, x4, x8 for temporal resampling results in resampled video frames with fixed intervals of 1, 3, and 7 frames relative to the original video. Although this method of customizing parameters can save some computation time, it cannot flexibly sample videos with scene changes, videos whose frame number cannot be divided evenly by the fixed step size, and videos with uneven motion amplitude. This has certain shortcomings in practical applications. In other words, if a fixed step size is used for sampling, it may be too small for scenes with small motion / change amplitude, resulting in redundant sampled video frames. For scenes with large motion / change amplitude, the fixed step size may be too large, causing the temporal interpolation technology at the decoding end to be unable to reconstruct the frames discarded in the original video due to temporal downsampling based on adjacent frames. Summary of the Invention

[0004] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a method for extracting keyframes from compressed video.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] According to a first aspect of the present invention, a method for extracting keyframes for compressing video is provided. The method includes: acquiring all video frames in a video and a preset number of required keyframes, wherein the last frame of the video is a fixed keyframe; extracting feature vectors of the video frames other than the last frame using a trained embedding model; determining the number of clusters according to the preset number; clustering the video frames according to the number of clusters and the feature vectors of the video frames other than the last frame to obtain a cluster to which each video frame belongs; extracting a non-fixed keyframe for compressing the video from each cluster, and using the fixed keyframe and the non-fixed keyframe as keyframes for compressing the video.

[0007] In some embodiments of the present invention, the step of extracting a non-fixed keyframe for video compression from each cluster includes: determining frame by frame whether the current video frame meets a preset extraction condition according to the cluster to which each video frame belongs, in the order from the first frame to the penultimate frame of the video; if the current video frame meets the preset extraction condition, then the current video frame is a non-fixed keyframe.

[0008] In some embodiments of the present invention, the preset extraction condition is that the current video is the video frame that appears first in the playback order among all video frames in the cluster to which it belongs.

[0009] In some embodiments of the present invention, the trained embedding model is trained once or multiple times in the following manner: acquiring multiple video samples, selecting a reference video from the multiple video samples; constructing a positive sample pair based on each video frame in the reference video and its next adjacent frame, and constructing a negative sample pair based on each video frame in the reference video and each video frame of other video samples in the multiple video samples besides the reference video; extracting features of each video frame in the multiple video samples using the feature extraction module of the embedding model; mapping the extracted features using the feature mapping module of the embedding model to obtain a feature vector for each video frame; calculating the sub-loss corresponding to each video frame based on the positive and negative sample pairs corresponding to each video frame using a preset loss function based on the feature vector; and updating the parameters of the embedding model based on the loss values ​​determined by the sub-losses corresponding to all video frames in the multiple video samples.

[0010] In some embodiments of the present invention, during training, the sub-loss corresponding to a video frame is configured to: shorten the distance between the video frame and the next adjacent frame in the positive sample, and widen the distance between the video frame and another video frame in the negative sample pair.

[0011] In some embodiments of the present invention, the preset loss function is:

[0012]

[0013] Where, x t Let x represent the t-th video frame in the reference video. t+1 f represents the (t+1)th video frame in the reference video, N represents the number of other video samples besides the reference video among the plurality of video samples, and f t x represents t eigenvectors, f t+1 x represents t+1 eigenvectors, Let c(f) represent the feature vector of the j-th video frame of the n-th other video sample, where J represents the number of video frames of the other video samples. t ,f t+1 The expression represents the cosine similarity between the feature vectors of two video frames in the positive sample pair corresponding to the t-th video frame in the benchmark video. Let represent the cosine similarity between the feature vectors of two video frames in the negative sample pair corresponding to the t-th video frame in the reference video.

[0014] According to a second aspect of the present invention, a video compression method is provided, the method comprising: acquiring keyframes extracted by the method of the first aspect of the present invention, and obtaining a first-order difference array of the keyframes based on the keyframes; compressing the video using a compression method based on the keyframes to obtain a video bitstream corresponding to the video; and processing and synthesizing the first-order difference array and the video bitrate to obtain a bitstream of the video.

[0015] According to a third aspect of the present invention, a method for reconstructing compressed video is provided, the method comprising: acquiring a bitstream of video obtained by the method of the second aspect of the present invention; decoding the bitstream to obtain decoded keyframes; reconstructing the decoded keyframes using an interpolation method based on a first-order difference array in the bitstream to obtain reconstructed video frames; and processing the reconstructed video frames using a video frame processing method to obtain a reconstructed video.

[0016] Compared with the prior art, the advantages of the present invention are as follows:

[0017] 1) By extracting feature vectors of video frames through a trained embedding model, not only is the computational complexity reduced, but the distance between feature vectors of video frames belonging to the same video is also reduced, while the distance between feature vectors of video frames not belonging to the same video is increased, which can effectively distinguish the spatial similarity of video frames.

[0018] 2) Clustering by embedding the feature vectors extracted by the model can better cluster similar video frames into the same cluster, so as to more flexibly fit the change range of video frames in different time periods and select more representative key frames in a non-equidistant manner, thus solving the problem that existing methods cannot flexibly sample.

[0019] 3) It can achieve better video compression and video reconstruction based on the flexibly extracted keyframes, with less information loss during the compression process. In the subsequent video reconstruction part, it can better predict the frames in the video that were not extracted based on the flexibly extracted keyframes, and better ensure the image quality of the decompressed video. Attached Figure Description

[0020] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0021] Figure 1 This is a flowchart illustrating a method for extracting keyframes from compressed video according to an embodiment of the present invention.

[0022] Figure 2 This is a schematic diagram of the embedded model structure according to an embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram of a device for extracting keyframes from compressed video according to an embodiment of the present invention.

[0024] Figure 4 This is a schematic diagram of the bitstream structure according to an embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram of a system architecture for implementing a keyframe extraction method, a video compression method, and a video reconstruction method according to another embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0027] As described in the background section, existing methods for sampling video frames involve uniform sampling at the same time intervals. For videos with shot transitions, videos whose frame count cannot be divided by a fixed step size, and videos with uneven motion amplitude, the fixed temporal sampling step size method cannot flexibly perform sampling, resulting in redundant sampled video frames. Consequently, the temporal interpolation technique at the decoding end cannot reconstruct the frames that were discarded during the sampling process of the original video based on adjacent frames.

[0028] The inventors, through research, believe that the above problems can be solved by first mapping video frames to an embedding space, then performing clustering operations on the embedded video frames, and finally referencing the clustering results to achieve flexible sampling. Therefore, this invention provides a method for extracting keyframes from compressed videos. The method includes: obtaining all video frames in the video and a preset number of required keyframes; wherein the last frame of the video is used as a fixed keyframe; using a trained embedding model to extract feature vectors from the video frames excluding the last frame; extracting feature vectors from video frames using a trained embedding model reduces computational load, while simultaneously narrowing the distance between feature vectors of video frames belonging to the same video and widening the distance between feature vectors of video frames not belonging to the same video, thus effectively distinguishing the spatial similarity of video frames. The number of clusters is determined according to the preset number; the video frames are clustered according to the number of clusters and the feature vectors of the video frames excluding the last frame to obtain the cluster to which each video frame belongs; clustering using feature vectors extracted by the embedding model better groups similar video frames into the same cluster, facilitating subsequent keyframe extraction. Extract a non-fixed keyframe from each cluster for video compression, and use both the fixed and non-fixed keyframes as keyframes for video compression. This allows for more flexible adaptation to the variation of video frames over different time periods, and the selection of more representative keyframes at non-equidistant intervals. After compression based on the selected keyframes, less information is lost, thus better ensuring the image quality of the subsequently decompressed video.

[0029] To better illustrate the technical solution of the present invention, a detailed description is provided below with reference to specific embodiments.

[0030] Figure 1 The diagram shown is a flowchart illustrating a method for extracting keyframes from compressed video proposed in this invention. The method includes the following steps:

[0031] S1. Obtain all video frames in the video and the preset number of keyframes required; wherein, the last frame of the video is used as a fixed keyframe;

[0032] S2. Use the trained embedding model to extract the feature vectors of the video frames other than the last frame.

[0033] S3. Determine the number of clusters according to the preset number, and cluster the video frames according to the number of clusters and the feature vectors of the video frames except the last frame to obtain the cluster to which each video frame belongs.

[0034] S4. Extract one non-fixed keyframe from each cluster for video compression, and use both the fixed keyframe and the non-fixed keyframe as keyframes for video compression. The method for extracting keyframes for video compression proposed in this invention will be described in detail below with reference to specific embodiments, examples, and accompanying drawings.

[0035] In step S1, all video frames in the video and the preset number of required keyframes are obtained, wherein the last frame of the video is used as a fixed keyframe.

[0036] According to one embodiment of the present invention, based on a relevant protocol, each video frame in a video has a specific sequence number according to the playback order. In order to perform video compression and video reconstruction based on the extracted keyframes, the first and last frames of each video must be extracted as keyframes. However, in order to better extract the required keyframes later, the first frame needs to participate in feature extraction and clustering. Therefore, the last frame of the video is used as a fixed keyframe. Illustratively, if the sequence numbers of all video frames in a video are 1, 2, ..., 20, then the last frame 20 is used as a fixed keyframe, and the remaining video frames 1-19 are used for subsequent feature extraction and clustering.

[0037] According to one embodiment of the present invention, the video can be a video from any scenario, such as a video from a driving scenario, a video from an industrial production inspection scenario, or a video from a monitoring scenario. Additionally, the video can be a segment of a complete video during video transmission, such as a segment of a video stream.

[0038] In step S2, the feature vectors of the video frames other than the last frame are extracted using a trained embedding model.

[0039] According to an embodiment of the present invention, the trained embedding model includes a feature extraction module and a feature mapping module. The video frames obtained in step S1 are input into the trained embedding model, and the feature extraction module of the trained embedding model is used to extract the features of the video frames. The feature mapping module of the trained embedding model is used to map the extracted features to obtain the feature vectors of each video frame.

[0040] According to an embodiment of the present invention, the trained embedding model is trained once or multiple times in the following manner: acquiring multiple video samples, selecting a reference video from the multiple video samples; constructing a positive sample pair based on each video frame in the reference video and its next adjacent frame, and constructing a negative sample pair based on each video frame in the reference video and each video frame of other video samples in the multiple video samples besides the reference video; extracting features of each video frame in the multiple video samples using the feature extraction module of the embedding model; mapping the extracted features using the feature mapping module of the embedding model to obtain a feature vector for each video frame; calculating a sub-loss corresponding to each video frame based on the positive and negative sample pairs corresponding to each video frame using a preset loss function based on the feature vector; and updating the parameters of the embedding model based on the loss values ​​determined by the sub-losses corresponding to all video frames in the multiple video samples.

[0041] According to one embodiment of the present invention, during training, the sub-loss corresponding to a video frame is configured to: shorten the distance between the video frame and its next adjacent frame in the positive sample, and widen the distance between the video frame and another video frame in its negative sample pair. In this invention, the distance between two video frames can be measured using cosine similarity. It should be understood that the distance between two video frames can also be measured using Manhattan distance, Euclidean distance, or Mahalanobis distance.

[0042] According to one embodiment of the present invention, the preset loss function is:

[0043]

[0044] Where, x t Let x represent the t-th video frame in the reference video. t+1 f represents the (t+1)th video frame in the reference video, N represents the number of other video samples besides the reference video among the plurality of video samples, and f t x represents t eigenvectors, f t+1 x represents t+1 eigenvectors, Let c(f) represent the feature vector of the j-th video frame of the n-th other video sample, where J represents the number of video frames of the other video samples. t ,f t+1 The expression represents the cosine similarity between the feature vectors of two video frames in the positive sample pair corresponding to the t-th video frame in the benchmark video. Let represent the cosine similarity between the feature vectors of two video frames in the negative sample pair corresponding to the t-th video frame in the reference video.

[0045] Schematic, the cosine similarity between the feature vectors of two video frames can be calculated as follows:

[0046]

[0047] Where T represents the number of video frames.

[0048] According to an example of the present invention, the acquired video samples include five video samples A, B, C, D, and E. Each video sample includes multiple video frames (e.g., 1, 2, 3, 4, 5, and 6). Any one of these video samples is used as the reference video, for example, video sample A is used as the reference video, and the remaining video samples B, C, D, and E are used as other video samples. The feature vectors of each video frame in the training set are extracted by an embedding model. For the reference video A (which includes video frames 1, 2, 3, 4, 5, and 6), the sub-loss of the 3rd frame in the reference video sample A is calculated. This requires calculating the cosine similarity between the feature vectors of the 3rd and 4th frames in the corresponding positive sample pair (3,4). If the feature vector of the 3rd frame is [1,2] and the feature vector of the 4th frame is [3,5], then the cosine similarity between them is: The cosine similarity between the feature vectors of the two video frames in the negative sample pair corresponding to the 3rd frame is calculated. For example, when other video samples are video B (a, b, c, d, e, f), the negative sample pair corresponding to its 3rd frame is: (3, a), (3, b), (3, c), (3, d), (3, e), (3, f). The loss value of the 3rd frame is determined based on the calculated cosine similarity. The loss value is determined by the sub-loss corresponding to all video frames in multiple video samples in the above way. The gradient is calculated based on the loss value to update the parameters of the embedding model in reverse, so that the trained model can better shorten the distance between the feature vectors of video frames belonging to the same video and widen the distance between the feature vectors of video frames not belonging to the same video, so that subsequent clustering can be performed according to the expectation to obtain better clustering results.

[0049] According to one embodiment of the present invention, deep learning modules such as VGG16 and ResNet can be used to extract feature vectors of video frames. However, in order to better distinguish the spatial similarity of video frames, the present invention constructs an embedding model for extracting feature vectors of video frames, such as... Figure 2As shown, the model includes a feature extraction module and a feature mapping module. The feature extraction module includes 5 convolutional layers and 1 pooling layer, and the feature mapping module includes 2 fully connected layers. Schematic, when a video frame is input to the feature extraction module, it first passes through the first convolutional layer to obtain a first feature (e.g., [64, 244, 244] in the figure). The first feature output from the first convolutional layer is used as the input to the second convolutional layer, and the second feature output from the second convolutional layer (e.g., [128, 112, 112] in the figure) is used as the input to the third convolutional layer. The third feature output from the third convolutional layer (e.g., [256, 56, 56] in the figure) is used as the input to the fourth convolutional layer, and the fourth feature output from the fourth convolutional layer (e.g., [512, 28, 28] in the figure) is used as the input to the fifth convolutional layer. The fifth feature output from the fifth convolutional layer (e.g., [512, 14, 14] in the figure) is used as the input to the pooling layer. This fifth feature undergoes dimensionality reduction by the pooling layer to obtain pooled features (e.g., [512, 7, 7] in the figure). The features of the video frame output from the feature extraction module are input to the first fully connected layer in the feature mapping module. After mapping, the first feature vector is output (e.g., [25088, l1] in the figure). The first feature vector output from the first fully connected layer is used as the input to the second fully connected layer. The second feature vector output from the second fully connected layer is used as the final extracted feature vector of the video frame (e.g., [l1, l2] in the figure). It should be understood that the above is merely an illustrative example. Those skilled in the art can increase or decrease the number of convolutional and fully connected layers, for example, by reducing one or two convolutional layers and adding one fully connected layer, to obtain other embodiments.

[0050] In step S3, the number of clusters is determined according to the preset number, and the video frames are clustered according to the number of clusters and the feature vectors of the video frames except the last frame to obtain the cluster to which each video frame belongs.

[0051] According to one embodiment of the present invention, the number of clusters is determined according to the preset number. Since the last frame of the video is a fixed key frame, the number of clusters is the preset number minus 1. The video frames are clustered using the K-means clustering algorithm based on the number of clusters and the feature vectors of the video frames other than the last frame, to obtain the cluster to which each video frame belongs.

[0052] According to one example of the present invention, the preset quantity is 8, so the number of clusters can be determined to be 7. The K-means clustering algorithm is used to determine the cluster center of each cluster based on the number of clusters and the feature vectors of the video frames excluding the last frame. Then, the distance between the feature vector of each video frame and each cluster center is calculated. Based on the calculated distance, each video frame is assigned to the cluster of the nearest cluster center, until all video frames except the last frame are clustered, thus obtaining the clustering result. It should be understood that using clustering algorithms to determine the cluster centers and the calculation process are well-known techniques to those skilled in the art and will not be elaborated upon here.

[0053] Optionally, in some embodiments of the present invention, existing hierarchical clustering algorithms, density clustering algorithms, spectral clustering algorithms, etc., can be used to cluster video frames.

[0054] In step S4, a non-fixed keyframe for compressing the video is extracted from each cluster, and the fixed keyframe and the non-fixed keyframe are used as keyframes for compressing the video.

[0055] According to one embodiment of the present invention, based on the clustering results obtained in step S3, a non-fixed keyframe for video compression is extracted from each cluster. The extraction method includes: judging frame by frame whether the current video frame meets the preset extraction conditions according to the cluster to which each video frame belongs, in the order from the first frame to the second to last frame of the video; if the current video frame meets the preset extraction conditions, then the current video frame is a non-fixed keyframe, wherein the preset extraction condition is that the current video is the video frame that appears first in the playback order of all video frames in the cluster to which it belongs.

[0056] According to an example of the present invention, for a video sample comprising 25 frames, the number of clusters is determined to be 7, that is, the video frames from frame 1 to 24 are clustered. The clustering results may be as follows: First cluster: 1, 2, 3, 4, 5; Second cluster: 6, 7, 8, 9; Third cluster: 10, 11, 12, 13; Fourth cluster: 14, 15, 16; Fifth cluster: 17, 18, 19; Sixth cluster: 20, 21; Seventh cluster: 22, 23, 24. According to preset extraction conditions, the non-fixed keyframes can be extracted as: 1, 6, 10, 14, 17, 20, 22. It should be understood that the above is only an example, and the present invention is not limited thereto.

[0057] According to one embodiment of the invention, the non-fixed keyframes and fixed keyframes extracted in the above manner are used as keyframes for compressing the video. Illustratively, based on the above example, the final keyframes extracted for video compression are: 1, 6, 10, 14, 17, 20, 22, 25.

[0058] It is worth noting that during the keyframe extraction process, the number of clusters may not be consistent with the number of non-fixed keyframes extracted. Clustering can be performed again by appropriately lowering the K value to make the number of extracted keyframes as consistent as possible with the number of keyframes required.

[0059] To better implement the keyframe extraction method for compressed video proposed in this invention, a device for extracting keyframes from compressed video is provided to implement the extraction method, such as... Figure 3 As shown, the device includes a parameter receiving module, an embedding module, a clustering module, and an extraction module; wherein, the parameter receiving module is used to acquire all video frames in the video and a preset number of required keyframes; the embedding module is used to extract feature vectors of video frames other than the last frame in the video using a trained embedding model; the clustering module is used to cluster the video frames according to the determined number of clusters and the feature vectors of video frames other than the last frame; the extraction module is used to extract a non-fixed keyframe for video compression from each cluster according to preset extraction conditions.

[0060] Based on the above-mentioned keyframe extraction method for compressed video, feature vectors of video frames can be extracted through a trained embedding model to distinguish the spatial similarity of video frames. This allows for better clustering of similar video frames into the same cluster, enabling flexible selection of keyframes based on the non-equidistant time intervals of video content changes. Furthermore, the keyframes selected by the above method are more closely aligned with the actual situation of the video, solving the problem of inflexible sampling in existing methods.

[0061] According to a second aspect of the present invention, a video compression method is also provided, the method comprising: acquiring keyframes extracted by the method of the first aspect of the present invention, and obtaining a first-order difference array of the keyframes based on the keyframes; compressing the video using a compression method based on the keyframes to obtain a video bitstream corresponding to the video; and processing and synthesizing the first-order difference array and the video bitrate to obtain a bitstream of the video.

[0062] According to one embodiment of the present invention, a first-order difference array of the keyframes is obtained based on the keyframes. For example, if the video frame numbers of the original video are 0, 1, 2, ..., 14, 15, and the first and last frames of the video are extracted as keyframes for the compressed video, then the extracted keyframes are: 0, 4, 7, 9, 13, 15. The first-order difference array of the keyframes can then be obtained as: 4, 3, 2, 4, 2. It should be understood that the difference value between every two video frames represents the number of video frames to be inserted + 1.

[0063] According to one embodiment of the present invention, the video is compressed using a compression method based on the keyframes to obtain a video bitstream corresponding to the video. The video bitstream includes, but is not limited to, compression information such as video frame data, motion vectors, residual data, video frame size, frame rate, and quantization parameters. Optionally, compression methods such as H.264, H.266, and AVS can be used to compress the video based on the extracted keyframes.

[0064] According to one embodiment of the present invention, the first-order difference array and the video bitrate are sequentially concatenated using character concatenation to obtain the video bitstream, the structure of which is as follows: Figure 4 As shown, the number of bits in the first-order difference array and the video bitrate is not limited.

[0065] The proposed video compression method can compress the video based on the extracted keyframes. Furthermore, since the keyframes are embedded and mapped, the computational complexity of the compression process is reduced, and the computational space and resources are decreased.

[0066] According to a third aspect of the present invention, a method for reconstructing compressed video is also provided, the method comprising the following steps: acquiring a bitstream of video obtained by the method described in the second aspect of the present invention; decoding the bitstream to obtain decoded keyframes; reconstructing the decoded keyframes using an interpolation method based on a first-order difference array in the bitstream to obtain reconstructed video frames; and processing the reconstructed video frames using a video frame processing method to obtain a reconstructed video.

[0067] According to one embodiment of the present invention, the number of video frames to be reconstructed between each two keyframes can be obtained from the first-order difference array in the bitstream. The video frames lost after compression can be reconstructed by using interpolation based on the relevant information of each two keyframes after decoding (e.g., video frame data, motion vectors, residual data, video frame size, frame rate, quantization parameters, and other related compression information) to obtain the reconstructed video frames.

[0068] According to one embodiment of the present invention, based on the reconstructed video frames, some existing video frame processing methods can be used to process the video frames to obtain the reconstructed video. Illustratively, existing processing methods include: deblocking filtering, transformation, color space conversion, etc.

[0069] Based on the above-mentioned method for reconstructing compressed video, it is possible to reconstruct frames that were discarded due to temporal downsampling in the original video well from the bitstream obtained by compressing the extracted keyframes.

[0070] In another embodiment of the present invention, a method for extracting keyframes for video compression can further perform spatial domain sampling and region of interest sampling on the extracted keyframes to obtain keyframes with lower bitrates for video compression. The method further includes the following steps: adjusting the extracted video keyframes according to a preset video frame size to obtain adjusted keyframes; and highlighting the adjusted keyframes based on salient targets in the video using a saliency detection method to obtain the final keyframes for video compression. Adjusting and highlighting the extracted keyframes can reduce the bitrate of video compression and reduce the computational complexity of the compression process.

[0071] Accordingly, in order to compress video using keyframes extracted according to another embodiment of the present invention, the compression method includes: acquiring the keyframes extracted by the above method, and related information for video reconstruction generated during the adjustment and highlighting of the keyframes, and obtaining a first-order difference array of the keyframes; compressing the video using a compression method based on the keyframes to obtain a video bitstream corresponding to the video; and processing and synthesizing the first-order difference array, the related information for video reconstruction, and the video bitrate to obtain a bitstream of the video.

[0072] Accordingly, in order to reconstruct the video based on the bitstream obtained by the above compression method, the reconstruction method includes: obtaining the bitstream of the video obtained by the above compression method; decoding the bitstream to obtain decoded keyframes; reconstructing the decoded keyframes using interpolation based on relevant information in the bitstream and a first-order difference array to obtain reconstructed video frames; and processing the reconstructed video frames using a video frame processing method to obtain the reconstructed video.

[0073] Schematic illustration, in order to coherently demonstrate the keyframe extraction method, video compression method, and video reconstruction method in another embodiment of the present invention, such as Figure 5As shown in the diagram, the system architecture for implementing the above method includes: a temporal sampling module, a spatial sampling module, a region of interest (ROI) sampling module, an encoding module, a synthesis module, a decoding module, a ROI reconstruction module, a spatial reconstruction module, a temporal reconstruction module, and a post-processing module. The temporal sampling module deploys the following modules: Figure 3 The extraction device shown is used to map and cluster video frames to extract keyframes; the spatial sampling module is used to adjust the extracted keyframes of the video according to the preset size of the video frames to obtain adjusted keyframes; the region of interest sampling module is used to highlight the adjusted keyframes according to salient targets in the video using a saliency detection method to obtain the final keyframes for video compression; the encoding module is used to receive the final keyframes for video compression, and compress the video according to the keyframes to obtain the corresponding video bitstream; the synthesis module is used to process and synthesize the first-order difference array determined according to the keyframes, relevant information for video reconstruction, and the video bitrate to obtain the bitstream of the video; the decoding module... The system receives a bitstream of video, decodes the bitstream to obtain decoded keyframes; the region of interest (ROI) reconstruction module reconstructs the ROI of the decoded keyframes based on relevant information for video reconstruction generated by the ROI sampling module, resulting in a ROI-reconstructed video frame; the spatial domain reconstruction module reconstructs the ROI-reconstructed video frame based on relevant information for video reconstruction generated by the spatial domain sampling module, resulting in a spatially reconstructed video frame; the temporal domain reconstruction module reconstructs the spatially reconstructed video frame based on the first-order difference array in the bitstream, resulting in a reconstructed video frame; and the post-processing module processes the reconstructed video frame using methods for processing video frames, resulting in a reconstructed video.

[0074] In summary, the method for extracting keyframes from compressed video proposed in this invention has the following advantages:

[0075] 1) By extracting feature vectors of video frames through a trained embedding model, not only is the computational complexity reduced, but the distance between feature vectors of video frames belonging to the same video is also reduced, while the distance between feature vectors of video frames not belonging to the same video is increased, which can effectively distinguish the spatial similarity of video frames.

[0076] 2) Clustering by embedding the feature vectors extracted by the model can better cluster similar video frames into the same cluster, so as to more flexibly fit the change range of video frames in different time periods and select more representative key frames in a non-equidistant manner, thus solving the problem that existing methods cannot flexibly sample.

[0077] 3) It can achieve better video compression and video reconstruction based on the flexibly extracted keyframes, with less information loss during the compression process. In the subsequent video reconstruction part, it can better predict the frames in the video that were not extracted based on the flexibly extracted keyframes, and better ensure the image quality of the decompressed video.

[0078] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0079] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0080] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0081] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for extracting keyframes from compressed video, characterized in that, The method includes the following steps: Obtain all video frames in the video and the preset number of keyframes required, wherein the last frame of the video is used as a fixed keyframe; The feature vectors of video frames (excluding the last frame) in the video are extracted using a trained embedding model. The trained embedding model is trained once or multiple times as follows: multiple video samples are acquired, and a reference video is selected from these samples; a positive sample pair is constructed based on each video frame in the reference video and its next adjacent frame, and a negative sample pair is constructed based on each video frame in the reference video and each video frame in the other video samples (excluding the reference video); features of each video frame in the multiple video samples are extracted using the feature extraction module of the embedding model; the extracted features are mapped using the feature mapping module of the embedding model to obtain the feature vector of each video frame; based on the positive and negative sample pairs corresponding to each video frame, a sub-loss corresponding to each video frame is calculated using a preset loss function based on the feature vector; the parameters of the embedding model are updated based on the loss values ​​determined by the sub-losses corresponding to all video frames in the multiple video samples; wherein, during training, the sub-loss corresponding to a video frame is configured to: shorten the distance between the video frame and its next adjacent frame in the positive sample, and widen the distance between the video frame and another video frame in its negative sample pair; the preset loss function is: ； in, Indicates the first in the benchmark video One video frame, Indicates the first in the benchmark video One video frame, This indicates the number of video samples other than the reference video among the plurality of video samples. express eigenvectors, express eigenvectors, Indicates the first The first of the other video samples Feature vectors of each video frame This indicates the number of video frames in other video samples. Indicates the first in the benchmark video The cosine similarity between the feature vectors of two video frames in a positive sample pair corresponding to each video frame. Indicates the first in the benchmark video Cosine similarity between the feature vectors of two video frames in a negative sample pair corresponding to a video frame; The number of clusters is determined according to the preset number. The video frames are clustered according to the number of clusters and the feature vectors of the video frames except the last frame to obtain the cluster to which each video frame belongs. Extract a non-fixed keyframe from each cluster for video compression, and use the fixed keyframe and the non-fixed keyframe as keyframes for video compression.

2. The method according to claim 1, characterized in that, Extracting a non-fixed keyframe from each cluster for video compression includes: Following the order from the first frame to the second-to-last frame of the video, each video frame is judged frame by frame according to the cluster to which it belongs to determine whether the current video frame meets the preset extraction conditions. If the current video frame meets the preset extraction conditions, then the current video frame is a non-fixed keyframe.

3. The method according to claim 2, characterized in that, The preset extraction condition is that the current video is the first video frame in the playback order of all video frames in its cluster.

4. A video compression method, characterized in that, The method includes: Obtain the keyframes extracted by any of the methods described in claims 1-3, and obtain the first-order difference array of the keyframes based on the keyframes; The video is compressed using a compression method based on the keyframes to obtain the corresponding video bitstream. The first-order difference array and the video bitrate are processed and combined to obtain the bitstream of the video.

5. A method for reconstructing compressed video, characterized in that, The method includes the following steps: Obtain the bitstream of the video obtained by the method described in claim 4, and decode the bitstream to obtain the decoded keyframes; Based on the first-order difference array in the bitstream, the decoded keyframes are reconstructed using interpolation to obtain the reconstructed video frames; The reconstructed video frames are processed using a method that processes video frames to obtain the reconstructed video.

6. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1 to 5.

7. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the electronic device to perform the steps of the method as described in any one of claims 1 to 5.

Images