A video key frame extraction method based on dual-stream information and sparse representation
Through the method of dual-stream information and sparse representation, combined with video space and time stream information, video keyframes are extracted, which solves the problem of lack of motion information utilization in the prior art, improves the accuracy and efficiency of keyframe extraction, and reduces the compression rate.
Patent Information
- Application Number
- CN202210616931.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-01
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-06-01
AI Technical Summary
The existing video keyframe extraction method based on sparse representation cannot fully consider the motion information of objects in the video, resulting in the extracted information only coming from spatial information and lacking effective utilization of motion information.
Using the method of double-stream information and sparse representation, the double-stream information matrix is obtained by constructing the video spatial stream matrix and time stream matrix, and the dual-stream feature matrix is extracted in combination with the VGG16 network, and the sparse representation model is used to calculate the sparse coefficient matrix to extract keyframes, taking into account video in a comprehensive way. Space and motion information of the frame.
The calculation accuracy of the keyframe extraction algorithm is improved, the number of extracted keyframes is reduced, the compression rate is reduced, the original video information can be better represented, and the video comprehension ability and extraction efficiency are improved.
Smart Images

Figure CN114913466B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision key frame extraction, and in particular to a video key frame extraction method based on dual-stream information and sparse representation. Background Art
[0002] With the rapid development of information technology, the Internet and multimedia technology are widely used. A large amount of video information is generated in people's daily life and work. The structure of video data is complex and diverse. Video summarization has become the main research content in the field of video understanding, which will effectively improve the efficiency of video retrieval and reduce the difficulty of video data storage. Here, key frame extraction is an important issue in video summarization. Since the amount of video data is huge and contains a lot of redundant information, how to effectively extract video key frame information is very important. Video signals are composed of continuous image sequences, which are rich in content, expressive, and have a large amount of information. There is a strong correlation between adjacent frames, and there is usually a lot of redundancy. Therefore, it is very important to use key frame extraction methods. Key frame extraction is to select a small number of frames with the largest amount of information to approximate the original video, which can alleviate the pressure of high-dimensional video signal processing and improve the efficiency of video understanding.
[0003] Existing image feature extraction methods often just pass the original image information through the neural network, and then stack the results, or stack these frames as a whole and input them into the convolutional neural network. In order to achieve a spatiotemporal learning goal. Traditional methods are better at learning local information, but not so good at learning the motion information before and after the video frame. Dual-stream information can well represent the spatial information and motion information of the original video frame. Using dual-stream information in key frame extraction technology can effectively improve the model's learning effect on video frames, improve the model's ability to understand videos, and improve the accuracy of key frame extraction.
[0004] At present, key frame extraction based on sparse models has attracted great attention due to its outstanding advantages, simplicity and mature mathematical analysis. The sparse representation of signals has a perfect mathematical representation. By describing the signal through a dictionary and a sparse coefficient matrix, a more concise representation of complex signals can be obtained, thereby improving the signal processing performance. It is worth noting that the effectiveness of the key frame extraction method based on sparse representation mainly depends on the sparsity constraint, which is usually based on the L1 norm. The sparse modeling representation selection (SMRS) method uses the L1 norm to calculate the sparse sparse matrix corresponding to the key frame. Although the sparse modeling representation selection method can effectively obtain key frames, the existing sparse representation-based methods cannot extract effective key frames. The extracted information only comes from the spatial information of the video, and does not fully consider the motion information of the objects in the video.
[0005] Therefore, it is necessary to develop more effective methods to obtain better video key frames. Summary of the invention
[0006] The purpose of the present invention is to provide a video key frame extraction method based on dual-stream information and sparse representation to solve the problems existing in the above-mentioned prior art, improve the calculation accuracy of the key frame extraction algorithm, and at the same time reduce the number of extracted key frames and lower the compression rate.
[0007] To achieve the above object, the present invention provides the following solutions:
[0008] A video key frame extraction method based on dual-stream information and sparse representation, comprising:
[0009] Splitting the video file to be extracted to obtain image frames, and constructing a video spatial flow matrix and a video temporal flow matrix based on the image frames;
[0010] Obtaining a dual-stream information matrix through the video spatial stream matrix and the video temporal stream matrix, performing feature extraction on the dual-stream information matrix to obtain a dual-stream feature matrix;
[0011] Inputting the dual-stream feature matrix into a sparse representation model, calculating a sparse coefficient matrix, and obtaining a key frame index based on the sparse coefficient matrix;
[0012] The key frames in the video file to be extracted are extracted through the key frame index.
[0013] Preferably, constructing the video spatial stream matrix includes:
[0014] Pixel points in the image frames are extracted, and the pixel points are arranged in sequence to obtain spatial flow feature vectors of different image frames, and the spatial flow feature vectors of all the image frames are combined to form the video spatial flow matrix.
[0015] Preferably, constructing the video time flow matrix includes:
[0016] Pixel points in the image frames are extracted, and according to the movement of the pixel points between different image frames, the optical flows of different image frames are obtained by the Farneback method, the optical flows of the different image frames are arranged and combined in sequence to obtain the time flow feature vectors of the different image frames, and the time flow feature vectors of all the image frames are combined to form the video time flow matrix.
[0017] Preferably, obtaining the dual-stream information matrix includes:
[0018] The video spatial stream matrix and the video temporal stream matrix are concatenated column by column to obtain the dual-stream information matrix.
[0019] Preferably, obtaining the dual-stream feature matrix includes: extracting the dual-stream information matrix through a VGG16 network to obtain the dual-stream feature matrix.
[0020] Preferably, the sparse representation model is:
[0021]
[0022] Among them, f(C) is the sparse representation model function, Y is the two-stream feature matrix, C is the sparse coefficient matrix, τ is the constraint parameter of the sparse matrix, ||·|| represents the norm operation function, ||·|| F represents the Frobenius norm, and st is the constraint condition.
[0023] Preferably, the method for calculating the sparse coefficient matrix based on the sparse representation model is:
[0024]
[0025] Among them, C is the sparse coefficient matrix, Y is the two-stream feature matrix, τ is the constraint parameter of the sparse matrix, and ||·|| represents the norm operation function.
[0026] Preferably, the form of calculating the sparse coefficient matrix solution is:
[0027]
[0028] Where Γ represents the permutation matrix, I k is a k-dimensional identity matrix, and the elements in Δ are in [0,1).
[0029] Preferably, obtaining the key frame index based on the sparse coefficient matrix includes: extracting non-zero rows in the sparse coefficient matrix, wherein the non-zero rows in the sparse coefficient matrix are the key frame index.
[0030] Preferably, the method further comprises evaluating the accuracy of the extracted key frames, based on the video key frame compression rate and F-measure as evaluation indicators, wherein the F-measure is used to measure the accuracy of the extracted key frames;
[0031] The method for calculating the video key frame compression rate is:
[0032]
[0033] Summary length is the compression rate of the video key frame, N select is the number of key frames extracted, N whole The total number of frames in the video.
[0034] The beneficial effects of the present invention are:
[0035] The present invention provides a video key frame extraction method based on dual-stream information and sparse representation, which can efficiently extract fewer key frames in a video, reduce the number of extracted key frames, reduce the compression rate of key frame extraction, and better represent the information of the original video. By utilizing the dual-stream information of the video, the spatial information and motion information in the video frame are comprehensively extracted, and then the sparse modeling representation learning method is used to obtain accurate key frames, while improving the calculation speed of the key frame extraction algorithm. The method is simple to operate, and a video signal is input. The video key frame can be obtained by sequentially performing the steps of the video key frame extraction method based on dual-stream information and sparse representation. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0037] Figure 1 A flow chart of a video key frame extraction method based on dual-stream information and sparse representation according to an embodiment of the present invention;
[0038] Figure 2 Schematic diagram of a video key frame extraction model based on dual-stream information and sparse representation according to an embodiment of the present invention. DETAILED DESCRIPTION
[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0041] like Figure 1-2 As shown, the present invention provides a video key frame extraction method based on dual-stream information and sparse representation, which specifically includes:
[0042] The video is split into image frames, each frame is represented as each column of the matrix, forming a video spatial flow matrix. Through the video image frames, the optical flow of the video image is obtained by the Farneback method, and the optical flow of each frame is used as a column to form the video time flow matrix. Then the spatial flow matrix and the time flow matrix are spliced by column to obtain the dual-flow matrix of the video, and the above matrix is extracted by VGG16 to obtain the dual-flow feature matrix Y of the video. By constructing the feature matrix of the video, the original video can be represented in a matrix form that can be sparsely represented.
[0043] In this embodiment, the data set uses a video signal, each video is composed of multiple video frames, there is redundant information between adjacent video frames, and each video frame is an image containing information.
[0044] Extract the pixels in each video frame, and obtain the optical flow of the video through the Farnback method based on the movement of the pixels between frames. Arrange the pixels from left to right and from top to bottom to form a spatial flow feature vector, that is, to obtain the spatial flow feature vector of the corresponding video frame. Combine the spatial flow feature vectors of all frames to form a video spatial flow matrix. Arrange the optical flow of all frames from left to right and from top to bottom to form a feature vector, that is, to obtain the temporal flow feature vector of the corresponding video frame. Combine the temporal flow vectors of all video frames to form a video temporal flow matrix. Concatenate the spatial flow matrix and the temporal flow matrix by column to obtain the dual-flow matrix of the video. Extract the dual-flow matrix through VGG16 to obtain the video feature Y, that is,
[0045] Y=[y1,y2,…,y i ,…,y N ],y i (1≤i≤N) is the feature vector of the i-th video frame.
[0046] A sparse representation model is constructed, which can obtain a sparse coefficient matrix and then efficiently extract the key frames of the video.
[0047] Initial matrices Y and D, where Y is the video feature matrix and D is the dictionary matrix. Substitute matrices Y and D into the following formula (1) to calculate the sparse coefficient matrix C.
[0048]
[0049] Among them, f(C) is the sparse representation model function, Y is the video feature matrix, C is the sparse coefficient matrix, D is the dictionary matrix, τ is the constraint parameter of the sparse matrix, and generally takes a value greater than 0, and ||·|| represents the norm operation function.
[0050] The original signal matrix can be represented by a linear combination of a few column vectors in the matrix. For example, if only the second and fourth rows in the sparse matrix are non-zero elements, the original signal matrix can be represented as a linear combination (product) of the second and fourth columns in the matrix and the second and fourth rows in the sparse matrix.
[0051] The video dual-stream signal matrix Y replaces the original signal matrix Y and dictionary matrix D in formula (1), and the constructed sparse representation model is shown in formula (2):
[0052]
[0053] Among them, Y is the video matrix signal, C is the sparse coefficient matrix, τ is the constraint of the sparse matrix, and τ is the constraint parameter of the sparse matrix, which is generally greater than 0. By bringing in the video feature matrix, the constructed sparse representation model is used for key frame extraction.
[0054] Calculate the sparse coefficient matrix, such as Figure 2 As shown, the non-zero rows of the sparse coefficient matrix represent the indices of the key frames.
[0055] Since the product of the dictionary (i.e., the video matrix) and the sparse coefficient matrix is the video signal matrix, the key frame solution problem can constitute an optimization sparse representation problem, that is, to find a sparse coefficient matrix that satisfies the sparse constraint. In this embodiment, the solution for calculating the sparse coefficient matrix X is shown in formula (3):
[0056]
[0057] Among them, Y is the video feature matrix, C is the sparse coefficient matrix, τ is the sparse constraint coefficient, which is generally greater than 0, and ||·|| represents the norm function.
[0058] Solve equation (3) and the final result C is shown in equation (4):
[0059]
[0060] Where Γ represents the permutation matrix, I k is a k-dimensional identity matrix, and the elements in Δ are in [0,1).
[0061] The calculated C is a sparse coefficient matrix, most of the elements in the sparse matrix are zero, and the non-zero rows represent the key frame index in the video. If the i and j rows in C are non-zero rows, the key frame index of the video is {i, j}. Select the corresponding key frame of the video according to the obtained key frame index. If the key frame index of the video is {i, j}, the key frame of the video is the i-th frame video frame image and the j-th frame video frame image. Associating the key frame index with the key frame can quickly and accurately bind the key frame, thereby improving the accuracy and efficiency of key frame extraction.
[0062] In order to test the effect of the extracted key frames, the test set is used to verify it. The video key frame compression rate and F-measure are used as evaluation indicators. Given a video, its key frame compression rate (Summary length) is shown in formula (5):
[0063]
[0064] Among them, N select is the number of key frames extracted, N whole The key frame compression ratio is the ratio of the extracted key frames to the total number of frames in the video. Its unit is %. The smaller the value, the stronger the video compression level.
[0065] At the same time, in this embodiment, F-measure is used to measure the accuracy of the extracted key frames. The definition of F-measure is shown in formula (6):
[0066]
[0067] Among them, P and R are precision and recall respectively. The higher the F-measure is, the better the extracted key frames are, and the more accurately they can reflect the content of the original video.
[0068] In order to fully extract the motion information hidden in the video frame, the present invention introduces optical flow features. This feature is extracted from the original image of the video frame, and the sparse representation of the video signal expression is used to generate a small number of (sparse) video key frames that can express motion features. Therefore, the present invention has a better summary effect on video data containing a large amount of motion information. In addition, by inputting the original video, accurate video key frames can be obtained through the above steps without the need for other additional operations, which is relatively convenient and simple.
[0069] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.
Claims
1. A video key frame extraction method based on dual-stream information and sparse representation, characterized in that: include: Splitting the video file to be extracted to obtain image frames, and constructing a video spatial flow matrix and a video temporal flow matrix based on the image frames; Constructing the video spatial stream matrix includes: Extracting pixel points in the image frames, arranging the pixel points in sequence to obtain spatial stream feature vectors of different image frames, and combining the spatial stream feature vectors of all the image frames to form the video spatial stream matrix; Constructing the video time flow matrix includes: Extracting pixel points in the image frames, obtaining optical flows of different image frames by using the Farneback method according to the movement of the pixel points between different image frames, arranging and combining the optical flows of different image frames in sequence to obtain time flow feature vectors of different image frames, and combining the time flow feature vectors of all image frames to form the video time flow matrix; Obtaining a dual-stream information matrix through the video spatial stream matrix and the video temporal stream matrix, performing feature extraction on the dual-stream information matrix to obtain a dual-stream feature matrix; Acquiring the dual-stream information matrix includes: splicing the video spatial stream matrix and the video temporal stream matrix by columns to obtain the dual-stream information matrix; Inputting the dual-stream feature matrix into a sparse representation model, calculating a sparse coefficient matrix, and obtaining a key frame index based on the sparse coefficient matrix; The sparse representation model is: in, is the sparse representation model function, is the two-stream feature matrix, is a sparse coefficient matrix, is the constraint parameter of the sparse matrix, represents the norm operation function, F represents the Frobenius norm, is a constraint condition; The key frames in the video file to be extracted are extracted through the key frame index.
2. The video key frame extraction method based on dual-stream information and sparse representation according to claim 1 is characterized in that: Obtaining the dual-stream feature matrix includes: extracting the dual-stream information matrix through a VGG16 network to obtain the dual-stream feature matrix.
3. The video key frame extraction method based on dual-stream information and sparse representation according to claim 1 is characterized in that: The method for calculating the sparse coefficient matrix based on the sparse representation model is: in, is a sparse coefficient matrix, is the two-stream feature matrix, is the constraint parameter of the sparse matrix, Represents the norm operation function.
4. The video key frame extraction method based on dual-stream information and sparse representation according to claim 3 is characterized in that: The form of calculating the sparse coefficient matrix solution is: in, represents the permutation matrix, for dimensional identity matrix, The elements in are in [0,1).
5. The video key frame extraction method based on dual-stream information and sparse representation according to claim 1 is characterized in that: Based on the sparse coefficient matrix, a key frame index is obtained, including: extracting non-zero rows in the sparse coefficient matrix, wherein the non-zero rows in the sparse coefficient matrix are the key frame index.
6. The video key frame extraction method based on dual-stream information and sparse representation according to claim 1 is characterized in that: The method further includes evaluating the accuracy of the extracted key frames, based on the video key frame compression rate and F-measure as evaluation indicators, wherein the F-measure is used to measure the accuracy of the extracted key frames; The method for calculating the video key frame compression rate is: is the video key frame compression rate, is the number of key frames extracted, The total number of frames in the video.
