Video key frame extraction method and system based on shot segmentation and VGG neural network

Abstract

The invention relates to the technical field of video processing, in particular to a video key frame extraction method and system based on shot segmentation and a VGG neural network, and the method comprises the steps: firstly, extracting LBP-HF features and Canny features of continuous video frames, and obtaining shot boundary frames; then calculating the inter-frame similarity of boundary frames of adjacent shots, and dividing each shot into an active area or an inactive area by setting a threshold value; selecting a shot boundary frame as a coarse extraction key frame set for an inactive area; for the active area, sequentially inputting the video frames in the shot into a VGG19 convolutional neural network for feature vector description, extracting image depth features, calculating the similarity between the video frames, setting a threshold value to determine key frames, and adding the obtained key frames of the active area into a coarse extraction key frame set; and finally, removing key frames with high similarity in the roughly extracted key frame set to obtain a final video key frame set. According to the method, the key frame extraction efficiency and accuracy are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to a video key frame extraction method and system based on shot segmentation and VGG neural network. Background Art

[0002] At present, the existing methods for extracting key frames of videos mostly use the method of extracting the underlying image features to extract key frames. Commonly used methods include: shot segmentation-based methods, information entropy-based methods, clustering-based methods, motion analysis-based methods, and local feature-based methods. With the development of deep learning, deep convolutional networks can greatly improve the ability to extract image features and have advantages in extracting global features and contextual information of images. Therefore, deep learning methods can also be applied to the extraction of video key frames. For example, the SegNet neural network is used to learn and identify the key frames of video sequences, but this method cannot adaptively extract the number of video key frames, and the deep learning-based method requires a large number of labeled training sets, which requires a lot of human intervention and prior knowledge.

[0003] In summary, traditional key frame extraction algorithms cannot make full use of the semantic elements of the video (such as objects of interest, motion, events, etc.), ignore the time series and dynamic information between video frames, and do not combine with methods with better performance such as deep learning. Therefore, there are a lot of information missing or redundancy, and high time complexity. Summary of the invention

[0004] The present invention aims to solve the problems that traditional key frame extraction algorithms cannot make full use of the semantic elements of the video, there is a large amount of information missing or redundant phenomena, and the time complexity is high. A video key frame extraction method and system based on shot segmentation and VGG neural network are proposed. The shots are divided into active areas and inactive areas to extract video key frames, which effectively improves the efficiency of extracting key frames; the global deep semantic features of video frames are extracted by using the VGG19 convolutional neural network, which can better reflect the content of the original video.

[0005] To achieve the above purpose, the technical solution adopted is: A video key frame extraction method based on shot segmentation and VGG neural network includes the following steps: Firstly, LBP-HF features are extracted from continuous video frames, and the video data is divided into several shots in combination with the Canny edge operator to obtain shot boundary frames. Then the inter-frame similarity of the boundary frames of adjacent shots is calculated, and each shot is divided into an active area or an inactive area by setting a threshold; Then, for the inactive area, since the content of the video frame in the shot is single or does not change much, the shot boundary frame is selected as the coarse extraction key frame set; for the active area, the video frames in the shot are sequentially input into the VGG19 convolutional neural network for feature vector description, image depth features are extracted, the similarity between video frames is calculated, the threshold is set to determine the key frame, and the key frame of the active area is added to the coarse extraction key frame set; Finally, the key frames with large similarity in the coarsely extracted key frame set are removed to obtain the final video key frame set.

[0006] According to the video key frame extraction method based on shot segmentation and VGG neural network of the present invention, further, extracting LBP-HF features from continuous video frames to obtain proposed shot boundary frames includes: Each video frame is divided into several blocks; starting from the first frame, the LBP-HF features of each video frame are calculated, and the difference in LBP-HF features between the current frame and the previous frame in each block is calculated using the Euclidean distance, and the differences in all blocks are accumulated as the total difference between the two frames; a threshold is set according to the standard deviation of these differences; if the current total difference is greater than the threshold, the frame is considered to be a possible shear point, and the video frame is added to the proposed shot boundary frame set, otherwise it is not retained; finally, all the proposed shot boundary frames of the video are obtained.

[0007] According to the video key frame extraction method based on shot segmentation and VGG neural network of the present invention, further, obtaining the final shot boundary frame by using the Canny edge operator includes: In the proposed shot boundary frame set, starting from the first frame proposed shot boundary frame, the Canny edge difference between the current frame and the previous frame on each block is calculated using Euclidean distance, and the differences of all blocks are accumulated as the total difference between the two frames; a threshold is set according to the standard deviation of these differences; if the current total difference is greater than the threshold, the frame is used as a shot boundary frame, otherwise it is not retained; and the final shot boundary frame in the video is obtained.

[0008] According to the video key frame extraction method based on shot segmentation and VGG neural network of the present invention, further, the VGG19 convolutional neural network is stacked by convolutional layers, pooling layers and fully connected layers, which are used to extract global depth features of the image.

[0009] According to the video key frame extraction method based on shot segmentation and VGG neural network of the present invention, further, for the video frames of the active area, calculating the similarity between the video frames includes: Each input video frame image is converted into a global feature vector through the VGG19 convolutional neural network. The inter-frame similarity is calculated using the Euclidean distance. The smaller the distance calculated between the current and the next video frames, the higher the inter-frame similarity between the two frames. When the similarity is less than the set threshold, the next video frame is selected as the key frame.

[0010] According to the video key frame extraction method based on shot segmentation and VGG neural network of the present invention, further, by calculating the hash values ​​of the key frames in the coarsely extracted key frame set, the key frames with large similarity are removed.

[0011] According to the video key frame extraction method based on shot segmentation and VGG neural network of the present invention, further, the method also includes evaluating the extraction quality of the video key frame, and the evaluation indicators are video missampling probability VSE and fidelity FID; the video missampling probability reflects the cumulative difference between the original video frame and the key frame set, and the smaller the VSE value, the smaller the redundancy of the extracted key frame; the fidelity reflects the richness of the original video information contained in the extracted key frame sequence, and the larger the FID value, it means that the selected key frame contains more comprehensive video information.

[0012] Furthermore, the present invention also proposes a video key frame extraction system based on shot segmentation and VGG neural network, comprising a shot boundary segmentation module, a shot classification module, a coarse video key frame extraction module and a redundant frame removal module, wherein: The shot boundary segmentation module is used to extract LBP-HF features from continuous video frames and divide the video data into several shots in combination with the Canny edge operator to obtain shot boundary frames; The shot classification module is used to calculate the inter-frame similarity of adjacent shot boundary frames and divide each shot into an active area or an inactive area by setting a threshold; The coarse video key frame extraction module selects the shot boundary frames as the coarse extraction key frame set for the inactive area because the content of the video frame in the shot is single or does not change much. For the active area, the video frames in the shot are sequentially input into the VGG19 convolutional neural network for feature vector description, image depth features are extracted, the similarity between video frames is calculated, the threshold is set to determine the key frame, and the key frame of the active area is added to the coarse extraction key frame set. The redundant frame removal module is used to remove key frames with large similarity in the coarsely extracted key frame set to obtain the final video key frame set.

[0013] The beneficial effects achieved by adopting the above technical solution are: 1. The present invention uses a shot segmentation method based on LBP-HF and Canny edge difference to divide video shots and obtain shot boundary frames, and robustly segments the video at the place where the content changes greatly; then calculates the inter-frame similarity of adjacent shot boundary frames to divide each shot into an active area or an inactive area. For the video frames in the inactive area, there is no need to perform similarity calculation, and the shot boundary frames are directly used as key frames, which shortens the key frame extraction time. In addition, the use of the Canny edge operator can avoid the interference of burst lighting and camera motion conditions on the extraction of shot boundary frames.

[0014] 2. In order to solve the problems of insufficient video semantic information and poor expression ability of video frame features in video key frame extraction, the present invention uses VGG19 convolutional neural network to extract deep features of video frames. Compared with extracting the underlying information of the image, this method can better extract the global features of the image and make full use of the video semantic information.

[0015] 3. Two evaluation indicators are used to measure the quality of video key frame extraction, including video missampling probability VSE and fidelity FID. The key frame evaluation indicators are quantified to make the evaluation system clearer and more complete.

[0016] 4. When a video key frame video set is used for 3D reconstruction of static scenes, the overall reconstruction time is greatly reduced and the efficiency of reconstruction is robustly improved. Compared with the method of equally spaced frame extraction, the reconstruction of video key frames has more advantages in the recovery and expression of details. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings of the embodiments of the present invention, wherein the drawings are only used to illustrate some embodiments of the present invention, but not to limit all embodiments of the present invention thereto.

[0018] Figure 1 1 is a flow chart of a method for extracting video key frames based on shot segmentation and VGG neural network according to an embodiment of the present invention; Figure 2 is a flow chart of video shot segmentation based on LBP-HF and Canny edge difference according to an embodiment of the present invention; Figure 3 Schematic diagram of the LBP operator of an embodiment of the present invention; Figure 4 Schematic diagram of the structure of the VGG19 convolutional neural network of an embodiment of the present invention; Figure 5 is the segmentation result of the Bike2 video shot in an embodiment of the present invention; Figure 6 is a feature map of a video frame according to an embodiment of the present invention; Figure 7 is a key frame image extracted from a video in an embodiment of the present invention; Figure 8 These are the static 3D reconstruction results of six types of video data according to the embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings of specific embodiments of the present invention to clearly and completely describe the exemplary scheme of the embodiment of the present invention. Unless otherwise defined, the technical terms or scientific terms used in the present invention should be the common meanings understood by people with ordinary skills in the field.

[0020] like Figure 1 As shown, this embodiment discloses a video key frame extraction method based on shot segmentation and VGG neural network, comprising the following steps: Step S101: extract LBP-HF features from continuous video frames, and divide the video data into several shots in combination with the Canny edge operator to obtain shot boundary frames.

[0021] Video shot segmentation is the basic step of key frame extraction. Accurate video shot segmentation can improve the fidelity of video key frame extraction and effectively improve the efficiency of key frame extraction. This solution uses a video shot segmentation method based on LBP-HF and Canny edge difference. The specific process is as follows: Figure 2 shown.

[0022] The process of calculating LBP-HF features is as follows: Step ①, preprocess the sequence images; first convert the color image into HSV space and take the brightness component of the image; then reduce the size of each frame to 60×60, and then divide the video frame into 10×10 blocks; finally, use the adaptive Wiener filter to improve the recognition ability of LBP-HF.

[0023] Step ②: After preprocessing the video frame, calculate the LBP-HF features of each block in each frame. Compared with the traditional LBP operator, the LBP-HF operator used in this scheme not only inherits the significant advantages such as grayscale and rotation invariance, but also can adapt to texture features of different scales and extract more refined texture features of the image.

[0024] First, extract LBP texture features from video frames. Figure 3 As shown, if ( x , y ) is the center pixel coordinate, p is the number of adjacent points, R is the radius, and its neighboring pixel coordinates can be expressed as , then the LBP calculation formula for the circular area is:

[0025] Next, the LBP texture histogram obtained from the video frame is subjected to discrete Fourier transform, as follows:

[0026]

[0027] In the formula, U p ( n,r ) is represented as the LBP feature of the acquired image, Figure 3 The lower part is the LBP equivalent mode feature map of 8 sampling points in a 3×3 neighborhood. n Indicates the number of 1s in the binary pattern of the feature variable (corresponding to Figure 3 ), r Represents the rotation mode of the feature variable (corresponding to Figure 3 Columns in ); h ( U p ( n,r )) is the number of LBP features in the image, | H ( n,u )| is the amplitude corresponding to each LBP feature, then the calculation formula of LBP-HF feature can be expressed as.

[0028] Starting from the first frame, the LBP-HF features of each video frame are calculated, and the difference in LBP-HF features of each block between the current frame (starting from the second frame) and the previous frame is calculated using the Euclidean distance, and the differences of all blocks are accumulated as the total difference between the two frames; a threshold is set according to the standard deviation of these differences; if the current total difference is greater than the threshold, the frame is considered to be a possible shear point, and the video frame is added to the proposed shot boundary frame set, otherwise it is not retained; finally, all the proposed shot boundary frames of the video are obtained.

[0029] In order to further remove the unprocessed lighting and motion effects in the previous stage and improve the accuracy of detecting mutation frames, the proposed shot boundary frames are screened by the Canny edge operator, and the final shot boundary frames include: In the proposed shot boundary frame set, starting from the first proposed shot boundary frame, the Canny edge difference between the current frame (starting from the second frame) and the previous frame on each block is calculated using the Euclidean distance, and the differences of all blocks are accumulated as the total difference between the two frames; the threshold is set according to the standard deviation of these differences; if the current total difference is greater than the threshold, the frame is used as the shot boundary frame, otherwise it is not retained. All the remaining proposed shot boundary frames are judged in this way, and some video frames in the proposed shot boundary frames that are affected by motion and mistakenly classified as content mutations are eliminated. Finally, the mutation frames in the video are extracted as shot boundary frames through LBP-HF features and canny features, which provide reliable and accurate initial values ​​for the selection of video key frames.

[0030] Step S102: Calculate the inter-frame similarity of adjacent shot boundary frames, and divide each shot into an active area or an inactive area by setting a threshold. When the inter-frame similarity of adjacent shot boundary frames is greater than the threshold, the shot between the two shot boundary frames is an inactive area, otherwise it is an active area.

[0031] Step S103: For the inactive area, since the content of the video frame in the shot is single or does not change much, the shot boundary frame is selected as the coarse extraction key frame set; for the active area, the video frames in the shot are sequentially input into the VGG19 convolutional neural network for feature vector description, image depth features are extracted, the similarity between video frames is calculated, the threshold is set to determine the key frame, and the key frame of the active area is added to the coarse extraction key frame set.

[0032] VGG19 convolutional neural network is a feedforward neural network that can recognize large-scale image sets. Unlike other convolutional networks, it uses a very small convolution kernel (3×3) to build convolutional neural networks of various depths, reducing the amount of computation. It is composed of a stack of convolutional layers, pooling layers, and fully connected layers. Its structure is as follows: Figure 4 As shown in Figure 2, this scheme uses VGG-19 as a video frame image feature extractor to extract the global deep features of the image.

[0033] The VGG19 convolutional neural network is used as a feature extractor, integrating the characteristics of video frame extraction to perform video frame feature vector analysis. For video image data, if the input video frame sequence is F={ y 1, y 2,… y N},in N is the number of video frames, then the convolution calculation formula is:

[0034]

[0035] In the formula,y conv is the feature map output by the convolutional layer, Mat is the grayscale image matrix, W is the convolution kernel, b is the bias value. δ is an activation function, which can reduce the correlation between parameters and reduce the amount of calculation. x is the output value of the convolution layer. The expression formula of the pooling layer is:

[0036] Each input video frame image is converted into a 1×4096-dimensional global feature vector through the VGG19 convolutional neural network. The inter-frame similarity is calculated using the Euclidean distance. The smaller the distance calculated between the current and the next video frames, the higher the inter-frame similarity between the two frames. When the similarity is less than the set threshold, the next video frame is selected as the key frame. For example, for active areas, after extracting features from the video frames of each shot, the distance between the second, third, fourth, and first frames is calculated in turn with the first frame as a reference, and converted into inter-frame similarity, and a threshold is set. When the similarity is less than the threshold, it is determined that the frame is significantly different from the first frame, and the frame is selected as the key frame (for example, the eighth frame). Next, with the eighth frame as a reference, the similarity between the ninth, tenth, eleventh, and eighth frames is calculated in turn, and so on, and key frames are selected.

[0037] The expression of Euclidean distance is:

[0038] In the formula, F The feature vector representing the image, n is the dimension of the video frame feature vector, dist ( F a ,F b ) represents the inter-frame similarity of video frames.

[0039] Step S104: Remove redundant frames based on the perceptual hash algorithm. By calculating the hash values ​​of the coarse video key frames and removing the key frames with large similarity, the final video key frame set is obtained, further reducing the risk of increased redundancy caused by the close interval between some shot boundary frames and key frames.

[0040] Image hashing converts the original digital image into a string of characters or digital sequences through certain rules. By comparing the differences between the digital sequences, the similarity between two images can be simply and quickly compared. Image hashing algorithm has the advantages of robustness and security, so it is also called image summary or image fingerprint. It has broad application prospects in digital information processing fields such as image retrieval and information security. The image similarity calculation formula based on perceptual hashing algorithm is as follows:

[0041] In the formula, Sim is the image similarity, h is the Hamming distance of the fingerprint string.

[0042] Step S105: Evaluate the extraction quality of the video key frames.

[0043] The quality of video key frame extraction can be evaluated from two aspects: key frame redundancy and key frame fidelity. The evaluation indicators are video sampling error probability VSE (video sampling error, VSE) and fidelity FID (fidelity, FID). The video error sampling probability VSE reflects the cumulative difference between the original video frame and the key frame set. When the VSE value is smaller, it proves that the extracted key frame redundancy is smaller. The VSE calculation formula is:

[0044] Among them, the original video sequence is f(m) , the key frame sequence is y(p) , the edge of the video interval where the key frame is located is b 1 , b 2 … b n+1 .

[0045] Fidelity FID reflects the richness of the original video information contained in the extracted key frame sequence. The larger the FID value, the more comprehensive the video information contained in the selected key frame, and the better the representation effect of the original video content. The calculation formula of FID is:

[0046] In the formula, D ( F m , y ( p )) represents any frame F m With keyframe collection y ( p ), D maxexpress D ( F m , y ( p ))'s maximum possible value.

[0047] Corresponding to the above method, this embodiment proposes a video key frame extraction system based on shot segmentation and VGG neural network, including a shot boundary segmentation module, a shot classification module, a coarse video key frame extraction module and a redundant frame removal module, wherein: The shot boundary segmentation module is used to extract LBP-HF features from continuous video frames and divide the video data into several shots in combination with the Canny edge operator to obtain shot boundary frames.

[0048] The shot classification module is used to calculate the inter-frame similarity of adjacent shot boundary frames and divide each shot into an active area or an inactive area by setting a threshold.

[0049] The coarse video key frame extraction module selects the shot boundary frames as the coarse extraction key frame set for the inactive area because the content of the video frames in the shot is single or does not change much. For the active area, the video frames in the shot are sequentially input into the VGG19 convolutional neural network for feature vector description, image depth features are extracted, the similarity between video frames is calculated, the threshold is set to determine the key frame, and the key frame of the active area is added to the coarse extraction key frame set.

[0050] The redundant frame removal module is used to remove key frames with large similarity in the coarsely extracted key frame set to obtain the final video key frame set.

[0051] In order to verify the effectiveness of this scheme, further explanation is given below in combination with experimental data.

[0052] (1) Shot segmentation results The present invention conducts experiments on multiple drone video data in the UAV-123 dataset. The shot boundary segmentation method based on LBP-HF and Canny edge difference is used to process the experimental data; among them, the shot segmentation results of the Bike2 experimental data are as follows: Figure 5 And as shown in Table 1. Figure 5 In the figure, the continuous curve is the LBP-HF value of the continuous frames of the Bike2 video, the red dotted line is the boundary frame divided into shots, and Table 1 shows the obtained video shot segmentation results.

[0053]

[0054] (2) VGG19 convolutional neural network feature vector generation results Select experimental data, input the video frames in each shot in the data into the VGG19 convolutional neural network and perform feature extraction. Figure 6 As shown in (a), the Bike2 dataset is selected and the video images in the dataset are input into the VGG19 convolutional neural network, and 36 feature result maps are calculated. The first five feature maps are shown in Figure 6(b) to Figure 6(f).

[0055] The entire data set is processed by the VGG19 convolutional neural network feature method, and finally a 4096-dimensional feature vector of the video frame image can be obtained. The VGG19 convolutional neural network converts the input video frame into a one-dimensional feature vector and improves the efficiency of the inter-frame similarity calculation. The global deep features calculated by the deep learning convolution layer can not only output the visual feature map of any layer, but also will not lose the global information of the image. Therefore, the image deep global features extracted by the VGG19 convolutional neural network can make better use of video semantic information than the underlying features of the image, greatly improving the quality of the video key frames.

[0056] (3) Video key frame extraction results Based on the shot segmentation and VGG neural network feature vector generation, the Bike2, Bike3, and Truck1 datasets are processed and key frames are extracted, such as Figure 7 As shown. Figure 7 Taking (a) as an example, the horizontal axis represents the sequence of inter-frame distances (Distance) between every two video frames, and the vertical axis is the inter-frame similarity (Similarity) calculated based on the distance. When the inter-frame similarity between the shot boundary frames is greater than or equal to the threshold, the shot boundary frames are selected as the proposed key frame set; otherwise, the inter-frame similarity between the video sequence frames in the shot and the video boundary frames is calculated one by one, and the video frames that are less than or equal to the threshold are selected to be added to the key frame set. The blue numbers in the figure represent the final extracted video key frame set.

[0057] In order to further verify the effectiveness of the method of the present invention, two methods, a key frame extraction method based on shot boundaries and a key frame extraction method based on optical flow and K-means clustering, were used for comparison with the method of the present invention, which were respectively recorded as method one and method two. Table 2 shows the number and time of video key frames extracted by the three methods. When extracting video key frames of similar numbers, the extraction efficiency of the method of the present invention is better than that of methods one and two, and can effectively reduce the extraction time, as shown in the Bike2 and Bike3 datasets; when targeting video data with fast target motion and fast scene changes, the extraction efficiency is slightly lower than that of methods one and two, as shown in the Truck1 dataset.

[0058]

[0059] Table 3 shows the average VSE and FID values ​​obtained by testing the three methods on multiple videos. The average VSE value and the average FID value of the method of the present invention are basically the same as those of the first two methods, and are partially better than the first two methods. The performance varies according to different data sets, indicating that the missed detection rate and fidelity rate of the method of the present invention basically meet the requirements.

[0060]

[0061] (4) Static scene 3D reconstruction results The video data is divided into 1031 frames according to the frame rate of the video, and the key frames of the video sequence are extracted using a method based on shot segmentation and VGG neural network; according to the set threshold, the number of extracted video key frames is divided into 35 frames and 75 frames. Secondly, the same number of video frames is extracted from the video using the equal interval method. Finally, the static scene 3D reconstruction experiment is carried out on the equal interval video set and the video key.

[0062] It can be seen that when the number of video frames is 35, the reconstruction of the video key frames is more detailed in terms of local details, such as Figure 8 (c) The reconstruction result of the video key frame can show the complete white car, while the reconstruction result of the equal interval method completely loses the information of this object. In addition, the reconstruction result of the video key frame can also show the complete boundary of the blue area, while the reconstruction result of the equal interval method has some missing.

[0063] Unless otherwise specifically stated, the components, steps, numerical expressions and values ​​set forth in these embodiments do not limit the scope of the present invention.

[0064] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0065] The units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.

[0066] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present invention is not limited to any specific form of combination of hardware and software.

[0067] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A video key frame extraction method based on shot segmentation and VGG neural network, characterized in that: The following steps are involved: Firstly, LBP-HF features are extracted from continuous video frames, and the video data is divided into several shots in combination with the Canny edge operator to obtain shot boundary frames. Then the inter-frame similarity of the boundary frames of adjacent shots is calculated, and each shot is divided into an active area or an inactive area by setting a threshold; Then, for the inactive area, since the content of the video frame in the shot is single or does not change much, the shot boundary frame is selected as the coarse extraction key frame set; for the active area, the video frames in the shot are sequentially input into the VGG19 convolutional neural network for feature vector description, image depth features are extracted, the similarity between video frames is calculated, the threshold is set to determine the key frame, and the key frame of the active area is added to the coarse extraction key frame set; Finally, the key frames with large similarity in the coarsely extracted key frame set are removed to obtain the final video key frame set.

2. The video key frame extraction method based on shot segmentation and VGG neural network according to claim 1 is characterized in that: Extracting LBP-HF features from continuous video frames to obtain the proposed shot boundary frames includes: Each video frame is divided into several blocks; starting from the first frame, the LBP-HF features of each video frame are calculated, and the difference in LBP-HF features between the current frame and the previous frame in each block is calculated using the Euclidean distance, and the differences in all blocks are accumulated as the total difference between the two frames; a threshold is set according to the standard deviation of these differences; if the current total difference is greater than the threshold, the frame is considered to be a possible shear point, and the video frame is added to the proposed shot boundary frame set, otherwise it is not retained; finally, all the proposed shot boundary frames of the video are obtained.

3. The video key frame extraction method based on shot segmentation and VGG neural network according to claim 2 is characterized in that: The final shot boundary frame obtained by the Canny edge operator includes: In the proposed shot boundary frame set, starting from the first frame proposed shot boundary frame, the Canny edge difference between the current frame and the previous frame on each block is calculated using Euclidean distance, and the differences of all blocks are accumulated as the total difference between the two frames; a threshold is set according to the standard deviation of these differences; if the current total difference is greater than the threshold, the frame is used as a shot boundary frame, otherwise it is not retained; and the final shot boundary frame in the video is obtained.

4. The video key frame extraction method based on shot segmentation and VGG neural network according to claim 1 is characterized in that: The VGG19 convolutional neural network is composed of a stack of convolutional layers, pooling layers, and fully connected layers, and is used to extract global deep features of images.

5. The video key frame extraction method based on shot segmentation and VGG neural network according to claim 4 is characterized in that: For video frames in the active area, calculating the similarity between video frames includes: Each input video frame image is converted into a global feature vector through the VGG19 convolutional neural network. The inter-frame similarity is calculated using the Euclidean distance. The smaller the distance calculated between the current and the next video frames, the higher the inter-frame similarity between the two frames. When the similarity is less than the set threshold, the next video frame is selected as the key frame.

6. The video key frame extraction method based on shot segmentation and VGG neural network according to claim 1 is characterized in that: By calculating the hash values ​​of the key frames in the roughly extracted key frame set, the key frames with large similarity are removed.

7. The video key frame extraction method based on shot segmentation and VGG neural network according to claim 1 is characterized in that: The method also includes evaluating the extraction quality of video key frames, and the evaluation indicators are video error sampling probability VSE and fidelity FID; the video error sampling probability reflects the cumulative difference between the original video frame and the key frame set. When the VSE value is smaller, it proves that the extracted key frame redundancy is smaller; the fidelity reflects the richness of the original video information contained in the extracted key frame sequence. When the FID value is larger, it means that the selected key frame contains more comprehensive video information.

8. A video key frame extraction system based on shot segmentation and VGG neural network, characterized in that: It includes a shot boundary segmentation module, a shot classification module, a coarse video key frame extraction module and a redundant frame removal module, wherein: The shot boundary segmentation module is used to extract LBP-HF features from continuous video frames and divide the video data into several shots in combination with the Canny edge operator to obtain shot boundary frames; The shot classification module is used to calculate the inter-frame similarity of adjacent shot boundary frames and divide each shot into an active area or an inactive area by setting a threshold; The coarse video key frame extraction module selects the shot boundary frames as the coarse extraction key frame set for the inactive area because the content of the video frame in the shot is single or does not change much. For the active area, the video frames in the shot are sequentially input into the VGG19 convolutional neural network for feature vector description, image depth features are extracted, the similarity between video frames is calculated, the threshold is set to determine the key frame, and the key frame of the active area is added to the coarse extraction key frame set. The redundant frame removal module is used to remove key frames with large similarity in the coarsely extracted key frame set to obtain the final video key frame set.

Images