Video key frame extraction method and system based on VGG16 model
Through the video keyframe extraction method of VGG16 model, the limitations of fixed time interval sampling and simple motion detection in traditional methods are solved, and the deep understanding and efficient extraction of video content are achieved, which significantly improves the accuracy and computing efficiency of keyframes. .
Patent Information
- Application Number
- CN202411320370.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2025-01-10
AI Technical Summary
Traditional video keyframe extraction methods rely on fixed time interval sampling and simple motion detection, and cannot accurately capture important events and scene changes in the video, and lack semantic understanding, resulting in the extracted snapshots that cannot fully represent the main content of the video.
Using the video keyframe extraction method based on the VGG16 model, through image preprocessing, feature extraction, similarity calculation and fine-tuning convolutional neural network model, frames with significant content changes are selected as keyframes, and clustering algorithms and multi-threaded processing are introduced. Improve computing efficiency.
It significantly improves the accuracy and efficiency of keyframe extraction, can automatically analyze video content, identify key events and scene changes, and extract keyframes representing the main content of the video.
Smart Images
Figure CN119295990A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of image processing and video analysis, and in particular to a method and system for extracting video key frames based on a VGG16 model. Background Art
[0002] With the rapid development of digital technology, the importance of video data in daily life continues to increase. With its rich content and dynamic characteristics, video has become one of the mainstream forms of information transmission and entertainment. However, the management and utilization of video content face multiple challenges, such as shooting equipment limitations, environmental changes, and post-editing requirements. Traditional video key frame extraction methods usually use fixed time interval sampling or simple motion detection, but have obvious limitations when processing complex videos.
[0003] Problems with existing technologies:
[0004] 1. Limitations of fixed time interval sampling: Traditional key frame extraction methods usually rely on sampling at fixed time intervals. This method ignores the actual changes in the video content, resulting in the key frames possibly failing to accurately capture important events and scene changes in the video.
[0005] 2. Shortcomings of simple motion detection: Algorithms based on simple motion detection perform poorly when dealing with static scenes or subtle changes, and cannot effectively identify and extract key information in the video.
[0006] 3. Lack of semantic understanding: Existing technologies often lack a deep understanding of the semantics of video content and are unable to identify and extract key frames from the content level, resulting in the extracted snapshots possibly not fully representing the main content of the video. Summary of the invention
[0007] In view of the above problems, the present invention is proposed to provide a method and system for extracting video key frames based on the VGG16 model, which overcomes the above problems or at least partially solves the above problems.
[0008] According to one aspect of the present invention, a method for extracting key frames from a video based on a VGG16 model is provided, the key frame extraction method comprising:
[0009] Collect video files, perform image preprocessing, and obtain preprocessed images;
[0010] Extract features from the preprocessed image using a pretrained convolutional neural network model to obtain a feature vector;
[0011] Processing the feature vector;
[0012] For each pair of consecutive image frames, the similarity between the feature vectors of the image frames is calculated;
[0013] Select key frames based on similarity;
[0014] Fine-tune convolutional neural network models for optimization and scaling.
[0015] Optionally, the collecting of video files and performing image preprocessing to obtain preprocessed images specifically includes:
[0016] Collect video files, perform video decoding, and convert video streams into continuous image frames;
[0017] The image frames are resized and normalized to ensure that the image sizes input to the convolutional neural network are uniform.
[0018] Optionally, extracting features from the preprocessed image using a pretrained convolutional neural network model to obtain a feature vector specifically includes:
[0019] Use the pre-trained convolutional neural network model and the VGG model to extract deep features of the image;
[0020] Remove the classification layer of the model and keep only the feature extraction layer, which is used to extract feature vectors from the input image.
[0021] Optionally, the processing of the feature vector specifically includes:
[0022] Normalize the extracted feature vector;
[0023] Perform dimensionality reduction on feature vectors.
[0024] Optionally, for each pair of consecutive image frames, calculating the similarity between the feature vectors of the image frames specifically includes:
[0025] For each pair of consecutive image frames, the similarity between the feature vectors is calculated;
[0026] The similarity measurement methods of cosine similarity and Euclidean distance are used to determine whether two frames belong to the same scene or contain similar content according to the similarity threshold.
[0027] Optionally, selecting a key frame according to the similarity specifically includes:
[0028] According to the similarity calculation results, frames with significant content changes are selected as key frames;
[0029] If the similarity between consecutive frames is lower than a preset threshold, they are considered to represent multiple scenes or events;
[0030] A clustering algorithm is introduced to cluster the feature vectors, and the frame corresponding to each cluster center is selected as the key frame.
[0031] Optionally, the fine-tuning of the convolutional neural network model for optimization and expansion specifically includes:
[0032] Fine-tune the convolutional neural network model to adapt to a variety of video content and application scenarios;
[0033] Introduce multi-threading or asynchronous processing technology to improve the computing efficiency of the system when processing large amounts of video data.
[0034] The present invention also provides a video key frame extraction system based on the VGG16 model, applying the above-mentioned video key frame extraction method based on the VGG16 model, the key frame extraction system comprises:
[0035] An image preprocessing module is used to collect video files and perform image preprocessing to obtain preprocessed images;
[0036] A feature extraction module, used to extract features from the preprocessed image using a pretrained convolutional neural network model to obtain a feature vector;
[0037] A feature vector processing module, used for processing the feature vector;
[0038] A similarity calculation module, used for calculating the similarity between the feature vectors of the image frames for each pair of consecutive image frames;
[0039] The key frame filtering module is used to select key frames based on similarity;
[0040] The optimization and expansion module is used to fine-tune the convolutional neural network model for optimization and expansion.
[0041] The present invention provides a method and system for extracting key frames from a video based on a VGG16 model. The method comprises: collecting video files and performing image preprocessing to obtain preprocessed images; extracting features from the preprocessed images using a pretrained convolutional neural network model to obtain feature vectors; processing the feature vectors; calculating the similarity between the feature vectors of each pair of continuous image frames; selecting key frames based on the similarity; and fine-tuning the convolutional neural network model for optimization and expansion. The accuracy of key frame extraction is improved, and the computational efficiency of key frame extraction is improved.
[0042] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0044] Figure 1 A flowchart of a method for extracting key frames from a video based on a VGG16 model provided by an embodiment of the present invention;
[0045] Figure 2 A block diagram of a video key frame extraction system based on a VGG16 model provided in an embodiment of the present invention;
[0046] Figure 3 A schematic diagram of the VGG16 model provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0047] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0048] The terms "comprises" and "having" and any variations thereof in the description embodiments, claims and drawings of the present invention are intended to cover non-exclusive inclusions, for example, including a series of steps or units.
[0049] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.
[0050] The present invention aims to solve the limitations of the prior art in video key frame extraction by adopting an automatic key frame extraction system based on convolutional neural network (CNN).
[0051] like Figure 1 As shown, a method and system for extracting key frames from videos based on a VGG16 model are provided, wherein the key frame extraction method comprises: acquiring video files, and performing image preprocessing to obtain preprocessed images; performing feature extraction on the preprocessed images using a pretrained convolutional neural network model to obtain feature vectors; processing the feature vectors; for each pair of consecutive image frames, calculating the similarity between the feature vectors of the image frames; selecting key frames based on the similarity; and fine-tuning the convolutional neural network model for optimization and expansion.
[0052] like Figure 2As shown, a video key frame extraction system based on the VGG16 model includes:
[0053] Image preprocessing module: Input video files, first decode the video, and convert the video stream into continuous image frames. Resize and normalize the image frames to ensure that the image size input to the convolutional neural network is uniform, which is convenient for subsequent processing.
[0054] Feature extraction module: Using a pre-trained convolutional neural network model, the present invention uses the VGG model as a feature extractor. The model has been trained on a large-scale image dataset and can extract deep features of the image. The classification layer of the model is removed, and only the feature extraction layer is retained to extract feature vectors from the input image.
[0055] Feature vector processing module: normalize the extracted feature vectors to reduce the scale differences between different feature vectors. Perform dimension reduction on the feature vectors to reduce the computational complexity and improve the processing speed.
[0056] Similarity calculation module: For each pair of consecutive image frames, calculate the similarity between their feature vectors. Use cosine similarity, Euclidean distance or other similarity measurement methods. Based on the similarity threshold, determine whether the two frames belong to the same scene or contain similar content.
[0057] Key frame screening module: Based on the similarity calculation results, frames with significant content changes are selected as key frames. If the similarity between consecutive frames is lower than the preset threshold, they are considered to represent different scenes or events. A clustering algorithm is introduced to cluster the feature vectors, and the frame corresponding to each cluster center is selected as the key frame.
[0058] Optimization and expansion module: In order to improve the performance and adaptability of the system, the convolutional neural network model is fine-tuned to better adapt to specific video content and application scenarios. Consider introducing multi-threading or asynchronous processing technology to improve the system's computing efficiency when processing large amounts of video data.
[0059] Model Introduction:
[0060] VGG16 (Visual Geometry Group 16) is a deep convolutional neural network. Figure 3 The VGG16 model has achieved excellent results in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) and has been widely used in image recognition, classification, and feature learning tasks due to its powerful feature extraction capabilities.
[0061] Model features:
[0062] 1. Deep structure: VGG16 consists of 13 convolutional layers and fully connected layers, including 16 weight layers, hence the name VGG16. This deep structure enables the model to learn more complex feature representations.
[0063] 2. Small convolution kernel: VGG16 uses 3x3 convolution kernels in all convolution layers, which gives the model an advantage in terms of the number of parameters and computational efficiency.
[0064] 3. Repeating modules: In the design of VGG16, the convolutional layers are organized into repeated modules, each module consists of two convolutional layers. This design simplifies the structure of the model and improves training efficiency.
[0065] 4. Pre-training and fine-tuning: VGG16 is usually pre-trained on large-scale image datasets (such as ImageNet), and then can be fine-tuned on specific tasks to adapt to different image recognition needs.
[0066] Model structure
[0067] The structure of the VGG16 model includes:
[0068] Convolutional layers: The first few layers of the model consist of multiple convolutional layers, each followed by a maximum pooling layer (MaxPooling) to extract local features of the image.
[0069] Fully connected layers: After the convolutional layers, the model contains several fully connected layers to map the extracted features to the final class labels.
[0070] Classification layer: The last fully connected layer is a softmax layer that outputs category probabilities.
[0071] Pre-training tasks:
[0072] VGG16 is usually pre-trained on the ImageNet dataset, which contains 1.2 million images in 1000 categories. The pre-training tasks include:
[0073] Image classification: The model is trained to recognize the categories of objects in images.
[0074] Fine-tuning and application:
[0075] VGG16 model fine-tuning
[0076] 1. Optimize activation function
[0077] 1.1 Introduce non-monotonic activation function, which is mathematically defined as
[0078] f(x)=xtanh(ln(1+e x ))
[0079] 1.2 Reference Image Block Collection
[0080] VGG16 model application
[0081] After pre-training, the VGG16 model is fine-tuned to adapt to different image recognition tasks by removing the last fully connected layer and adding task-specific output layers. VGG16 has achieved excellent performance on multiple image recognition tasks, including: object detection, scene recognition, and image segmentation.
[0082] Beneficial effects: Deep learning technology is used to automatically analyze video content, identify key events and scene changes, and extract key frames that best represent the main content and structure of the video. Compared with traditional methods, this system significantly improves the accuracy and efficiency of key frame extraction, providing a new technical means for the management and utilization of video content.
[0083] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for extracting key frames from a video based on a VGG16 model, characterized in that: The key frame extraction method comprises: Collect video files, perform image preprocessing, and obtain preprocessed images; Extract features from the preprocessed image using a pretrained convolutional neural network model to obtain a feature vector; Processing the feature vector; For each pair of consecutive image frames, the similarity between the feature vectors of the image frames is calculated; Select key frames based on similarity; Fine-tune convolutional neural network models for optimization and scaling.
2. The method for extracting key frames from a video based on a VGG16 model according to claim 1, characterized in that: The collecting of video files and performing image preprocessing to obtain preprocessed images specifically includes: Collect video files, perform video decoding, and convert video streams into continuous image frames; The image frames are resized and normalized to ensure that the image sizes input to the convolutional neural network are uniform.
3. The method for extracting key frames from a video based on a VGG16 model according to claim 1, characterized in that: The step of extracting features from the preprocessed image using a pretrained convolutional neural network model to obtain a feature vector specifically includes: Use the pre-trained convolutional neural network model and the VGG model to extract deep features of the image; Remove the classification layer of the model and keep only the feature extraction layer, which is used to extract feature vectors from the input image.
4. The method for extracting key frames from a video based on a VGG16 model according to claim 1, characterized in that: The processing of the feature vector specifically includes: Normalize the extracted feature vector; Perform dimensionality reduction on feature vectors.
5. The method for extracting key frames from a video based on a VGG16 model according to claim 1, characterized in that: The step of calculating the similarity between the feature vectors of each pair of consecutive image frames specifically includes: For each pair of consecutive image frames, the similarity between the feature vectors is calculated; The similarity measurement methods of cosine similarity and Euclidean distance are used to determine whether two frames belong to the same scene or contain similar content according to the similarity threshold.
6. The method for extracting key frames from a video based on a VGG16 model according to claim 1, characterized in that: The step of selecting a key frame according to the similarity specifically includes: According to the similarity calculation results, frames with significant content changes are selected as key frames; If the similarity between consecutive frames is lower than a preset threshold, they are considered to represent multiple scenes or events; A clustering algorithm is introduced to cluster the feature vectors, and the frame corresponding to each cluster center is selected as the key frame.
7. The method for extracting key frames from a video based on a VGG16 model according to claim 1, characterized in that: The fine-tuning of the convolutional neural network model for optimization and expansion specifically includes: Fine-tune the convolutional neural network model to adapt to a variety of video content and application scenarios; Introduce multi-threading or asynchronous processing technology to improve the computing efficiency of the system when processing large amounts of video data.
8. A video key frame extraction system based on the VGG16 model, using the video key frame extraction method based on the VGG16 model described in any one of claims 1 to 7, characterized in that: The key frame extraction system comprises: An image preprocessing module is used to collect video files and perform image preprocessing to obtain preprocessed images; A feature extraction module, used to extract features from the preprocessed image using a pretrained convolutional neural network model to obtain a feature vector; A feature vector processing module, used for processing the feature vector; A similarity calculation module, used for calculating the similarity between the feature vectors of the image frames for each pair of consecutive image frames; The key frame filtering module is used to select key frames based on similarity; The optimization and expansion module is used to fine-tune the convolutional neural network model for optimization and expansion.