Key frame extraction method, device and storage medium
By combining local features and deep feature extraction methods, the similarity of the fused frame images is calculated and weighted to extract keyframes in the video, which solves the problem of poor keyframe extraction effect in the prior art, and achieves higher accuracy and robustness.
Patent Information
- Application Number
- CN202411314338.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-20
- Publication Date
- 2025-02-14
- Estimated Expiration
- 2044-09-20
AI Technical Summary
The existing keyframe extraction methods have limited effects when processing complex video content, and relying on hand-designed features leads to inaccurate and robust enough for the extracted keyframes.
Using a combination of local feature extraction and deep feature extraction, the local feature cosine similarity and depth feature cosine similarity between frame images and adjacent frame images and weighted fusion to obtain the total similarity to extract the key frames in the video. .
It improves the accuracy and robustness of keyframe extraction, can capture the characteristics of video frames more effectively, and enhances the processing ability of complex dynamic background videos.
Smart Images
Figure CN118840699B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a key frame extraction method, a device and a storage medium, and belongs to the technical field of image processing. Background Art
[0002] With the rapid growth of digital video content, how to effectively manage and retrieve digital video content has become a key research topic in the field of computer vision. Key frame extraction technology extracts representative frames from videos, reduces the amount of data and retains the core content of the video, which is of great significance to applications such as video surveillance and content retrieval.
[0003] Existing key frame extraction methods mainly include methods based on shot boundaries, content analysis, motion analysis and clustering, but these methods have limited effects when processing complex video content due to their reliance on manually designed features. Summary of the invention
[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a key frame extraction method, device and storage medium, which can improve the accuracy and robustness of key frame extraction. To achieve the above purpose, the present invention is implemented by adopting the following technical solutions:
[0005] In a first aspect, the present invention provides a key frame extraction method, comprising:
[0006] Obtain the frame image of the video to be processed;
[0007] Perform local feature extraction and depth feature extraction on the acquired frame images frame by frame to obtain local features and depth features of each frame image;
[0008] Calculate the local feature cosine similarity and deep feature cosine similarity between each frame image and its adjacent frame image, and weightedly fuse the calculated local feature cosine similarity and deep feature cosine similarity to obtain the total similarity between each frame image and its adjacent frame image;
[0009] In response to the total similarity between a certain frame image and its adjacent frame images being a local minimum of a sequence consisting of the total similarities of all frame images in the video to be processed and their adjacent frame images, and the total similarity is less than a preset adaptive threshold, this frame is extracted as a key frame of the video to be processed.
[0010] In combination with the first aspect, optionally, extracting local features and deep features from the acquired frame images frame by frame to obtain local features and deep features of each frame image includes:
[0011] Scale-invariant feature transformation and bag-of-words model are used to extract local features of the acquired frame images frame by frame to obtain local features of each frame image;
[0012] The encoder of the pre-trained deep learning model is used to extract deep features of the acquired frame images frame by frame to obtain the deep features of each frame image.
[0013] In combination with the first aspect, optionally, the extracting local features of the acquired frame images frame by frame using the scale-invariant feature transformation and the bag-of-words model to obtain local features of each frame image includes:
[0014] The scale-invariant feature transformation algorithm is used to detect the key points of the acquired frame images frame by frame, and the key points and descriptors of the key points of each frame image are obtained;
[0015] The key points of each frame image are clustered using the K-means algorithm to obtain an index map of the key points of each frame image;
[0016] Retrieving the cluster center closest to the descriptor of the key point of each frame image in the index map, and obtaining the visual vocabulary corresponding to the descriptor of the key point of each frame image;
[0017] Based on the visual vocabulary corresponding to the descriptors of all key points in each frame of image, a bag-of-words vector is formed for each frame of image;
[0018] Based on the frequencies of various visual words in the bag-of-words vector of each frame of image, a feature vector of each frame of image is obtained as the local feature of each frame of image.
[0019] In combination with the first aspect, optionally, the deep learning model includes an encoder and a decoder;
[0020] The encoder includes D convolution blocks, each convolution block includes two convolution submodules, the first convolution submodule includes a dilated convolution filter, a ReLU activation function layer and a batch normalization layer, and the second convolution submodule includes a dilated convolution filter, a ReLU activation function layer, an average pooling layer and a batch normalization layer;
[0021] The decoder includes N deconvolution blocks, each deconvolution block includes two deconvolution sub-modules, the first deconvolution sub-module includes a bilinear interpolation layer, a convolution filter, a ReLU activation function layer and a batch normalization layer, and the second deconvolution sub-module includes a transposed convolution layer, a ReLU activation function layer and a batch normalization layer.
[0022] In combination with the first aspect, optionally, the deep learning model is trained by the following method:
[0023] Get the images in the training set;
[0024] Adding the standard deviation to the images in the training set is Gaussian noise is used to obtain a noisy image;
[0025] Input the noisy image into a pre-built deep learning model, use the encoder of the pre-built deep learning model to learn the deep features of the input image, and obtain the deep features of the training image;
[0026] The deep features of the training image are input into the decoder of the pre-built deep learning model to reconstruct the image and obtain a reconstructed image containing noise;
[0027] Calculate the loss function of the images in the training set without noise and the reconstructed images with noise;
[0028] Determine whether the loss function meets the preset loss value; if not, adjust the weights and bias of the encoder and decoder in the pre-built deep learning model, output the reconstructed image containing noise, and calculate the loss function; if satisfied, output the optimal weights and bias of the encoder and decoder in the pre-built deep learning model to obtain a trained deep learning model.
[0029] In combination with the first aspect, optionally, the loss function is a mean square error loss function, which is calculated by the following formula:
[0030] ,
[0031] in, L MSE is the mean square error loss function, M is the number of images in the training set, X i For the first i images in the training set, Y i The first i images in the training set.
[0032] In combination with the first aspect, optionally, the calculated local feature cosine similarity and deep feature cosine similarity are weightedly fused to obtain the total similarity between each frame image and its adjacent frame image, which is calculated by the following formula:
[0033] ,
[0034] in, S Total is the total similarity between each frame image and the adjacent frame images, α is the weight factor of the local features of each frame image and the adjacent frame images, S SIFT is the cosine similarity of the local features between each frame image and the adjacent frame images, βis the weight factor of the depth feature of each frame image and the adjacent frame image, S DEEP is the cosine similarity of the deep features between each frame image and the adjacent frame images;
[0035] Among them, the cosine similarity is calculated by the following formula:
[0036] ,
[0037] in, S is the cosine similarity, v i For the i The feature vector of the frame image, v i+1 For the i +1 Feature vector of the image frame.
[0038] In combination with the first aspect, optionally, the preset adaptive threshold is calculated by the following formula:
[0039] ,
[0040] in, is the adaptive threshold, n is the total number of frame images in the video to be processed, f i For the i Frame image, f i+1 For the i +1 frame image, S Total ( f i , f i+1 ) is the i The frame image and the adjacent i +1 The total similarity of the image.
[0041] In a second aspect, the present invention provides a key frame extraction device, comprising:
[0042] Acquisition module: used to acquire frame images of the video to be processed;
[0043] Feature extraction module: used to extract local features and deep features of the acquired frame images frame by frame to obtain local features and deep features of each frame image;
[0044] Calculation module: used to calculate the local feature cosine similarity and deep feature cosine similarity between each frame image and its adjacent frame image, and weightedly fuse the calculated local feature cosine similarity and deep feature cosine similarity to obtain the total similarity between each frame image and its adjacent frame image;
[0045] Extraction module: used for extracting a frame as a key frame of the video to be processed in response to the total similarity between a certain frame image and its adjacent frame images being a local minimum of a sequence consisting of the total similarities between all frame images in the video to be processed and their adjacent frame images, and the total similarity is less than a preset adaptive threshold.
[0046] In a third aspect, the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of the key frame extraction method described in the first aspect.
[0047] Compared with the prior art, the key frame extraction method, device and storage medium provided by the embodiment of the present invention have the following beneficial effects:
[0048] The present invention extracts local features and deep features from the acquired frame images frame by frame to obtain local features and deep features of each frame image. The present invention can fully capture the features of each frame image.
[0049] The present invention calculates the local feature cosine similarity and deep feature cosine similarity between each frame image and its adjacent frame images, and the present invention can quantify the similarity between each frame image and its adjacent frame images at the feature level;
[0050] The present invention weightedly fuses the calculated local feature cosine similarity and deep feature cosine similarity to obtain the total similarity between each frame image and its adjacent frame image; the total similarity provided by the present invention can more accurately reflect the actual situation, not only improves the efficiency of feature extraction, but also enhances the accuracy of feature extraction; the present invention integrates local features and deep features, can fully capture the features of each frame image, and can improve the efficiency and accuracy of feature extraction when processing video frame images with complex dynamic backgrounds; based on feature fusion, the present invention effectively improves the accuracy and robustness of key frame extraction by combining scale-invariant feature transformation and convolutional denoising autoencoder technology;
[0051] In response to the total similarity between a certain frame image and its adjacent frame images being a local minimum value of a sequence consisting of the total similarities between all frame images in the video to be processed and their adjacent frame images, and the total similarity being less than a preset adaptive threshold, the present invention extracts this frame as a key frame of the video to be processed; the present invention can ensure that the extracted key frame has high uniqueness and representativeness in the video to be processed;
[0052] Compared with the existing key frame extraction algorithm, the present invention can adaptively process the frame images of the video and accurately extract the key frames without human intervention; by considering the fusion of local features and deep features, not only the accuracy of key frame extraction is improved, but also the robustness of the algorithm is enhanced, and it can cope with various complex video contents and dynamic backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 is a flow chart of a key frame extraction method provided in Embodiment 1 of the present invention;
[0054] Figure 2 It is a structural schematic diagram of an encoder of a pre-trained deep learning model in a key frame extraction method provided in Embodiment 1 of the present invention;
[0055] Figure 3 It is a structural diagram of a decoder of a pre-trained deep learning model in a key frame extraction method provided in Example 1 of the present invention. DETAILED DESCRIPTION
[0056] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.
[0057] Embodiment 1:
[0058] like Figure 1 As shown, an embodiment of the present invention provides a key frame extraction method, including:
[0059] Obtain the frame image of the video to be processed;
[0060] Perform local feature extraction and depth feature extraction on the acquired frame images frame by frame to obtain local features and depth features of each frame image;
[0061] Calculate the local feature cosine similarity and deep feature cosine similarity between each frame image and its adjacent frame image, and weightedly fuse the calculated local feature cosine similarity and deep feature cosine similarity to obtain the total similarity between each frame image and its adjacent frame image;
[0062] In response to the total similarity between a certain frame image and its adjacent frame images being a local minimum of a sequence consisting of the total similarities of all frame images in the video to be processed and their adjacent frame images, and the total similarity is less than a preset adaptive threshold, this frame is extracted as a key frame of the video to be processed.
[0063] The specific steps include:
[0064] Step 1: Get the frame image of the video to be processed.
[0065] Step 1.1: Get the video to be processed.
[0066] Step 1.2: The video to be processed is read according to the video frame rate, saved as a frame image, and the frame image of the video to be processed is obtained.
[0067] Initialize video sharing N Frame, each frame image is recorded as f i ,in, .
[0068] Step 2: Perform local feature extraction and depth feature extraction on the acquired frame images frame by frame to obtain the local features and depth features of each frame image.
[0069] Step 2.1: Use scale-invariant feature transformation and bag-of-words model to extract local features of each frame image, and obtain the local features of each frame image.
[0070] Step 2.1.1: Use the scale-invariant feature transformation algorithm to detect the key points of each frame image, and obtain the key points and key point descriptors of each frame image. F i .
[0071] Step 2.1.2: Use the K-means algorithm to cluster the key points of each frame image to obtain an index map of the key points of each frame image.
[0072] In this embodiment, the number of cluster centers .
[0073] Step 2.1.3: Retrieve the cluster center closest to the descriptor of the key point of each frame image in the index map, and obtain the visual vocabulary corresponding to the descriptor of the key point of each frame image.
[0074] Step 2.1.4: Based on the visual vocabulary corresponding to the descriptors of all key points in each frame of image, form a bag-of-words vector for each frame of image.
[0075] Step 2.1.5: Based on the frequencies of various visual words in the bag-of-words vector of each frame of image, obtain the feature vector of each frame of image as the local feature of each frame of image.
[0076] Step 2.2: Use the encoder of the pre-trained deep learning model to extract deep features of the acquired frame images frame by frame to obtain the deep features of each frame image.
[0077] A deep learning model consists of an encoder and a decoder.
[0078] like Figure 2As shown, the encoder includes D convolution blocks, each convolution block includes two convolution sub-modules, the first convolution sub-module includes a dilated convolution filter, a ReLU activation function layer and a batch normalization layer, and the second convolution sub-module includes a dilated convolution filter, a ReLU activation function layer, an average pooling layer and a batch normalization layer.
[0079] Specifically, the number, size and expansion coefficient of the dilated convolution filter are selected and set according to actual conditions. The number of D convolution blocks is selected and set according to actual conditions.
[0080] In this embodiment, Figure 2 As shown in Figure 1, the encoder of the deep learning model consists of three convolution blocks, and the dilation coefficients of the dilated convolution filters contained in the three convolution blocks are 1, 2, and 5 respectively.
[0081] like Figure 3 As shown, the decoder includes N deconvolution blocks, each deconvolution block includes two deconvolution sub-modules, the first deconvolution sub-module includes a bilinear interpolation layer, a convolution filter, a ReLU activation function layer and a batch normalization layer, and the second deconvolution sub-module includes a transposed convolution layer, a ReLU activation function layer and a batch normalization layer.
[0082] Specifically, the number of N deconvolution blocks is selected and set according to actual conditions.
[0083] In this embodiment, Figure 3 As shown, the decoder of the deep learning model consists of 3 deconvolution blocks.
[0084] Step 2.2.1: Get the images in the training set.
[0085] Step 2.2.2: Add a standard deviation of Gaussian noise is added to obtain a noisy image.
[0086] In this embodiment, the standard deviation of the Gaussian noise The value of is 30.
[0087] Step 2.2.3: Input the noisy image into the pre-built deep learning model, use the encoder of the pre-built deep learning model to learn the deep features of the input image, and obtain the deep features of the training image.
[0088] Step 2.2.4: Input the deep features of the training image into the decoder of the pre-built deep learning model for image reconstruction to obtain a reconstructed image containing noise.
[0089] Step 2.2.5: Calculate the loss function for the images in the training set without noise and the reconstructed images with noise.
[0090] The loss function is the mean square error loss function, which is calculated by the following formula:
[0091] ,
[0092] in, L MSE is the mean square error loss function, M is the number of images in the training set, X i For the first i images in the training set, Y i The first i images in the training set.
[0093] Step 2.2.6: Determine whether the loss function satisfies the preset loss value.
[0094] If it is not satisfied, adjust the weights and bias of the encoder and decoder in the pre-built deep learning model, output the reconstructed image containing noise, and calculate the loss function.
[0095] If satisfied, the optimal weights and biases of the encoder and decoder in the pre-built deep learning model are output to obtain a trained deep learning model.
[0096] Although this embodiment uses a pre-trained deep learning model encoder when performing deep feature extraction on the acquired frame images frame by frame, it is still necessary to train the deep learning model encoder.
[0097] The purpose of training the encoder of a deep learning model is that, first, the decoder is required for image reconstruction when training a pre-built deep learning model, and image reconstruction cannot be completed without either the encoder or the decoder. Second, the existence of the decoder forces the encoder to learn a compact representation of the data; without the decoder, the encoder does not need to consider how to reconstruct the data back to its original form, and the encoder's output will not contain enough information.
[0098] This embodiment performs local feature extraction and deep feature extraction on the acquired frame images frame by frame, and can fully capture the features of each frame image.
[0099] Step 3: Calculate the local feature cosine similarity and deep feature cosine similarity between each frame image and its adjacent frame image, and weightedly fuse the calculated local feature cosine similarity and deep feature cosine similarity to obtain the total similarity between each frame image and its adjacent frame image.
[0100] Step 3.1: Calculate the local feature cosine similarity and deep feature cosine similarity between each frame image and the adjacent frame images.
[0101] The cosine similarity is calculated by the following formula:
[0102] ,
[0103] in, S is the cosine similarity, v i For the i The feature vector of the frame image, v i+1 For the i +1 Feature vector of the image frame.
[0104] This embodiment calculates the cosine similarity, which can quantify the similarity between each frame image and its adjacent frame images at the feature level.
[0105] Step 3.2: Weightedly fuse the calculated local feature cosine similarity and deep feature cosine similarity to obtain the total similarity between each frame image and its adjacent frame images.
[0106] The total similarity between each frame image and its adjacent frame images is calculated by the following formula:
[0107] ,
[0108] in, S Total is the total similarity between each frame image and the adjacent frame images, α is the weight factor of the local features of each frame image and the adjacent frame images, S SIFT is the cosine similarity of the local features between each frame image and the adjacent frame images, β is the weight factor of the depth feature of each frame image and the adjacent frame image, S DEEP It is the cosine similarity of the deep features between each frame image and the adjacent frame images.
[0109] After experiments, it was found that α and β When the ratio is 2:1, the key frames extracted by the key frame extraction method provided in this embodiment can obtain the highest F1 score, which means that the performance of the key frame extraction method is optimal.
[0110] Specifically, α and β The relationship between the ratio setting and the F1 score is shown in Table 1:
[0111] Table 1 α and β Relationship between ratio setting and F1 score
[0112]
[0113] In Table 1, n_cluster Indicates the number of cluster centers.
[0114] This embodiment obtains the total similarity between each frame image and its adjacent frame images, which can more accurately reflect the actual situation, and not only improves the efficiency of feature extraction, but also enhances the accuracy of feature extraction.
[0115] This embodiment combines local features and deep features, which can fully capture the features of each frame image and improve the efficiency and accuracy of feature extraction when processing video frame images with complex dynamic backgrounds. Based on feature fusion, by combining scale-invariant feature transformation and convolutional denoising autoencoder technology, the accuracy and robustness of key frame extraction are effectively improved.
[0116] Step 4: In response to the total similarity between a certain frame image and its adjacent frame images being a local minimum of a sequence consisting of the total similarities of all frame images in the video to be processed and their adjacent frame images, and the total similarity is less than a preset adaptive threshold, extract this frame as a key frame of the video to be processed.
[0117] The preset adaptive threshold is calculated by the following formula:
[0118] ,
[0119] in, is the adaptive threshold, n is the total number of frame images in the video to be processed, f i For the i Frame image, f i+1 For the i +1 frame image, S Total ( f i , f i+1 ) is the relationship between the i-th frame image and the adjacent i-th frame image. i +1 The total similarity of the image.
[0120] The two determination conditions provided in this embodiment for extracting key frames can ensure that the extracted key frames have high uniqueness and representativeness in the video to be processed.
[0121] Embodiment 2:
[0122] The embodiment of the present invention uses a key frame extraction method described in Example 1 and an existing Frame Difference method and SIFT feature extraction method to extract key frames on the standard test data set TVsum, and calculates the F1 score of the extraction results of each method, as shown in Table 2.
[0123] Table 2 F1 scores of extraction results of each method
[0124]
[0125] It can be seen from Table 2 that the method in Example 1 has a higher F1 score on the standard test data set TVsum than the method mentioned in the prior art (the higher the F1 score, the better the effect).
[0126] Compared with the existing key frame extraction algorithm, the key frame extraction method described in Example 1 can adaptively process the frame images of the video and accurately extract the key frames without human intervention; by considering the fusion of local features and deep features, not only the accuracy of key frame extraction is improved, but also the robustness of the algorithm is enhanced, and it can cope with various complex video content and dynamic backgrounds.
[0127] Embodiment 3:
[0128] An embodiment of the present invention provides a key frame extraction device, comprising:
[0129] Acquisition module: used to acquire frame images of the video to be processed;
[0130] Feature extraction module: used to extract local features and deep features of the acquired frame images frame by frame to obtain local features and deep features of each frame image;
[0131] Calculation module: used to calculate the local feature cosine similarity and deep feature cosine similarity between each frame image and its adjacent frame image, and weightedly fuse the calculated local feature cosine similarity and deep feature cosine similarity to obtain the total similarity between each frame image and its adjacent frame image;
[0132] Extraction module: used for extracting a frame as a key frame of the video to be processed in response to the total similarity between a certain frame image and its adjacent frame images being a local minimum of a sequence consisting of the total similarities between all frame images in the video to be processed and their adjacent frame images, and the total similarity is less than a preset adaptive threshold.
[0133] Embodiment 4:
[0134] An embodiment of the present invention provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the key frame extraction method described in Embodiment 1 are implemented.
[0135] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0136] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0137] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0138] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0139] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A key frame extraction method, characterized in that: include: Obtain the frame image of the video to be processed; Perform local feature extraction and depth feature extraction on the acquired frame images frame by frame to obtain local features and depth features of each frame image; The extracting of local features and depth features of each frame of the acquired frame images to obtain local features and depth features of each frame of the image includes: Scale-invariant feature transformation and bag-of-words model are used to extract local features of the acquired frame images frame by frame to obtain local features of each frame image; The scale-invariant feature transformation and the bag-of-words model are used to extract local features from the acquired frame images frame by frame to obtain local features of each frame image, including: The scale-invariant feature transformation algorithm is used to detect the key points of the acquired frame images frame by frame, and the key points and descriptors of the key points of each frame image are obtained; The key points of each frame image are clustered using the K-means algorithm to obtain an index map of the key points of each frame image; Retrieving the cluster center closest to the descriptor of the key point of each frame image in the index map, and obtaining the visual vocabulary corresponding to the descriptor of the key point of each frame image; Based on the visual vocabulary corresponding to the descriptors of all key points in each frame of image, a bag-of-words vector is formed for each frame of image; Based on the frequencies of various visual words in the bag-of-words vector of each frame of image, a feature vector of each frame of image is obtained as the local feature of each frame of image; The encoder of the pre-trained deep learning model is used to extract deep features from the acquired frame images frame by frame to obtain the deep features of each frame image; The deep learning model includes an encoder and a decoder; The encoder includes D convolution blocks, each convolution block includes two convolution submodules, the first convolution submodule includes a dilated convolution filter, a ReLU activation function layer and a batch normalization layer, and the second convolution submodule includes a dilated convolution filter, a ReLU activation function layer, an average pooling layer and a batch normalization layer; The decoder includes N deconvolution blocks, each deconvolution block includes two deconvolution submodules, the first deconvolution submodule includes a bilinear interpolation layer, a convolution filter, a ReLU activation function layer and a batch normalization layer, and the second deconvolution submodule includes a transposed convolution layer, a ReLU activation function layer and a batch normalization layer; The deep learning model is trained by the following method: Get the images in the training set; Adding the standard deviation to the images in the training set is Gaussian noise is used to obtain a noisy image; Input the noisy image into a pre-built deep learning model, use the encoder of the pre-built deep learning model to learn the deep features of the input image, and obtain the deep features of the training image; The deep features of the training image are input into the decoder of the pre-built deep learning model to reconstruct the image and obtain a reconstructed image containing noise; Calculate the loss function of the images in the training set without noise and the reconstructed images with noise; Determine whether the loss function satisfies a preset loss value; if not, adjust the weights and biases of the encoder and decoder in the pre-built deep learning model, output a noisy reconstructed image, and calculate the loss function; if satisfied, output the optimal weights and biases of the encoder and decoder in the pre-built deep learning model to obtain a trained deep learning model; Calculate the local feature cosine similarity and deep feature cosine similarity between each frame image and its adjacent frame image, and weightedly fuse the calculated local feature cosine similarity and deep feature cosine similarity to obtain the total similarity between each frame image and its adjacent frame image; In response to the total similarity between a certain frame image and its adjacent frame images being a local minimum of a sequence consisting of the total similarities of all frame images in the video to be processed and their adjacent frame images, and the total similarity is less than a preset adaptive threshold, this frame is extracted as a key frame of the video to be processed.
2. The key frame extraction method according to claim 1, characterized in that: The loss function is the mean square error loss function, which is calculated by the following formula: ， in, L MSE is the mean square error loss function, M is the number of images in the training set, X i For the first i images in the training set, Y i The first i images in the training set.
3. The key frame extraction method according to claim 1, characterized in that: The calculated local feature cosine similarity and deep feature cosine similarity are weightedly fused to obtain the total similarity between each frame image and its adjacent frame image, which is calculated by the following formula: ， in, S Total is the total similarity between each frame image and the adjacent frame images, α is the weight factor of the local features of each frame image and the adjacent frame images, S SIFT is the cosine similarity of the local features between each frame image and the adjacent frame images, β is the weight factor of the depth feature of each frame image and the adjacent frame image, S DEEP is the cosine similarity of the deep features between each frame image and the adjacent frame images; Among them, the cosine similarity is calculated by the following formula: ， in, S is the cosine similarity, v i For the i The feature vector of the frame image, v i+1 For the i +1 Feature vector of the image frame.
4. The key frame extraction method according to claim 1, characterized in that: The preset adaptive threshold is calculated by the following formula: ， in, is the adaptive threshold, n is the total number of frame images in the video to be processed, f i For the i Frame image, f i+1 For the i +1 frame image, S Total ( f i , f i+1 ) is the i The frame image and the adjacent i +1 The total similarity of the image.
5. A key frame extraction device, characterized in that: include: Acquisition module: used to acquire the frame image of the video to be processed; Feature extraction module: used to extract local features and deep features of the acquired frame images frame by frame, and obtain the local features and deep features of each frame image; The extracting of local features and depth features of each frame of the acquired frame images to obtain local features and depth features of each frame of the image includes: Scale-invariant feature transformation and bag-of-words model are used to extract local features of the acquired frame images frame by frame to obtain local features of each frame image; The scale-invariant feature transformation and the bag-of-words model are used to extract local features from the acquired frame images frame by frame to obtain local features of each frame image, including: The scale-invariant feature transformation algorithm is used to detect the key points of the acquired frame images frame by frame, and the key points and descriptors of the key points of each frame image are obtained; The key points of each frame image are clustered using the K-means algorithm to obtain an index map of the key points of each frame image; Retrieving the cluster center closest to the descriptor of the key point of each frame image in the index map, and obtaining the visual vocabulary corresponding to the descriptor of the key point of each frame image; Based on the visual vocabulary corresponding to the descriptors of all key points in each frame of image, a bag-of-words vector is formed for each frame of image; Based on the frequencies of various visual words in the bag-of-words vector of each frame of image, a feature vector of each frame of image is obtained as the local feature of each frame of image; The encoder of the pre-trained deep learning model is used to extract deep features from the acquired frame images frame by frame to obtain the deep features of each frame image; The deep learning model includes an encoder and a decoder; The encoder includes D convolution blocks, each convolution block includes two convolution submodules, the first convolution submodule includes a dilated convolution filter, a ReLU activation function layer and a batch normalization layer, and the second convolution submodule includes a dilated convolution filter, a ReLU activation function layer, an average pooling layer and a batch normalization layer; The decoder includes N deconvolution blocks, each deconvolution block includes two deconvolution submodules, the first deconvolution submodule includes a bilinear interpolation layer, a convolution filter, a ReLU activation function layer and a batch normalization layer, and the second deconvolution submodule includes a transposed convolution layer, a ReLU activation function layer and a batch normalization layer; The deep learning model is trained by the following method: Get the images in the training set; Adding the standard deviation to the images in the training set is Gaussian noise is used to obtain a noisy image; Input the noisy image into a pre-built deep learning model, use the encoder of the pre-built deep learning model to learn the deep features of the input image, and obtain the deep features of the training image; The deep features of the training image are input into the decoder of the pre-built deep learning model to reconstruct the image and obtain a reconstructed image containing noise; Calculate the loss function of the images in the training set without noise and the reconstructed images with noise; Determine whether the loss function satisfies a preset loss value; if not, adjust the weights and biases of the encoder and decoder in the pre-built deep learning model, output a noisy reconstructed image, and calculate the loss function; if satisfied, output the optimal weights and biases of the encoder and decoder in the pre-built deep learning model to obtain a trained deep learning model; Calculation module: used to calculate the local feature cosine similarity and deep feature cosine similarity between each frame image and its adjacent frame image, and weightedly fuse the calculated local feature cosine similarity and deep feature cosine similarity to obtain the total similarity between each frame image and its adjacent frame image; Extraction module: used for extracting a frame as a key frame of the video to be processed in response to the total similarity between a certain frame image and its adjacent frame images being a local minimum of a sequence consisting of the total similarities between all frame images in the video to be processed and their adjacent frame images, and the total similarity is less than a preset adaptive threshold.
6. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of the key frame extraction method described in any one of claims 1-4 are implemented.