Video key frame acquisition method and device, electronic equipment and storage medium

Abstract

The invention relates to a video key frame obtaining method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a plurality of candidate image frames corresponding to a to-be-processed video, carrying out the preprocessing of each candidate image frame, inputting the preprocessed candidate image frame into a pre-constructed image feature extraction model, obtaining a feature vector corresponding to each candidate image frame, and obtaining a feature vector corresponding to each candidate image frame; performing clustering processing on the feature vector corresponding to each candidate image frame through the target clustering parameter based on a preset clustering algorithm to obtain at least one target clustering result, and performing calculation based on the feature vector corresponding to each candidate image frame and the center vector corresponding to the target clustering result to obtain a target clustering result; and determining a target image frame of the to-be-processed video from the plurality of candidate image frames based on a calculation result. By adopting the technical scheme, the image frames with large scene and element differences can be screened from the continuous image frames, and the screened image frames are labeled and applied to the training of the automatic driving perception algorithm, so that the training efficiency can be greatly improved.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology, and in particular to a method, device, electronic device and storage medium for acquiring a video key frame. Background Art

[0002] Generally speaking, the core of autonomous driving technology is to enable the car to autonomously perceive the environment, make decisions and perform corresponding driving operations through various sensors and algorithms. As one of the important sources of information in the autonomous driving system, the continuity and real-time nature of video data are crucial to the decision-making of the system.

[0003] At present, in most visual perception algorithms, the input is images, and the timing information (i.e., continuity information) provided by continuous frames (videos) is useless. A large number of images of the same scenes and elements have little effect on visual perception. Therefore, it is necessary to extract key frames from the video to reduce the burden of subsequent annotation and improve model training efficiency.

[0004] In the related art, images are screened by locating moving areas by comparing pixel differences between two or more consecutive frames. This method is sensitive to changes in illumination, easily disturbed by noise, and has limited screening effects, and the screened image scenes have small differences. Summary of the invention

[0005] In order to solve the above technical problem or at least partially solve the above technical problem, at least one embodiment of the present disclosure provides a video key frame acquisition method, device, electronic device and storage medium.

[0006] In a first aspect, the present disclosure provides a method for acquiring a video key frame, comprising:

[0007] Obtain multiple candidate image frames corresponding to the video to be processed;

[0008] After preprocessing each of the candidate image frames, inputting the pre-built image feature extraction model to obtain a feature vector corresponding to each of the candidate image frames;

[0009] Based on a preset clustering algorithm, clustering is performed on the feature vector corresponding to each of the candidate image frames by using a target clustering parameter to obtain at least one target clustering result;

[0010] Calculation is performed based on the feature vector corresponding to each of the candidate image frames and the center vector corresponding to the target clustering result, and a target image frame of the video to be processed is determined from the multiple candidate image frames based on the calculation result.

[0011] In a second aspect, the present disclosure provides a video key frame acquisition device, comprising:

[0012] An acquisition module, used to acquire multiple candidate image frames corresponding to the video to be processed;

[0013] A preprocessing module, used for preprocessing each of the candidate image frames;

[0014] An extraction module, used for inputting each of the preprocessed candidate image frames into a pre-built image feature extraction model to obtain a feature vector corresponding to each of the candidate image frames;

[0015] A clustering module, configured to perform clustering processing on the feature vector corresponding to each of the candidate image frames by using a target clustering parameter based on a preset clustering algorithm to obtain at least one target clustering result;

[0016] A processing module is used to perform calculations based on the feature vector corresponding to each of the candidate image frames and the center vector corresponding to the target clustering result, and determine the target image frame of the video to be processed from the multiple candidate image frames based on the calculation results.

[0017] In a third aspect, the present disclosure provides an electronic device, including: a processor and a memory;

[0018] The processor is used to execute any of the video key frame acquisition methods provided by the embodiments of the present disclosure by calling the program or instruction stored in the memory.

[0019] In a fourth aspect, the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instruction, wherein the program or instruction enables a computer to execute any of the video key frame acquisition methods provided in the embodiments of the present disclosure.

[0020] In a fifth aspect, the present disclosure provides a computer program product, which is used to execute any of the video key frame acquisition methods provided by the embodiments of the present disclosure.

[0021] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has at least the following advantages:

[0022] In the disclosed embodiment, multiple candidate image frames corresponding to the video to be processed are obtained, each candidate image frame is pre-processed and then input into a pre-built image feature extraction model to obtain a feature vector corresponding to each candidate image frame, the feature vector corresponding to each candidate image frame is clustered based on a preset clustering algorithm and a target clustering parameter to obtain at least one target clustering result, calculation is performed based on the feature vector corresponding to each candidate image frame and the center vector corresponding to the target clustering result, and the target image frame of the video to be processed is determined from the multiple candidate image frames based on the calculation result. By adopting the above technical solution, image frames with large scene and element differences can be quickly and accurately screened out from continuous image frames, and the screened image frames are annotated and applied to the training of the autonomous driving perception algorithm, which can greatly improve the training efficiency while ensuring the training effect, meet the use requirements of various application scenarios of autonomous driving, and use the clustering algorithm to classify the image frames corresponding to the video to screen out key frames, reduce the processing of redundant data, and further improve the processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0025] Figure 1 A flowchart of a method for acquiring a video key frame provided by an exemplary embodiment of the present disclosure;

[0026] Figure 2 A schematic flow chart of a method for acquiring a video key frame provided by another exemplary embodiment of the present disclosure;

[0027] Figure 3 A schematic diagram of the structure of a video key frame acquisition device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to more clearly understand the above-mentioned purposes, features and advantages of the present disclosure, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understandable that the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments, and the specific embodiments described herein are only used to explain the present disclosure, rather than to limit the present disclosure. In the absence of conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in the field belong to the scope of protection of the present disclosure.

[0029] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0030] The core of autonomous driving technology is to enable cars to autonomously perceive the environment, make decisions and perform corresponding driving operations through various sensors and algorithms. As one of the important sources of information in autonomous driving systems, the continuity and real-time nature of video data are crucial to the decision-making of the system. At present, in most visual perception algorithms, the input is images, and the timing information (i.e., continuity information) provided by continuous frames (videos) is useless. A large number of images of the same scene and elements have little effect on visual perception. Therefore, it is necessary to extract key frames in the video to reduce the burden of subsequent annotation and improve the efficiency of model training.

[0031] There are many types of video data processing commonly used in existing autonomous driving systems. As an example, FDM (Frame Difference Method) is used to obtain video key frames. Specifically, the frame difference method is a dynamic target detection method based on simple visual principles. It locates the moving area by comparing the pixel differences between two or more consecutive frames to screen image frames; more specifically, two or more consecutive frames are extracted from the video stream, and differential calculation is used to obtain the pixel-level difference between the consecutive frames, and the differential result is compared with a preset threshold to determine the moving area, and finally the image frames exceeding the threshold are extracted as key frames; the above method is simple to implement, with low computational complexity, and is suitable for real-time systems; however, it is sensitive to changes in illumination, easily interfered by noise, and has limited screening effect and small scene differences.

[0032] As another example, background subtraction is used to obtain video key frames. Specifically, background subtraction achieves target detection by separating foreground targets and backgrounds. More specifically, at the beginning of the video, a background model is established, usually by averaging frames or a Gaussian mixture model over a period of time. The current image frame is compared with the background model, the difference is identified as the foreground target, and morphological operations such as dilation and erosion are performed on the extracted foreground to remove noise and disconnected targets. The above method has good adaptability to static backgrounds and can handle background changes well. However, the performance decreases under dynamic backgrounds or lighting changes, and the requirements for updating the background model are high. In addition, the screening effect is limited and the scene difference is small.

[0033] Therefore, the existing video key frame acquisition method cannot filter out image frames with large scene and element differences from continuous frames.

[0034] In response to the above problems, the present disclosure provides a method for acquiring video key frames, which is applied to the screening of mass-produced feedback videos. The screened image frames are annotated and applied to the training of autonomous driving perception algorithms. It is necessary to screen image frames of different scenes and elements as much as possible. The greater the difference, the greater the improvement in model training efficiency. Specifically, multiple candidate image frames corresponding to the video to be processed are obtained, each candidate image frame is pre-processed and then input into a pre-built image feature extraction model to obtain a feature vector corresponding to each candidate image frame, and the feature vector corresponding to each candidate image frame is clustered by target clustering parameters based on a preset clustering algorithm to obtain at least one target clustering result, and calculations are performed based on the feature vector corresponding to each candidate image frame and the center vector corresponding to the target clustering result, and the target image frame of the video to be processed is determined from the multiple candidate image frames based on the calculation results. By adopting the above technical solution, image frames with large differences in scenes and elements can be quickly and accurately screened out from continuous image frames. The screened image frames are labeled and applied to the training of autonomous driving perception algorithms. While ensuring the training effect, the training efficiency can be greatly improved to meet the usage requirements of various application scenarios of autonomous driving. The clustering algorithm is used to classify the image frames corresponding to the video to screen out key frames, reduce the processing of redundant data, and further improve processing efficiency.

[0035] The specific implementation methods of the video key frame acquisition method, device, electronic device and storage medium disclosed in the present invention are explained in detail below with reference to the accompanying drawings.

[0036] Figure 1 A flowchart of a video key frame acquisition method provided for an exemplary embodiment of the present disclosure is provided. The method can be executed by a video key frame acquisition device provided for an embodiment of the present disclosure. The video key frame acquisition device can be implemented using software and / or hardware and can generally be integrated in an electronic device.

[0037] like Figure 1 As shown, the video key frame acquisition method may include the following steps:

[0038] Step 101: Acquire multiple candidate image frames corresponding to the video to be processed.

[0039] Among them, the video to be processed can be one or more. In the application scenario of autonomous driving, the video to be processed is usually multiple videos; the multiple candidate image frames refer to continuous image frames corresponding to the video to be processed.

[0040] In the embodiments of the present disclosure, there are many ways to obtain multiple candidate image frames corresponding to the video to be processed. As an example, there is one video to be processed, and the video to be processed is processed by a video frame extraction tool such as FFmpeg, OpenCV, etc., and continuous image frames corresponding to the video to be processed are obtained as multiple candidate image frames; as another example, there are multiple videos to be processed, and video frame extraction is performed on each video to be processed to obtain continuous image frames corresponding to each video to be processed, and all continuous image frames are used as multiple candidate image frames.

[0041] In the embodiment of the present disclosure, after the video to be processed is obtained, a video frame extraction process may be performed on the video to be processed to obtain a plurality of candidate image frames.

[0042] Step 102 , pre-process each candidate image frame and input it into a pre-built image feature extraction model to obtain a feature vector corresponding to each candidate image frame.

[0043] In the disclosed embodiment, after acquiring a plurality of candidate image frames, the candidate image frames may be preprocessed, and features may be extracted from each of the preprocessed candidate image frames to obtain a feature vector corresponding to each candidate image frame.

[0044] There are many ways to preprocess the candidate image frames. As an example, after adjusting the candidate image frame to an image frame of a certain size, the pixels of the candidate image frame are adjusted, such as scaling the pixels to a certain pixel range, and the pixel values ​​of each color channel of the candidate image frame can also be adjusted to a certain range, thereby ensuring that each pixel point of the image frame is on the same scale, avoiding pixels with larger values ​​from having a disproportionate impact on the results in distance calculation, thereby improving the stability and accuracy of subsequent calculations.

[0045] As an example, after adjusting the candidate image frame to an image frame of a certain size, the candidate image frame is input into a pre-trained preprocessing model to obtain a preprocessed candidate image frame; wherein the preprocessing model can be trained according to actual application preprocessing requirements.

[0046] In an embodiment of the present disclosure, a pre-built image feature extraction model is obtained. The pre-built image feature extraction model can be set up as needed. As an example, multiple network layers are obtained and connected to obtain a network model, and the network model is trained based on image samples to obtain an image feature extraction model. As another example, a pre-trained ResNet (Residual Networks) model is obtained, and the fully connected layers in the ResNet model are deleted to obtain multiple network layers. The multiple network layers are serialized based on a preset serialization function to obtain a serialized model. The processing mode of the serialized model is set to an evaluation mode to obtain an image feature extraction model. The ResNet model is usually an image classification model, not for feature vector extraction. After removing the last fully connected layer, the ResNet model becomes a new feature extractor, so that the previous network layers of ResNet can be used to extract high-level features of image frames, thereby obtaining the feature vector corresponding to each candidate image frame for subsequent clustering processing.

[0047] In the disclosed embodiment, after obtaining a pre-built image feature extraction model, the pre-processed candidate image frames are input into the image feature extraction model for feature extraction to obtain a feature vector corresponding to each candidate image frame.

[0048] Step 103 : clustering the feature vectors corresponding to each candidate image frame based on a preset clustering algorithm and target clustering parameters to obtain at least one target clustering result.

[0049] Among them, the preset clustering algorithm can be a K-means clustering algorithm, or an FCM (Fuzzy C-Means) clustering algorithm, etc., which is selected and set according to the application scenario; different clustering algorithms correspond to different target clustering parameters. For example, in the scenario of the K-means clustering algorithm, the target clustering parameters may include the target cluster number, the target iteration number and the target error threshold; among them, the target cluster number can be determined according to the number of videos to be processed, and the target cluster number can be adjusted according to the square error corresponding to the number of clusters, so as to further improve the accuracy of the clustering results and ultimately improve the video key frame acquisition effect; the target iteration number and the target error threshold can also be automatically adjusted according to the changes in the environment or scene in the video, so as to further improve the subsequent processing efficiency and effect.

[0050] In an embodiment of the present disclosure, the feature vector corresponding to each candidate image frame is clustered based on a preset clustering algorithm using target clustering parameters to obtain at least one target clustering result; wherein, it can be understood that the target clustering result can be understood as aggregating the feature vector corresponding to each candidate image frame into one category, which can be that the feature vectors corresponding to all candidate image frames are aggregated into one category, that is, one target clustering result is obtained, or the feature vectors corresponding to all candidate image frames can be aggregated into multiple categories, that is, multiple target clustering results are obtained.

[0051] Among them, different clustering algorithms perform clustering processing on the feature vector corresponding to each candidate image frame through the target clustering parameters based on the clustering algorithm, and obtain the target clustering results in different ways. As an example, the target clustering parameters include: the number of target clusters, the target number of iterations and the target error threshold. The feature vectors of the target number of clusters are obtained as the initial clustering centers, the distance between each feature vector and each initial clustering center is calculated, each feature vector is assigned to the initial clustering center corresponding to the minimum distance, at least one initial clustering result is obtained, the current clustering center corresponding to each initial clustering result is determined, and the error value between the current clustering center and the initial clustering center is calculated; at the current number of iterations When the number of iterations is less than the target number, or the error value is greater than the target error threshold, the current cluster center is used as the new cluster center for clustering processing until the current number of iterations is greater than or equal to the target number of iterations, or the error value is less than or equal to the target error threshold, and the latest clustering result is determined to be the target clustering result; as another example, the target clustering parameters include: the target number of clusters, the target number of iterations, and the target membership, obtaining the initial number of clusters and the initial membership of each data point to each cluster, calculating the cluster center according to the current membership, updating the membership according to the current cluster center value, until the maximum number of iterations is reached or the membership change is less than the target membership, etc., and determining the latest clustering result to be the target clustering result.

[0052] In the embodiment of the present disclosure, after acquiring the feature vector corresponding to each candidate image frame, the feature vector corresponding to each candidate image frame may be clustered using target clustering parameters based on a preset clustering algorithm to obtain at least one target clustering result.

[0053] Step 104 , performing calculation based on the feature vector corresponding to each candidate image frame and the center vector corresponding to the target clustering result, and determining the target image frame of the to-be-processed video from the plurality of candidate image frames based on the calculation result.

[0054] In the disclosed embodiment, each target clustering result has a corresponding cluster center, and the vector corresponding to the cluster center is the center vector corresponding to the target clustering result. The vector distance such as the vector cosine value between the feature vector corresponding to each candidate image frame and the center vector corresponding to the target clustering result to which the feature vector belongs can be calculated as the calculation result, and all the calculation results are sorted, and the candidate image frame corresponding to the minimum settlement result is used as the target image frame of the video to be processed. It can be understood that there are as many target image frames as there are target clustering results.

[0055] In the disclosed embodiment, after obtaining the target clustering result, calculations are performed based on the feature vector corresponding to each candidate image frame and the center vector corresponding to the target clustering result, and the target image frame of the video to be processed is determined from multiple candidate image frames based on the calculation results.

[0056] The video key frame acquisition method of the disclosed embodiment obtains multiple candidate image frames corresponding to the video to be processed, pre-processes each candidate image frame and inputs it into a pre-built image feature extraction model to obtain a feature vector corresponding to each candidate image frame, clusters the feature vector corresponding to each candidate image frame through a target clustering parameter based on a preset clustering algorithm to obtain at least one target clustering result, calculates based on the feature vector corresponding to each candidate image frame and the center vector corresponding to the target clustering result, and determines the target image frame of the video to be processed from the multiple candidate image frames based on the calculation result. The above technical solution can quickly and accurately filter out image frames with large scene and element differences from continuous image frames, and the filtered image frames are annotated and applied to the training of the autonomous driving perception algorithm. While ensuring the training effect, the training efficiency can be greatly improved to meet the use requirements of various application scenarios of autonomous driving, and the clustering algorithm is used to classify the image frames corresponding to the video to filter out key frames, reduce the processing of redundant data, and further improve the processing efficiency.

[0057] In an optional embodiment of the present disclosure, each candidate image frame is preprocessed, including: adjusting each candidate image frame to a target size, obtaining an initial pixel value corresponding to each candidate image frame after adjustment, and calculating a ratio of the initial pixel value to a preset maximum pixel value as a target pixel value corresponding to each candidate image frame, obtaining an initial channel pixel value of each color channel corresponding to each candidate image frame after adjustment, and calculating the target channel pixel value of each color channel corresponding to each candidate image frame based on the initial channel pixel value, the average channel pixel value and the standard channel pixel difference.

[0058] In the disclosed embodiment, the target size can be set according to the specific application scenario. Usually, the target size is smaller than the current image size of the candidate image frame. For example, the candidate image frame is scaled to 224x224 pixels. Usually, the target size is the standard input size of a pre-trained model (such as ResNet).

[0059] That is, the image size of each candidate image frame is first scaled, and then the candidate image frames after image size adjustment are pixel normalized, that is, the pixel value of each candidate image frame is adjusted to a certain range, and the pixel value of each color channel corresponding to each candidate image frame is also adjusted to a certain range.

[0060] In the embodiment of the present disclosure, the preset maximum pixel value is usually 255, the initial pixel value corresponding to each candidate image frame after adjustment is obtained, and the ratio of the initial pixel value to the preset maximum pixel value is calculated as the target pixel value corresponding to each candidate image frame. For example, if the initial pixel value corresponding to the candidate image frame is 0, the ratio of the initial pixel value to the preset maximum pixel value 255 is calculated to be 0, and 0 is used as the target pixel value corresponding to the candidate image frame; for another example, if the initial pixel value corresponding to the candidate image frame is 51, the ratio of the initial pixel value to the preset maximum pixel value 255 is calculated to be 0.2, and 0.2 is used as the target pixel value corresponding to the candidate image frame; for another example, if the initial pixel value corresponding to the candidate image frame is 255, the ratio of the initial pixel value to the preset maximum pixel value 255 is calculated to be 1, and 1 is used as the target pixel value corresponding to the candidate image frame; thereby, the pixel value range of the candidate image frame can be scaled from [0, 255] to [0, 1].

[0061] Furthermore, the initial channel pixel value of each color channel corresponding to each candidate image frame after adjustment is obtained, and the target channel pixel value of each color channel corresponding to each candidate image frame is calculated based on the initial channel pixel value, the average channel pixel value and the standard channel pixel difference value; wherein each candidate image frame has three color channels corresponding to RGB, and each color channel is subjected to the above processing. The average channel pixel value and the standard channel pixel difference value corresponding to each color channel of each candidate image frame can be obtained based on the image statistics of the ImageNet dataset (a large visualization database for visual object recognition software research), and the candidate image frames can be normalized to a specific range, so that the subsequent image feature extraction model processing is more robust and effective.

[0062] Specifically, the input continuous frame images (multiple candidate image frames) are preprocessed, that is, the candidate image frames are first scaled to 224x224 pixels, and then the pixel value range is scaled from [0, 255] to [0, 1]. The converted data type is usually float32, that is, tensor format (a data storage and processing result). Finally, the candidate image frames are standardized, that is, each color channel (R, G, B) of the candidate image frames will be subtracted from the mean value, and then divided by the standard deviation value, so that each color channel of the candidate image frames will be standardized at the same scale, so that the subsequent model can better utilize the acquired feature vector when performing feature extraction.

[0063] Therefore, scaling the pixel value range of the candidate image frame from [0, 255] to [0, 1] can help improve the performance and stability of the algorithm. By scaling the pixel value to the range of [0, 1], it can ensure that all features (i.e., each pixel point of the image) are on the same scale, thereby avoiding the disproportionate impact of features with larger values ​​on the results in the distance calculation. Especially in downstream clustering processing, such as K-means aggregation tasks, a smaller value range can reduce rounding errors in floating-point operations, improve the numerical stability of calculations, and comply with the calculation logic of K-means; and the normalized data helps to speed up the convergence of the K-means algorithm, because all features are on the same scale, and the algorithm will be more efficient when adjusting the cluster center during the iteration process.

[0064] In an optional embodiment of the present disclosure, the method further includes: obtaining a pre-trained ResNet model, and deleting the fully connected layer in the ResNet model to obtain multiple network layers, serializing the multiple network layers based on a preset serialization function to obtain a serialized model, setting the processing mode of the serialized model to an evaluation mode, and obtaining an image feature extraction model.

[0065] In the disclosed embodiment, a pre-trained ResNet model, such as a ResNet-18 model, is used to load the pre-trained ResNet-18 model weights; wherein, ResNet18 is a deep learning architecture, which is one of the variants of residual networks (Residual Networks, referred to as ResNet). Its core idea is to introduce a residual learning framework to solve the degradation problem in deep neural network training, and includes an 18-layer deep convolutional neural network.

[0066] Specifically, remove the last fully connected layer of the ResNet model (the fully connected layer is the network layer used for classification in the ResNet model), obtain all sublayers except the last sublayer (i.e., the fully connected layer), and then combine these sublayers into a serialized model through a preset serialization function such as the "torch.nn.Sequential" function. The last layer of the serialized model is not a fully connected layer, and the serialized model is set to evaluation mode. That is to say, after the ResNet model is trained, it is used to convert certain specific layers in the ResNet model (such as Dropout and BatchNorm layers, etc.) from training mode to evaluation mode, so that these layers can be used correctly during evaluation or testing. Therefore, in evaluation mode, the model will not update weights.

[0067] Therefore, ResNet itself is an image classification model, not for feature vector extraction. Removing the last fully connected layer turns the ResNet model into a new feature extractor. The first few layers of the ResNet model can be used to extract high-level features of the input data (such as images) for downstream aggregation tasks. Therefore, the ResNet-18 model introduces residual connections. Different calculation gradients help alleviate the gradient vanishing problem in deep neural network training, making it possible to effectively train deeper networks, so that it can still maintain high accuracy and stability in the face of complex environments and data noise. The structural design of the ResNet-18 model allows it to be easily expanded or modified, such as creating different variants by reducing the number of layers, improving the calculation speed, and ensuring the real-time performance of data processing by optimizing the algorithm structure and calculation process.

[0068] In an optional embodiment of the present disclosure, the target clustering parameters include: a target cluster number, a target iteration number and a target error threshold, and clustering processing is performed on the feature vector corresponding to each candidate image frame based on a preset clustering algorithm through the target clustering parameters to obtain at least one target clustering result, including: obtaining feature vectors of the target cluster number as initial cluster centers; calculating the distance between each feature vector and each initial cluster center, assigning each feature vector to the initial cluster center corresponding to the minimum distance, and obtaining at least one initial clustering result; determining the current cluster center corresponding to each initial clustering result, and calculating the error value between the current cluster center and the initial cluster center; when the current iteration number is less than the target iteration number, or the error value is greater than the target error threshold, clustering processing is performed on the current cluster center as the new cluster center until the current iteration number is greater than or equal to the target iteration number, or the error value is less than or equal to the target error threshold, and determining the latest clustering result as the target clustering result.

[0069] In the disclosed embodiment, all candidate image frames in the continuous frames are loaded, preprocessed, features are extracted using a pre-trained model (such as ResNet), and the extracted feature vector is returned, that is, the feature vector corresponding to each candidate image frame is obtained. In order to save memory and computing resources, the ResNet model does not need to calculate gradients.

[0070] Specifically, a K-means clustering algorithm is executed for clustering processing on the extracted feature vectors, wherein K-means is an unsupervised machine learning algorithm for dividing data points (all feature vectors) into K clusters, wherein K is a positive integer greater than 0, and each data point is assigned to a nearest cluster center; the number of clusters to be formed is set, the maximum number of iterations of the clustering algorithm is set, and the convergence tolerance of the clustering algorithm is set. When the change of the cluster center is less than the convergence tolerance, the algorithm will stop iterating.

[0071] In the embodiments of the present disclosure, the target number of clusters can be the number of videos to be processed. In order to further improve the clustering effect, in some embodiments, the number of videos to be processed is obtained, the number of videos is used as the initial number of clusters, and multiple candidate numbers of clusters are determined according to preset coefficients and the initial number of clusters. A target straight line is constructed based on the sum of square errors corresponding to the initial number of clusters and the sum of square errors corresponding to each candidate number of clusters, and the target number of clusters is determined from multiple candidate numbers of clusters based on the target straight line.

[0072] Among them, the preset coefficient can be selected and set according to the needs of the application scenario. For example, the initial number of clusters is 100, and the number of candidate clusters is 110, 120 and 130. The sum of square errors corresponding to each 100, 110, 120 and 130 is obtained, and the target straight line is constructed with the number of clusters as the horizontal axis. The slope corresponding to each number of clusters is obtained, and the number of clusters corresponding to the maximum slope is used as the target number of clusters, that is, the number of videos to be processed n is used as the initial number of clusters (that is, the number of clusters), and then the sum of square errors is accumulated and calculated at intervals of 0.1k. Based on the Elbow Method, the point where the sum of square errors decreases significantly (the number of candidate clusters) is automatically selected as the target n value (target number of clusters).

[0073] In the embodiments of the present disclosure, the number of iterations and the error threshold can also be selected and set according to application needs. In some embodiments, a candidate parameter set is obtained; wherein the candidate parameter set includes multiple candidate numbers of iterations and multiple candidate error thresholds; the candidate parameter set is divided into multiple parameter subsets based on a preset grid search algorithm, and any parameter subset of the multiple parameter subsets is used as a test parameter set, and all parameter subsets except the test parameter set are used as training sets to train a parameter model, and the test parameter set is input into the parameter model to obtain a test result, and the test result and the preset scoring rule are scored based on the grid search algorithm to obtain a scoring result corresponding to each parameter subset, and a target parameter subset is determined from the multiple parameter subsets based on the scoring result; wherein the target parameter subset includes a target number of iterations and a target error threshold.

[0074] Specifically, the grid search automated parameter adjustment method is used. Grid search is used to search between different maximum number of iterations (target number of iterations) and convergence tolerance values ​​(target error value), and the clustering effect of each parameter combination is evaluated. That is, GridSearch will cross-validate each set of parameters according to the provided parameter grid param_grid and scoring rules. That is, for each set of parameters, GridSearch will divide the data set into several subsets (usually k subsets in k-fold cross validation), and then use the data except one subset to train the parameter model, and test the model on the retained subset. This process will be repeated, and different subsets will be replaced each time as test sets, and others as training sets. In this way, among all parameter combinations, GridSearch will find the set of parameters with the highest average score, and save these parameters in the best_params_ attribute as the target parameter subset (including target number of iterations and target error threshold).

[0075] Therefore, parameters can be automatically adjusted according to environmental changes to adapt to different screening requirements, and the Elbow Method is introduced to calculate the sum of square errors of different numbers of clusters and select the point where the square error and the rate of decrease slow down significantly as the target number of clusters.

[0076] In an embodiment of the present disclosure, after determining the target number of clusters, the target number of iterations and the target error threshold, the feature vectors of the target number of clusters can be used as the initial cluster centers, and the distance between each feature vector and each initial cluster center is calculated, and each feature vector is assigned to the initial cluster center corresponding to the minimum distance to obtain one or more initial clustering results, and then the error value between the current cluster center of the initial clustering result and the initial cluster center is determined. When the error value is greater than the preset error threshold or the current number of iterations is less than the target number of iterations, the current cluster center of the initial clustering result is used as the new initial cluster center to repeat the above process until the current number of iterations is greater than or equal to the target number of iterations, or the error value is less than or equal to the target error threshold, and the latest clustering result is determined to be the target clustering result.

[0077] Therefore, the clustering algorithm is used to classify the video frames, filter out the key frames, reduce the processing of redundant data, and filter out the image frames of different scenes and elements, and the differences between the image frames are large.

[0078] In an optional embodiment of the present disclosure, calculations are performed based on the feature vector corresponding to each candidate image frame and the center vector corresponding to the target clustering result, and the target image frame of the video to be processed is determined from multiple candidate image frames based on the calculation results, including: calculating the vector distance between the feature vector corresponding to each candidate image frame and the center vector corresponding to the target clustering result to which the feature vector belongs, obtaining the vector distance corresponding to each candidate image frame, and taking the candidate image frame corresponding to the minimum vector distance as the target image frame of the video to be processed.

[0079] In the disclosed embodiment, after clustering processing, the feature vector of each candidate image frame has a corresponding target clustering result, and each target clustering result has a target clustering center. The center vector of the target clustering center is obtained, and the vector distance between the feature vector and the center vector corresponding to the target clustering result is calculated. That is, all feature vectors included in the target clustering result have corresponding vector distances with the center vector. The candidate image frame corresponding to the minimum vector distance can be selected as the target image frame of the video to be processed, and as many target image frames as there are target clustering results can be obtained.

[0080] Specifically, the input feature vector is inferred, that is, the cluster center is calculated and the distribution of the feature vector and the cluster center is determined, that is, the center point of each cluster or the image frame closest to the center point is selected as the key frame (target image frame) to complete the continuous frame screening. In this way, image frames with large scene and element differences can be screened out from the continuous frames.

[0081] Based on the description of the foregoing embodiments, in most visual perception algorithms, the timing information (i.e., continuity information) provided by continuous frames (videos) is useless, and a large number of pictures of the same scenes and elements have little effect on visual perception. Therefore, it is necessary to extract key frames in the video to reduce the burden of subsequent annotation and improve model training efficiency. The existing technology only supports the screening of pictures of different targets (elements) but does not support the screening of pictures of different scenes. Therefore, the disclosed embodiments propose a video key frame acquisition method that can screen out image frames with large differences in scenes and elements from continuous frames.

[0082] Figure 2 A flowchart of a video key frame acquisition method provided for another exemplary embodiment of the present disclosure is provided. The method can be executed by a video key frame acquisition device provided for an embodiment of the present disclosure. The video key frame acquisition device can be implemented using software and / or hardware and can generally be integrated in an electronic device.

[0083] like Figure 2 As shown, the video key frame acquisition method may include the following steps:

[0084] Step 201, obtain a pre-trained ResNet model, and delete the fully connected layer in the ResNet model to obtain multiple network layers, serialize the multiple network layers based on a preset serialization function to obtain a serialized model, set the processing mode of the serialized model to an evaluation mode, and obtain an image feature extraction model.

[0085] In the disclosed embodiment, the last fully connected layer of the ResNet model is removed, all network layers except the fully connected layer are obtained, and then the network layers are combined into a serialized model through a preset serialization function such as the "torch.nn.Sequential" function. The last layer of the serialized model is no longer a fully connected layer, and the serialized model is set to evaluation mode. That is, after the ResNet model is trained, some specific layers in the ResNet model (such as Dropout and BatchNorm layers, etc.) are converted from training mode to evaluation mode, so that these layers can be used correctly during evaluation or testing. Therefore, in the evaluation mode, the model will not update the weights.

[0086] Therefore, ResNet itself is an image classification model, not used for feature vector extraction. Removing the last fully connected layer turns the ResNet model into a new feature extractor. The first few layers of the ResNet model can be used to extract high-level features of input data (such as images) for downstream aggregation tasks.

[0087] Step 202, obtain multiple candidate image frames corresponding to the video to be processed, adjust each candidate image frame to a target size, obtain an initial pixel value corresponding to each candidate image frame after adjustment, and calculate the ratio of the initial pixel value to a preset maximum pixel value as the target pixel value corresponding to each candidate image frame.

[0088] In the embodiment of the present disclosure, the target size can be set according to the specific application scenario. Usually, the target size is smaller than the current image size of the candidate image frame.

[0089] In the disclosed embodiment, the image size of each candidate image frame is firstly scaled, and then the candidate image frames after image size adjustment are pixel normalized, that is, the pixel values ​​of each candidate image frame are adjusted to within a certain range.

[0090] In the embodiment of the present disclosure, the initial pixel value corresponding to each candidate image frame after adjustment is obtained, and the ratio of the initial pixel value to the preset maximum pixel value is calculated as the target pixel value corresponding to each candidate image frame. For example, if the initial pixel value corresponding to the candidate image frame is 0, the ratio of the initial pixel value to the preset maximum pixel value 255 is calculated to be 0, and 0 is used as the target pixel value corresponding to the candidate image frame; for another example, if the initial pixel value corresponding to the candidate image frame is 51, the ratio of the initial pixel value to the preset maximum pixel value 255 is calculated to be 0.5, and 0.5 is used as the target pixel value corresponding to the candidate image frame; for another example, if the initial pixel value corresponding to the candidate image frame is 255, the ratio of the initial pixel value to the preset maximum pixel value 255 is calculated to be 1, and 1 is used as the target pixel value corresponding to the candidate image frame; thereby, the pixel value range of the candidate image frame can be scaled from [0, 255] to [0, 1].

[0091] Step 203, obtaining the initial channel pixel value of each color channel corresponding to each candidate image frame after adjustment, and calculating the target channel pixel value of each color channel corresponding to each candidate image frame based on the initial channel pixel value, the average channel pixel value and the standard channel pixel difference.

[0092] In the disclosed embodiment, each candidate image frame has three corresponding RGB color channels, and each color channel is subjected to the above-mentioned processing. The average channel pixel value and the standard channel pixel difference corresponding to each color channel of each candidate image frame can be obtained based on the image statistics of the ImageNet data set (a large visualization database for visual object recognition software research), and the candidate image frame can be normalized to a specific range, so that the subsequent image feature extraction model processing is more robust and effective.

[0093] In the disclosed embodiment, the mean value is subtracted from each color channel (R, G, B) of the candidate image frame and then divided by the standard deviation value, so that each color channel of the candidate image frame is standardized on the same scale, so that the subsequent model can better utilize the acquired feature vector when performing feature extraction.

[0094] Step 204: input each preprocessed candidate image frame into a pre-built image feature extraction model to obtain a feature vector corresponding to each candidate image frame.

[0095] In the disclosed embodiment, after obtaining a pre-built image feature extraction model, the pre-processed candidate image frames are input into the image feature extraction model for feature extraction to obtain a feature vector corresponding to each candidate image frame.

[0096] Step 205, obtain the feature vectors of the target cluster quantity as the initial cluster center, calculate the distance between each feature vector and each initial cluster center, assign each feature vector to the initial cluster center corresponding to the minimum distance, obtain at least one initial clustering result, determine the current cluster center corresponding to each initial clustering result, and calculate the error value between the current cluster center and the initial cluster center.

[0097] Step 206, when the current number of iterations is less than the target number of iterations, or the error value is greater than the target error threshold, the current cluster center is used as the new cluster center for clustering processing until the current number of iterations is greater than or equal to the target number of iterations, or the error value is less than or equal to the target error threshold, and the latest clustering result is determined to be the target clustering result.

[0098] In the embodiments of the present disclosure, the target clustering result can be understood as aggregating the feature vectors corresponding to each candidate image frame into one category. The feature vectors corresponding to all candidate image frames can be aggregated into one category, that is, one target clustering result is obtained, or the feature vectors corresponding to all candidate image frames can be aggregated into multiple categories, that is, multiple target clustering results are obtained.

[0099] In an embodiment of the present disclosure, after determining the target number of clusters, the target number of iterations and the target error threshold, the feature vectors of the target number of clusters can be used as the initial cluster centers, and the distance between each feature vector and each initial cluster center is calculated, and each feature vector is assigned to the initial cluster center corresponding to the minimum distance to obtain one or more initial clustering results, and then the error value between the current cluster center of the initial clustering result and the initial cluster center is determined. When the error value is greater than the preset error threshold or the current number of iterations is less than the target number of iterations, the current cluster center of the initial clustering result is used as the new initial cluster center to repeat the above process until the current number of iterations is greater than or equal to the target number of iterations, or the error value is less than or equal to the target error threshold, and the latest clustering result is determined to be the target clustering result.

[0100] Step 207, calculate the vector distance between the feature vector corresponding to each candidate image frame and the central vector corresponding to the target clustering result to which the feature vector belongs, obtain the vector distance corresponding to each candidate image frame, and use the candidate image frame corresponding to the minimum vector distance as the target image frame of the video to be processed.

[0101] In the disclosed embodiment, after clustering processing, the feature vector of each candidate image frame has a corresponding target clustering result, and each target clustering result has a target clustering center. The center vector of the target clustering center is obtained, and the vector distance between the feature vector and the center vector corresponding to the target clustering result is calculated. That is, all feature vectors included in the target clustering result have corresponding vector distances with the center vector. The candidate image frame corresponding to the minimum vector distance can be selected as the target image frame of the video to be processed, and as many target image frames as there are target clustering results can be obtained.

[0102] The video key frame acquisition method of the embodiment of the present disclosure obtains a pre-trained ResNet model, deletes the fully connected layer in the ResNet model, obtains multiple network layers, serializes the multiple network layers based on a preset serialization function to obtain a serialization model, sets the processing mode of the serialization model to an evaluation mode, and obtains an image feature extraction model; obtains multiple candidate image frames corresponding to the video to be processed, adjusts each candidate image frame to a target size, obtains an initial pixel value corresponding to each candidate image frame after adjustment, and calculates a ratio of the initial pixel value to a preset maximum pixel value as a target pixel value corresponding to each candidate image frame; obtains an initial channel pixel value of each color channel corresponding to each candidate image frame after adjustment, and calculates a target channel pixel value of each color channel corresponding to each candidate image frame based on the initial channel pixel value, the average channel pixel value, and the standard channel pixel difference; inputs each pre-processed candidate image frame into a pre-constructed image feature extraction model to obtain each The characteristic vectors corresponding to the candidate image frames; obtaining the characteristic vectors of the target clustering quantity as the initial clustering center, calculating the distance between each characteristic vector and each initial clustering center, assigning each characteristic vector to the initial clustering center corresponding to the minimum distance, obtaining at least one initial clustering result, determining the current clustering center corresponding to each initial clustering result, and calculating the error value between the current clustering center and the initial clustering center; when the current number of iterations is less than the target number of iterations, or the error value is greater than the target error threshold, using the current clustering center as the new clustering center for clustering processing until the current number of iterations is greater than or equal to the target number of iterations, or the error value is less than or equal to the target error threshold, determining the latest clustering result as the target clustering result; calculating the vector distance between the characteristic vector corresponding to each candidate image frame and the center vector corresponding to the target clustering result to which the characteristic vector belongs, obtaining the vector distance corresponding to each candidate image frame, and using the candidate image frame corresponding to the minimum vector distance as the target image frame of the video to be processed. Therefore, it has an efficient data screening method, uses clustering algorithms to classify video frames, screens out key frames, and reduces the processing of redundant data; it is adaptive and can automatically adjust parameters according to environmental changes to adapt to different screening requirements. The Elbow Method is introduced to calculate the sum of square errors of different numbers of clusters, and select the point where the square error and the rate of decline are significantly slowed down as the final number of clusters; it is robust and can still maintain high accuracy and stability in the face of complex environments and data noise. The ResNet model introduces residual connections to help alleviate the gradient vanishing problem in deep neural network training, so that deeper networks can be effectively trained; it is real-time and ensures the real-time performance of data processing by optimizing the algorithm structure and calculation process. The structural design of the ResNet model allows it to be easily expanded or modified, such as creating different variants by reducing the number of layers to improve the calculation speed.

[0103] In order to implement the above embodiments, the present disclosure further provides a video key frame acquisition device, which can be implemented by software and / or hardware.

[0104] Figure 3 A schematic diagram of the structure of a video key frame acquisition device provided by an embodiment of the present disclosure, such as Figure 3 As shown, the video key frame acquisition device 30 may include: an acquisition module 310 , a pre-processing module 320 , an extraction module 330 , a clustering module 340 and a processing module 350 .

[0105] The acquisition module 310 is used to acquire multiple candidate image frames corresponding to the video to be processed;

[0106] A preprocessing module 320, configured to preprocess each of the candidate image frames;

[0107] An extraction module 330, configured to input each of the preprocessed candidate image frames into a pre-built image feature extraction model to obtain a feature vector corresponding to each of the candidate image frames;

[0108] A clustering module 340 is used to perform clustering processing on the feature vector corresponding to each of the candidate image frames based on a preset clustering algorithm and a target clustering parameter to obtain at least one target clustering result;

[0109] The processing module 350 is used to perform calculations based on the feature vector corresponding to each of the candidate image frames and the center vector corresponding to the target clustering result, and determine the target image frame of the video to be processed from the multiple candidate image frames based on the calculation results.

[0110] Optionally, the preprocessing module 320 is specifically configured to:

[0111] Adjusting each of the candidate image frames to a target size, obtaining an initial pixel value corresponding to each of the adjusted candidate image frames, and calculating a ratio of the initial pixel value to a preset maximum pixel value as a target pixel value corresponding to each of the candidate image frames;

[0112] The initial channel pixel value of each color channel corresponding to each of the candidate image frames after adjustment is obtained, and the target channel pixel value of each color channel corresponding to each of the candidate image frames is calculated based on the initial channel pixel value, the average channel pixel value and the standard channel pixel difference value.

[0113] Optionally, the device further comprises:

[0114] An acquisition and deletion module is used to acquire a pre-trained ResNet model and delete a fully connected layer in the ResNet model to obtain multiple network layers;

[0115] A serialization module, used for serializing the multiple network layers based on a preset serialization function to obtain a serialization model;

[0116] A setting module is used to set the processing mode of the serialization model to an evaluation mode to obtain the image feature extraction model.

[0117] Optionally, the target clustering parameters include: target cluster number, target iteration number and target error threshold, and the clustering module 340 includes:

[0118] A first acquisition unit, used to acquire the feature vector of the target cluster quantity as an initial cluster center;

[0119] A calculation unit, used to calculate the distance between each of the feature vectors and each of the initial clustering centers, and assign each of the feature vectors to the initial clustering center corresponding to the minimum distance, so as to obtain at least one initial clustering result;

[0120] A determination calculation unit is used to determine a current cluster center corresponding to each of the initial clustering results, and calculate an error value between the current cluster center and the initial cluster center;

[0121] A computing unit, for, when the current number of iterations is less than the target number of iterations, or the error value is greater than the target error threshold, using the current cluster center as a new cluster center for clustering processing until the current number of iterations is greater than or equal to the target number of iterations, or the error value is less than or equal to the target error threshold, and determining that the latest clustering result is the target clustering result.

[0122] Optionally, the clustering module 340 further includes:

[0123] A second acquisition unit is used to acquire the number of videos to be processed, and use the number of videos as the initial cluster number;

[0124] A determination unit, configured to determine a plurality of candidate cluster numbers according to a preset coefficient and the initial cluster number;

[0125] A determination unit is constructed, which is used to construct a target straight line based on the square error sum corresponding to the initial cluster number and the square error sum corresponding to each candidate cluster number, and determine the target cluster number from the multiple candidate cluster numbers based on the target straight line.

[0126] Optionally, the clustering module 340 further includes:

[0127] A third acquisition unit is used to acquire a candidate parameter set; wherein the candidate parameter set includes multiple candidate iteration numbers and multiple candidate error thresholds;

[0128] A training and testing unit, used for dividing the candidate parameter set into a plurality of parameter subsets based on a preset grid search algorithm, taking any parameter subset among the plurality of parameter subsets as a test parameter set, taking all parameter subsets except the test parameter set as training sets to train a parameter model, and inputting the test parameter set into the parameter model to obtain a test result;

[0129] A scoring determination unit is used to score the test results and preset scoring rules based on the grid search algorithm to obtain a scoring result corresponding to each of the parameter subsets, and determine a target parameter subset from the multiple parameter subsets based on the scoring results; wherein the target parameter subset includes the target number of iterations and the target error threshold.

[0130] Optionally, the processing module 350 is specifically configured to:

[0131] Calculating the vector distance between the feature vector corresponding to each of the candidate image frames and the center vector corresponding to the target clustering result to which the feature vector belongs, to obtain the vector distance corresponding to each of the candidate image frames;

[0132] The candidate image frame corresponding to the minimum vector distance is used as the target image frame of the video to be processed.

[0133] The video key frame acquisition device provided in the embodiment of the present disclosure can execute the video key frame acquisition method provided in the embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method. The contents not described in detail in the embodiment of the device of the present disclosure can refer to the description in any method embodiment of the present disclosure.

[0134] The disclosed embodiment also provides an electronic device, including a processor and a memory; the processor calls the program or instructions stored in the memory to execute the steps of the aforementioned video key frame acquisition method embodiments, which will not be repeated here to avoid repeated description.

[0135] The embodiments of the present disclosure also provide a computer-readable storage medium, which is non-transitory and stores programs or instructions. The programs or instructions enable a computer to execute the steps of the aforementioned video key frame acquisition method embodiments. To avoid repeated description, they are not repeated here.

[0136] The embodiments of the present disclosure also provide a computer program product, which is used to execute the steps of the embodiments of the aforementioned video key frame acquisition method.

[0137] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0138] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for obtaining a video key frame, characterized in that: include: Obtain multiple candidate image frames corresponding to the video to be processed; After preprocessing each of the candidate image frames, inputting the pre-built image feature extraction model to obtain a feature vector corresponding to each of the candidate image frames; Based on a preset clustering algorithm, clustering is performed on the feature vector corresponding to each of the candidate image frames by using a target clustering parameter to obtain at least one target clustering result; Calculation is performed based on the feature vector corresponding to each of the candidate image frames and the center vector corresponding to the target clustering result, and a target image frame of the video to be processed is determined from the multiple candidate image frames based on the calculation result.

2. The method according to claim 1, characterized in that The preprocessing of each candidate image frame comprises: Adjusting each of the candidate image frames to a target size, obtaining an initial pixel value corresponding to each of the adjusted candidate image frames, and calculating a ratio of the initial pixel value to a preset maximum pixel value as a target pixel value corresponding to each of the candidate image frames; The initial channel pixel value of each color channel corresponding to each of the candidate image frames after adjustment is obtained, and the target channel pixel value of each color channel corresponding to each of the candidate image frames is calculated based on the initial channel pixel value, the average channel pixel value and the standard channel pixel difference value.

3. The method according to claim 1, characterized in that The method further comprises: Obtain a pre-trained ResNet model, and delete the fully connected layer in the ResNet model to obtain multiple network layers; Performing serialization processing on the multiple network layers based on a preset serialization function to obtain a serialization model; The processing mode of the serialization model is set to an evaluation mode to obtain the image feature extraction model.

4. The method according to claim 1, characterized in that: in, The target clustering parameters include: the number of target clusters, the number of target iterations and the target error threshold. The preset clustering algorithm performs clustering processing on the feature vectors corresponding to each candidate image frame through the target clustering parameters to obtain at least one target clustering result, including: Obtaining the characteristic vector of the target cluster quantity as the initial cluster center; Calculating the distance between each of the feature vectors and each of the initial clustering centers, assigning each of the feature vectors to the initial clustering center corresponding to the minimum distance, and obtaining at least one initial clustering result; Determine a current cluster center corresponding to each of the initial clustering results, and calculate an error value between the current cluster center and the initial cluster center; When the current number of iterations is less than the target number of iterations, or the error value is greater than the target error threshold, the current cluster center is used as the new cluster center for clustering processing until the current number of iterations is greater than or equal to the target number of iterations, or the error value is less than or equal to the target error threshold, and the latest clustering result is determined to be the target clustering result.

5. The method according to claim 4, characterized in that The method further comprises: Obtaining the number of videos to be processed, and using the number of videos as the initial clustering number; Determining multiple candidate cluster numbers according to a preset coefficient and the initial cluster number; A target straight line is constructed based on the sum of square errors corresponding to the initial cluster number and the sum of square errors corresponding to each of the candidate cluster numbers, and the target cluster number is determined from the multiple candidate cluster numbers based on the target straight line.

6. The method according to claim 4, characterized in that The method further comprises: Acquire a candidate parameter set; wherein the candidate parameter set includes multiple candidate iteration times and multiple candidate error thresholds; Divide the candidate parameter set into multiple parameter subsets based on a preset grid search algorithm, use any parameter subset among the multiple parameter subsets as a test parameter set, use all parameter subsets excluding the test parameter set as training sets to train a parameter model, and input the test parameter set into the parameter model to obtain a test result; The test results and preset scoring rules are scored based on the grid search algorithm to obtain scoring results corresponding to each of the parameter subsets, and a target parameter subset is determined from the multiple parameter subsets based on the scoring results; wherein the target parameter subset includes the target number of iterations and the target error threshold.

7. The method according to claim 1, characterized in that The step of calculating based on the feature vector corresponding to each of the candidate image frames and the center vector corresponding to the target clustering result, and determining the target image frame of the video to be processed from the multiple candidate image frames based on the calculation result, includes: Calculating the vector distance between the feature vector corresponding to each of the candidate image frames and the center vector corresponding to the target clustering result to which the feature vector belongs, to obtain the vector distance corresponding to each of the candidate image frames; The candidate image frame corresponding to the minimum vector distance is used as the target image frame of the video to be processed.

8. A video key frame acquisition device, characterized in that: The device comprises: An acquisition module, used to acquire multiple candidate image frames corresponding to the video to be processed; A preprocessing module, used for preprocessing each of the candidate image frames; An extraction module, used for inputting each of the preprocessed candidate image frames into a pre-built image feature extraction model to obtain a feature vector corresponding to each of the candidate image frames; A clustering module, configured to perform clustering processing on the feature vector corresponding to each of the candidate image frames by using a target clustering parameter based on a preset clustering algorithm to obtain at least one target clustering result; A processing module is used to perform calculations based on the feature vector corresponding to each of the candidate image frames and the center vector corresponding to the target clustering result, and determine the target image frame of the video to be processed from the multiple candidate image frames based on the calculation results.

9. An electronic device, characterized in that: include: Processor and memory; The processor is used to execute the video key frame acquisition method according to any one of claims 1 to 7 by calling the program or instruction stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute the video key frame acquisition method according to any one of claims 1 to 7.

Images