Key frame selection method and device

Abstract

The present invention discloses a key frame selection method and device, and relates to the field of computer technology. A specific implementation of the method includes: receiving a key frame selection request, determining a target video according to the selection request, and obtaining multiple video frames corresponding to the target video; filtering the multiple video frames according to the feature information of the multiple video frames according to preset filtering rules, wherein the filtering rules include spatial dimension filtering rules and temporal dimension filtering rules; clustering the multiple video frames after filtering to obtain at least one type of video frames, and selecting the key frame corresponding to the target video according to the center of at least one type of video frames. This implementation can filter out low-quality video frames in the spatial dimension and the temporal dimension, and can also remove redundant information from the clustering perspective, so that key frames can be selected quickly and accurately.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a key frame selection method and device. Background Art

[0002] Videos are composed of thousands of video frames, which contain more information than pictures. However, there is a lot of redundant information between video frames, and video key frames can be regarded as pictures that can reflect the main content of the video. Therefore, how to select key frames is of great significance. At present, the following three methods are mainly used to select video frames: 1. Feature-based key frame selection method; 2. Clustering-based key frame selection method; 3. Comprehensive algorithm-based key frame selection method.

[0003] However, the existing methods for selecting video frames have the following problems: the feature-based key frame selection method has the problems of slow selection speed and large amount of calculation; the clustering-based key frame selection method has the problems of large errors and large consumption of computing resources; the comprehensive algorithm-based key frame selection method has the problem of serious frame missing; the above three methods have redundant information in the selected key frames. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a key frame selection method and device, which can filter out low-quality video frames in spatial and temporal dimensions, and can also remove redundant information from a clustering perspective, so that key frames can be selected quickly and accurately.

[0005] To achieve the above objective, according to one aspect of an embodiment of the present invention, a key frame selection method is provided.

[0006] The key frame selection method of an embodiment of the present invention includes: receiving a key frame selection request, determining a target video according to the selection request, and obtaining multiple video frames corresponding to the target video; filtering the multiple video frames according to preset filtering rules and feature information of the multiple video frames, wherein the filtering rules include spatial dimension filtering rules and temporal dimension filtering rules; clustering the multiple video frames after filtering to obtain at least one type of video frames, and selecting the key frame corresponding to the target video according to the center of the at least one type of video frames.

[0007] Optionally, the filtering process is performed on the multiple video frames according to the feature information of the multiple video frames according to preset filtering rules, including: filtering the multiple video frames according to the first feature information of the multiple video frames according to the preset spatial dimension filtering rules to filter out low-quality video frames in the spatial dimension to obtain multiple optional video frames; filtering the multiple optional video frames according to the second feature information corresponding to the multiple optional video frames according to the preset time dimension filtering rules to filter out low-quality video frames in the time dimension.

[0008] Optionally, the first feature information includes: picture brightness value, picture sharpness value and picture balance value; and, filtering the multiple video frames according to the first feature information of the multiple video frames according to the preset spatial dimension filtering rule, including: for one video frame among the multiple video frames, if the first feature information of the one video frame meets at least one of the following options, filtering out the one video frame: the picture brightness value of the one video frame is not greater than a preset brightness threshold; the picture sharpness value of the one video frame is not greater than a preset sharpness threshold; the picture balance value of the one video frame is not less than a preset balance threshold.

[0009] Optionally, the method also includes: calculating the image brightness value of the video frame according to the RGB value of the color channel of the video frame; introducing the pixel gradient of the video frame according to the grayscale value of the video frame to calculate the sharpness value of the video frame; and performing histogram equalization processing on the video frame to obtain the grayscale level after the histogram equalization processing, and then calculating the image equalization value of the video frame according to the grayscale level after the histogram equalization processing.

[0010] Optionally, the second characteristic information includes: an inter-frame differential value and a lens edge change rate; and, according to the preset time dimension filtering rule, the multiple optional video frames are filtered according to the second characteristic information corresponding to the multiple optional video frames, including: for an optional video frame among the multiple optional video frames, if the second characteristic information corresponding to the one optional video frame meets at least one of the following options, the one optional video frame is filtered out: the inter-frame differential value corresponding to the one optional video frame is greater than a preset inter-frame differential threshold; the lens edge change rate corresponding to the one optional video frame is greater than a preset lens edge change threshold.

[0011] Optionally, the method also includes: calculating a first difference between the one optional video frame and a previous optional video frame corresponding to the one optional video frame, and calculating a second difference between the one optional video frame and a subsequent optional video frame corresponding to the one optional video frame; normalizing the first difference and the second difference, and calculating an inter-frame differential value corresponding to the one optional video frame using the first difference and the second difference after normalization.

[0012] Optionally, the method also includes: dilating an edge picture of a subsequent optional video frame corresponding to the one optional video frame, and calculating a fade-in edge change rate between the edge picture of the subsequent optional video frame after the dilation and the edge picture of the one optional video frame; dilating an edge picture of a previous optional video frame corresponding to the one optional video frame, and calculating a fade-out edge change rate between the edge picture of the previous optional video frame after the dilation and the edge picture of the one optional video frame; and selecting the maximum value of the fade-in edge change rate and the fade-out edge change rate as the lens edge change rate corresponding to the one optional video frame.

[0013] Optionally, the calculation of the fade-in edge change rate between the edge picture of the subsequent optional video frame after the dilation processing and the edge picture of the one optional video frame includes: performing an AND operation on the edge picture of the subsequent optional video frame after the dilation processing and the edge picture of the one optional video frame, and then summing them to obtain a first pixel point sum value; performing a dot division on the first pixel point sum value and the pixel point sum value corresponding to the edge picture of the one optional video frame, and then calculating the difference with 1 to obtain the fade-in edge change rate corresponding to the one optional video frame.

[0014] Optionally, the calculation of the fade-out edge change rate between the edge picture of the previous optional video frame after the dilation processing and the edge picture of the one optional video frame includes: performing an AND operation on the edge picture of the previous optional video frame after the dilation processing and the edge picture of the one optional video frame, and then summing them to obtain a second pixel point sum value; dividing the second pixel point sum value by the pixel point sum value corresponding to the edge picture of the one optional video frame, and then calculating the difference with 1 to obtain the fade-out edge change rate corresponding to the one optional video frame.

[0015] Optionally, clustering the multiple video frames after filtering to obtain at least one category of video frames, and selecting key frames corresponding to the target video according to the center of the at least one category of video frames, including: determining the number of shots based on the multiple video frames after filtering based on a preset step size; clustering the multiple video frames after filtering according to the number of shots using a clustering algorithm to obtain at least one category of video frames, and determining the center of the at least one category of video frames; for each category of video frames, selecting the video frame closest to the center of that category of video frames as the key frame corresponding to that category of video frames, so as to obtain the key frame corresponding to the target video.

[0016] To achieve the above objective, according to another aspect of an embodiment of the present invention, a key frame selection device is provided.

[0017] The key frame selection device of the embodiment of the present invention includes: an acquisition module, which is used to receive a key frame selection request, determine a target video according to the selection request, and obtain multiple video frames corresponding to the target video; a filtering module, which is used to filter the multiple video frames according to preset filtering rules and feature information of the multiple video frames, wherein the filtering rules include spatial dimension filtering rules and time dimension filtering rules; a selection module, which is used to cluster the multiple video frames after filtering to obtain at least one type of video frames, and select the key frame corresponding to the target video according to the center of the at least one type of video frames.

[0018] Optionally, the filtering module is also used to: filter the multiple video frames according to the first feature information of the multiple video frames according to the preset spatial dimension filtering rules, so as to filter out low-quality video frames in the spatial dimension and obtain multiple optional video frames; filter the multiple optional video frames according to the second feature information corresponding to the multiple optional video frames according to the preset time dimension filtering rules, so as to filter out low-quality video frames in the time dimension.

[0019] Optionally, the first feature information includes: picture brightness value, picture sharpness value and picture balance value; and the filtering module is also used to: for one video frame among the multiple video frames, if the first feature information of the one video frame meets at least one of the following options, then filter out the one video frame: the picture brightness value of the one video frame is not greater than a preset brightness threshold; the picture sharpness value of the one video frame is not greater than a preset sharpness threshold; the picture balance value of the one video frame is not less than a preset balance threshold.

[0020] Optionally, the device also includes a calculation module, which is used to: calculate the image brightness value of the video frame according to the RGB value of the color channel of the video frame; introduce the pixel gradient of the video frame according to the grayscale value of the video frame to calculate the sharpness value of the video frame; and perform histogram equalization processing on the video frame to obtain the grayscale level after the histogram equalization processing, and then calculate the image equalization value of the video frame according to the grayscale level after the histogram equalization processing.

[0021] Optionally, the second characteristic information includes: an inter-frame differential value and a lens edge change rate; and the filtering module is also used to: for an optional video frame among the multiple optional video frames, if the second characteristic information corresponding to the one optional video frame meets at least one of the following options, then filter out the one optional video frame: the inter-frame differential value corresponding to the one optional video frame is greater than a preset inter-frame differential threshold; the lens edge change rate corresponding to the one optional video frame is greater than a preset lens edge change threshold.

[0022] Optionally, the calculation module is also used to: calculate a first difference between the optional video frame and a previous optional video frame corresponding to the optional video frame, and calculate a second difference between the optional video frame and a subsequent optional video frame corresponding to the optional video frame; normalize the first difference and the second difference, and calculate the inter-frame differential value corresponding to the optional video frame using the first difference and the second difference after normalization.

[0023] Optionally, the calculation module is also used to: perform dilation processing on the edge picture of the next optional video frame corresponding to the one optional video frame, and calculate the fade-in edge change rate between the edge picture of the next optional video frame after the dilation processing and the edge picture of the one optional video frame; perform dilation processing on the edge picture of the previous optional video frame corresponding to the one optional video frame, and calculate the fade-out edge change rate between the edge picture of the previous optional video frame after the dilation processing and the edge picture of the one optional video frame; select the maximum value of the fade-in edge change rate and the fade-out edge change rate as the lens edge change rate corresponding to the one optional video frame.

[0024] Optionally, the calculation module is also used to: perform an AND operation on the edge image of the subsequent optional video frame after the expansion processing and the edge image of the optional video frame, and then sum them to obtain a first pixel point sum value; perform a dot division on the first pixel point sum value and the pixel point sum value corresponding to the edge image of the optional video frame, and then calculate the difference with 1 to obtain the fade-in edge change rate corresponding to the optional video frame.

[0025] Optionally, the calculation module is also used to: perform an AND operation on the edge image of the previous optional video frame after the expansion process and the edge image of the optional video frame, and then sum them to obtain a second pixel point sum value; perform a dot division on the second pixel point sum value and the pixel point sum value corresponding to the edge image of the optional video frame, and then calculate the difference with 1 to obtain the fade-out edge change rate corresponding to the optional video frame.

[0026] Optionally, the selection module is also used to: determine the number of shots based on a preset step size and according to the multiple video frames after filtering; use a clustering algorithm to cluster the multiple video frames after filtering according to the number of shots to obtain at least one type of video frames, and determine the center of at least one type of video frames; for each type of video frame, select the video frame closest to the center of that type of video frame as the key frame corresponding to that type of video frame, so as to obtain the key frame corresponding to the target video.

[0027] To achieve the above objective, according to another aspect of an embodiment of the present invention, an electronic device is provided.

[0028] An electronic device according to an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the key frame selection method according to the embodiment of the present invention.

[0029] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer readable medium is provided.

[0030] A computer-readable medium according to an embodiment of the present invention stores a computer program, and when the program is executed by a processor, the key frame selection method according to the embodiment of the present invention is implemented.

[0031] An embodiment of the above invention has the following advantages or beneficial effects: multiple video frames corresponding to the target video can be filtered according to preset spatial dimension filtering rules and temporal dimension filtering rules, and low-quality video frames can be filtered out in the spatial dimension and temporal dimension, solving the problems of large errors and large computing resource consumption in the prior art; clustering technology is used to classify the multiple video frames after filtering, and then key frames are selected based on the center of each type of video frame, which can remove redundant information from a clustering perspective, solving the problem of redundant information in the selected key frames in the prior art, so that key frames can be selected quickly and accurately, further solving the problem of missing frames in the prior art.

[0032] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0034] Figure 1 is a schematic diagram of the main steps of the key frame selection method according to an embodiment of the present invention;

[0035] Figure 2 is a schematic diagram of main steps of filtering multiple video frames according to an embodiment of the present invention;

[0036] Figure 3 is a schematic diagram of a main process of selecting a key frame from a plurality of video frames after filtering according to an embodiment of the present invention;

[0037] Figure 4 is a schematic diagram of the main process of the key frame selection method according to an embodiment of the present invention;

[0038] Figure 5 is a schematic diagram of main modules of a key frame selection device according to an embodiment of the present invention;

[0039] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0040] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0042] Video keyframes can be seen as images that can reflect the main content of the video. For many application scenarios, how to select keyframes from thousands of video frames is of great significance. For example, in the video review scenario, when reviewing the content of a video file, it was originally necessary to review the entire video lens, but through keyframe selection technology, only the selected keyframe images can be reviewed, which can reduce costs, improve efficiency, and reduce the interference of other redundant information. In addition to video review, the application scenarios of keyframe selection technology can also include video retrieval, video structuring, video summarization, etc.

[0043] In the prior art, key frame selection methods mainly include: feature-based key frame selection method, clustering-based key frame selection method and comprehensive algorithm-based key frame selection method. Among them, the feature-based key frame selection method has the problems of slow selection speed and large amount of calculation, and there is redundant information in the selected key frames, resulting in low quality of pictures corresponding to the key frames; the key frames selected by the clustering-based key frame selection method have large errors, and it is necessary to use a network model to extract features, resulting in large consumption of computing resources, and there is redundant information in the selected key frames, resulting in low quality of pictures corresponding to the key frames; the key frame selection method based on the comprehensive algorithm is faster in actual application, but the frame leakage is more serious. And this method needs to use the underlying encoding and decoding method, which will cause redundant information in the selected key frames, thereby causing blurred pictures.

[0044] In order to solve the above problems, an embodiment of the present invention provides a key frame selection method, which can realize key frame selection quickly and accurately. Figure 1 FIG. 1 is a schematic diagram of the main steps of the key frame selection method according to an embodiment of the present invention. Figure 1 As shown, the main steps of the key frame selection method may include steps S101 to S103.

[0045] Step S101: receiving a key frame selection request, determining a target video according to the selection request, and acquiring a plurality of video frames corresponding to the target video.

[0046] The target video refers to the video for which key frames need to be selected. In an embodiment of the present invention, the target video can be determined directly according to the key frame selection request. Then, an open source tool can be used to intercept the target video at a certain interval to obtain multiple video frames corresponding to the target video, such as 24 video frames for 1 second of video. On the other hand, the open source tool can be OpenCV, which consists of a series of C functions and a small number of C++ classes, and implements many general algorithms in image processing and computer vision.

[0047] Step S102: Filter multiple video frames according to preset filtering rules and feature information of multiple video frames. The filtering rules may include: spatial dimension filtering rules and temporal dimension filtering rules. Feature information refers to features of a video frame, such as brightness and darkness of a picture.

[0048] Considering the spatial dimension (i.e., the image content), multiple video frames include darker video frames, blurred video frames, and unbalanced video frames. Blurry video frames are low-quality video frames, and their image content is blurred; darker video frames and unbalanced video frames will introduce errors in the recognition of image content and increase the amount of calculation. Among them, if the image content quality corresponding to a video frame is low and cannot reflect the main content of the video, then the video frame is a low-quality video frame. Therefore, in the key frame selection method, it is necessary to filter out darker video frames, blurred video frames, and unbalanced video frames. In an embodiment of the present invention, multiple video frames can be filtered according to the spatial dimension filtering rules in combination with the feature information of the video frames, so as to filter out low-quality video frames in the spatial dimension.

[0049] Each video is composed of different lenses, that is, when each video switches from one lens to another, a lens conversion frame will appear. On the other hand, there are some special effects during the lens conversion, such as fade-in and fade-out (i.e., the gradual appearance and disappearance of the video screen; the screen changes from dark to bright and finally becomes completely clear, the beginning of this shot is called fade-in, also called gradual appearance; the screen changes from bright to dark and completely disappears, the end of this shot is called fade-out, also called gradual disappearance). Lens conversion frames and video frames corresponding to the special effects part during lens conversion do not belong to the range of high-quality video frames. Considering the time dimension, these video frames can be filtered out by comparing the previous and next video frames. In an embodiment of the present invention, multiple video frames can be filtered according to the time dimension filtering rules and combined with the feature information of the video frames, so as to filter out low-quality video frames in the time dimension.

[0050] Step S103: clustering the filtered video frames to obtain at least one type of video frames, and selecting key frames corresponding to the target video according to the center of the at least one type of video frames.

[0051] The above step S102 can filter out low-quality video frames from the spatial dimension and the temporal dimension to obtain high-quality video frames. Step S103 is mainly used to remove redundant frames from the obtained high-quality video frames, and finally obtain key frames. Specifically, a clustering algorithm can be first used to process the filtered multiple video frames to obtain at least one type of video frames; then, the center of each type of video frame is used to select the key frames corresponding to each type of video frame from each type of video frame, that is, the key frames corresponding to the target video.

[0052] The key frame selection technology of the embodiment of the present invention can filter the multiple video frames corresponding to the target video according to the preset spatial dimension filtering rules and the temporal dimension filtering rules, and can filter out low-quality video frames in the spatial dimension and the temporal dimension, thereby solving the problems of large errors and large computing resource consumption in the prior art. The clustering technology is used to classify the multiple video frames after filtering, and then the key frames are selected based on the center of each type of video frame, thereby removing redundant information from the clustering perspective, solving the problem of redundant information in the selected key frames in the prior art. Therefore, the key frames can be selected quickly and accurately, further solving the problem of frame missing in the prior art.

[0053] Filtering out low-quality video frames from multiple video frames is the basis for subsequent key frame selection using clustering technology. Figure 2 FIG. 1 is a schematic diagram of the main steps of filtering multiple video frames according to an embodiment of the present invention. Figure 2 As shown, the main steps of filtering multiple video frames may include:

[0054] Step S201, filtering the multiple video frames according to the preset spatial dimension filtering rule and the first feature information of the multiple video frames, so as to filter out low-quality video frames in the spatial dimension and obtain multiple optional video frames;

[0055] Step S202 , filtering the multiple optional video frames according to the preset time dimension filtering rule and the second feature information corresponding to the multiple optional video frames, so as to filter out low-quality video frames in the time dimension.

[0056] The filtering process of multiple video frames in the embodiment of the present invention mainly includes filtering process from the spatial dimension and filtering process from the temporal dimension. As mentioned above, considering from the spatial dimension (i.e., the content of the picture), multiple video frames include darker video frames, blurred video frames, and unbalanced video frames. Since there are a large number of motion-blurred video frames, as well as darker video frames and unbalanced video frames in the video, in order to reduce the amount of calculation, as a preferred embodiment of the present invention, low-quality video frames can be first filtered out from the spatial dimension, and then low-quality video frames can be filtered out from the temporal dimension. On the other hand, for low-quality video frames in the temporal dimension, they can be filtered out by comparing the previous and next video frames. Therefore, filtering out low-quality video frames from the spatial dimension first can ensure that the filtered video frames are high-quality video frames to a certain extent (i.e., in the spatial dimension), and then filtering out low-quality video frames from the temporal dimension, thereby reducing the error of key frame selection. Next, the process of filtering low-quality video frames from the spatial dimension and the temporal dimension is specifically described.

[0057] (I) Filtering low-quality video frames from the spatial dimension

[0058] In an embodiment of the present invention, the first feature information may include: picture brightness value, picture sharpness value and picture balance value. And, according to the preset spatial dimension filtering rule, according to the first feature information of multiple video frames, filtering multiple video frames may include: for a video frame among multiple video frames, if the first feature information of a video frame meets at least one of the following options, then filtering out a video frame: the picture brightness value of a video frame is not greater than the preset brightness threshold; the picture sharpness value of a video frame is not greater than the preset sharpness threshold; the picture balance value of a video frame is not less than the preset balance threshold. In other words, the picture brightness value, picture sharpness value and picture balance value of the video frame can be used to filter out the darker video frames, blurred video frames and unbalanced video frames among the multiple video frames corresponding to the target video. The filtering process is described in detail below.

[0059] a. Filter darker video frames

[0060] Since darker video frames will introduce errors in the recognition of image content and increase the amount of calculation, they are low-quality video frames, so it is necessary to filter out the darker video frames in the multiple video frames corresponding to the target video. In an embodiment of the present invention, filtering can be performed according to the image brightness value of the video frame (that is, the brightness of the image corresponding to the video frame). The video frame is composed of three channels of RGB (red, green, and blue). Different weights are assigned to each channel to obtain a three-channel brightness map corresponding to the video frame. Then, the pixels of the three-channel brightness map are normalized and averaged. Finally, the image brightness value of the video frame is obtained. The calculation formula is as follows:

[0061]

[0062] Where y1 represents the brightness value of a video frame; i and j represent the pixel points of the image corresponding to the video frame; m and n represent the width and height of the image corresponding to the video frame respectively; I r ,I g and I b They represent the red channel, green channel and blue channel corresponding to the video frame respectively; f1, f2 and f3 represent the weights corresponding to different channels respectively. On the other hand, the range of the brightness value of the video frame calculated by formula (1) is between [0,1], and the larger the brightness value of the image, the brighter the image corresponding to the video frame.

[0063] For a video frame, the image brightness value of the video frame can be calculated according to formula (1), and then it is determined whether the image brightness value of the video frame is not greater than a preset brightness threshold T1 (which can be but is not limited to 0.08); if so, it is confirmed that the video frame is a low-quality video frame and the video frame is filtered out; if not, it is confirmed that the brightness verification of the video frame has passed.

[0064] b. Filter blurred video frames

[0065] Since the image content corresponding to the blurred video frame is blurred and belongs to a low-quality video frame, it is necessary to filter out the blurred video frames in the multiple video frames corresponding to the target video, that is, filtering can be performed according to the blurriness of the image content. From the analysis of the image content corresponding to the video frame, if the sharpness value of the image content is higher, the contrast of the image details is higher, and the image is clearer. Conversely, if the sharpness value of the image content is lower, the contrast of the image details is lower, and the image is blurrier. Therefore, the image sharpness value of the video frame can be used to determine the blurriness of the image content corresponding to the video frame. On the other hand, for the determination of blurred video frames, the influence of color information on the blurriness can be weakened, so grayscale information can be used for determination; in addition, the pixel gradient represents the horizontal and vertical changes of the pixel points, and the adjacent pixels of the blurred image change less, that is, the pixel gradient is smaller. In summary, in the calculation of the image sharpness value of the video frame, the pixel gradient can be introduced as a multiplicative factor of the grayscale value of the video frame. The calculation formula is as follows:

[0066]

[0067] In order to improve the calculation speed of the algorithm, formula (2) can be simplified to the following formula:

[0068]

[0069] Where y2 represents the sharpness value of a video frame; i and j represent the pixel points of the image corresponding to the video frame; m and n represent the width and height of the image corresponding to the video frame respectively; I gray (i, j) represents the grayscale value of the pixel point of the image corresponding to the video frame; Δx and Δy represent the horizontal and vertical gradients of the pixel point of the image corresponding to the video frame, respectively. The image sharpness value of the video frame calculated by formula (3) is in the range of [0, 1], and the larger the image sharpness value, the clearer the image corresponding to the video frame.

[0070] For a video frame, the image sharpness value of the video frame can be calculated according to formula (3), and then it is determined whether the image sharpness value of the video frame is not greater than a preset brightness threshold T2 (which can be but is not limited to 0.1); if so, it is confirmed that the video frame is a low-quality video frame and the video frame is filtered out; if not, it is confirmed that the blurriness verification of the video frame has passed.

[0071] c. Filtering unbalanced video frames

[0072] Since unbalanced video frames will introduce errors in the recognition of image content and increase the amount of calculation, they are low-quality video frames, so it is necessary to filter out the unbalanced video frames in the multiple video frames corresponding to the target video. In an embodiment of the present invention, filtering can be performed according to the picture balance value of the video frame (that is, the degree of balance or uniformity of the picture corresponding to the video frame). In an embodiment of the present invention, a video frame can first be subjected to histogram equalization processing to obtain the grayscale after equalization, and then the picture balance value of a video frame can be calculated based on the grayscale after the histogram equalization processing.

[0073] Among them, the picture histogram (i.e. a two-dimensional statistical chart whose two coordinates are the statistical sample and the measurement of a certain attribute corresponding to the sample) is a histogram used to represent the brightness distribution in a digital picture. It plots the number of pixels of each brightness value in the picture, which can be used to characterize the distribution of pixel values ​​of the picture. Histogram equalization is a method of achieving image enhancement by realizing grayscale mapping with the help of histogram transformation. It is a very good method for normalizing the grayscale value of the image and can enhance the contrast of the image. Therefore, after the histogram equalization processing of the video frame, the overall contrast of the picture can be better observed, that is, the degree of balance of the picture corresponding to the video frame can be better observed. The following formula can be used to calculate the picture balance value of the video frame:

[0074]

[0075] Wherein, y3 represents the picture equalization value of a video frame; I gray Indicates the grayscale value of the picture corresponding to the video frame; norm_hist(I gray ) indicates that the image corresponding to the video frame is subjected to histogram equalization. Since the histogram equalization is a prior art, it will not be described in detail here. After the grayscale value of the image is subjected to histogram equalization, a new grayscale can be obtained. For example, the grayscale range of the image corresponding to a certain video frame is an integer in [0,7]. After the histogram equalization is performed, the new grayscale is [1,3,5,6,7]. It should be noted that the above example is only for the sake of understanding and no normalization is performed. Then, the new grayscale is sorted from large to small, and the sum of the grayscales of the top f4% is selected as the image equalization value of the video frame, where f4 is a preset parameter value. For example, f4 is selected as 5, that is, the sum of the grayscales of the top 5% is selected as the image equalization value of the video frame. The range of the image equalization value of the video frame calculated by formula (4) is between [0,1], and the larger the image equalization value, the more unbalanced the image corresponding to the video frame is in grayscale.

[0076] For a video frame, the picture balance value of the video frame can be calculated according to formula (4), and then it is determined whether the picture balance value of the video frame is not less than a preset balance threshold T3 (which can be but is not limited to 0.8); if so, it is determined that the video frame is a low-quality video frame and the video frame is filtered out; if not, it is determined that the balance degree verification of the video frame has passed.

[0077] In summary, in the method of filtering multiple video frames from the spatial dimension, the darker video frames can be filtered out using the image brightness value of the video frame, the blurred video frames can be filtered out using the image sharpness value of the video frame, and the unbalanced video frames can be filtered out using the image balance value of the video frame. The specific filtering order is: (1) Each video frame can be analyzed, and if the video frame belongs to a darker video frame, a blurred video frame or an unbalanced video frame, the video frame is directly filtered out; (2) The darker video frame, the blurred video frame and the unbalanced video frame can also be regarded as three different types, and the low-quality video frames in the spatial dimension are filtered out according to different types, such as first filtering out all the darker video frames in the multiple video frames, then filtering out all the blurred video frames in the multiple video frames, and finally filtering out all the unbalanced video frames in the multiple video frames. On the other hand, considering that there are a large number of blurred video frames caused by motion in the multiple video frames corresponding to the target video, in order to reduce the amount of calculation, as a preferred embodiment of the present invention, the blurred video frames can be filtered out first, then the darker video frames can be filtered out, and finally the unbalanced video frames can be filtered out. Of course, three different types of filtering orders may also be set according to actual needs, which is not limited in the embodiment of the present invention.

[0078] In an embodiment of the present invention, darker video frames, blurred video frames or unbalanced video frames are filtered out from the spatial dimension, which can reduce the number of video frames included in multiple video frames, and further, can reduce the amount of calculation for subsequent key frame selection. On the other hand, low-quality video frames are first filtered out from the spatial dimension, which can ensure to a certain extent (i.e., in the spatial dimension) that the filtered video frames are high-quality video frames, providing a basis for subsequent filtering of low-quality video frames from the temporal dimension, thereby reducing the error in key frame selection.

[0079] (II) Filtering low-quality video frames from the temporal dimension

[0080] In an embodiment of the present invention, the second characteristic information may include: an inter-frame differential value and a lens edge change rate. Also, according to a preset time dimension filtering rule, filtering multiple optional video frames according to the second characteristic information corresponding to multiple optional video frames may include: for an optional video frame among multiple optional video frames, if the second characteristic information corresponding to an optional video frame meets at least one of the following options, then filtering out an optional video frame: an inter-frame differential value corresponding to an optional video frame is greater than a preset inter-frame differential threshold; a lens edge change rate corresponding to an optional video frame is greater than a preset lens edge change threshold. In other words, the inter-frame differential value and lens edge change rate of the video frame can be used to filter out the lens conversion frame and the video frame corresponding to the special effect part during the lens conversion. The filtering process is described in detail below.

[0081] d. Filter out the lens conversion frames

[0082] Each video is obtained by combining different lenses, and the video frames at the lens turn point are low-quality video frames. After filtering out the low-quality video frames from the spatial dimension, multiple optional video frames are obtained, and then the inter-frame differential value can be used to determine whether an optional video frame is a low-quality video frame. Among them, the inter-frame differential value can represent the difference change between a video frame and the previous video frame and the next video frame. In an embodiment of the present invention, the inter-frame differential value can be calculated according to the following method: calculate the first difference between an optional video frame and the previous optional video frame corresponding to an optional video frame, and calculate the second difference between an optional video frame and the next optional video frame corresponding to an optional video frame; normalize the first difference and the second difference, and use the normalized first difference and second difference to calculate the inter-frame differential value corresponding to an optional video frame. The specific calculation formula is as follows:

[0083]

[0084] Among them, y4(k) represents the inter-frame differential value corresponding to the k-th optional video frame; m and n represent the width and height of the picture corresponding to the video frame respectively; I(k) represents the picture pixel corresponding to the k-th optional video frame; I(k-1) represents the picture pixel corresponding to the k-1-th optional video frame (i.e., the previous optional video frame corresponding to the k-th optional video frame); I(k+1) represents the picture pixel corresponding to the k+1-th optional video frame (i.e., the next optional video frame corresponding to the k-th optional video frame); norm represents normalization processing.

[0085] If the inter-frame differential value corresponding to an optional video frame is larger, it means that the possibility that the optional video frame is a shot transition frame is greater. For an optional video frame, the inter-frame differential value corresponding to the optional video frame can be calculated according to formula (5), and then it is determined whether the inter-frame differential value corresponding to the optional video frame is greater than a preset inter-frame differential threshold T4; if so, it is determined that the optional video frame is a low-quality video frame and the optional video frame is filtered out.

[0086] It should be noted that the setting of T4 can be combined with the inter-frame differential value setting corresponding to all optional video frames. Specifically, firstly, the inter-frame differential values ​​corresponding to all optional video frames are calculated, and the calculated inter-frame differential values ​​are sorted from large to small; then, the optional video frames with the highest sorting are selected, such as the first 10% of the optional video frames in the sorting; then, T4 is set according to the inter-frame differential values ​​corresponding to the selected optional video frames, and the optional video frames with inter-frame differential values ​​greater than T4 are filtered out.

[0087] e. Filter out the video frames corresponding to the special effects during the lens transition

[0088] During the lens conversion, there is a fade-in and fade-out special effect, and the video frame corresponding to the fade-in and fade-out special effect is at the beginning or end of a certain lens, which is a low-quality video frame in the time dimension and needs to be filtered out. In order to filter out the video frames corresponding to the fade-in and fade-out special effects, the concept of lens edge change rate is introduced, that is, for the fade-in and fade-out scenes, the edges of the front and rear video frames are extracted respectively, and then the edge changes are analyzed to determine whether it is a fade-in and fade-out special effect. In an embodiment of the present invention, the lens edge change rate corresponding to an optional video frame can be calculated according to the following method: the edge picture of the next optional video frame corresponding to an optional video frame is expanded, and the fade-in edge change rate between the edge picture of the next optional video frame after the expansion process and the edge picture of an optional video frame is calculated; the edge picture of the previous optional video frame corresponding to an optional video frame is expanded, and the fade-out edge change rate between the edge picture of the previous optional video frame after the expansion process and the edge picture of an optional video frame is calculated; the maximum value of the fade-in edge change rate and the fade-out edge change rate is selected as the lens edge change rate corresponding to an optional video frame. The specific implementation can be:

[0089] (1) dilating an edge image of a subsequent optional video frame corresponding to an optional video frame, performing an AND operation on the edge image after the dilation process and the edge image of the optional video frame, and then summing the two images to obtain a first pixel point sum value;

[0090] (2) dividing the first pixel point sum value by the pixel point sum value corresponding to the edge image of an optional video frame, and then calculating the difference between the first pixel point sum value and 1 to obtain a fade-in edge change rate corresponding to the optional video frame;

[0091] (3) dilating an edge image of a previous optional video frame corresponding to an optional video frame, performing an AND operation on the edge image after the dilation process and the edge image of the optional video frame, and then summing them to obtain a second pixel point sum value;

[0092] (4) dividing the second pixel point sum value by the pixel point sum value corresponding to the edge image of an optional video frame, and then calculating the difference between the second pixel point sum value and 1 to obtain a fade-out edge change rate corresponding to the optional video frame;

[0093] (5) The maximum value of the fade-in edge change rate and the fade-out edge change rate is selected as the lens edge change rate corresponding to an optional video frame.

[0094] Among them, (1) and (2) are used to calculate the fade-in edge change rate. Specifically, the fade-in edge change rate corresponding to the kth optional video frame can be calculated as follows: first, the edge image of the k+1th optional video frame is expanded, that is, the k+1th optional video frame is blurred. For the fade-in scene, the change is from blurry to clear. Therefore, the edge image of the k+1th optional video frame is expanded to simulate the fade-in result; then, the edge image of the k+1th optional video frame after the expansion process is ANDed with the edge image of the kth optional video frame, that is, the intersection between the edge contour of the kth optional video frame and the blurred edge contour of the k+1th optional video frame is obtained; then, the pixel points obtained after the AND operation are summed to obtain the first pixel point sum value, and the third pixel point sum value corresponding to the edge image of the kth optional video frame is calculated; finally, the first pixel point sum value is divided by the third pixel point sum value, and then the difference is calculated with 1 to obtain the fade-in edge change rate corresponding to the kth optional video frame. The specific calculation formula is as follows:

[0095]

[0096] Among them, y in (k) represents the fade-in edge change rate corresponding to the kth optional video frame; edge (k) represents the pixel corresponding to the edge image of the kth optional video frame; I edge_dilate (k+1) represents the pixel point obtained after dilating the edge image of the k+1th optional video frame. If the overlap between the edge contour of the kth optional video frame and the blurred edge contour of the k+1th optional video frame is smaller, the calculated fade-in edge change rate is greater, that is, the kth optional video frame is more likely to be the video frame corresponding to the special effect part during the lens conversion.

[0097] (3) and (4) are used to calculate the fade-out edge change rate. Specifically, the fade-out edge change rate corresponding to the kth optional video frame can be calculated as follows: first, the edge image of the k-1th optional video frame is expanded, that is, the k-1th optional video frame is blurred. For the fade-out scene, the change is from clear to blurred. Therefore, the edge image of the k-1th optional video frame is expanded to simulate the fade-out result; then, the edge image of the k-1th optional video frame after the expansion is ANDed with the edge image of the kth optional video frame, that is, the intersection between the edge contour of the kth optional video frame and the blurred edge contour of the k-1th optional video frame is obtained; then, the pixel points obtained after the AND operation are summed to obtain the second pixel point sum value, and the third pixel point sum value corresponding to the edge image of the kth optional video frame is calculated; finally, the second pixel point sum value is divided by the third pixel point sum value, and then the difference is calculated with 1 to obtain the fade-out edge change rate corresponding to the kth optional video frame. The specific calculation formula is as follows:

[0098]

[0099] Among them, y out (k) represents the fade-out edge change rate corresponding to the kth optional video frame; edge (k) represents the pixel corresponding to the edge image of the kth optional video frame; I edge_dilate (k-1) represents the pixel point obtained after dilating the edge image of the k-1th optional video frame. If the overlap between the edge contour of the kth optional video frame and the blurred edge contour of the k-1th optional video frame is smaller, the calculated fade-out edge change rate is greater, that is, the kth optional video frame is more likely to be the video frame corresponding to the special effect part during the lens conversion.

[0100] Through the above analysis, it is known that the larger the fade-in edge change rate or fade-out edge change rate corresponding to a selectable video frame is, the greater the possibility that the selectable video frame is the video frame corresponding to the special effect part during the lens transition. Therefore, in (5), the maximum value of the fade-in edge change rate and the fade-out edge change rate is selected as the lens edge change rate corresponding to the selectable video frame.

[0101] For an optional video frame, the fade-in edge change rate and the fade-out edge change rate corresponding to the optional video frame can be calculated according to formula (6) and formula (7), and then the maximum value of the fade-in edge change rate and the fade-out edge change rate is selected as the lens edge change rate corresponding to the optional video frame. Then, it is determined whether the lens edge change rate corresponding to the optional video frame is greater than a preset lens edge change threshold T5; if so, the optional video frame is determined to be a low-quality video frame and the optional video frame is filtered out.

[0102] In summary, in the method of filtering multiple optional video frames from the time dimension, the lens conversion frames can be filtered out using the inter-frame differential value of the optional video frame, and the video frames corresponding to the special effects part during the lens conversion period can be filtered out using the lens edge change rate of the optional video frame. The specific filtering order is: (1) Each optional video frame can be analyzed, and if the optional video frame belongs to the lens conversion frame or the video frame corresponding to the special effects part during the lens conversion period, the optional video frame is directly filtered out; (2) the lens conversion frame or the video frame corresponding to the special effects part during the lens conversion period can also be regarded as two different types, and the low-quality video frames in the time dimension are filtered out according to the different types, such as first filtering out all the lens conversion frames in the multiple optional video frames, and then filtering out all the video frames corresponding to the special effects part during the lens conversion period in the multiple optional video frames.

[0103] In an embodiment of the present invention, by filtering out the shot conversion frames or the video frames corresponding to the special effects part during the shot conversion from the time dimension, the number of video frames included in the multiple video frames can be reduced. Further, the amount of calculation for subsequent key frame selection can be reduced, thereby improving the efficiency of key frame selection.

[0104] Low-quality video frames are filtered out from multiple video frames from the spatial dimension and the temporal dimension, thereby reducing the number of video frames. Then, key frames are selected from the filtered multiple video frames, which can improve the efficiency of key frame selection and reduce key frame selection errors. Figure 3 FIG. 1 is a schematic diagram of the main process of selecting key frames from a plurality of video frames after filtering according to an embodiment of the present invention. Figure 3 As shown, the main process of selecting key frames from multiple video frames after filtering may include:

[0105] Step S301, based on a preset step length, the number of shots is determined according to a plurality of video frames after filtering;

[0106] Step S302, using a clustering algorithm, clustering the filtered multiple video frames according to the number of shots to obtain at least one type of video frames, and determining the center of at least one type of video frames;

[0107] Step S303 : for each type of video frame, a video frame closest to the center of the video frame of this type is selected as a key frame corresponding to the video frame of this type, so as to obtain a key frame corresponding to the target video.

[0108] Among them, the preset step size refers to the preset time step size. After filtering the low-quality video frames in the target video from the spatial dimension and the time dimension, the filtered multiple video frames are analyzed from the perspective of the lens. There will be a large frame interval before each lens, that is, there will be a certain time step between different lenses, so the filtered multiple video frames can be decomposed into several video lenses in combination with the preset time step. Then, the clustering algorithm can be used to cluster the filtered multiple video frames according to the number of lenses to obtain at least one type of video frames and determine the center of at least one type of video frames. Among them, the number of lenses is the same as the number of classes of the obtained video frames. Then, for each type of video frame, the video frame closest to the center of the video frame of this type is selected as the key frame corresponding to the video frame of this type, and finally the key frame corresponding to the target video can be obtained. On the other hand, the underlying features of the image content, such as the color histogram, can be used to extract features from each filtered video frame, and the filtered multiple time frames can be clustered based on the K-means clustering algorithm to obtain K types of video frames, where K is equal to the number of lenses.

[0109] Figure 4 FIG. 1 is a schematic diagram of the main process of the key frame selection method according to an embodiment of the present invention. Figure 4 As shown, the main process of the key frame selection method may include: first, obtaining multiple video frames corresponding to the target video; then, filtering out darker video frames, blurred video frames and unbalanced video frames from the multiple video frames from the spatial dimension; then, filtering out the lens conversion frames and the video frames corresponding to the special effects part during the lens conversion from the multiple video frames from the temporal dimension; then, clustering the multiple filtered video frames to obtain at least one category of video frames; finally, for each category of video frames, selecting the video frame closest to the center of the video frame of that category as the key frame corresponding to the video frame of that category, to obtain the key frame corresponding to the target video.

[0110] The key frame selection method of the embodiment of the present invention can filter the multiple video frames corresponding to the target video according to the preset spatial dimension filtering rules and the temporal dimension filtering rules, and can filter out low-quality video frames in the spatial dimension and the temporal dimension, solving the problems of large errors and large computing resource consumption in the prior art; clustering technology is used to classify the multiple video frames after filtering, and then the key frames are selected based on the center of each type of video frame, which can remove redundant information from the clustering perspective, solving the problem of redundant information in the selected key frames in the prior art, so that the key frames can be selected quickly and accurately, further solving the problem of missing frames in the prior art.

[0111] Figure 5 Schematic diagram of the main modules of the key frame selection device according to an embodiment of the present invention. Figure 5As shown, the key frame selection device 500 of the embodiment of the present invention may include: an acquisition module 501, a filtering module 502 and a selection module 503.

[0112] The acquisition module 501 may be used to: receive a key frame selection request, determine a target video according to the selection request, and obtain multiple video frames corresponding to the target video; the filtering module 502 may be used to: filter multiple video frames according to the preset filtering rules and feature information of the multiple video frames; the selection module 503 may be used to: cluster the multiple video frames after filtering to obtain at least one type of video frames, and select the key frames corresponding to the target video according to the center of at least one type of video frames. The filtering rules may include: spatial dimension filtering rules and temporal dimension filtering rules.

[0113] In an embodiment of the present invention, the filtering module 502 can also be used to: filter the multiple video frames according to the first feature information of the multiple video frames according to the preset spatial dimension filtering rules, so as to filter out low-quality video frames in the spatial dimension and obtain multiple optional video frames; filter the multiple optional video frames according to the second feature information corresponding to the multiple optional video frames according to the preset time dimension filtering rules, so as to filter out low-quality video frames in the time dimension.

[0114] In an embodiment of the present invention, the first feature information may include: a picture brightness value, a picture sharpness value, and a picture balance value. Furthermore, the filtering module 502 may also be used to: for a video frame among multiple video frames, if the first feature information of a video frame meets at least one of the following options, then filter out the video frame: the picture brightness value of a video frame is not greater than a preset brightness threshold; the picture sharpness value of a video frame is not greater than a preset sharpness threshold; the picture balance value of a video frame is not less than a preset balance threshold.

[0115] In the embodiment of the present invention, the key frame selection device may further include: a calculation module (not shown in the figure). The calculation module may be used to: calculate the image brightness value of a video frame according to the RGB value of the color channel of a video frame; introduce the pixel gradient of a video frame according to the gray value of a video frame to calculate the sharpness value of a video frame; and perform histogram equalization processing on a video frame to obtain the gray level after the histogram equalization processing, and then calculate the image equalization value of a video frame according to the gray level after the histogram equalization processing.

[0116] In the embodiment of the present invention, the second feature information may include: an inter-frame differential value and a lens edge change rate. Furthermore, the filtering module 502 may also be used to: for an optional video frame among the multiple optional video frames, if the second feature information corresponding to the optional video frame meets at least one of the following options, then filter out the optional video frame: the inter-frame differential value corresponding to the optional video frame is greater than a preset inter-frame differential threshold; the lens edge change rate corresponding to the optional video frame is greater than a preset lens edge change threshold.

[0117] In an embodiment of the present invention, the calculation module can also be used to: calculate a first difference between an optional video frame and a previous optional video frame corresponding to the optional video frame, and calculate a second difference between an optional video frame and a subsequent optional video frame corresponding to the optional video frame; normalize the first difference and the second difference, and use the normalized first difference and second difference to calculate an inter-frame differential value corresponding to an optional video frame.

[0118] In the embodiment of the present invention, the calculation module can also be used to: perform dilation processing on the edge picture of the next optional video frame corresponding to an optional video frame, and calculate the fade-in edge change rate between the edge picture of the next optional video frame after the dilation processing and the edge picture of the optional video frame; perform dilation processing on the edge picture of the previous optional video frame corresponding to an optional video frame, and calculate the fade-out edge change rate between the edge picture of the previous optional video frame after the dilation processing and the edge picture of the optional video frame; select the maximum value of the fade-in edge change rate and the fade-out edge change rate as the lens edge change rate corresponding to an optional video frame.

[0119] In the embodiment of the present invention, the calculation module can also be used to: perform an AND operation on the edge image of the next optional video frame after the expansion process and the edge image of an optional video frame, and then sum them to obtain a first pixel point sum value; perform a dot division on the first pixel point sum value and the pixel point sum value corresponding to the edge image of an optional video frame, and then calculate the difference with 1 to obtain a fade-in edge change rate corresponding to an optional video frame.

[0120] In an embodiment of the present invention, the calculation module can also be used to: perform an AND operation on the edge image of the previous optional video frame after the expansion process and the edge image of an optional video frame, and then sum them to obtain a second pixel point sum value; perform a dot division on the second pixel point sum value and the pixel point sum value corresponding to the edge image of an optional video frame, and then calculate the difference with 1 to obtain the fade-out edge change rate corresponding to the optional video frame.

[0121] In an embodiment of the present invention, the selection module 503 can also be used to: determine the number of shots based on a preset step size and according to a plurality of video frames after filtering; use a clustering algorithm to cluster the plurality of video frames after filtering according to the number of shots to obtain at least one category of video frames, and determine the center of at least one category of video frames; for each category of video frames, select the video frame closest to the center of the video frame of that category as the key frame corresponding to the video frame of that category, so as to obtain the key frame corresponding to the target video.

[0122] The key frame selection device of the embodiment of the present invention can filter the multiple video frames corresponding to the target video according to the preset spatial dimension filtering rules and the temporal dimension filtering rules, and can filter out low-quality video frames in the spatial dimension and the temporal dimension, thereby solving the problems of large errors and large computing resource consumption in the prior art. The clustering technology is used to classify the multiple video frames after filtering, and then the key frames are selected based on the center of each type of video frame, thereby removing redundant information from the clustering perspective, solving the problem of redundant information in the selected key frames in the prior art, thereby enabling the key frames to be selected quickly and accurately, further solving the problem of frame omission in the prior art.

[0123] Figure 6 An exemplary system architecture 600 is shown to which the key frame selection method or key frame selection device according to the embodiment of the present invention can be applied.

[0124] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0125] The user can use the terminal devices 601, 602, 603 to interact with the server 605 through the network 604 to receive or send messages, etc. For example, the terminal devices 601, 602, 603 can be used to select key frames from multiple video frames.

[0126] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0127] The server 605 may be a server that provides various services, for example, a background management server that provides support when a user selects key frames using the terminal devices 601 , 602 , and 603 (for example only).

[0128] It should be noted that the key frame selection method provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the key frame selection device is generally disposed in the server 605 .

[0129] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0130] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0131] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0132] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0133] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0134] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0135] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0136] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes an acquisition module, a filtering module and a selection module. The names of these modules do not constitute a limitation on the modules themselves in some cases. For example, the acquisition module may also be described as "a module that receives a key frame selection request, determines a target video according to the selection request, and obtains multiple video frames corresponding to the target video".

[0137] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes: receiving a key frame selection request, determining a target video according to the selection request, and obtaining multiple video frames corresponding to the target video; filtering the multiple video frames according to the preset filtering rules and the feature information of the multiple video frames, wherein the filtering rules include spatial dimension filtering rules and time dimension filtering rules; clustering the multiple video frames after filtering to obtain at least one type of video frames, and selecting the key frames corresponding to the target video according to the center of at least one type of video frames.

[0138] According to the technical solution of the embodiments of the present invention, multiple video frames corresponding to the target video can be filtered according to preset spatial dimension filtering rules and temporal dimension filtering rules, and low-quality video frames can be filtered out in the spatial dimension and temporal dimension, solving the problems of large errors and large computing resource consumption in the prior art. Clustering technology is used to classify the multiple video frames after filtering, and then key frames are selected based on the center of each type of video frame, which can remove redundant information from a clustering perspective and solve the problem of redundant information in the selected key frames in the prior art. Therefore, key frames can be selected quickly and accurately, further solving the problem of frame omission in the prior art.

[0139] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A key frame selection method, characterized in that: include: Receiving a key frame selection request, determining a target video according to the selection request, and acquiring a plurality of video frames corresponding to the target video; According to a preset filtering rule, and according to the feature information of the multiple video frames, the multiple video frames are filtered, wherein the filtering rule includes a spatial dimension filtering rule and a temporal dimension filtering rule; specifically, the method includes: for one of the multiple optional video frames filtered by the temporal dimension filtering rule: if the lens edge change rate corresponding to the optional video frame is greater than a preset lens edge change threshold, filtering out the one optional video frame; Performing clustering processing on the multiple video frames after filtering processing to obtain at least one type of video frames, and selecting key frames corresponding to the target video according to the center of the at least one type of video frames; The step of determining the lens edge change rate comprises: performing expansion processing on an edge picture of a subsequent optional video frame corresponding to the one optional video frame, and calculating a fade-in edge change rate between the edge picture of the subsequent optional video frame after the expansion processing and the edge picture of the one optional video frame; Performing dilation processing on an edge picture of a previous optional video frame corresponding to the one optional video frame, and calculating a fade-out edge change rate between the edge picture of the previous optional video frame after the dilation processing and the edge picture of the one optional video frame; The maximum value of the fade-in edge change rate and the fade-out edge change rate is selected as the lens edge change rate corresponding to the one optional video frame.

2. The method according to claim 1, characterized in that The filtering process on the multiple video frames according to the preset filtering rules and the feature information of the multiple video frames includes: According to a preset spatial dimension filtering rule, and based on the first feature information of the multiple video frames, the multiple video frames are filtered to filter out low-quality video frames in the spatial dimension, thereby obtaining multiple optional video frames; According to a preset time dimension filtering rule, and based on the second feature information corresponding to the multiple optional video frames, the multiple optional video frames are filtered to filter out low-quality video frames in the time dimension.

3. The method according to claim 2, characterized in that The first feature information includes: picture brightness value, picture sharpness value and picture balance value; and, The filtering process of the plurality of video frames according to the preset spatial dimension filtering rule and the first feature information of the plurality of video frames includes: For one video frame among the multiple video frames, if the first feature information of the one video frame meets at least one of the following options, the one video frame is filtered out: the image brightness value of the one video frame is not greater than a preset brightness threshold; the image sharpness value of the one video frame is not greater than a preset sharpness threshold; the image balance value of the one video frame is not less than a preset balance threshold.

4. The method according to claim 3, characterized in that The method further comprises: Calculate the image brightness value of the video frame according to the RGB value of the color channel of the video frame; According to the grayscale value of the one video frame, the pixel gradient of the one video frame is introduced to calculate the sharpness value of the one video frame; and, A histogram equalization process is performed on the one video frame to obtain a grayscale after the histogram equalization process, and then a picture equalization value of the one video frame is calculated according to the grayscale after the histogram equalization process.

5. The method according to claim 2, characterized in that: The second feature information further includes: an inter-frame differential value; and The filtering process of the plurality of optional video frames according to the preset time dimension filtering rule and the second feature information corresponding to the plurality of optional video frames further includes: For one of the multiple selectable video frames, if the inter-frame differential value corresponding to the one selectable video frame is greater than a preset inter-frame differential threshold, the one selectable video frame is filtered out.

6. The method according to claim 5, characterized in that The method further comprises: Calculating a first difference between the one selectable video frame and a previous selectable video frame corresponding to the one selectable video frame, and calculating a second difference between the one selectable video frame and a subsequent selectable video frame corresponding to the one selectable video frame; The first difference and the second difference are normalized, and the inter-frame differential value corresponding to the one optional video frame is calculated using the normalized first difference and the second difference.

7. The method according to claim 1, characterized in that The step of calculating the fade-in edge change rate between the edge picture of the next optional video frame after the dilation process and the edge picture of the one optional video frame comprises: Performing an AND operation on the edge picture of the latter optional video frame after the dilation process and the edge picture of the one optional video frame, and then summing them to obtain a first pixel point sum value; The first pixel point sum value is divided by the pixel point sum value corresponding to the edge picture of the one optional video frame, and then the difference is calculated with 1 to obtain the fade-in edge change rate corresponding to the one optional video frame.

8. The method according to claim 1, characterized in that: The calculating of the fade-out edge change rate between the edge picture of the previous optional video frame after the dilation process and the edge picture of the optional video frame comprises: Perform an AND operation on the edge picture of the previous optional video frame after the dilation process and the edge picture of the optional video frame, and then sum them to obtain a second pixel point sum value; The second pixel point sum value is divided by the pixel point sum value corresponding to the edge picture of the one optional video frame, and then the difference is calculated with 1 to obtain the fade-out edge change rate corresponding to the one optional video frame.

9. The method according to any one of claims 1 to 8, characterized in that: The step of clustering the filtered video frames to obtain at least one type of video frames, and selecting key frames corresponding to the target video according to the center of the at least one type of video frames, comprises: Based on a preset step length, determining the number of shots according to the plurality of video frames after filtering; Using a clustering algorithm, clustering the plurality of video frames after filtering according to the number of shots to obtain at least one type of video frames, and determining the center of the at least one type of video frames; For each type of video frame, a video frame closest to the center of the video frame of this type is selected as a key frame corresponding to the video frame of this type, so as to obtain a key frame corresponding to the target video.

10. A key frame selection device, characterized in that: include: An acquisition module, configured to receive a key frame selection request, determine a target video according to the selection request, and acquire a plurality of video frames corresponding to the target video; A filtering module is used to filter the multiple video frames according to the preset filtering rules and the feature information of the multiple video frames; specifically comprising: for one of the multiple optional video frames filtered by the time dimension filtering rules: if the lens edge change rate corresponding to the optional video frame is greater than a preset lens edge change threshold, the optional video frame is filtered out; the lens edge change rate determination step comprises: dilating the edge picture of the next optional video frame corresponding to the optional video frame, and calculating the fade-in edge change rate between the edge picture of the next optional video frame after the dilating process and the edge picture of the optional video frame; dilating the edge picture of the previous optional video frame corresponding to the optional video frame, and calculating the fade-out edge change rate between the edge picture of the previous optional video frame after the dilating process and the edge picture of the optional video frame; selecting the maximum value of the fade-in edge change rate and the fade-out edge change rate as the lens edge change rate corresponding to the optional video frame; The selection module is used to perform clustering processing on the multiple video frames after filtering processing to obtain at least one type of video frames, and select key frames corresponding to the target video according to the center of the at least one type of video frames.

11. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

12. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images