A keyframe extraction method, apparatus, electronic device, and storage medium

Abstract

This invention provides a data filtering method, apparatus, electronic device, and storage medium, relating to the field of video processing technology. The method includes: weighted fusion of the semantic content of the video before and after a video frame based on a preset one-dimensional convolution kernel and the differences between two adjacent consecutive video frames in the target video; gradient change point extraction of the gradient changes in the fused video semantic content according to a preset gradient change point extraction rule to obtain target gradient change points; and using the video frame corresponding to the target gradient change point as a keyframe of the target video. This invention uses a preset one-dimensional convolution kernel to perform convolution, weightedly fusing the semantic content of the current video frame and the video frames before and after it, and using the video frame corresponding to the gradient change points of the fused video semantic content as a keyframe. This approach takes into account both the semantic content of the current video frame and the changes in the semantic content of the video frames before and after it, thus solving the technical problem of inaccurate keyframe extraction.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a keyframe extraction method, apparatus, electronic device, and storage medium. Background Technology

[0002] Video consists of continuously changing video frames. The frames that effectively represent the main content of the video are called keyframes. Keyframe extraction technology plays a crucial role in video processing. By extracting keyframes, redundant information can be effectively reduced, video processing efficiency can be improved, and subsequent tasks such as face recognition and object detection can be facilitated.

[0003] Existing keyframe extraction methods typically extract keyframes from videos at a fixed frame rate, for example, extracting two video frames every 30 seconds; these two frames are then considered keyframes. However, the semantic content of a video does not change uniformly, and this method fails to consider changes in the semantic content of the video and between consecutive video frames, thus leading to inaccurate keyframe extraction. Summary of the Invention

[0004] In view of this, the present invention aims to propose a keyframe extraction method, apparatus, electronic device and storage medium. By using a preset one-dimensional convolution kernel to perform convolution, the semantic content of the current video frame and the video frames before and after the current video frame are weighted and fused. The video frames corresponding to the gradient change points of the fused video semantic content are used as keyframes. By taking into account the semantic content of the current video frame and the changes in the semantic content of the video frames before and after the current video frame, the technical problem of inaccurate keyframe extraction is solved.

[0005] According to a first aspect of the present invention, a keyframe extraction method is provided, the method comprising:

[0006] Based on the preset one-dimensional convolution kernel and the differences between two adjacent consecutive video frames in the target video, the video semantic content before and after the video frame is weighted and fused to obtain the fused video semantic content.

[0007] The gradient changes of the fused video semantic content are obtained based on the fused video semantic content;

[0008] According to the preset gradient change point extraction rules, the gradient change points of the semantic content of the fused video are extracted to obtain the target gradient change points.

[0009] The video frames corresponding to the target gradient change points are used as keyframes of the target video.

[0010] Optionally, before the step of weighted fusing of video semantic content before and after a video frame based on a pre-set one-dimensional convolutional kernel and the differences between two adjacent consecutive video frames in the target video to obtain the fused video semantic content, the method further includes:

[0011] Sequentially acquire two consecutive adjacent video frames from the target video;

[0012] The structural similarity index of the two adjacent consecutive video frames is obtained according to the structural similarity algorithm, and the structural similarity index is stored in the structural similarity index array;

[0013] The structural similarity indices are normalized sequentially to obtain the difference indices between two adjacent consecutive video frames, and the difference indices are stored in the difference index array.

[0014] Optionally, the step of weighted fusion of video semantic content before and after a video frame based on a pre-set one-dimensional convolutional kernel and the differences between two adjacent consecutive video frames in the target video to obtain fused video semantic content includes:

[0015] A one-dimensional convolution kernel is pre-set, and data is pre-filled at the beginning and end of the difference index array to obtain the difference index array to be convolved;

[0016] The one-dimensional convolution kernel performs a convolution operation on the difference index array to be convolved based on a preset stride to obtain a convolution array.

[0017] Optionally, obtaining the gradient change of the fused video semantic content based on the fused video semantic content includes:

[0018] The derivatives of two adjacent consecutive data points in the convolution array are calculated sequentially to obtain the rate of change values ​​of the two adjacent consecutive data points, and the rate of change values ​​are stored in the rate of change array.

[0019] Optionally, the gradient change point extraction rule is that in two adjacent consecutive rate of change values, the rate of change of the preceding term is greater than 0 and the rate of change of the following term is less than 0; or,

[0020] In the two consecutive consecutive rate of change values, the rate of change of the first term is less than 0 and the rate of change of the second term is greater than 0.

[0021] Optionally, the number of values ​​in the one-dimensional convolution kernel is odd; and,

[0022] The numerical summation within the one-dimensional convolution kernel equals 1; and,

[0023] The values ​​within the one-dimensional convolution kernel decrease from the center to both sides, exhibiting a normal distribution.

[0024] According to a second aspect of the present invention, a keyframe extraction apparatus is provided, the apparatus comprising:

[0025] The data fusion module is used to perform weighted fusion of the video semantic content before and after a video frame based on a preset one-dimensional convolution kernel and the differences between two adjacent consecutive video frames in the target video, so as to obtain the fused video semantic content.

[0026] The data gradient change determination module is used to obtain the gradient change of the fused video semantic content based on the fused video semantic content;

[0027] The gradient change point extraction module is used to extract gradient change points from the gradient changes of the semantic content of the fused video according to the preset gradient change point extraction rules, so as to obtain the target gradient change points.

[0028] The keyframe extraction module is used to extract the video frames corresponding to the target gradient change points as keyframes of the target video.

[0029] Optionally, the device further includes:

[0030] The video frame acquisition module is used to sequentially acquire two adjacent consecutive video frames in the target video.

[0031] The structural similarity index acquisition module is used to acquire the structural similarity index of two adjacent consecutive video frames according to the structural similarity algorithm, and store the structural similarity index into a structural similarity index array.

[0032] The difference index acquisition module is used to perform normalization processing on the structural similarity index in turn to obtain the difference index of the two adjacent consecutive video frames, and store the difference index in the difference index array.

[0033] Optionally, the data fusion module includes:

[0034] The data padding submodule is used to pre-set a one-dimensional convolution kernel and to pre-padded the beginning and end of the difference index array to obtain the difference index array to be convolved.

[0035] The data convolution submodule is used to perform convolution operations on the difference index array to be convolved based on a preset stride by the one-dimensional convolution kernel to obtain a convolution array.

[0036] Optionally, the data gradient change determination module includes:

[0037] The rate of change value determination submodule is used to perform derivative operations on two adjacent consecutive data in the convolution array in turn to obtain the rate of change value of the two adjacent consecutive data, and store the rate of change value into the rate of change array.

[0038] Optionally, the gradient change point extraction rule is that in two adjacent consecutive rate of change values, the rate of change of the preceding term is greater than 0 and the rate of change of the following term is less than 0; or,

[0039] In the two consecutive consecutive rate of change values, the rate of change of the first term is less than 0 and the rate of change of the second term is greater than 0.

[0040] Optionally, the number of values ​​in the one-dimensional convolution kernel is odd; and,

[0041] The numerical summation within the one-dimensional convolution kernel equals 1; and,

[0042] The values ​​within the one-dimensional convolution kernel decrease from the center to both sides, exhibiting a normal distribution.

[0043] According to a third aspect of the present invention, an electronic device is provided, comprising:

[0044] processor;

[0045] Memory used to store the processor's executable instructions;

[0046] The processor is configured to execute the instructions to implement the keyframe extraction method.

[0047] According to a fourth aspect of the present invention, a storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of a keyframe extraction method as described in the first aspect.

[0048] This invention provides a keyframe extraction method, apparatus, electronic device, and storage medium. It weights and fuses the semantic content of the video before and after a video frame based on a preset one-dimensional convolution kernel and the differences between two adjacent consecutive video frames in the target video, obtaining fused semantic content. The gradient changes of the fused semantic content are then obtained. Gradient change points are extracted from the gradient changes according to a preset gradient change point extraction rule, yielding target gradient change points. The video frame corresponding to the target gradient change point is then used as the keyframe of the target video. This invention uses a preset one-dimensional convolution kernel to weight and fuse the semantic content of the current video frame and the video frames before and after it, and uses the video frame corresponding to the gradient change point of the fused semantic content as the keyframe. This greatly incorporates the semantic content of the current video frame and the changes in semantic content of the video frames before and after it, avoiding the failure of existing technologies to consider changes in video semantic content and the semantic content of the video frames before and after it, thus solving the technical problem of inaccurate keyframe extraction.

[0049] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0050] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0051] Figure 1 This is a flowchart illustrating the steps of a keyframe extraction method according to an exemplary embodiment;

[0052] Figure 2 This is a flowchart illustrating the steps of another keyframe extraction method according to an exemplary embodiment;

[0053] Figure 3 yes Figure 1 The flowchart shown is a step 101 flowchart of a keyframe extraction method according to an exemplary embodiment.

[0054] Figure 4 This is a device block diagram illustrating a keyframe extraction apparatus according to an exemplary embodiment;

[0055] Figure 5This is a device block diagram illustrating another keyframe extraction apparatus according to an exemplary embodiment;

[0056] Figure 6 yes Figure 4 The diagram shown is a block diagram of a keyframe extraction apparatus according to an exemplary embodiment, illustrating a data fusion module 401.

[0057] Figure 7 This is a schematic diagram illustrating a one-dimensional convolution kernel according to an exemplary embodiment;

[0058] Figure 8 This is a schematic diagram illustrating a one-dimensional valid convolution according to an exemplary embodiment. Detailed Implementation

[0059] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0060] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0061] The following detailed description, in conjunction with the accompanying drawings, of a keyframe extraction method, apparatus, electronic device, and storage medium provided by the present invention through specific embodiments and application scenarios, will illustrate these features in detail.

[0062] The first embodiment of the present invention relates to a keyframe extraction method, the flowchart of which is shown below. Figure 1 As shown, it includes:

[0063] Step 101: Based on the preset one-dimensional convolution kernel and the differences between two adjacent consecutive video frames in the target video, the video semantic content before and after the video frame is weighted and fused to obtain the fused video semantic content.

[0064] Step 102: Obtain the gradient changes of the fused video semantic content based on the fused video semantic content.

[0065] It should be noted that, in this embodiment of the invention, when analyzing the semantic content changes of the current video frame, the semantic content of the current video frame and the semantic content of the video frames before and after it must be taken into account. Only in this way can the analysis of the semantic content changes of the current video frame be accurate. Therefore, this application uses a pre-set one-dimensional convolution kernel to perform convolution with the differences between two adjacent consecutive video frames in the target video, thereby weightedly fusing the semantic content of the current video frame and the video frames before and after it, greatly taking into account the semantic content of the current video frame and the video frames before and after it. After completing the weighted fusion, the gradient change of the fused video semantic content is further obtained based on the obtained fused video semantic content, that is, the semantic content change rate value between two adjacent consecutive video frames is obtained.

[0066] Step 103: Extract gradient change points from the gradient changes of the semantic content of the fused video according to the preset gradient change point extraction rules to obtain the target gradient change points.

[0067] It should be noted that, in this embodiment of the invention, the gradient change point extraction rule is that in two adjacent consecutive rate of change values, the rate of change of the preceding term is greater than 0 and the rate of change of the following term is less than 0, i.e., i n >0 and i n+1 <0; or, in two consecutive rate of change values, the rate of change of the preceding term is less than 0 and the rate of change of the following term is greater than 0, i.e., i n <0 and i n+1 >0. The rate of change array can be used to represent the gradient changes in the semantic content of the fused video.

[0068] Specifically, following the order from the rate of change array I = [i1, i2, ... i N-1 Extract i from ] n , n=1; if i n >0 and i n+1 <0, or, i n <0 and i n+1 If the value is greater than 0, then n is stored in the target gradient change point array Y, where n is the target gradient change point. n = n + 1, and so on, until n = N - 1, that is, until all gradient change points of the rate of change in the rate of change array are extracted.

[0069] Step 104: Use the video frames corresponding to the target gradient change points as keyframes of the target video.

[0070] It should be noted that, in the embodiments of the present invention, the video data has the characteristics of most areas remaining unchanged between consecutive frames and significant relative changes in local areas. Video frames with significant changes often contain rich target motion information. The selection of key frames is the key to solving the current video task. The selection of key frames is based on the degree of change in the video, rather than on the length of the video data. More key frames are selected in videos with significant changes, and fewer key frames are selected in video data without significant changes. Therefore, the selection of key frames is mainly based on the degree of change between video frames.

[0071] Specifically, the target gradient change point is used to indicate the degree of change between two adjacent consecutive video frames. Therefore, after determining the target gradient change point, this invention uses the video frame corresponding to the target gradient change point as the key frame of the target video, which takes into account the degree of change between video frames and improves the accuracy of key frame extraction.

[0072] This invention provides a keyframe extraction method that, based on a preset one-dimensional convolutional kernel and the differences between two adjacent consecutive video frames in a target video, weightedly fuses the semantic content of the video before and after a video frame to obtain fused semantic content. The method then obtains the gradient changes of the fused semantic content; extracts gradient change points according to a preset gradient change point extraction rule to obtain target gradient change points; and uses the video frame corresponding to the target gradient change point as the keyframe of the target video. This invention, by using a preset one-dimensional convolutional kernel, weightedly fuses the semantic content of the current video frame and the video frames before and after it, and uses the video frame corresponding to the gradient change point of the fused semantic content as the keyframe. This greatly incorporates the semantic content of the current video frame and the changes in semantic content of the video frames before and after it, avoiding the failure of existing technologies to consider changes in video semantic content and the semantic content of the video frames before and after it, thus solving the technical problem of inaccurate keyframe extraction.

[0073] Furthermore, such as Figure 2 As shown, in this embodiment of the invention, the following steps may be included before step 101.

[0074] Step 201: Sequentially acquire two adjacent consecutive video frames from the target video.

[0075] It should be noted that, in this embodiment of the invention, the target video is the video from which keyframes need to be extracted, and it consists of continuously changing video frames (i.e., video frame images). After acquiring and reading the target video, two adjacent consecutive video frames are sequentially acquired starting from the first video frame of the target video. Two adjacent consecutive video frames can be represented by f. n and fn+1 (n=1) represents the sequence. Therefore, two consecutive video frames in the target video can be [(f1, f2), (f2, f3), (f3, f4)...(f... n f n+1 )).

[0076] Step 202: Obtain the structural similarity index of two adjacent consecutive video frames according to the structural similarity algorithm, and store the structural similarity index into the structural similarity index array.

[0077] It should be noted that, in this embodiment of the invention, the structural similarity algorithm is mainly used to detect the similarity between two images of the same size, or to detect the degree of image distortion. By comparing the brightness, contrast, and structure of the two images respectively, and then weighting these three elements and representing them with a product, the structural similarity index is finally obtained, which can be represented by S(x,y), where x and y are images.

[0078] After sequentially acquiring two adjacent consecutive video frames from the target video, a structural similarity algorithm can be used to perform calculations on each of the acquired adjacent consecutive video frames, starting from the first video frame, to obtain the structural similarity index between the two adjacent consecutive video frames. This index can be represented by S(f). n f n+1 ) represents, and sequentially assigns S(f) to each video frame starting from the first video frame. n f n+1 The structural similarity indices are stored in the structural similarity index array S until the structural similarity indices between all adjacent consecutive video frames in the target video have been calculated. The structural similarity index indicates the similarity between two adjacent consecutive video frames. After the structural similarity indices between all adjacent consecutive video frames in the target video have been calculated, the resulting structural similarity index array S can be represented as [S(f1, f2), S(f2, f3), S(f3, f4)...S(f...]. n f n+1 )).

[0079] Step 203: Normalize the structural similarity index in turn to obtain the difference index between two consecutive video frames, and store the difference index in the difference index array.

[0080] It should be noted that, in this embodiment of the invention, the structural similarity indices in the structural similarity index array are normalized to the [0,1] interval sequentially starting from the first video frame. This yields the difference index between two adjacent consecutive video frames, which can be represented by d. NThe difference indices are stored sequentially in the difference index array D, starting from the first video frame, until all structural similarity indices in the structural similarity index array have been normalized. The difference index indicates the difference between two adjacent consecutive video frames. After calculating the difference indices between all adjacent consecutive video frames in the target video, the resulting difference index array D can be represented as [d1, d2, d3…d…]. N It should be noted that if the structural similarity index is S(f1, f2), then the specific process of normalizing the structural similarity index to the [0,1] interval to obtain the difference index d1 can be described as follows: 1＝ 1-S(f1, f2).

[0081] Furthermore, in embodiments of the present invention, such as Figure 3 As shown, step 101 may also include the following steps.

[0082] Step 301: Pre-set a one-dimensional convolution kernel, and pre-fill the beginning and end of the difference index array with data respectively to obtain the difference index array to be convolved;

[0083] It should be noted that, in this embodiment of the invention, a one-dimensional convolution kernel needs to be pre-set, which can be K1. 1×M This indicates that M is the number of values ​​within the one-dimensional convolution kernel, and the values ​​within the one-dimensional convolution kernel represent the weights at the current position. For example... Figure 7As shown, the setting of this one-dimensional convolution kernel must simultaneously satisfy the following definition: 1) The number of values ​​in the one-dimensional convolution kernel is odd; If the consecutive video frames are (f1, f2, f3), it is necessary to determine the difference of the second video frame f2. After completing step 203, only the difference between the first video frame f1 and the second video frame f2, and the difference between the second video frame f2 and the third video frame f3 are considered. However, from the perspective of video semantic content, the semantic content between consecutive video frames is related, that is, the semantic content of the current video frame is related to the semantic content of the video frames before and after it. Step 203 does not take into account the difference between the first video frame f1 and the third video frame f3 and the second video frame f2. Therefore, the number of values ​​in the convolution kernel must be set to odd. In this way, after performing convolution operations using the convolution kernel, the difference between the video frames before and after the current video frame and the current video frame can be taken into account. 2) The sum of the values ​​within the one-dimensional convolution kernel is equal to 1; to avoid an increase in overall information due to weight changes, the sum of the values ​​within the convolution kernel must be set to equal 1. 3) The values ​​within the one-dimensional convolution kernel decrease from the center to both sides, following a normal distribution; from the perspective of changes in video semantic content, video frames closer to the current video frame have a greater impact on the semantic content of the current video frame, while video frames farther away have a smaller impact. The values ​​within the convolution kernel represent the weight of the current position; the smaller the value, the smaller the impact of the video frame at that position on the semantic content of the current video frame, and the larger the value, the greater the impact of the video frame at that position on the semantic content of the current video frame. Therefore, the values ​​within the convolution kernel are set to decrease from the center to both sides, following a normal distribution.

[0084] Data padding defines how to handle sample boundaries. The purpose of data padding is to maintain the output size of the convolution operation equal to the input size. This is because if the convolution kernel is greater than 1, not padding will result in the output size of the convolution operation being smaller than the input size. In this embodiment of the invention, valid convolution is preferably used. The following uses an array of dissimilarity indices of length 5, D = [3, 4, 1, 5, 6], and a one-dimensional convolution kernel of length 3, K = [-2, 2, 1], as an example to illustrate the calculation process of valid convolution. Valid convolution only considers the case where D can completely cover K, that is, the case where K moves within D. Figure 8 As shown, the calculation process is as follows:

[0085] 3*(-2)+4*2+1*1=3,

[0086] 4*(-2)+1*2+5*1=-1,

[0087] 1*(-2)+5*2+6*1=14.

[0088] As shown in the calculation process above, the calculated convolution array is [3, -1, 14]. Therefore, the input size (number of values ​​in the difference index array) of the convolution operation is 5, and the output size (number of values ​​in the convolution array) is 3. The output size of the convolution operation is smaller than the input size. Since the values ​​in the difference index array are obtained sequentially from the first video frame (the number of values ​​in the difference index array equals the number of video frames), after performing a convolution operation on the difference index array, the number of values ​​in the resulting difference index array to be convolved should also correspond to the number of values ​​in the difference index array. Only in this way can we ensure that the keyframe corresponding to the position of the value in the array can be accurately obtained later. To ensure that the number of values ​​in the difference index array to be convolved corresponds to the number of values ​​in the difference index array, the values ​​in the difference index array should be padded so that the number of values ​​in the resulting difference index array to be convolved after the convolution operation is consistent with the number of values ​​in the difference index array.

[0089] The specific process for filling the values ​​in the difference index array is as follows: fill the beginning of the difference index array with (M-1) / 2 d1 values, and fill the end of the difference index array with (M-1) / 2 d1 values. N Where M is the number of values ​​in the one-dimensional convolution kernel, d1 is the first dissimilarity index in the dissimilarity index array, and d N This is the Nth difference index in the difference index array. The array obtained after data filling is the difference index array D to be convolved.

[0090] Step 302: The one-dimensional convolution kernel performs a convolution operation on the array of convolutional difference indices based on a preset stride to obtain a convolution array.

[0091] It should be noted that, in this embodiment of the invention, the stride is defined as the step size of the convolution kernel movement when traversing the image, and the preset stride size in this embodiment of the invention is 1.

[0092] Specifically, a one-dimensional convolution operation is performed on the array of dissimilarity exponents to be convolved, D, using a one-dimensional convolution kernel K with a stride of 1, to obtain a convolution array, which can be represented as D' = [d'1, d'2, ..., d']. N ], n=1. One-dimensional convolution typically has three types: full convolution, same convolution, and valid convolution. In this embodiment of the invention, valid convolution is preferably used.

[0093] Furthermore, in this embodiment of the invention, step 102 may also include the following steps: performing derivative operations on two adjacent consecutive data in the convolution array in turn to obtain the rate of change values ​​of the two adjacent consecutive data, and storing the rate of change values ​​in the rate of change array.

[0094] It should be noted that, in this embodiment of the invention, the convolution array starts from n=1, i.e., d'1, and for two adjacent consecutive convolution values ​​[(d'1,d'2),(d'2,d'3),(d'3,d'4)...,(d' N-1 ,d' N By performing differentiation, we obtain the rate of change (i) of two consecutive convolution values. n i n =(d' n+1 -d' n The preset step size is used to store the rate of change values ​​in the rate of change array I, n = n + 1, until n = N, that is, the differentiation operation of all adjacent two consecutive convolution values ​​in the convolution array is completed. After the differentiation operation of all adjacent two consecutive convolution values ​​in the convolution array is completed, the resulting rate of change array I can be represented as [i1, i2, ... i N-1 ].

[0095] The second embodiment of the present invention relates to a keyframe extraction device, the block diagram of which is shown below. Figure 4 As shown, it includes:

[0096] The data fusion module 401 is used to perform weighted fusion of the video semantic content before and after the video frame based on the preset one-dimensional convolution kernel and the difference between two adjacent consecutive video frames in the target video, so as to obtain the fused video semantic content.

[0097] The data gradient change determination module 402 is used to obtain the gradient change of the semantic content of the fused video based on the semantic content of the fused video.

[0098] The gradient change point extraction module 403 is used to extract gradient change points from the gradient changes of the semantic content of the fused video according to the preset gradient change point extraction rules, so as to obtain the target gradient change points.

[0099] The keyframe extraction module 404 is used to extract the video frames corresponding to the target gradient change points as keyframes of the target video.

[0100] Optionally, such as Figure 5 As shown, the device also includes:

[0101] The video frame acquisition module 501 is used to sequentially acquire two adjacent consecutive video frames in the target video.

[0102] The structural similarity index acquisition module 502 is used to acquire the structural similarity index of two adjacent consecutive video frames according to the structural similarity algorithm, and store the structural similarity index into the structural similarity index array.

[0103] The difference index acquisition module 503 is used to normalize the structural similarity index in turn to obtain the difference index of two adjacent consecutive video frames, and store the difference index into the difference index array.

[0104] Optionally, such as Figure 6 As shown, the data fusion module 401 includes:

[0105] The data padding submodule 601 is used to pre-set the one-dimensional convolution kernel and pre-padded the head and tail of the difference index array to obtain the difference index array to be convolved.

[0106] The data convolution submodule 602 is used to perform convolution operations on the array of discrepancy exponents to be convolved based on a preset stride by a one-dimensional convolution kernel, so as to obtain a convolution array.

[0107] Optionally, the data gradient change determination module includes:

[0108] The rate of change value determination submodule is used to perform derivative operations on two adjacent consecutive data in the convolution array in turn to obtain the rate of change values ​​of the two adjacent consecutive data, and store the rate of change values ​​into the rate of change array.

[0109] Optionally, the gradient change point extraction rule is that in two adjacent consecutive rate of change values, the rate of change of the preceding term is greater than 0 and the rate of change of the following term is less than 0; or,

[0110] In two consecutive rate of change values, the rate of change of the first term is less than 0 and the rate of change of the second term is greater than 0.

[0111] Optionally, the number of values ​​in the one-dimensional convolution kernel is odd; and,

[0112] The sum of the values ​​within a one-dimensional convolution kernel equals 1; and,

[0113] The values ​​within a one-dimensional convolution kernel decrease from the center outwards, exhibiting a normal distribution.

[0114] This invention provides a keyframe extraction device that, based on a preset one-dimensional convolution kernel and the differences between two adjacent consecutive video frames in a target video, performs weighted fusion of the semantic content of the video before and after a video frame to obtain fused semantic content. It then obtains the gradient changes of the fused semantic content; extracts gradient change points according to a preset gradient change point extraction rule to obtain target gradient change points; and uses the video frame corresponding to the target gradient change point as the keyframe of the target video. This invention, by using a preset one-dimensional convolution kernel, weightedly fuses the semantic content of the current video frame and the video frames before and after it, and uses the video frame corresponding to the gradient change point of the fused semantic content as the keyframe. This greatly incorporates the semantic content of the current video frame and the changes in semantic content of the video frames before and after it, avoiding the failure of existing technologies to consider changes in video semantic content and the semantic content of the video frames before and after it, thus solving the technical problem of inaccurate keyframe extraction.

[0115] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0116] A third embodiment of the present invention relates to an electronic device. For example, the electronic device may be provided as a server. The electronic device may include one or more processors, and a memory for storing processor-executable instructions, such as application programs. The processor is configured to execute a keyframe extraction method as described above.

[0117] A fourth embodiment of the present invention relates to a storage medium storing a computer program that, when executed by a processor, implements the steps of a keyframe extraction method as described in the first aspect.

[0118] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0119] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0120] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

Claims

1. A key frame extraction method characterized by, The method comprises: obtaining two adjacent continuous video frames in a target video in sequence; obtaining a structural similarity index of the two adjacent continuous video frames according to a structural similarity algorithm, and storing the structural similarity index into a structural similarity index array; normalizing the structural similarity index in sequence to obtain a difference index of the two adjacent continuous video frames, and storing the difference index into a difference index array; performing weighted fusion on video semantic contents before and after a video frame based on a preset one-dimensional convolution kernel and the difference of the two adjacent continuous video frames in the target video to obtain fused video semantic contents; obtaining gradient changes of the fused video semantic contents based on the fused video semantic contents; extracting gradient change points of the gradient changes of the fused video semantic contents according to a preset gradient change point extraction rule to obtain a target gradient change point; taking a video frame corresponding to the target gradient change point as a key frame of the target video; wherein the performing weighted fusion on the video semantic contents before and after the video frame based on the preset one-dimensional convolution kernel and the difference of the two adjacent continuous video frames in the target video to obtain the fused video semantic contents comprises: pre-setting a one-dimensional convolution kernel, and pre-filling data at a head and a tail of the difference index array to obtain a difference index array to be convolved; performing convolution operation on the difference index array to be convolved based on a preset step length to obtain a convolution array; wherein the obtaining the gradient changes of the fused video semantic contents based on the fused video semantic contents comprises: performing derivation operation on two adjacent continuous data in the convolution array in sequence to obtain a change rate value of the two adjacent continuous data, and storing the change rate value into a change rate array.

2. The method of claim 1, wherein, the gradient change point extraction rule is that a change rate value of a previous item in two adjacent continuous change rate values is greater than 0 and a change rate value of a next item is less than 0; or the change rate value of the previous item in the two adjacent continuous change rate values is less than 0 and the change rate value of the next item is greater than 0.

3. The method of claim 1, wherein, the number of values in the one-dimensional convolution kernel is odd; and the values in the one-dimensional convolution kernel are accumulated to equal 1; and the values in the one-dimensional convolution kernel decrease from the middle to both sides, and present normal distribution.

4. An apparatus for key frame extraction, characterized by, The device comprises: a video frame obtaining module configured to obtain two adjacent continuous video frames in a target video in sequence; a structural similarity index obtaining module configured to obtain a structural similarity index of the two adjacent continuous video frames according to a structural similarity algorithm, and store the structural similarity index into a structural similarity index array; a difference index obtaining module configured to normalize the structural similarity index in sequence to obtain a difference index of the two adjacent continuous video frames, and store the difference index into a difference index array; a data fusion module configured to perform weighted fusion on video semantic contents before and after a video frame based on a preset one-dimensional convolution kernel and the difference of the two adjacent continuous video frames in the target video to obtain fused video semantic contents; and a gradient change point extraction module configured to extract gradient change points of the gradient changes of the fused video semantic contents according to a preset gradient change point extraction rule to obtain a target gradient change point. The data gradient change determination module is configured to obtain gradient changes of the fused video semantic content based on the fused video semantic content. The gradient change point extraction module is configured to extract gradient change points of the gradient changes of the fused video semantic content according to a preset gradient change point extraction rule, to obtain target gradient change points. The key frame extraction module is configured to take video frames corresponding to the target gradient change points as key frames of the target video. The data fusion module includes: The data padding submodule is configured to preset a one-dimensional convolution kernel, and prefill data at the head and tail of the difference index array, to obtain a difference index array to be convolved. The data convolution submodule is configured to perform convolution operation on the difference index array to be convolved based on a preset step length, to obtain a convolution array. The data gradient change determination module includes: The change rate value determination submodule is configured to sequentially perform derivative operation on two adjacent continuous data in the convolution array, to obtain change rate values of the two adjacent continuous data, and store the change rate values in a change rate array.

5. An electronic device, comprising: The processor is configured to execute the instructions to implement the key frame extraction method according to any one of claims 1 to 3. The storage medium stores a computer program, and the computer program is executed by the processor to implement the key frame extraction method according to any one of claims 1 to 3. ​ ​ 6. A storage medium, characterized by ​

Images