A video key frame extraction method and system based on CNN

Abstract

The present invention discloses a method and system for extracting video keyframes based on a CNN, which relates to the field of computer vision. The method comprises: determining a first video input source based on information about a first target video segment; obtaining video background features and video recognition object features by performing feature analysis on the first video input source; performing video similarity analysis based on the video background features and the video recognition object features to obtain a first similarity coefficient; generating a target response function based on the first similarity coefficient, and using the output first convolution kernel data to build a keyframe feature extraction model; inputting the first key feature and the first target video segment information into the keyframe feature extraction model to obtain a keyframe video extraction result. The method solves the technical problems of low keyframe extraction accuracy and low flexibility in the prior art, achieving a targeted construction of a convolution data model, and improving the accuracy and flexibility of keyframe extraction.

Description

Technical Field

[0001] The present invention relates to fields related to computer vision, and in particular to a method and system for extracting video key frames based on CNN. Background Art

[0002] Because videos contain more information than images, but a sequence contains too much redundant information, extracting video keyframes is crucial for many tasks. With the continuous development of multimedia video and its enrichment of people's lives, keyframes have become an important part of multimedia video processing. Therefore, how to accurately identify video keyframes has become a current research topic.

[0003] Since the information in the video is dynamic and changing, higher technical requirements are placed on the accurate extraction of video key frames. However, the current key frame extraction technology still has great limitations. Changes based on the video will affect the accuracy of key frame extraction, and thus cannot meet the needs of people in today's society for key frame extraction.

[0004] Currently, the existing technology has the technical problem of being unable to accurately extract key frames according to the changing content of the video and having low flexibility. Summary of the Invention

[0005] In response to the defects in the existing technology, the purpose of this application is to provide a CNN-based video key frame extraction method and system to solve the technical problems in the existing technology that key frames cannot be accurately extracted according to the changing content of the video and the flexibility is not high. The method achieves the technical effect of performing feature analysis on the video source, further analyzing the changes in video motion, and building a corresponding CNN convolution data model to extract key frame features, thereby improving the extraction accuracy and flexibility.

[0006] On the one hand, the present application provides a video key frame extraction method based on CNN, the method comprising: obtaining first target video segment information; determining a first video input source based on the first target video segment information; obtaining video background features and video recognition object features by performing feature analysis on the first video input source; performing video similarity analysis based on the video background features and the video recognition object features to obtain a first similarity coefficient; building a target response function based on the first similarity coefficient, and determining first convolution kernel data based on the target response function; building a key frame feature extraction model according to the first convolution kernel data; obtaining a first key feature; inputting the first key feature and the first target video segment information into the key frame feature extraction model, and obtaining first output information based on the key frame feature extraction model, wherein the first output information is a key frame video extraction result based on the first key feature.

[0007] On the other hand, the present application also provides a CNN-based video key frame extraction system, the system comprising: a first obtaining unit, the first obtaining unit being used to obtain first target video segment information; a first determining unit, the first determining unit being used to determine a first video input source based on the first target video segment information; a second obtaining unit, the second obtaining unit being used to obtain video background features and video recognition object features by performing feature analysis on the first video input source; a third obtaining unit, the third obtaining unit being used to perform video similarity analysis based on the video background features and the video recognition object features to obtain a first similarity coefficient; a second determining unit, the second determining unit being used to build a target response function based on the first similarity coefficient, and to determine first convolution kernel data based on the target response function; a first building unit, the first building unit being used to build a key frame feature extraction model according to the first convolution kernel data; a fourth obtaining unit, the fourth obtaining unit being used to obtain a first key feature; a first output unit, the first output unit being used to input the first key feature and the first target video segment information into the key frame feature extraction model, and to obtain first output information based on the key frame feature extraction model, wherein the first output information is a key frame video extraction result based on the first key feature.

[0008] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above methods when executing the program.

[0009] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in any one of the above methods when executed by a processor.

[0010] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0011] By obtaining the first target video segment information to be extracted and analyzing the video input source based on the first target video segment information, further analyzing the background features and main identification object features of the video source, the motion similarity of each video segment in the first target video segment information is analyzed based on the features obtained by the analysis, and a first similarity coefficient is obtained. The first similarity coefficient is then used as the main analysis parameter for building a CNN mathematical model, and the first convolution kernel data is obtained by building a target response function. Furthermore, the extracted key features are obtained, and the key frame feature extraction model is calculated based on the obtained key features and the first target video segment information to obtain the output key frame video extraction result based on the first key features. This solves the technical problems in the prior art of being unable to accurately extract key frames based on the changing content of the video and having low flexibility, and achieves the technical effect of improving the extraction accuracy and flexibility by performing feature analysis on the video source, further analyzing the video motion changes, and building a corresponding CNN convolution data model for key frame feature extraction.

[0012] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0014] Figure 1 This is a flow chart of a CNN-based video key frame extraction method according to an embodiment of the present application;

[0015] Figure 2 This is a schematic diagram of a process for obtaining the first convolution kernel data in a CNN-based video key frame extraction method according to an embodiment of the present application;

[0016] Figure 3 This is a flowchart of a key feature combination analysis method for extracting key frames from a video based on CNN according to an embodiment of the present application;

[0017] Figure 4 This is a structural diagram of a video key frame extraction system based on CNN in an embodiment of the present application;

[0018] Figure 5 This is a schematic diagram of the structure of an exemplary electronic device according to an embodiment of the present application.

[0019] Explanation of the accompanying drawings: first obtaining unit 11, first determining unit 12, second obtaining unit 13, third obtaining unit 14, second determining unit 15, first building unit 16, fourth obtaining unit 17, first output unit 18, bus 300, receiver 301, processor 302, transmitter 303, memory 304, bus interface 305. DETAILED DESCRIPTION

[0020] The embodiments of the present application provide a CNN-based video key frame extraction method and system, which solves the technical problems in the prior art of being unable to accurately extract key frames according to the changing content of the video and having low flexibility. It achieves the technical effect of performing feature analysis on the video source, thereby performing further analysis based on the changes in video motion, and building a corresponding CNN convolutional data model to extract key frame features, thereby improving the extraction accuracy and flexibility.

[0021] Below, the exemplary embodiments of the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described herein.

[0022] The acquisition, storage, use, and processing of data in this application's technical solution comply with relevant national laws and regulations.

[0023] The shot-based key frame extraction algorithm is the first to be developed in the field of video retrieval and is currently the most mature general method. This algorithm generally uses some technical means to segment the video file according to shot changes, and then selects the first and last two frames in each shot of the video as key frames. However, due to the variability of the content in the video, it is currently impossible to accurately analyze the segmentation process of the video shot, making it difficult to meet people's current needs for video positioning retrieval, resulting in the inability to accurately extract key frames according to the changing content of the video and low flexibility.

[0024] In response to the above technical problems, the overall idea of ​​the technical solution provided by this application is as follows:

[0025] This application proposes a CNN-based video keyframe extraction method and system. By obtaining the first target video segment information to be extracted and analyzing the video input source based on the first target video segment information, the background features and main identification object features of the video source are further analyzed. Based on the features obtained by the analysis, the motion similarity of each video segment in the first target video segment information is analyzed to obtain a first similarity coefficient. The first similarity coefficient is then used as the main analysis parameter for building a CNN mathematical model. The first convolution kernel data is obtained by building a target response function. Furthermore, the extracted key features are obtained. Based on the obtained key features and the first target video segment information, a keyframe feature extraction model is calculated to obtain an output keyframe video extraction result based on the first key features. This solves the technical problem in the prior art that keyframes cannot be accurately extracted according to the changing content of the video and the flexibility is low. The technical effect of improving the extraction accuracy and flexibility is achieved by analyzing the features of the video source, further analyzing the video motion changes, and building a corresponding CNN convolution data model for keyframe feature extraction.

[0026] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0027] Example 1

[0028] like Figure 1 As shown, the embodiment of the present application provides a video key frame extraction method based on CNN, the method comprising:

[0029] Step S100: obtaining first target video segment information;

[0030] Specifically, the shot-based key frame extraction algorithm is the first to be developed in the field of video retrieval and is currently the most mature general method. The general implementation process of this algorithm is: first, the source video file is segmented according to the shot changes using certain technical means, and then the first and last two frames in each shot of the video are selected as key frames. However, due to the variability of the content in the video, it is currently impossible to accurately analyze the segmentation process of the video shot, making it difficult to meet people's current needs for video positioning retrieval, resulting in the inability to accurately extract key frames according to the changing content of the video and low flexibility.

[0031] To solve the above problems, a CNN-based video keyframe extraction method is proposed to build a corresponding CNN convolutional data model for keyframe feature extraction, thereby improving the accuracy of the video extraction results. Convolutional Neural Networks (CNN) have been widely used in computer vision in recent years, and convolutional networks are used to implement model construction. Furthermore, firstly, a first target video segment information is obtained, wherein the first target video segment information is any video segment information of the video keyframe to be extracted, so that the next step of video analysis is carried out based on this video, and video analysis data is intended to be provided.

[0032] Step S200: determining a first video input source according to the first target video segment information;

[0033] Step S300: Obtaining video background features and video recognition object features by performing feature analysis on the first video input source;

[0034] Specifically, scene information of the video is collected according to the information of the first target video segment, and the scene is used as a video input source for scene feature analysis, and then its video background features and video recognition object features are determined, wherein the video background features are background features of the video, for example, in a lane video, its background features are based on lane recognition, and its vehicles are recognition objects; or in a certain construction site, its site background features are buildings, and its main recognition objects are construction workers wearing safety helmets, etc., feature recognition is performed on the background and objects respectively, so that feature analysis can be completed according to the video background features and video recognition object features, and the first target video segment can be refined and distinguished.

[0035] Step S400: performing video similarity analysis based on the video background features and the video recognition object features to obtain a first similarity coefficient;

[0036] Specifically, a video similarity analysis is performed based on the video background features and the video recognition object features to obtain a first similarity coefficient, wherein, since the video background features are recognition results obtained by extracting video background features based on the first target video segment information, and the video recognition object features are recognition results obtained by extracting video object features based on the first target video segment information, further, a video similarity analysis is performed based on the video background features and the video recognition object features to obtain a first similarity coefficient, so that the video in the first target video segment can be analyzed based on the first similarity coefficient.

[0037] Furthermore, the first similarity coefficient includes background similarity and object similarity, wherein the corresponding background similarity coefficient and object similarity coefficient are obtained, and then the uniformity of the video background is analyzed by the degree of background similarity, and the background changes of the first target video segment are identified; the changes of the main characters in the first target video are analyzed by the object similarity coefficient, and therefore, the first video segment information is analyzed accordingly based on the first similarity coefficient, so as to bidirectionally analyze the degree of change to determine the similarity of the video content, and provide basic decision data for the subsequent construction of the convolutional neural network.

[0038] Step S500: constructing a target response function according to the first similarity coefficient, and determining first convolution kernel data according to the target response function;

[0039] Step S600: Building a key frame feature extraction model according to the first convolution kernel data;

[0040] Further, such as Figure 2 As shown, the target response function is constructed according to the first similarity coefficient, and the first convolution kernel data is determined according to the target response function. In this embodiment of the application, step S500 further includes:

[0041] Step S510: obtaining first resource configuration environment information;

[0042] Step S520: performing computing capacity analysis based on the first resource configuration environment information to obtain first computing performance;

[0043] Step S530: Using the first computing performance as a constraint variable, the first similarity coefficient as an input variable, and the convolution kernel data as an output result, constructing the target response function, wherein the target response function is used to output a response of the convolution kernel data;

[0044] Step S540: Obtain the first convolution kernel data according to the target response function, wherein the first convolution kernel data includes the convolution kernel size and the number of convolution kernels.

[0045] Specifically, when the first similarity coefficient is obtained, it is determined whether the first similarity coefficient is within the preset similarity coefficient. If it is within the preset similarity coefficient, it indicates that the current similarity of the first target video segment information is relatively large, that is, the required key frame extraction feature points are relatively small. If it is not within the preset similarity coefficient, it indicates that the current similarity of the first target video segment information is relatively small, that is, the required key frame extraction feature points are relatively large. Therefore, in order to further refine the output of the key frame extraction feature points, the target response function is constructed according to the first similarity coefficient, and the response output of the convolution kernel data is performed according to the target response function, and then the corresponding key frame feature extraction model is regenerated for key frame extraction of the first target video segment information.

[0046] Furthermore, the target response function is constructed according to the first similarity coefficient by determining the first convolution kernel data through variable input and response output. Since the convolutional neural network performs deep convolution for feature extraction based on the convolution kernel size, the first similarity coefficient is used as the input variable and the convolution kernel data is used as the output result to construct the target response function. According to the target response function, the first convolution kernel data is obtained, wherein the first convolution kernel data includes the convolution kernel size and the number of convolution kernels.

[0047] When building the target response function, since the size of the convolution kernel is related to the computing performance of the model, that is, the larger the convolution kernel, the more convolution layers it has, which will affect the computing performance of the model, therefore, by obtaining the first resource configuration environment information, and then performing computing capacity analysis based on the first resource configuration environment information, the first computing performance is obtained, wherein the first resource configuration environment information is analyzed based on the hardware environment and software environment implemented by this system, so as to determine the conditions that can be provided by the computing environment, and thus make the generated mathematical model usable. After obtaining the first computing performance, the first computing performance is output as a constraint variable of the target response function to improve the accuracy of the first convolution kernel data output by the target response function, and then build an accurate and high-quality key frame feature extraction model to complete accurate extraction.

[0048] Step S700: obtaining a first key feature;

[0049] Step S800: Input the first key feature and the first target video segment information into the key frame feature extraction model, and obtain first output information according to the key frame feature extraction model, wherein the first output information is a key frame video extraction result based on the first key feature.

[0050] Specifically, the first key feature is a key identification feature obtained based on the first target video segment information, which can be determined according to the main identification point selected by the user. Furthermore, the first key feature and the first target video segment information are input into the key frame feature extraction model, and according to the key frame feature extraction model, the first output information is obtained, that is, the key frame video extraction result about the first key feature is output.

[0051] Furthermore, the key frame feature extraction model is a convolutional neural network model. Convolutional Neural Networks (CNN) is a type of feedforward neural network that includes convolution calculations and has a deep structure. It has representation learning capabilities and can perform translation-invariant classification of input information according to its hierarchical structure. Since the network avoids complex pre-processing of the image, the original image can be directly input. Among them, the key frame feature extraction model is a model built based on the first convolution kernel data, which can effectively extract key frame features based on changes in the information of the first target video segment, thereby achieving the technical effect of performing feature analysis on the video source, and then further analyzing the changes in video motion, and building a corresponding convolutional neural network model to extract key frame features, thereby improving the extraction accuracy and flexibility.

[0052] Furthermore, step S600 in the embodiment of the present application further includes:

[0053] Step S610: obtaining a test key feature set, wherein the test key feature set is sample data for model testing;

[0054] Step S620: obtaining a preset key video frame set according to the test key feature set, wherein the test key feature set corresponds to the preset key video frame set in a one-to-one manner;

[0055] Step S630: Inputting the test key feature set into the key frame feature extraction model, and obtaining second output information according to the key frame feature extraction model, wherein the second output information is a key frame video extraction result based on the test key feature set;

[0056] Step S640: Compare the second output information with the preset key video frame set to obtain the first model accuracy.

[0057] Furthermore, step S640 in this embodiment of the application further includes:

[0058] Step S641: determining whether the accuracy of the first model is within the preset model accuracy;

[0059] Step S642: If the accuracy of the first model is not within the preset model accuracy, performing quantitative analysis and category analysis on the test key feature set to obtain a first sample quantization coefficient and a first sample feature strength;

[0060] Step S643: Obtaining first adjustment information according to the first sample quantization coefficient and the first sample feature strength;

[0061] Step S644: Adjust the test key feature set based on the first adjustment information.

[0062] Specifically, after building the key frame feature extraction model, in order to ensure the usability of the key frame feature extraction model, the model is tested on data to ensure the data recognition accuracy of the model. First, a test key feature set is obtained, wherein the test key feature set is sample data used for model testing, and the test key feature set is determined according to different extraction objectives, so that the model is tested based on the test key feature set.

[0063] On the one hand, a preset key video frame set is obtained based on the test key feature set, wherein the preset key video frame set is one-to-one mapped data set based on the test key feature set; on the other hand, the test key feature set is input into the key frame feature extraction model, and according to the key frame feature extraction model, a key frame video extraction result based on the test key feature set is obtained, and this is used as the second output information. Furthermore, the key frame video extraction result of the test key feature set is compared with the preset key video frame set, and the accuracy test result of the model is output according to the comparison result.

[0064] Furthermore, after outputting the first model accuracy, it is determined whether the first model accuracy is within the preset model accuracy. If the first model accuracy is not within the preset model accuracy, it is first necessary to perform a sample accuracy analysis on the sample data used for model testing. The process is as follows:

[0065] Quantitative analysis and category analysis are performed on the test key feature set respectively to obtain a first sample quantization coefficient and a first sample feature strength, that is, to ensure that the sample quantity and sample feature strength of the test key feature set meet the standards, that is, when the first sample quantization coefficient and the first sample feature strength do not meet the test requirements, first adjustment information is obtained, and the test key feature set is adjusted based on the first adjustment information, so as to ensure the accuracy and reliability of the key frame feature extraction model.

[0066] Furthermore, step S540 in this embodiment of the present application further includes:

[0067] Step S541: Obtaining a first grouping instruction according to the convolution kernel size and the number of convolution kernels in the first convolution kernel data;

[0068] Step S542: grouping the first convolution kernel data based on the first grouping instruction to generate multiple groups of convolution kernel data;

[0069] Step S543: Building the key frame feature extraction model based on the multiple sets of convolution kernel data.

[0070] Specifically, the first convolution kernel data includes the convolution kernel size and the number of convolution kernels. The convolution kernel size determines the feature size of the output layer, and the number of convolution kernels determines the number of output channels. Accordingly, the larger the convolution kernel in the first convolution kernel data, the better the feature extraction effect. However, large convolution kernels lead to a surge in computational complexity, which is not conducive to increasing the depth of the model and also reduces computational performance. Specifically, as the number of layers increases, the model's performance deteriorates. This is largely because as the number of layers increases, the gradient vanishes more and more severely, making it difficult for backpropagation to train shallow layers of the network. Therefore, the first convolution kernel data is grouped using the first grouping instruction to generate multiple groups of convolution kernel data. For example, an input is processed simultaneously by 1×1, 3×3, and 5×5 convolution kernels, and the resulting features are then combined to obtain better features. Building the keyframe feature extraction model based on these multiple groups of convolution kernel data can reduce computational complexity while not affecting the feature extraction effect corresponding to the first convolution kernel data, thereby improving the convolution processing effect.

[0071] Furthermore, step S600 in the embodiment of the present application further includes:

[0072] Step S610: Building the key frame feature extraction model, wherein the key frame feature extraction model includes a video preprocessing layer, a key frame extraction layer, and a key frame output layer;

[0073] Step S620: inputting the first key feature and the first target video segment information into the key frame extraction model;

[0074] Step S630: performing video preprocessing on the first target video segment information according to the video preprocessing layer, performing key frame extraction on the key frame according to the preprocessed first target video segment information and the first key feature, and outputting the key frame extraction result through the key frame output layer.

[0075] Specifically, the key frame feature extraction model is constructed, wherein the key frame feature extraction model is constructed by performing a convolutional neural network model based on the first convolution kernel data, and its logical function layer includes a video preprocessing layer, a key frame extraction layer and a key frame output layer. The first target video segment information is preprocessed according to the video preprocessing layer, such as video clarity processing, color correction processing, etc., and then the preprocessed video segment information is input into the key frame extraction layer. The key frame extraction layer extracts and identifies key frames based on the preprocessed video segment information and the first key features, and outputs the key frame extraction results through the key frame output layer. The key frame extraction layer is the main functional logic layer of the key frame feature extraction model, which processes the preprocessed video information according to the first convolution kernel data, and completes the convolution kernel calculation of the image in the video to make the extracted video key frames more accurate.

[0076] Further, such as Figure 3 As shown, the step S700 of obtaining the first key feature in the embodiment of the present application further includes:

[0077] Step S710: Obtaining first video coverage according to the first key feature, wherein the first video coverage is based on the feature coverage of the first key feature in the first target video segment information;

[0078] Step S720: determining whether to activate a feature combination instruction according to the first video coverage;

[0079] Step S730: If the feature combination instruction is activated, a second key feature is obtained;

[0080] Step S740: performing feature combination based on the first key feature and the second key feature, and outputting a combined key feature;

[0081] Step S750: Input the combined key features and the first target video segment information into the key frame feature extraction model, and obtain third output information according to the key frame feature extraction model, wherein the third output information is a key frame video extraction result based on the combined key features output.

[0082] Specifically, the first video coverage is obtained by analyzing the feature coverage degree based on the first key feature in the first target video segment information according to the first key feature. When the first video coverage is low, the current video key frame extraction based on the first key feature is not comprehensive enough, and thus it is necessary to conduct a comprehensive analysis in combination with new key features, that is, to determine whether to activate the feature combination instruction according to the first video coverage, and if the feature combination instruction is activated, obtain the second key feature, and perform feature combination based on the first key feature and the second key feature, and output the combined key feature, that is, input the combined key feature and the first target video segment information into the key frame feature extraction model, and obtain the key frame video extraction result based on the output of the combined key feature according to the key frame feature extraction model, that is, the third output information.

[0083] Activating the feature combination instruction and obtaining the second key feature is when the first video coverage calculated by the first key feature is not high and it is difficult to meet the feature extraction requirements, thereby activating the feature combination instruction to perform feature combination. Therefore, when the feature combination instruction is activated, the second key feature is obtained, wherein the second key feature is different from the first key feature, thereby achieving a method of jointly analyzing video key frames based on combined features, thereby improving the accuracy of the output information of the key frame feature extraction model.

[0084] Compared with the prior art, the present invention has the following beneficial effects:

[0085] 1. By obtaining the first target video segment information to be extracted and analyzing the video input source based on the first target video segment information, further analyzing the background features and main identification object features of the video source, the motion similarity of each video segment in the first target video segment information is analyzed based on the features obtained by the analysis to obtain a first similarity coefficient. The first similarity coefficient is then used as the main analysis parameter for building a CNN mathematical model, and the first convolution kernel data is obtained by building a target response function. Furthermore, the extracted key features are obtained, and the key frame feature extraction model is calculated based on the obtained key features and the first target video segment information to obtain an output key frame video extraction result based on the first key features. This solves the technical problem in the prior art of being unable to accurately extract key frames based on the changing content of the video and having low flexibility. It achieves the technical effect of improving extraction accuracy and flexibility by analyzing the video source features, further analyzing the video motion changes, and building a corresponding CNN convolution data model for key frame feature extraction.

[0086] 2. Since the first convolution kernel data is grouped by the first grouping instruction to generate multiple groups of convolution kernel data, the key frame feature extraction model is built based on the multiple groups of convolution kernel data. This can reduce the computational complexity without affecting the feature extraction effect corresponding to the first convolution kernel data, thereby improving the convolution processing effect.

[0087] 3. By analyzing the feature coverage degree based on the first key feature in the first target video segment information to obtain the first video coverage, and activating the feature combination instruction according to the first video coverage, the second key feature is combined with the first key feature, thereby achieving a method of jointly analyzing the video key frames based on the combined features, thereby improving the accuracy of the output information of the key frame feature extraction model.

[0088] Example 2

[0089] Based on the same inventive concept as the CNN-based video key frame extraction method in the aforementioned embodiment, the present invention also provides a CNN-based video key frame extraction system, such as Figure 4 As shown, the system includes:

[0090] A first obtaining unit 11, wherein the first obtaining unit 11 is configured to obtain first target video segment information;

[0091] a first determining unit 12, configured to determine a first video input source according to the first target video segment information;

[0092] A second obtaining unit 13, configured to obtain video background features and video recognition object features by performing feature analysis on the first video input source;

[0093] a third obtaining unit 14 configured to perform video similarity analysis based on the video background features and the video recognition object features to obtain a first similarity coefficient;

[0094] a second determining unit 15, configured to construct a target response function according to the first similarity coefficient, and determine first convolution kernel data according to the target response function;

[0095] A first building unit 16, configured to build a key frame feature extraction model according to the first convolution kernel data;

[0096] a fourth obtaining unit 17, wherein the fourth obtaining unit 17 is configured to obtain a first key feature;

[0097] The first output unit 18 is used to input the first key feature and the first target video segment information into the key frame feature extraction model, and obtain first output information according to the key frame feature extraction model, wherein the first output information is the key frame video extraction result based on the first key feature.

[0098] Furthermore, the system further comprises:

[0099] a fifth obtaining unit, configured to obtain first resource configuration environment information;

[0100] a first computing unit, configured to perform computing capacity analysis based on the first resource configuration environment information to obtain first computing performance;

[0101] a second building unit, the second building unit being configured to use the first computing performance as a constraint variable, the first similarity coefficient as an input variable, and the convolution kernel data as an output result to build the target response function, wherein the target response function is configured to output a response of the convolution kernel data;

[0102] A sixth obtaining unit, wherein the sixth obtaining unit is used to obtain the first convolution kernel data according to the target response function, wherein the first convolution kernel data includes the convolution kernel size and the number of convolution kernels.

[0103] Furthermore, the system further comprises:

[0104] a seventh obtaining unit, the seventh obtaining unit being configured to obtain a test key feature set, wherein the test key feature set is sample data for performing model testing;

[0105] an eighth obtaining unit, configured to obtain a preset key video frame set according to the test key feature set, wherein the test key feature set corresponds to the preset key video frame set in a one-to-one manner;

[0106] a second input unit, configured to input the test key feature set into the key frame feature extraction model, and obtain second output information according to the key frame feature extraction model, wherein the second output information is a key frame video extraction result based on the test key feature set;

[0107] A ninth obtaining unit is configured to compare the second output information with the preset key video frame set to obtain the first model accuracy.

[0108] Furthermore, the system further comprises:

[0109] a first judging unit, configured to judge whether the accuracy of the first model is within a preset model accuracy;

[0110] a tenth obtaining unit, configured to, if the first model accuracy is not within the preset model accuracy, perform quantitative analysis and category analysis on the test key feature set to obtain a first sample quantization coefficient and a first sample feature strength;

[0111] an eleventh obtaining unit, configured to obtain first adjustment information according to the first sample quantization coefficient and the first sample feature strength;

[0112] A first adjustment unit is configured to adjust the test key feature set based on the first adjustment information.

[0113] Furthermore, the system further comprises:

[0114] a twelfth obtaining unit, configured to obtain a first grouping instruction according to a convolution kernel size and a number of convolution kernels in the first convolution kernel data;

[0115] a first grouping unit, configured to group the first convolution kernel data based on the first grouping instruction to generate multiple groups of convolution kernel data;

[0116] A third building unit is used to build the key frame feature extraction model based on the multiple groups of convolution kernel data.

[0117] Furthermore, the system further comprises:

[0118] A fourth building unit, the fourth building unit is used to build the key frame feature extraction model, wherein the key frame feature extraction model includes a video preprocessing layer, a key frame extraction layer and a key frame output layer;

[0119] a third input unit, configured to input the first key feature and the first target video segment information into the key frame extraction model;

[0120] The second output unit is used to perform video preprocessing on the first target video segment information according to the video preprocessing layer, the key frame extraction layer performs key frame extraction based on the preprocessed first target video segment information and the first key feature, and outputs the key frame extraction result through the key frame output layer.

[0121] Furthermore, the system further comprises:

[0122] a thirteenth obtaining unit, configured to obtain a first video coverage according to the first key feature, wherein the first video coverage is a feature coverage degree based on the first key feature in the first target video segment information;

[0123] a second judgment unit, configured to judge whether to activate a feature combination instruction according to coverage of the first video;

[0124] a fourteenth obtaining unit, configured to obtain a second key feature if the feature combination instruction is activated;

[0125] a third output unit, configured to perform feature combination based on the first key feature and the second key feature, and output the combined key feature;

[0126] A fifteenth obtaining unit is used to input the combined key feature and the first target video segment information into the key frame feature extraction model, and obtain third output information according to the key frame feature extraction model, wherein the third output information is a key frame video extraction result based on the combined key feature output.

[0127] The foregoing Figure 1 The various variations and specific examples of the CNN-based video key frame extraction method in Example 1 are also applicable to the CNN-based video key frame extraction system in this embodiment. Through the above detailed description of the CNN-based video key frame extraction method, those skilled in the art can clearly understand the implementation method of the CNN-based video key frame extraction system in this embodiment, so for the sake of brevity of the specification, it will not be described in detail here.

[0128] Example 3

[0129] Reference below Figure 5 To describe the electronic device of this application.

[0130] Figure 5 The figure shows a schematic structural diagram of an electronic device according to the present application.

[0131] Based on the inventive concept of a CNN-based video key frame extraction method in the aforementioned embodiment, the present invention also provides a CNN-based video key frame extraction system, which stores a computer program that, when executed by a processor, implements the steps of any method of the CNN-based video key frame extraction system described above.

[0132] Among them, Figure 5In the embodiment of the present invention, a bus architecture (represented by bus 300) is shown. Bus 300 may include any number of interconnected buses and bridges, and bus 300 links together various circuits including one or more processors represented by processor 302 and memory represented by memory 304. Bus 300 may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 305 provides an interface between bus 300 and receiver 301 and transmitter 303. Receiver 301 and transmitter 303 may be the same component, namely a transceiver, which provides a unit for communicating with various other systems over a transmission medium. Processor 302 is responsible for managing bus 300 and general processing, while memory 304 may be used to store data used by processor 302 when performing operations.

[0133] The present application provides a method for extracting video keyframes based on a CNN, the method comprising: obtaining first target video segment information; determining a first video input source based on the first target video segment information; obtaining video background features and video recognition object features by performing feature analysis on the first video input source; performing video similarity analysis based on the video background features and the video recognition object features to obtain a first similarity coefficient; constructing a target response function based on the first similarity coefficient, determining first convolution kernel data based on the target response function; constructing a keyframe feature extraction model based on the first convolution kernel data; obtaining a first key feature; inputting the first key feature and the first target video segment information into the keyframe feature extraction model, and obtaining first output information based on the keyframe feature extraction model, wherein the first output information is a keyframe video extraction result based on the first key feature. The method solves the technical problems in the prior art of being unable to accurately extract keyframes based on the changing content of the video and having low flexibility, and achieves the technical effect of improving extraction accuracy and flexibility by performing feature analysis on the video source, thereby further analyzing the video motion changes, and constructing a corresponding CNN convolution data model for keyframe feature extraction.

[0134] Those skilled in the art will understand that the various numerical numbers such as the first and second involved in this application are only for the convenience of description, and are not used to limit the scope of the embodiments of the present application, nor do they represent the order of precedence. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one" refers to one or more. At least two refers to two or more. "At least one", "any one" or similar expressions refer to any combination of these items, including any combination of single items (individuals) or plural items (individuals). For example, at least one item (individual, kind) of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0135] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable systems. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0136] The various illustrative logic units and circuits described in the embodiments of the present application can be implemented or operated by the design of a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic system, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor, alternatively, the general-purpose processor can also be any traditional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing systems, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a digital signal processor core, or any other similar configuration to implement.

[0137] Although the present application has been described with reference to specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and drawings are merely illustrative of the present application as defined herein and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, the present application is intended to include such modifications and variations as fall within the scope of the present application and its equivalents.

Claims

1. A video key frame extraction method based on CNN, characterized in that: The method comprises: Obtaining first target video segment information; determining a first video input source according to the first target video segment information; Obtaining video background features and video recognition object features by performing feature analysis on the first video input source; Performing video similarity analysis based on the video background features and the video recognition object features to obtain a first similarity coefficient; Building a target response function according to the first similarity coefficient, and determining first convolution kernel data according to the target response function; According to the first convolution kernel data, a key frame feature extraction model is built; Obtain the first key feature; Inputting the first key feature and the first target video segment information into the key frame feature extraction model, and obtaining first output information according to the key frame feature extraction model, wherein the first output information is a key frame video extraction result based on the first key feature; The method further comprises: constructing a target response function according to the first similarity coefficient, and determining first convolution kernel data according to the target response function. Obtaining first resource configuration environment information; Performing computing capacity analysis based on the first resource configuration environment information to obtain first computing performance; Using the first computing performance as a constraint variable, the first similarity coefficient as an input variable, and the convolution kernel data as an output result, to construct the target response function, wherein the target response function is used to output a response of the convolution kernel data; According to the target response function, the first convolution kernel data is obtained, wherein the first convolution kernel data includes the convolution kernel size and the number of convolution kernels.

2. The method according to claim 1, wherein The method further comprises: Obtaining a test key feature set, wherein the test key feature set is sample data for performing model testing; Obtaining a preset key video frame set according to the test key feature set, wherein the test key feature set corresponds to the preset key video frame set one-to-one; Inputting the test key feature set into the key frame feature extraction model, and obtaining second output information according to the key frame feature extraction model, wherein the second output information is a key frame video extraction result based on the test key feature set; Comparing the second output information with the preset key video frame set to obtain the first model accuracy.

3. The method according to claim 2, wherein The method further comprises: Determining whether the accuracy of the first model is within a preset model accuracy; If the accuracy of the first model is not within the preset model accuracy, performing quantitative analysis and category analysis on the test key feature set to obtain a first sample quantization coefficient and a first sample feature strength; Obtaining first adjustment information according to the first sample quantization coefficient and the first sample feature strength; The test key feature set is adjusted based on the first adjustment information.

4. The method according to claim 1, wherein The method further comprises: Obtaining a first grouping instruction according to the convolution kernel size and the number of convolution kernels in the first convolution kernel data; Grouping the first convolution kernel data based on the first grouping instruction to generate multiple groups of convolution kernel data; Based on the multiple sets of convolution kernel data, the key frame feature extraction model is built.

5. The method according to claim 1, wherein The method further comprises: Building the key frame feature extraction model, wherein the key frame feature extraction model includes a video preprocessing layer, a key frame extraction layer and a key frame output layer; Inputting the first key feature and the first target video segment information into the key frame feature extraction model; The first target video segment information is preprocessed according to the video preprocessing layer, the key frame extraction layer extracts key frames according to the preprocessed first target video segment information and the first key feature, and outputs the key frame extraction result through the key frame output layer.

6. The method according to claim 1, wherein The method of obtaining the first key feature further comprises: Obtaining first video coverage according to the first key feature, wherein the first video coverage is based on a feature coverage degree of the first key feature in the first target video segment information; Determining whether to activate a feature combination instruction according to the first video coverage; If the feature combination instruction is activated, a second key feature is obtained; Performing feature combination based on the first key feature and the second key feature, and outputting a combined key feature; The combined key feature and the first target video segment information are input into the key frame feature extraction model, and third output information is obtained according to the key frame feature extraction model, wherein the third output information is a key frame video extraction result based on the combined key feature output.

7. A video key frame extraction system based on CNN, characterized in that: The system is used to perform the method according to any one of claims 1 to 6, and the system includes: a first obtaining unit, configured to obtain first target video segment information; a first determining unit, configured to determine a first video input source according to the first target video segment information; a second obtaining unit, configured to obtain video background features and video recognition object features by performing feature analysis on the first video input source; a third obtaining unit, configured to perform video similarity analysis based on the video background feature and the video recognition object feature to obtain a first similarity coefficient; a second determining unit, configured to construct a target response function according to the first similarity coefficient, and determine first convolution kernel data according to the target response function; A first building unit, configured to build a key frame feature extraction model according to the first convolution kernel data; a fourth obtaining unit, configured to obtain the first key feature; A first output unit, wherein the first output unit is used to input the first key feature and the first target video segment information into the key frame feature extraction model, and obtain first output information according to the key frame feature extraction model, wherein the first output information is a key frame video extraction result based on the first key feature.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images