A method, device, terminal and storage medium for identifying empty shot material

Abstract

An embodiment of the present invention provides a method, device, terminal and storage medium for identifying empty shot materials, wherein the method includes: extracting a target frame image to be identified from a target video file; inputting the target frame image into a trained empty shot recognition model, and obtaining an empty shot recognition result obtained by processing the target frame image by the empty shot recognition model; wherein the empty shot recognition model includes a convolutional neural network layer and at least two classifiers with different functions, and the target frame image is input into each classifier after being processed by the convolutional neural network layer, and the empty shot recognition result is obtained according to the recognition results of the at least two classifiers with different functions; when the empty shot recognition result is an empty shot image, the target frame image is saved to a preset storage path; the empty shot material in the target video file can be automatically extracted, and the extracted empty shot material can be saved to the preset storage path, thereby overcoming the problem of difficulty in collecting empty shot materials in related technologies.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, terminal and storage medium for identifying empty shot material. Background Art

[0002] Empty footage, which refers to special effects images of buildings and landscapes without human subjects, is an important tool for storytelling and emotional expression in film and television production. Therefore, empty footage is also an essential reference for user creation and director previews.

[0003] In the related art, one way to collect empty shot materials is through user shooting, which requires a high level of shooting skills from the user and is limited to places that the user has actually visited, resulting in great limitations in the collected empty shot materials; the other way is that when the user sees an empty shot in the video while watching the video, the empty shot materials can be obtained by manually taking screenshots or screen captures. During the operation, the time of taking the screenshot or screen capture needs to be accurately set, which is relatively difficult. If the time point is not accurate, the obtained empty shot materials may be incomplete or have impurities, and are limited to the videos that the user has watched. Summary of the Invention

[0004] The present invention provides a method, device, terminal and computer-readable storage medium for identifying hollow shot materials, so as to solve the problem of difficulty in collecting hollow shot materials in the prior art to a certain extent.

[0005] According to a first aspect of the present invention, a method for identifying empty shot material is provided, the method comprising:

[0006] Extract the target frame image to be identified from the target video file;

[0007] Inputting the target frame image into a trained empty shot recognition model, and obtaining an empty shot recognition result obtained by processing the target frame image by the empty shot recognition model; wherein the empty shot recognition model includes a convolutional neural network layer and at least two classifiers with different functions, and the target frame image is input into each classifier after being processed by the convolutional neural network layer, and the empty shot recognition result is obtained according to the recognition results of the at least two classifiers with different functions;

[0008] When the empty shot recognition result is an empty shot image, the target frame image is saved in a preset storage path.

[0009] Optionally, the empty shot recognition model includes a scene classifier and a purity classifier; the scene classifier is used to identify the category of the scene contained in the image; the purity classifier is used to identify whether the image is an empty shot image; the inputting of the target frame image into the trained empty shot recognition model and obtaining the empty shot recognition result obtained by the empty shot recognition model from processing the target frame image include:

[0010] Inputting the implicit spatial features obtained by the convolutional neural network layer from processing the target frame image into the scene classifier for processing to determine the scene category corresponding to the target frame image;

[0011] When the scene category meets the preset scene category, the implicit space feature is input into the purity classifier, and the empty shot recognition result obtained by the purity classifier processing the implicit space feature is obtained.

[0012] Optionally, the empty shot recognition model includes a landmark classifier and a purity classifier; the landmark classifier is used to identify landmark type images; the purity classifier is used to identify whether an image is an empty shot image; the inputting of the target frame image into the trained empty shot recognition model and obtaining the empty shot recognition result obtained by the empty shot recognition model from processing the target frame image include:

[0013] Inputting the implicit spatial features obtained by the convolutional neural network layer from processing the target frame image into the landmark classifier for processing to determine whether the target frame image is a landmark type image;

[0014] When the target frame image is a landmark type image, the implicit spatial feature is input into the purity classifier, and an empty shot recognition result obtained by the purity classifier processing the implicit spatial feature is obtained.

[0015] Optionally, the empty shot recognition model includes a landmark classifier, a scene classifier, and a purity classifier; the landmark classifier is used to identify images of landmark types; the scene classifier is used to identify the category of scenes contained in an image; the purity classifier is used to identify whether an image is an empty shot image; the inputting of the target frame image into the trained empty shot recognition model and obtaining the empty shot recognition result obtained by the empty shot recognition model from processing the target frame image include:

[0016] Inputting the implicit spatial features obtained by the convolutional neural network layer from processing the target frame image into the landmark classifier for processing to determine whether the target frame image is a landmark type image;

[0017] When the target frame image is a landmark type image, the implicit spatial features are input into the scene classifier, and the scene classifier processes the implicit spatial features to determine the scene category corresponding to the target frame image;

[0018] When the scene category meets the preset scene category, the implicit space feature is input into the purity classifier, and the empty shot recognition result obtained by the purity classifier processing the implicit space feature is obtained.

[0019] Optionally, when the empty shot recognition result is an empty shot image, after saving the target frame image to a preset storage path, the method further includes:

[0020] Acquire all empty-shot images under the target lens corresponding to the key frame from the preset storage path;

[0021] When the proportion of all empty mirror images under the target lens in all frame images corresponding to the target lens is greater than a preset value, all frame images under the target lens are saved as empty mirror images in the preset storage path.

[0022] Optionally, the empty shot recognition result further includes a scene category; when the empty shot recognition result is an empty shot image, the target frame image is saved to a preset storage path, further comprising:

[0023] According to the scene category corresponding to the target frame image, the target frame image is saved in a sub-path corresponding to the scene category in the preset storage path.

[0024] Optionally, extracting a target frame image to be identified from a target video file includes:

[0025] Extract target frame images from target video files according to target intervals; or

[0026] A target frame image corresponding to a shot is extracted from the shot contained in the target video file.

[0027] According to a second aspect of the present invention, there is provided an apparatus for identifying empty shot material, the apparatus comprising:

[0028] An image extraction module is used to extract a target frame image to be identified from a target video file;

[0029] An empty shot recognition module is used to input the target frame image into a trained empty shot recognition model and obtain an empty shot recognition result obtained by processing the target frame image by the empty shot recognition model; wherein the empty shot recognition model includes a convolutional neural network layer and at least two classifiers with different functions, the target frame image is processed by the convolutional neural network layer and input into each classifier respectively, and the empty shot recognition result is obtained based on the recognition results of the at least two classifiers with different functions;

[0030] The empty shot saving module is used to save the target frame image to a preset storage path when the empty shot recognition result is an empty shot image.

[0031] Optionally, the empty shot recognition model includes a scene classifier and a purity classifier; the scene classifier is used to identify the category of the scene contained in the image; the purity classifier is used to identify whether the image is an empty shot image; the empty shot recognition module includes:

[0032] A scene classifier processing module, configured to input the implicit spatial features obtained by the convolutional neural network layer processing the target frame image into the scene classifier for processing to determine the scene category corresponding to the target frame image;

[0033] The scene classifier-based processing result recognition module is used to input the implicit space feature into the purity classifier when the scene category meets the preset scene category, and obtain the empty shot recognition result obtained by the purity classifier processing the implicit space feature.

[0034] Optionally, the empty shot recognition model includes a landmark classifier and a purity classifier; the landmark classifier is used to identify images of landmark types; the purity classifier is used to identify whether an image is an empty shot image; the empty shot recognition module includes:

[0035] A landmark classifier processing module, configured to input the implicit spatial features obtained by the convolutional neural network layer processing the target frame image into the landmark classifier for processing to determine whether the target frame image is a landmark type image;

[0036] The landmark classifier-based processing result recognition module is used to input the implicit spatial features into the purity classifier when the target frame image is a landmark type image, and obtain the empty shot recognition result obtained by the purity classifier processing the implicit spatial features.

[0037] Optionally, the empty shot recognition model includes a landmark classifier, a scene classifier, and a purity classifier; the landmark classifier is used to identify images of landmark types; the scene classifier is used to identify the category of scenes contained in the image; the purity classifier is used to identify whether the image is an empty shot image; the empty shot recognition module includes:

[0038] A landmark classifier processing module, configured to input the implicit spatial features obtained by the convolutional neural network layer processing the target frame image into the landmark classifier for processing to determine whether the target frame image is a landmark type image;

[0039] A scene classifier processing module based on landmark results, configured to input the implicit spatial features into the scene classifier when the target frame image is a landmark type image, and have the scene classifier process the implicit spatial features to determine the scene category corresponding to the target frame image;

[0040] The purity classifier processing module is used to input the implicit space feature into the purity classifier when the scene category meets the preset scene category, and obtain the empty shot recognition result obtained by the purity classifier processing the implicit space feature.

[0041] Optionally, the empty shot storage module further includes:

[0042] The empty shot acquisition module of the target shot is used to acquire all empty shot images under the target shot corresponding to the key frame from the preset storage path;

[0043] The empty shot keeping module of the target lens is used to save all the frame images under the target lens as empty shot images to the preset storage path when the proportion of all the empty shot images under the target lens in all the frame images corresponding to the target lens is greater than a preset value.

[0044] Optionally, the empty shot recognition result also includes a scene category; the empty shot saving module is further used to save the target frame image to the preset storage path and the sub-path corresponding to the scene category according to the scene category corresponding to the target frame image.

[0045] Optionally, the image extraction module is configured to extract target frame images from a target video file according to a target interval; or to extract target frame images corresponding to shots contained in the target video file.

[0046] According to the third aspect of the present invention, a terminal is provided, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements any of the above-mentioned methods for identifying empty shot materials when executed by the processor.

[0047] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned methods for identifying empty shot materials is implemented.

[0048] Compared with the background technology, the embodiments of the present invention have the following advantages:

[0049] An embodiment of the present invention extracts a target frame image to be identified from a target video file; inputs the target frame image into a trained empty shot recognition model, which includes a convolutional neural network layer and at least two classifiers with different functions; the target frame image is processed by the convolutional neural network layer and input into each classifier respectively, and an empty shot recognition result is obtained according to the recognition results of the at least two classifiers with different functions; obtains the empty shot recognition result obtained by the empty shot recognition model on the target frame image; when the empty shot recognition result is an empty shot image, the target frame image is saved to a preset storage path; it is possible to automatically extract the empty shot material in the target video file and save the extracted empty shot material to the preset storage path, thereby overcoming the problem of difficulty in collecting empty shot materials in the related art.

[0050] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0052] Figure 1 This is a schematic flow chart of the steps of a method for identifying empty shot material provided by an embodiment of the present invention;

[0053] Figure 2 1 is a schematic diagram of a model structure corresponding to an exemplary method for identifying empty shot material in an embodiment of the present invention;

[0054] Figure 3It is a schematic structural block diagram of an empty shot material recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0056] In response to the problem of difficulty in collecting empty shot materials in related technologies, an embodiment of the present invention provides a method for identifying empty shot materials, which extracts a target frame image to be identified from a target video file; inputs the target frame image into a trained empty shot recognition model, which includes a convolutional neural network layer and at least two classifiers with different functions. The target frame image is processed by the convolutional neural network layer and input into each classifier respectively, and an empty shot recognition result is obtained based on the recognition results of the at least two classifiers with different functions; and the empty shot recognition result obtained by the empty shot recognition model on the target frame image is obtained; when the empty shot recognition result is an empty shot image, the target frame image is saved to a preset storage path; the empty shot material in the target video file can be automatically extracted, and the extracted empty shot material is saved to the preset storage path, thereby overcoming the problem of difficulty in collecting empty shot materials in related technologies.

[0057] Figure 1 This is a flowchart of an exemplary method for identifying empty shot material provided by an embodiment of the present invention. The method can be applied to electronic devices to accurately identify video frames and video clips in the target video file to be identified as empty shots. The electronic device can be hardware or software. When the electronic device is hardware, it can be implemented as a cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the electronic device is software, it can be installed in the hardware devices listed above. Figure 1 As shown, the empty shot material recognition method provided in this embodiment may include the following steps:

[0058] Step 101: extract a target frame image to be identified from a target video file.

[0059] The target video file can be considered to be the video from which the blank footage is to be extracted. The target video file can be a video file of a TV series, movie, documentary, or other genre. Specifically, the target video file can be specified by the user, or it can be a video file currently being played. The target frame image to be identified is the image to be identified as blank footage.

[0060] The process of extracting the target frame image to be identified from the target video file may include: extracting the key frames of the target video file by equal interval extraction, shot key frame extraction, etc. to obtain the target frame image, without specific limitation.

[0061] Step 102: Input the target frame image into the trained empty shot recognition model, and obtain the empty shot recognition result obtained by the empty shot recognition model processing the target frame image; wherein, the empty shot recognition model includes a convolutional neural network layer and at least two classifiers with different functions; the target frame image is processed by the convolutional neural network layer and input into each classifier respectively, and the empty shot recognition result is obtained according to the recognition results of at least two classifiers with different functions.

[0062] The empty shot recognition model is used to identify whether an image is empty shot material, that is, to identify whether the scene in the image is an empty shot. Exemplarily, the empty shot recognition model can be a convolutional neural network (CNN) model, which can adopt a variety of typical image classification backbone networks, such as AlexNet, VGG, ResNet, RegNet, Transformer, etc., without specific limitation.

[0063] By inputting the target frame image to be identified into the empty shot recognition model, and the empty shot recognition model processes the input target frame image to output the corresponding empty shot recognition result, the empty shot image can be automatically, efficiently and accurately identified.

[0064] Among them, the empty mirror recognition model includes a convolutional neural network layer and at least two classifiers with different functions. After the convolutional neural network layer extracts the features of the target frame image, multiple classifiers with different functions recognize the image from multiple aspects, which can improve the accuracy of the recognition results and reduce the requirements for the generalization performance of the classifier, thereby reducing the training difficulty of the empty mirror recognition model.

[0065] Step 103: When the empty shot recognition result is an empty shot image, the target frame image is saved in a preset storage path.

[0066] In this embodiment, when the empty shot recognition result is an empty shot image, that is, the picture in the target frame image to be identified is an empty shot, the target frame image is saved to a preset storage path, thereby automatically collecting the empty shot material in the target video file so that the user can retrieve it when needed.

[0067] The embodiment of the present invention extracts a target frame image to be identified from a target video file; inputs the target frame image into a trained empty shot recognition model, and obtains an empty shot recognition result obtained by processing the target frame image by the empty shot recognition model; wherein, the empty shot recognition model includes a convolutional neural network layer and at least two classifiers with different functions, and the target frame image is input into each classifier after being processed by the convolutional neural network layer, and the empty shot recognition result is obtained according to the recognition results of the at least two classifiers with different functions; when the empty shot recognition result is an empty shot image, the target frame image is saved to a preset storage path; the empty shot material in the target video file can be automatically extracted, and the extracted empty shot material can be saved to the preset storage path. At the same time, by using the empty shot recognition model including the convolutional neural network layer and at least two classifiers with different functions, the efficiency and accuracy of empty shot recognition can be effectively improved, and the training difficulty of the empty shot recognition model can be reduced.

[0068] Next, the hollow mirror material recognition method in this exemplary embodiment will be further described.

[0069] In step 101, a target frame image to be identified is extracted from a target video file.

[0070] The target video file can be considered as the video from which the blank shot material is to be extracted. The target video file can be a video of a TV series, movie, documentary, or other type. Specifically, the target video file can be specified by the user, or it can be a video currently being played. The target frame image to be identified refers to the image to be identified as blank shot material. For example, methods such as equal-interval extraction and shot key frame extraction can be used to extract the key frames of the target video file to obtain the target frame image.

[0071] Exemplarily, a video input interface to be identified can be displayed to the user, and the user selects the target video file for which empty shot materials need to be collected and inputs it into the video input interface. The target video file can be one or multiple. When there are multiple target video files, the empty shot material can be identified for each target video file in turn in a single-threaded manner, or the empty shot material can be identified for multiple target video files at the same time in a multi-threaded manner. The process of identifying the empty shot material for each target video file can be considered to be the same. The following will be described exemplarily using one target video file as an example.

[0072] In one example, the process of extracting the target frame image to be identified from the target video file may include:

[0073] Extract target frame images from the target video file according to the target interval.

[0074] In this example, target frame images can be extracted from the target video file according to the target interval. Each time a video frame is extracted, the video frame is used as the target frame image to be currently identified, and a subsequent process of identifying whether the target frame image is a blank image is performed. The target interval can be a target time interval, for example, extracting a video frame from the target video file every n seconds as the target frame image to be currently identified, where n can be set or modified according to actual needs. In addition, the target interval can also be a target number of video frame intervals, that is, extracting a video frame every m video frames as the target frame image to be currently identified, where m can be set or modified according to actual needs.

[0075] In another example, the process of extracting the target frame image to be identified from the target video file may include:

[0076] A target frame image corresponding to the shot is extracted from the shot contained in the target video file.

[0077] In this example, the target video file may include multiple shots, wherein the similarity between adjacent video frames within the same shot may be greater than or equal to a first threshold. Specifically, shot detection technology can be used to determine the shots contained in the target video file and the video frames corresponding to each shot. For each shot, one or more video frames can be extracted as target frame images to be identified. Each time a video frame is extracted, the video frame is used as the current target frame image to be identified, and a subsequent process is performed to identify whether the target frame image is an empty shot image.

[0078] For each shot, the specific process for extracting the target frame image corresponding to that shot from the target video file can be performed by extracting the target frame image from the video clip corresponding to that shot according to the target interval. The process of extracting the target frame image according to the target interval can be found in the previous description and will not be repeated here. The target frame image can also be extracted from the video clip corresponding to the shot in combination with the similarity of adjacent video frames; for example, when the similarity of adjacent video frames is less than a preset similarity threshold, the two adjacent video frames can be extracted simultaneously as the target frame image to be identified.

[0079] In step 102, the target frame image is input into the trained empty shot recognition model, and the empty shot recognition result obtained by the empty shot recognition model processing the target frame image is obtained; the empty shot recognition model includes a convolutional neural network layer and at least two classifiers with different functions.

[0080] The empty shot recognition model is used to identify whether an image contains empty shots, that is, to determine whether the image contains empty shots. By inputting the target frame image to be identified into the empty shot recognition model, the empty shot recognition model processes the input target frame image and outputs the corresponding empty shot recognition result, achieving automatic, efficient, and accurate recognition of empty shots.

[0081] The empty mirror recognition model in this embodiment includes a convolutional neural network layer and at least two classifiers with different functions. By using multiple classifiers with different functions to recognize images from multiple aspects, the accuracy of the recognition results can be improved, and the requirements for the generalization performance of the classifier can be reduced, thereby reducing the training difficulty of the empty mirror recognition model.

[0082] In some optional embodiments of the present invention, the above-mentioned empty shot recognition model includes a scene classifier and a purity classifier, that is, the empty shot recognition model in this embodiment includes a convolutional neural network layer, a scene classifier, and a purity classifier. The convolutional neural network layer processes the target frame image to obtain implicit spatial features. The purity classifier is used to identify whether the image is an empty shot image. Specifically, the purity classifier is used to identify empty shot materials with a certain degree of picture integrity. It can be understood that in addition to not containing people, empty shot materials are generally medium-long perspective images. In this embodiment, the purity classifier is obtained by training the initial purity classifier using the first training sample. The first training sample is an empty shot image marked with an empty shot label. The scene classifier is used to identify the category of the scene contained in the image. It can be obtained by training the initial scene classifier using the second training sample. The second training sample is an image marked with a scene category.

[0083] The purity classifier can be implemented using a support vector machine (SVM) or a fully connected layer. In a specific implementation, a convolutional neural network layer can be used to extract features from the image in the first training sample to obtain the corresponding latent spatial features. The latent spatial features are then used as input data for the initial purity classifier (i.e., the purity classifier to be trained), and the empty mirror label is used as the target value output by the initial purity classifier to train the parameters of the initial purity classifier to obtain a trained purity classifier.

[0084] The scene classifier can be implemented using a fully connected layer. Exemplarily, the scene classifier can be obtained by training the initial scene classifier with the second training sample, where the second training sample is an image with the scene category marked, and cross entropy or additive angle interval is used as a metric function, and gradient back propagation is used to learn the parameters of the convolutional neural network model. Specifically, the image in the second training sample can be first feature extracted through a convolutional neural network layer to obtain the corresponding latent spatial features. The latent spatial features are then used as the input data of the initial scene classifier (i.e., the scene classifier to be trained), and the scene type label is used as the target value output by the initial scene classifier to train the parameters of the initial scene classifier to obtain a trained scene classifier.

[0085] In this embodiment, the inputs of the scene classifier and the purity classifier are both implicit spatial features obtained by processing the target frame image by the convolutional neural network layer; therefore, the underlying model structure and parameters in the empty shot recognition model can be shared in the scene classifier and the purity classifier to reduce the training difficulty of the empty shot recognition model, and at the same time, the recognition effect of the empty shot recognition model can be improved.

[0086] The above-mentioned inputting the target frame image into the trained empty shot recognition model and obtaining the empty shot recognition result obtained by the empty shot recognition model processing the target frame image includes:

[0087] The implicit spatial features obtained by the convolutional neural network layer processing the target frame image are input into the scene classifier for processing to determine the scene category corresponding to the target frame image;

[0088] When the scene category meets the preset scene category, the implicit space feature is input into the purity classifier, and the empty shot recognition result obtained by the purity classifier processing the implicit space feature is obtained.

[0089] In this embodiment, the target frame image is first processed using a convolutional neural network to obtain implicit spatial features. These implicit spatial features are then processed using a scene classifier to determine the scene category corresponding to the target frame image. If the scene category corresponding to the target frame image matches a preset scene category, the implicit spatial features are further processed using a purity classifier, which then outputs an empty shot recognition result. The empty shot recognition result can also include the scene category information of the target frame image.

[0090] For example, the preset scene categories can be architecture and / or scenery. The architecture category can include aerial photography of cities, building exteriors, bridges, traditional Chinese architecture, Western-style churches, and square sculptures, while the scenery category can include images of oceans, beaches, mountains, canyons, bamboo forests, and deserts. It will be appreciated that images corresponding to the preset scene categories are generally empty-frame images. For example, if the scene category is portrait, it clearly does not match the preset scene categories, and the image in the portrait category can be directly determined to be a non-empty-frame image.

[0091] This embodiment combines scene categories with target frame images to perform empty shot recognition, which can perform an extra layer of filtering on the target frame images to be recognized, and exclude a portion of target frame images of non-empty shot types as early as possible. Combined with the feature of the purity classifier that recognizes the integrity of the image, the images that meet the preset scene categories recognized by the scene classifier are further recognized and filtered, which can improve the efficiency and accuracy of empty shot recognition. Specifically, the target frame image is processed by the convolutional neural network layer of the empty shot recognition model, and the implicit spatial features obtained are input into the scene classifier for processing. The scene category output by the scene classifier determines whether the implicit spatial features need to be input into the purity classifier for processing to output the empty shot recognition result.

[0092] In some optional embodiments of the present invention, the above-mentioned empty shot recognition model includes a landmark classifier and a purity classifier. That is to say, the empty shot recognition model in this embodiment includes a convolutional neural network layer, a landmark classifier, and a purity classifier. The convolutional neural network layer processes the target frame image to obtain implicit spatial features, and the purity classifier is used to identify whether the image is an empty shot image. For details, please refer to the previous description.

[0093] The landmark classifier is used to identify images of landmark types, and the landmark classifier can be implemented using a fully connected layer. Exemplarily, the landmark classifier can be obtained by training the initial landmark classifier with the third training sample, and the third training sample is an image with the landmark category marked, and uses cross entropy or additive angle interval as a metric function, and adopts gradient back propagation to learn the parameters of the convolutional neural network model. Among them, the image of the landmark type can refer to an image with a certain picture distinction / scene recognition, including but not limited to images containing iconic areas or places, buildings and natural scenery. Specifically, the image in the third training sample can be first subjected to feature extraction through a convolutional neural network layer to obtain the corresponding implicit spatial features. The implicit spatial features are then used as the input data of the initial landmark classifier (i.e., the landmark classifier to be trained), and the landmark type label is used as the target value output by the initial landmark classifier to train the parameters of the initial landmark classifier to obtain a trained landmark classifier.

[0094] In this embodiment, the inputs of the landmark classifier and the purity classifier are both implicit spatial features obtained by processing the target frame image by the convolutional neural network layer; therefore, the underlying model structure and parameters in the empty mirror recognition model can be shared in the landmark classifier and the purity classifier to reduce the training difficulty of the empty mirror recognition model, and at the same time, the recognition effect of the empty mirror recognition model can be improved.

[0095] The above-mentioned inputting the target frame image into the trained empty shot recognition model and obtaining the empty shot recognition result obtained by the empty shot recognition model processing the target frame image includes:

[0096] The implicit spatial features obtained by the convolutional neural network layer processing the target frame image are input into the landmark classifier for processing to determine whether the target frame image is a landmark type image;

[0097] When the target frame image is a landmark type image, the implicit spatial features are input into the purity classifier, and the empty shot recognition result obtained by the purity classifier processing the implicit spatial features is obtained.

[0098] In this embodiment, the target frame image is first processed by a convolutional neural network to obtain implicit spatial features. Then, the implicit spatial features are processed by a landmark classifier to determine whether the target frame image is a landmark type image. When the target frame image is a landmark type image, the implicit spatial features are processed by a purity classifier, and the purity classifier outputs the empty mirror recognition result.

[0099] This embodiment combines the landmark type with the target frame image to perform empty shot recognition, and can perform an additional layer of filtering on the target frame image to be recognized, excluding a portion of the target frame images of non-empty shot type as early as possible. Then, combined with the feature of the purity classifier that recognizes the integrity of the image, the landmark type images recognized by the landmark classifier are further recognized and filtered to improve the efficiency and accuracy of empty shot recognition. Specifically, the target frame image is processed by the convolutional neural network layer of the empty shot recognition model, and the implicit spatial features obtained are input into the landmark classifier for processing. The landmark type judgment result output by the landmark classifier determines whether the implicit spatial features need to be input into the purity classifier for processing to output the empty shot recognition result.

[0100] In some optional embodiments of the present invention, the above-mentioned empty shot recognition model includes a landmark classifier, a scene classifier, and a purity classifier. In other words, the empty shot recognition model in this embodiment includes a convolutional neural network layer, a landmark classifier, a purity classifier, and a scene classifier. The relevant descriptions of the landmark classifier, scene classifier, and purity classifier can be found in the above description.

[0101] In this embodiment, the inputs of the landmark classifier, scene classifier and purity classifier are all implicit spatial features obtained by processing the target frame image by the convolutional neural network layer; therefore, the underlying model structure and parameters in the empty mirror recognition model can be shared among the landmark classifier, scene classifier and purity classifier to reduce the training difficulty of the empty mirror recognition model and at the same time, improve the recognition effect of the empty mirror recognition model.

[0102] The above-mentioned inputting the target frame image into the trained empty shot recognition model and obtaining the empty shot recognition result obtained by the empty shot recognition model processing the target frame image includes:

[0103] The implicit spatial features obtained by the convolutional neural network layer processing the target frame image are input into the landmark classifier for processing to determine whether the target frame image is a landmark type image;

[0104] When the target frame image is a landmark type image, the latent spatial features are input into the scene classifier, which processes the latent spatial features to determine the scene category corresponding to the target frame image;

[0105] When the scene category meets the preset scene category, the implicit space feature is input into the purity classifier, and the empty shot recognition result obtained by the purity classifier processing the implicit space feature is obtained.

[0106] In this embodiment, the target frame image is first processed using a convolutional neural network to obtain implicit spatial features. The implicit spatial features are then processed using a landmark classifier to determine whether the target frame image is a landmark-type image. If the target frame image is a landmark-type image, the implicit spatial features are then processed using a scene classifier to determine the scene category corresponding to the target frame image. If the scene category corresponding to the target frame image meets a preset scene category, the implicit spatial features are then processed using a purity classifier, which then outputs an empty shot recognition result. At this point, the empty shot recognition result may also include scene category information for the target frame image.

[0107] Optionally, in the process of training the initial landmark classifier, this embodiment can also perform multi-task learning in conjunction with the scene classifier loss function; it can also freeze the parameters of the convolutional neural network layer and only learn the landmark classifier parameters, thereby reducing the difficulty of classifier training and improving training efficiency.

[0108] This embodiment combines landmark types and scene categories to perform empty mirror recognition on target frame images, and can perform an additional layer of filtering on the target frame images to be recognized, directly judging non-landmark type target frame images as non-empty mirror images, and at the same time directly judging target frame images that do not conform to the preset scene categories as non-empty mirror images, so that most non-empty mirror type target frame images can be excluded as early as possible to improve the efficiency and accuracy of empty mirror recognition. It can be understood that this embodiment can reduce the requirements for the generalization performance of the classifier and alleviate the difficulty of classifier training by constraining the input sample distribution of the purity classifier through model cascade; thus, accurate empty mirror recognition can be achieved by collecting a small amount of training samples; and multiple classifiers can share the parameters of the convolutional neural network layer, so that the overall computational complexity is low and the recognition efficiency is high.

[0109] In step 103, when the empty shot recognition result is an empty shot image, the target frame image is saved in a preset storage path.

[0110] In this embodiment, when the empty shot recognition result is an empty shot image, that is, the picture in the target frame image to be identified is an empty shot, the target frame image is saved to a preset storage path, thereby automatically collecting the empty shot material in the target video file so that the user can retrieve it when needed.

[0111] Furthermore, when the empty shot recognition result includes a scene category, when the empty shot recognition result is an empty shot image, the target frame image is saved to a preset storage path, and the following steps may also be included:

[0112] According to the scene category corresponding to the target frame image, the target frame image is saved in a sub-path corresponding to the scene category in a preset storage path.

[0113] In this embodiment, when the empty mirror recognition result includes a scene category, in the process of saving the target frame image whose empty mirror recognition result is an empty mirror image, the empty mirror images belonging to the same scene category can also be saved in the same path. Exemplarily, the preset storage path can include multiple sub-paths related to the scene category. When saving the empty mirror image, the target sub-path for saving the empty mirror image can be first determined based on the scene category of the empty mirror image, and the empty mirror image can be saved in the target sub-path. This embodiment classifies and saves the empty mirror images according to the scene category, which is more convenient for users to use.

[0114] Furthermore, when the empty shot recognition result is an empty shot image, after saving the target frame image to a preset storage path, the following steps may be further included:

[0115] Obtain all empty mirror images under the target lens corresponding to the key frame from the preset storage path;

[0116] When the proportion of all empty mirror images under the target lens in all frame images corresponding to the target lens is greater than a preset value, all frame images under the target lens are saved as empty mirror images in a preset storage path.

[0117] In this embodiment, after the target frame image is saved in the preset storage path, all empty mirror images under the target lens corresponding to the target frame image can also be obtained from the preset storage path, and then the ratio of all empty mirror images under the target lens stored in the preset storage path in all frame images corresponding to the target lens is calculated, wherein all frame images under the target lens refer to the video frames contained in the video clip collected between the start time and the end time of the target lens. When the ratio is greater than a preset value (which can be set or modified as needed), it is considered that all frame images under the target lens are empty mirror images, and all frame images under the target lens are saved in the preset storage path, that is, the video clip corresponding to the target lens is saved in the preset storage path. By calculating the ratio of the number of empty mirror images under the same lens to the number of all video frames under the lens, when the ratio meets the preset conditions, all video frames under the lens are saved as empty mirror materials in the preset storage path, so that the acquisition of empty mirror materials not only includes single frame images, but also includes coherent video clips, thereby enriching the empty mirror materials saved in the preset storage path.

[0118] Exemplarily, when there are p empty shot images under the target lens stored in the preset storage path, and there are q frames of all the frame images under the target lens, when p / q>preset value α, the video clips captured between the start time and the end time of the target lens are saved as empty shot material clips in the preset storage path.

[0119] The embodiment of the present invention extracts a target frame image to be identified from a target video file; inputs the target frame image into a trained empty shot recognition model, and obtains an empty shot recognition result obtained by processing the target frame image by the empty shot recognition model; wherein the empty shot recognition model includes a convolutional neural network layer and at least two classifiers with different functions; when the empty shot recognition result is an empty shot image, the target frame image is saved to a preset storage path; it is possible to automatically extract the empty shot material from the target video file, and save the extracted empty shot material to the preset storage path. At the same time, by using the empty shot recognition model including the convolutional neural network layer and at least two classifiers with different functions, the efficiency and accuracy of empty shot recognition can be effectively improved, and the training difficulty of the empty shot recognition model can be reduced.

[0120] In order to facilitate those skilled in the art to understand the present invention, the following Figure 2 Provide illustrative explanations and illustrations.

[0121] Figure 2FIG. 1 is a flow chart showing a schematic diagram of a model structure corresponding to an exemplary method for identifying empty shot material in an embodiment of the present invention. Figure 2 As shown, the empty shot material recognition method in this embodiment first extracts the target frame image of the target video file to obtain the target frame image to be identified. After the target frame image is processed by the empty shot recognition model, the corresponding empty shot recognition result can be obtained. Specifically, the empty shot recognition model may include a convolutional neural network layer for extracting the features of the target frame image to obtain implicit spatial features; there are three ways to process the implicit spatial features to obtain the corresponding empty shot recognition results. The first is to directly process the implicit spatial features through a purity classifier to obtain the corresponding empty shot recognition result; the second is to process the implicit spatial features through a scene classifier to obtain the corresponding scene category, and then filter the scene category. When the scene category meets the conditions, the purity classifier is used to process the implicit spatial features to obtain the corresponding empty shot recognition result; the third is to first process the implicit spatial features through a landmark classifier, and when the processing result is non-landmark, the implicit spatial features are processed by the landmark classifier. When a landmark image is detected, the target frame image is directly determined to be a non-empty mirror image without subsequent processing. When the processing result is a landmark image, the purity classifier is used to process the implicit spatial features to obtain the corresponding empty mirror recognition result; the fourth is to first process the implicit spatial features through the landmark classifier. When the processing result is a non-landmark image, the target frame image is directly determined to be a non-empty mirror image without subsequent processing. When the processing result is a landmark image, the scene classifier is used to process the implicit spatial features to obtain the corresponding scene category, and then the scene category is screened. When the scene category meets the conditions, the purity classifier is used to process the implicit spatial features to obtain the corresponding empty mirror recognition result.

[0122] When the empty shot recognition result is an empty shot image, the empty shot image is saved; at the same time, the target video file can be combined with shot detection to perform frame-level point post-processing to obtain an empty shot material video clip, that is, through the frame number information of the target frame image, the target frame image is associated with the video shot point. If the ratio of the empty shot image contained in a shot to all the video frames in the shot is greater than the preset value, the video clip corresponding to the shot is saved as an empty shot material video clip.

[0123] Through the embodiments of the present invention, it is possible to automatically extract the empty shot materials (including empty shot images and empty shot video clips) in the target video file, and save the extracted empty shot materials to a preset storage path for user convenience, thereby solving the problem of difficulty in collecting empty shot materials.

[0124] It should be noted that for the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0125] Reference Figure 3 , shows a schematic structural block diagram of a device for identifying empty shot material according to an embodiment of the present invention. Corresponding to the above-mentioned method for identifying empty shot material, the device may specifically include the following modules:

[0126] An image extraction module is used to extract a target frame image to be identified from a target video file;

[0127] An empty shot recognition module is used to input a target frame image into a trained empty shot recognition model and obtain an empty shot recognition result obtained by processing the target frame image by the empty shot recognition model; wherein the empty shot recognition model includes a convolutional neural network layer and at least two classifiers with different functions. After the target frame image is processed by the convolutional neural network layer, it is input into each classifier respectively, and the empty shot recognition result is obtained based on the recognition results of the at least two classifiers with different functions;

[0128] The empty shot saving module is used to save the target frame image to a preset storage path when the empty shot recognition result is an empty shot image.

[0129] In an optional embodiment of the present invention, the empty shot recognition model includes a scene classifier and a purity classifier; the scene classifier is used to identify the category of the scene contained in the image; the purity classifier is used to identify whether the image is an empty shot image; the empty shot recognition module includes:

[0130] A scene classifier processing module is used to input the implicit spatial features obtained by the convolutional neural network layer from processing the target frame image into the scene classifier for processing to determine the scene category corresponding to the target frame image;

[0131] The scene classifier-based processing result recognition module is used to input the implicit space features into the purity classifier when the scene category meets the preset scene category, and obtain the empty shot recognition result obtained by the purity classifier processing the implicit space features.

[0132] In an optional embodiment of the present invention, the empty shot recognition model includes a landmark classifier and a purity classifier; the landmark classifier is used to identify images of landmark types; the purity classifier is used to identify whether an image is an empty shot image; the empty shot recognition module includes:

[0133] The landmark classifier processing module is used to input the implicit spatial features obtained by the convolutional neural network layer from the target frame image into the landmark classifier for processing to determine whether the target frame image is a landmark type image;

[0134] The landmark classifier processing result recognition module is used to input the implicit spatial features into the purity classifier when the target frame image is a landmark type image, and obtain the empty shot recognition result obtained by the purity classifier processing the implicit spatial features.

[0135] In an optional embodiment of the present invention, the empty shot recognition model includes a landmark classifier, a scene classifier, and a purity classifier; the landmark classifier is used to identify images of landmark types; the scene classifier is used to identify the category of scenes contained in the image; the purity classifier is used to identify whether the image is an empty shot image; the empty shot recognition module includes:

[0136] The landmark classifier processing module is used to input the implicit spatial features obtained by the convolutional neural network layer from the target frame image into the landmark classifier for processing to determine whether the target frame image is a landmark type image;

[0137] A scene classifier processing module based on landmark results is used to input the implicit spatial features into the scene classifier when the target frame image is a landmark type image, and the scene classifier processes the implicit spatial features to determine the scene category corresponding to the target frame image;

[0138] The purity classifier processing module is used to input the implicit spatial features into the purity classifier when the scene category meets the preset scene category, and obtain the empty shot recognition result obtained by the purity classifier processing the implicit spatial features.

[0139] In an optional embodiment of the present invention, the empty shot storage module further includes:

[0140] The empty shot acquisition module of the target shot is used to obtain all empty shot images under the target shot corresponding to the key frame from a preset storage path;

[0141] The empty shot holding module of the target lens is used to save all frame images under the target lens as empty shot images to a preset storage path when the proportion of all empty shot images under the target lens in all frame images corresponding to the target lens is greater than a preset value.

[0142] In an optional embodiment of the present invention, the empty shot recognition result also includes a scene category; the empty shot saving module is also used to save the target frame image to a preset storage path and a sub-path corresponding to the scene category according to the scene category corresponding to the target frame image.

[0143] In an optional embodiment of the present invention, the image extraction module is used to extract target frame images from the target video file according to the target interval; or to extract target frame images corresponding to the shots contained in the target video file.

[0144] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0145] An embodiment of the present invention also provides a terminal, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the various processes of the embodiment of the empty shot material recognition method described above are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0146] The present invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various steps of the above-described embodiment of the method for identifying empty shot material, achieving the same technical effect. To avoid repetition, the details are omitted here. The computer-readable storage medium may be, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0147] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0148] It is easy for those skilled in the art to think that any combination of the above embodiments is feasible, so any combination of the above embodiments is an implementation scheme of the present invention. However, due to space limitations, this specification will not describe them in detail here.

[0149] The method for identifying empty shot material provided herein is not inherently associated with any particular computer, virtual system, or other device. Various general-purpose systems may also be used in conjunction with the teachings herein. Based on the above description, it is apparent that the structure required to construct a system having the present invention's solution is suitable. Furthermore, the present invention is not directed to any particular programming language. It should be understood that the present invention described herein may be implemented using a variety of programming languages, and the description of specific languages ​​above is intended to disclose the best mode of implementation of the present invention.

[0150] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0151] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Accordingly, the claims that follow the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the present invention.

[0152] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0153] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

[0154] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components of the empty shot material recognition method according to an embodiment of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for implementing the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0155] It should be noted that the above embodiments illustrate rather than limit the invention, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

Claims

1. A method for identifying empty shot material, characterized in that: The method comprises: Extract the target frame image to be identified from the target video file; Inputting the target frame image into a trained empty shot recognition model, and obtaining an empty shot recognition result obtained by processing the target frame image by the empty shot recognition model; wherein the empty shot recognition model includes a convolutional neural network layer and at least two classifiers with different functions, and the target frame image is input into each classifier after being processed by the convolutional neural network layer, and the empty shot recognition result is obtained according to the recognition results of the at least two classifiers with different functions; When the empty shot recognition result is an empty shot image, the target frame image is saved in a preset storage path; The empty shot recognition model includes a landmark classifier and a purity classifier; the landmark classifier is used to identify landmark type images; the purity classifier is used to identify whether an image is an empty shot image; the target frame image is input into the trained empty shot recognition model, and the empty shot recognition result obtained by the empty shot recognition model after processing the target frame image is obtained, including: Inputting the implicit spatial features obtained by the convolutional neural network layer from processing the target frame image into the landmark classifier for processing to determine whether the target frame image is a landmark type image; When the target frame image is a landmark type image, the implicit spatial feature is input into the purity classifier, and an empty shot recognition result obtained by the purity classifier processing the implicit spatial feature is obtained.

2. The method according to claim 1, characterized in that The empty shot recognition model further includes a scene classifier; the scene classifier is used to identify the category of the scene contained in the image; the target frame image is input into the trained empty shot recognition model, and the empty shot recognition result obtained by the empty shot recognition model after processing the target frame image is obtained, including: Inputting the implicit spatial features obtained by the convolutional neural network layer from processing the target frame image into the landmark classifier for processing to determine whether the target frame image is a landmark type image; When the target frame image is a landmark type image, the implicit spatial features are input into the scene classifier, and the scene classifier processes the implicit spatial features to determine the scene category corresponding to the target frame image; When the scene category meets the preset scene category, the implicit space feature is input into the purity classifier, and the empty shot recognition result obtained by the purity classifier processing the implicit space feature is obtained.

3. The method according to claim 1, characterized in that When the empty shot recognition result is an empty shot image, after saving the target frame image to a preset storage path, the method further includes: Acquire all empty-shot images under the target lens corresponding to the key frame from the preset storage path; When the proportion of all empty mirror images under the target lens in all frame images corresponding to the target lens is greater than a preset value, all frame images under the target lens are saved as empty mirror images in the preset storage path.

4. The method according to claim 2, characterized in that The empty shot recognition result also includes a scene category; when the empty shot recognition result is an empty shot image, the target frame image is saved to a preset storage path, further comprising: According to the scene category corresponding to the target frame image, the target frame image is saved in a sub-path corresponding to the scene category in the preset storage path.

5. The method according to claim 1, wherein The step of extracting a target frame image to be identified from a target video file includes: Extract target frame images from target video files according to target intervals; or A target frame image corresponding to a shot is extracted from the shot contained in the target video file.

6. A device for identifying empty shot material, characterized in that: The device comprises: An image extraction module is used to extract a target frame image to be identified from a target video file; An empty shot recognition module is used to input the target frame image into a trained empty shot recognition model and obtain an empty shot recognition result obtained by processing the target frame image by the empty shot recognition model; wherein the empty shot recognition model includes a convolutional neural network layer and at least two classifiers with different functions, the target frame image is processed by the convolutional neural network layer and input into each classifier respectively, and the empty shot recognition result is obtained based on the recognition results of the at least two classifiers with different functions; An empty shot saving module is used to save the target frame image to a preset storage path when the empty shot recognition result is an empty shot image; The empty shot recognition model includes a landmark classifier and a purity classifier; the landmark classifier is used to identify images of landmark types; the purity classifier is used to identify whether an image is an empty shot image; the empty shot recognition module includes: A landmark classifier processing module, configured to input the implicit spatial features obtained by the convolutional neural network layer processing the target frame image into the landmark classifier for processing to determine whether the target frame image is a landmark type image; The landmark classifier-based processing result recognition module is used to input the implicit spatial features into the purity classifier when the target frame image is a landmark type image, and obtain the empty shot recognition result obtained by the purity classifier processing the implicit spatial features.

7. A terminal, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the empty shot material recognition method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the empty shot material recognition method according to any one of claims 1 to 5 is implemented.

Images