Image classification method, device, electronic device and storage medium
By cascading the results of the first and second classification models, the problems of slow frame-level image classification speed and poor recognition effect in the existing video classification technology are solved, and faster and more accurate video classification is achieved.
Patent Information
- Application Number
- CN202111543520.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-16
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-12-16
AI Technical Summary
Due to the large workload of frame-level image classification, the existing video classification technology has slow processing speed, poor recognition effect, and high error detection rate, which cannot meet the speed requirements for actual online use.
The first classification model is used to detect multiple image frames of the video, generate the first feature data, and use the second classification model to extract and classify the features. By cascading the results of the first and second classification models, filtering to improve the accuracy rate .
It improves the accuracy of category classification of image frames, reduces the error detection rate, meets the speed requirements for online use, and achieves faster and more accurate video classification.
Smart Images

Figure CN114241241B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an image classification method, device, electronic equipment and storage medium. Background Art
[0002] With the continuous development of video technology, especially the rise of short videos, there are more and more videos of various types, which inevitably brings considerable pressure to video review work.
[0003] At present, the mainstream video classification technology uses convolutional neural networks to achieve frame-level image classification, and then uses the frame-level classification results to obtain video classification results, thereby achieving video classification. However, since a video is composed of many frames, the workload of image frame classification is very large, so the processing speed will be very slow. In the process of business application, it is found that the current convolutional neural network still has poor recognition effect and high false positive rate in order to meet the speed requirements of actual online use. Summary of the invention
[0004] The purpose of the embodiments of the present invention is to provide an image classification method, device, electronic device and storage medium, which to a certain extent solve the technical problem that the recognition effect that can be achieved is still poor and the false detection rate is high in order to meet the speed requirements of actual online use.
[0005] In order to solve the above problem, in a first aspect of the implementation of the present invention, an image classification method is first provided, comprising:
[0006] Using a first classification model, respectively detecting categories of a plurality of image frames of a target video, obtaining a first classification result of each of the image frames, and first feature data generated in a feature extraction process;
[0007] For the image frame whose first classification result is the target category, using a feature extraction network of a second classification model to extract features from the image frame to obtain second feature data;
[0008] For the image frame whose first classification result is the target category, the classification network of the second classification model detects the category of the image frame based on the first feature data and the second feature data to obtain a second classification result that replaces the first classification result of the image frame.
[0009] Optionally, for the image frame whose first classification result is a target category, using a feature extraction network of a second classification model to extract features from the image frame to obtain second feature data includes:
[0010] Using the feature extraction network of the second classification model, extracting features from the image frames whose first classification results are the target category, to obtain intermediate feature data;
[0011] The intermediate feature data is weighted according to channel importance and space importance to obtain the second feature data.
[0012] Optionally, before using the first classification model to respectively detect the categories of multiple image frames of the target video to obtain the first classification results of each of the image frames and the first feature data generated in the feature extraction process, the method further includes:
[0013] Using the first classification model, detecting the categories of a plurality of image frame samples, and obtaining initial classification results of each of the image frame samples; wherein the image frame samples are correspondingly marked with sample categories;
[0014] For each of the image frame samples, comparing whether the initial classification result is consistent with the corresponding sample category;
[0015] Adding target image frame samples whose initial classification results are inconsistent with corresponding sample categories to the training sample data set of the second classification model, wherein the proportion of the target image frame samples in the training sample data set exceeds a set proportion.
[0016] Optionally, the method further comprises:
[0017] Inputting the image frame samples in the training sample data set into the feature extraction network of the first classification model to extract features, thereby obtaining first feature data of the samples;
[0018] Inputting the image frame samples in the training sample data set into the feature extraction network of the second classification model to extract features, thereby obtaining second feature data of the samples;
[0019] Outputting the sample first feature data and the sample second feature data to the classification network of the second classification model;
[0020] The feature extraction network and the classification network of the second classification model are trained using the sample categories marked for the image frame samples in the training sample data set until the loss value of the second classification model is less than a set threshold or the maximum number of iterations is reached, thereby obtaining a trained second classification model.
[0021] Optionally, the first classification model is used to respectively detect the categories of multiple image frames of the target video to obtain the first classification results of each of the image frames, and the first feature data generated in the feature extraction process includes:
[0022] Using at least one first convolutional layer of the feature extraction network of the first classification model to extract features from the image frame to obtain first feature data;
[0023] Using at least one second convolutional layer of the feature extraction network of the first classification model, perform feature extraction on the first feature data to obtain third feature data;
[0024] According to the third feature data, the classification network of the first classification model detects the category of the image frame to obtain the first classification result.
[0025] Optionally, before adopting the first classification model to respectively detect the categories of a plurality of image frames of the target video and obtaining a first classification result of each of the image frames and first feature data generated in a feature extraction process, the method further comprises: converting the sizes of the plurality of image frames into a first preset size;
[0026] Inputting a plurality of image frames of the first preset size into the first classification model;
[0027] Before extracting features from the image frame whose first classification result is the target category using a feature extraction network of a second classification model to obtain second feature data, the method further includes: converting the size of the image frame into a second preset size;
[0028] Inputting a plurality of image frames of the second preset size into the second classification model;
[0029] Wherein, the second preset size is greater than or equal to the first preset size.
[0030] Optionally, for the image frame whose first classification result is a target category, detecting the category of the image frame by a classification network of the second classification model according to the first feature data and the second feature data, and obtaining a second classification result that replaces the first classification result of the image frame includes:
[0031] Performing a fusion operation on the first feature data and the second feature data to obtain fourth feature data; wherein the fusion operation includes concatenation and / or addition;
[0032] According to the fourth feature data, the classification network of the second classification model detects the category of the image frame to obtain the second classification result.
[0033] Optionally, the method further comprises:
[0034] Determining, according to the classification results of the multiple image frames, the number of image frames whose classification results are the target category;
[0035] If the number of image frames whose classification results are the target category meets a preset condition, it is determined that the classification result of the target video is the target category.
[0036] In a second aspect of the present invention, there is also provided an image classification device, comprising:
[0037] A first classification module, used to detect the categories of multiple image frames of the target video respectively using a first classification model, and obtain a first classification result of each of the image frames, and first feature data generated in the feature extraction process;
[0038] A feature extraction module, configured to extract features of the image frame whose first classification result is a target category by using a feature extraction network of a second classification model to obtain second feature data;
[0039] The second classification module is used to detect the category of the image frame whose first classification result is the target category by the classification network of the second classification model according to the first feature data and the second feature data, so as to obtain the second classification result that replaces the first classification result of the image frame.
[0040] Optionally, the feature extraction module includes:
[0041] an intermediate feature extraction submodule, configured to extract features of image frames whose first classification results are the target category by using the feature extraction network of the second classification model, so as to obtain intermediate feature data;
[0042] The weighting submodule is used to weight the intermediate feature data according to the channel importance and the space importance to obtain the second feature data.
[0043] Optionally, the device further comprises:
[0044] A category detection module, used to detect the categories of multiple image frames of the target video respectively using the first classification model to obtain the first classification results of each of the image frames, and before the first feature data is generated in the feature extraction process, use the first classification model to detect the categories of multiple image frame samples to obtain the initial classification results of each of the image frame samples; wherein the image frame samples are marked with sample categories;
[0045] A comparison module, used for comparing the initial classification result with the corresponding sample category for each of the image frame samples to see whether they are consistent;
[0046] An adding module is used to add the target image frame samples whose initial classification results are inconsistent with the corresponding sample categories to the training sample data set of the second classification model, wherein the proportion of the target image frame samples in the training sample data set exceeds a set proportion.
[0047] Optionally, the device further comprises:
[0048] A first extraction module, used for inputting the image frame samples in the training sample data set into the feature extraction network of the first classification model to extract features, so as to obtain first feature data of the samples;
[0049] A second extraction module, used for inputting the image frame samples in the training sample data set into the feature extraction network of the second classification model to perform feature extraction, so as to obtain second feature data of the samples;
[0050] An output module, used for outputting the sample first feature data and the sample second feature data to the classification network of the second classification model;
[0051] A training module is used to train the feature extraction network and the classification network of the second classification model using the sample categories marked for the image frame samples in the training sample data set, until the loss value of the second classification model is less than a set threshold or reaches a maximum number of iterations, thereby obtaining a trained second classification model.
[0052] Optionally, the first classification module includes:
[0053] A first extraction submodule, configured to extract features from the image frame using at least one first convolutional layer of the feature extraction network of the first classification model to obtain first feature data;
[0054] A second extraction submodule, configured to extract features from the first feature data using at least one second convolutional layer of the feature extraction network of the first classification model to obtain third feature data;
[0055] A detection submodule is used to detect the category of the image frame according to the third feature data by the classification network of the first classification model to obtain the first classification result.
[0056] Optionally, the device further comprises: a first conversion module, configured to convert the sizes of the multiple image frames into a first preset size before respectively detecting the categories of the multiple image frames of the target video using the first classification model to obtain the first classification results of the respective image frames and before generating the first feature data in the feature extraction process;
[0057] A first input module, used for inputting a plurality of image frames of the first preset size into the first classification model;
[0058] The device further includes: a second conversion module, configured to convert the size of the image frame into a second preset size before extracting features from the image frame whose first classification result is the target category using a feature extraction network of a second classification model to obtain second feature data;
[0059] A second input module, adapted to input a plurality of image frames of the second preset size into the second classification model;
[0060] Wherein, the second preset size is greater than or equal to the first preset size.
[0061] Optionally, the second classification module includes:
[0062] A fusion submodule, used for performing a fusion operation on the first feature data and the second feature data to obtain fourth feature data; wherein the fusion operation includes concatenation and / or addition;
[0063] The second classification submodule is used to detect the category of the image frame according to the fourth feature data by the classification network of the second classification model to obtain the second classification result.
[0064] Optionally, the device further comprises:
[0065] A quantity determination module, used to determine the number of image frames whose classification results are the target category according to the classification results of the multiple image frames;
[0066] The video classification module is used to determine that the classification result of the target video is the target category if the number of image frames whose classification results are the target category meets a preset condition.
[0067] In another aspect of the present invention, there is provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;
[0068] Memory, used to store computer programs;
[0069] The processor is used to implement any of the above-mentioned method steps when executing the program stored in the memory.
[0070] In another aspect of the present invention, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes any of the above-described methods.
[0071] In yet another aspect of the present invention, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-described methods.
[0072] The embodiment of the present invention provides an image classification method, device, electronic device and storage medium. The method includes: using a first classification model to respectively detect the categories of multiple image frames of a target video, obtaining a first classification result of each of the image frames, and a first feature data generated in the feature extraction process; for the image frame whose first classification result is the target category, using a feature extraction network of a second classification model to extract features from the image frame to obtain second feature data; for the image frame whose first classification result is the target category, using a classification network of the second classification model to detect the category of the image frame according to the first feature data and the second feature data, to obtain a second classification result that replaces the first classification result of the image frame. In the classification stage, the classification results of the first classification model and the second classification model are cascaded, and the second classification result is used for filtering on the basis that the first classification result is the target category. Since the first feature data and the second feature data are directly used for fusion, feature complementarity is achieved at a lower cost, and the image frame is characterized more comprehensively and accurately, thereby improving the accuracy of the second classification result, and then improving the classification accuracy of the category of the image frame. BRIEF DESCRIPTION OF THE DRAWINGS
[0073] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.
[0074] Figure 1 A flowchart showing a method for image classification according to an embodiment of the present invention is shown;
[0075] Figure 2 A flowchart showing another method for image classification according to an embodiment of the present invention is shown;
[0076] Figure 3 A flowchart showing another method for image classification according to an embodiment of the present invention is shown;
[0077] Figure 4 A flowchart showing another method for image classification according to an embodiment of the present invention is shown;
[0078] Figure 5 A flowchart showing another method for image classification according to an embodiment of the present invention is shown;
[0079] Figure 6 A schematic diagram showing the image classification process of the present invention;
[0080] Figure 7A structural block diagram of an image classification device embodiment of the present invention is shown;
[0081] Figure 8 A schematic diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0082] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0083] Reference Figure 1 , shows a flowchart of an image classification method embodiment of the present invention, which may specifically include the following steps:
[0084] Step 101: Use a first classification model to detect the categories of multiple image frames of a target video respectively, and obtain a first classification result of each of the image frames, as well as first feature data generated in a feature extraction process.
[0085] In the embodiment of the present invention, the target video is a video to be classified, including a horizontal video, a vertical video, a video with black edges, a video with poor pixels, a long video, a short video, etc., or any other applicable video, which is not limited by the embodiment of the present invention.
[0086] In an embodiment of the present invention, the first classification model is a machine learning model that can be used to detect the category of an image after training. After providing an image, the trained first classification model can output the category of the image. The first classification model needs to be trained using image samples and labeled categories, and the obtained model is recorded as the first classification model. For example, the first classification model can adopt a DPN (Duall Path Networks) network architecture.
[0087] Among them, the categories of images can be divided into two or more categories, for example, risk and no risk, or violence and non-violence, or pornography and non-pornography, or smoking and drinking shots and no smoking and drinking shots, or pornography, gambling, drugs and normal, or violation, suspected violation and normal, or any other applicable categories, and the embodiments of the present invention do not limit this.
[0088] In the embodiment of the present invention, multiple image frames are extracted from the target video, and the extraction may be performed frame by frame, or at intervals of several frames, or at fixed intervals, or in any other applicable extraction manner, which is not limited in the embodiment of the present invention. For example, the target video is extracted at 1fps, that is, m frames are played per second, and 1 image frame is extracted for every m frames.
[0089] In an embodiment of the present invention, multiple image frames of the target video are respectively input into the first classification model, and the first classification model can output the classification results of each image frame, which are recorded as first classification results. For each input image frame, a first classification result is output. For example, the first classification model is used to detect the categories of image frames including risk or no risk. For an image frame, the output first classification result can be risk or no risk.
[0090] In an embodiment of the present invention, the processing of the image frame by the first classification model includes a feature extraction process, and the feature extraction process can generate feature data. By performing feature extraction on the input image frame, more concrete feature data can be obtained first, and then feature extraction is performed on the obtained more concrete feature data to obtain more abstract feature data. The first feature data is generated in the feature extraction process, and can be feature data generated in the middle of the feature extraction process, or can be feature data generated at the end of the feature extraction process, and the embodiment of the present invention does not limit this.
[0091] Step 102: for the image frame whose first classification result is the target category, a feature extraction network of a second classification model is used to extract features from the image frame to obtain second feature data.
[0092] In an embodiment of the present invention, for image frames whose first classification results are target categories, in order to avoid image frames being mistakenly detected as target categories, an embodiment of the present invention proposes to use a second classification model for secondary filtering. The second classification model is also a machine learning model, which can be used to detect the category of an image after training. After providing an image, the trained second classification model can output the category of the image. The second classification model also needs to be trained using image samples and labeled categories, and the obtained model is recorded as the second classification model. The second classification model can use the same network structure as the first classification model, or a different network structure from the first classification model, and the embodiment of the present invention does not limit this.
[0093] In an embodiment of the present invention, the second classification model includes a feature extraction network and a classification network, wherein the feature extraction network is used to extract features from the image frame, and the feature data finally obtained is recorded as the second feature data. The classification network is used to classify the image frame according to the second feature data.
[0094] In the embodiment of the present invention, the first classification model is used to detect the category of the image frame. Some image frames are correctly classified as the target category, but some image frames are incorrectly classified as the target category. The first classification model has a certain detection error rate for the target category. The second classification model is used to detect the category of the image frame detected as the target category again. Among the image frames that are incorrectly classified as the target category, some or all will be reclassified as other categories other than the target category. The second classification model reduces the situation where the image frame is incorrectly classified as the target category.
[0095] In an embodiment of the present invention, the input of the classification network of the second classification model is the first feature data and the second feature data. Since different features differ in the degree of trade-off between invariance and discrimination ability, different image data sets require different trade-offs, and therefore, different features have complementary characteristics. By inputting both the first feature data and the second feature data into the classification network of the second classification model, more features can be input into the classification network, thereby using more complementary features, thereby improving the classification effect of the classification network, reducing the detection error rate for the target category, and achieving a higher detection accuracy rate for the target category by the second classification model than the detection accuracy rate of the target category by the first classification model.
[0096] Step 103, for the image frame whose first classification result is a target category, the classification network of the second classification model detects the category of the image frame based on the first feature data and the second feature data, and obtains a second classification result that replaces the first classification result of the image frame.
[0097] In an embodiment of the present invention, for an image frame whose first classification result is a target category, the first feature data and the second feature data are input into a classification network of a second classification model, and the classification network detects the category of the image frame. The classification network can output a classification result of the image frame, which is recorded as a second classification result. A second classification result is output for each input image frame. For example, the second classification model is used to detect that the category of the image frame includes two categories, risk or no risk. For an image frame, the output second classification result can be risk or no risk.
[0098] In the embodiment of the present invention, the first classification model has classified the image frame to obtain a first classification result. Since the detection accuracy of the second classification model is higher, the second classification result is used to replace the first classification result as the category of the image frame.
[0099] The image classification method provided by the embodiment of the present invention adopts a first classification model to respectively detect the categories of multiple image frames of the target video, obtain the first classification results of each of the image frames, and the first feature data generated in the feature extraction process, and for the image frames whose first classification results are the target category, the feature extraction network of the second classification model is used to extract features from the image frames to obtain second feature data; for the image frames whose first classification results are the target category, the classification network of the second classification model detects the category of the image frame according to the first feature data and the second feature data, and obtains a second classification result that replaces the first classification result of the image frame. In the classification stage, the classification results of the first classification model and the second classification model are cascaded, and the second classification result is used for filtering on the basis that the first classification result is the target category. Since the first feature data is directly used to fuse with the second feature data, feature complementarity is achieved at a lower cost, and the image frame is characterized more comprehensively and accurately, thereby improving the accuracy of the second classification result, and then improving the classification accuracy of the category of the image frame.
[0100] In an optional embodiment of the present invention, Figure 2 As shown, step 102 includes:
[0101] Step 1021: Use the feature extraction network of the second classification model to extract features from the image frames whose first classification results are the target category to obtain intermediate feature data.
[0102] Step 1022: weighting the intermediate feature data according to channel importance and spatial importance to obtain the second feature data.
[0103] When extracting feature data from an image frame, each local information has a different influence on whether the image can be correctly recognized. Therefore, an attention mechanism is introduced to let the model know the importance of different local information in the image.
[0104] Attention mechanisms are divided into two categories: channel attention and spatial attention. Channel attention refers to weighted processing of feature data according to channel importance. Spatial attention refers to weighted processing of feature data according to spatial importance. In the process of feature extraction of image frames, intermediate feature data is obtained before entering the processing process of the attention mechanism. Then the intermediate feature data is processed by the attention mechanism. Each channel of the intermediate feature data represents a certain feature of the image frame, and the attention of the channel dimension refers to weighted processing of the channel dimension so that the model can learn what kind of features are meaningful. Then in order to make the features more effectively represented in the spatial dimension, the attention of the spatial dimension refers to weighted processing of the spatial dimension so that the model can identify where the key information is and where the invalid noise is in space.
[0105] For example, the second classification model uses SE-ResNeXt (Squeeze-and-Excitation Residual NeXt) with an Attention module. Assuming that the intermediate feature data has a dimension of 7x7x2048, the channel dimension is retained through spatial global average pooling and global maximum pooling. At this time, the dimension of the feature data is 1x1x2048, and then the weight coefficient is obtained through sigmoid (activation function). Finally, the weight coefficient of 1x1x2048 is multiplied by the intermediate feature data of 7x7x2048 to obtain the feature data weighted according to the channel importance, with a dimension of 7x7x2048. According to the feature data weighted by the channel importance, average pooling and maximum pooling of the channel dimension are performed respectively to retain the spatial dimension. The two pooled features are added together to obtain feature data of dimension 7x7x2, and then passed through a convolution layer with a convolution kernel of 7x7 to obtain the weight coefficient of the spatial dimension, with dimension 7x7x1. Finally, the feature data weighted according to the channel importance is multiplied by the weight coefficient of the spatial dimension to obtain the feature data weighted according to the spatial importance, i.e., the second feature data.
[0106] After the above-mentioned weighted processing, the model can learn the differences in features, so as to distinguish which features and where the features are important, thereby improving the accuracy of the second classification model in classifying images, especially when the target video may be any of horizontal videos, vertical videos, videos with black borders, videos with poor pixels, long videos, short videos, etc. For various videos, the introduction of the attention mechanism can improve which features in the image frames in different videos are important, thereby effectively improving the accuracy of image classification.
[0107] In an optional embodiment of the present invention, Figure 3 As shown, before the above step 101, the following steps may also be included:
[0108] Step 201: Use the first classification model to detect the categories of multiple image frame samples to obtain initial classification results of each of the image frame samples; wherein the image frame samples are correspondingly marked with sample categories.
[0109] Step 202: for each of the image frame samples, compare the initial classification result with the corresponding sample category to see whether they are consistent.
[0110] Step 203, adding the target image frame samples whose initial classification results are inconsistent with the corresponding sample categories to the training sample data set of the second classification model, wherein the proportion of the target image frame samples in the training sample data set exceeds the set proportion.
[0111] The first classification model has been trained, and multiple image frame samples are input into the first classification model. The first classification model detects the category of the image frame samples and obtains the classification results of each image frame sample, which are recorded as the initial classification results. The image frame samples are marked with sample categories. For each image frame sample, compare whether the initial classification result is consistent with the corresponding sample category, filter out the target image frame samples whose initial classification results are inconsistent with the corresponding sample category, and then add them to the training sample data set of the second classification model. And ensure that the proportion of the target image frame samples in the training sample data set exceeds the set proportion, wherein the set proportion can be obtained through repeated experiments, and the embodiment of the present invention does not limit this. For example, the proportion is set to 40%.
[0112] Since the training sample data set used to train the second classification model contains a large number of samples that cannot be correctly classified by the first classification model, the second classification model can be trained specifically for the weaknesses of the first classification model, thereby having a higher detection accuracy for image frames that are mistakenly detected as target categories, which can improve the detection accuracy of the second classification model for target categories and improve the classification accuracy of image frame categories.
[0113] In an optional embodiment of the present invention, in a specific implementation method of training a second classification model, it may also include: inputting the image frame samples in the training sample data set into the feature extraction network of the first classification model for feature extraction, and obtaining the first feature data of the samples; inputting the image frame samples in the training sample data set into the feature extraction network of the second classification model for feature extraction, and obtaining the second feature data of the samples; outputting the first feature data of the samples and the second feature data of the samples to the classification network of the second classification model; using the sample categories marked for the image frame samples in the training sample data set to train the feature extraction network and the classification network of the second classification model until the loss value of the second classification model is less than the set threshold or the maximum number of iterations is reached, thereby obtaining the trained second classification model.
[0114] When training the second classification model, the input is the image frame sample in the training sample data set. First, feature extraction is performed through the feature extraction network of the first classification model to obtain feature data, which is recorded as the first feature data of the sample. Then, feature extraction is performed through the feature extraction network of the second classification model to obtain feature data, which is recorded as the second feature data of the sample.
[0115] The input of the classification network of the second classification model is the first feature data of the sample and the second feature data of the sample. The feature extraction network and the classification network of the second classification model are trained by using the sample categories marked for the image frame samples in the training sample data set. That is to say, after the model outputs the predicted value of the image frame sample each time, it is compared with the actual value (i.e., the marked sample category), and the result of the comparison is input into the loss function to calculate the loss value. The convergence conditions of the model may be: the loss value is less than the set threshold, or the maximum number of iterations is reached. For example, a relatively small set threshold is set, and the size of the loss value is calculated at the same time during each training. When the loss value is less than the set threshold, the model can be considered to have converged, and then the training can be terminated. A relatively large maximum number of iterations is pre-set, such as 100 iterations, or 10,000 iterations, or 1,000,000 iterations, etc., which needs to be selected according to the actual situation, and the embodiment of the present invention does not limit this. After the model completes the specified number of trainings, the model training can be considered to be completed.
[0116] In an optional embodiment of the present invention, Figure 4 As shown, the above step 101 includes:
[0117] Step 1011: Use at least one first convolutional layer of the feature extraction network of the first classification model to extract features from the image frame to obtain first feature data.
[0118] Step 1012: Use at least one second convolutional layer of the feature extraction network of the first classification model to extract features from the first feature data to obtain third feature data.
[0119] Step 1013: Based on the third feature data, the classification network of the first classification model detects the category of the image frame to obtain the first classification result.
[0120] The feature extraction network of the first classification model includes multiple convolutional layers. After the image frame passes through a convolutional layer, feature data is obtained. After the feature data passes through the next convolutional layer, feature data is obtained again. The image frame is subjected to feature extraction through at least one convolutional layer to obtain the first feature data. The convolutional layer through which the first feature data is obtained is recorded as the first convolutional layer. Then the first feature data is subjected to feature extraction through at least one convolutional layer to obtain the third feature data. The convolutional layer through which the third feature data is obtained is recorded as the second convolutional layer. Finally, the input of the classification network of the first classification model is the third feature data.
[0121] The first feature data is feature data generated in the process of extracting the third feature data, and the first feature data is obtained from the feature extraction network of the first classification model for use by the subsequent second classification model.
[0122] Since more convolutional layers mean that the feature data is deeper feature data, the feature data will become more abstract. Therefore, the first feature data is shallower feature data than the third feature data, that is, more concrete feature data. The shallower first feature data and the second feature data are more complementary to each other, so more complementary features are used to improve the classification effect of the classification network and the detection accuracy of the target category.
[0123] In an optional embodiment of the present invention, before the above step 101, the following may be included: converting the size of the multiple image frames into a first preset size, and inputting the multiple image frames of the first preset size into a first classification model. Before the above step 102, the following may be included: converting the size of the image frames into a second preset size, and inputting the multiple image frames of the second preset size into a second classification model. The second preset size is greater than or equal to the first preset size.
[0124] Images input into the same classification model need to have a unified size. For the first classification model, the sizes of multiple image frames are converted into a first preset size. For the second classification model, the sizes of multiple image frames are converted into a second preset size.
[0125] Generally, the larger the image size, the greater the computational load of the classification model, but the classification accuracy can also be higher. Therefore, the second preset size can be greater than or equal to the first preset size, thereby ensuring that the accuracy of the classification result output by the second classification model is not lower than that of the first classification model.
[0126] In an optional embodiment of the present invention, Figure 5 As shown, the above step 103 may include:
[0127] Step 1031, performing a fusion operation on the first feature data and the second feature data to obtain fourth feature data; wherein the fusion operation includes concatenation and / or addition.
[0128] The first feature data and the second feature data are fused to obtain feature data, which is recorded as fourth feature data. The fusion operation may include at least one of the following: concat, i.e., directly connecting two features, add, i.e., combining two feature vectors into a complex vector, or any other applicable fusion operation, which is not limited in the embodiment of the present invention.
[0129] For example, the first feature data of 832 dimensions and the second feature data of 2048 dimensions are feature fused to obtain fourth feature data of 2080 dimensions.
[0130] Step 1032: Based on the fourth feature data, the classification network of the second classification model detects the category of the image frame to obtain the second classification result.
[0131] The fourth feature data is input into the classification network of the second classification model, and the classification network detects the category of the image frame to obtain a second classification result.
[0132] In an optional embodiment of the present invention, it may also include: determining the number of image frames whose classification results are target categories based on the classification results of multiple image frames; if the number of image frames whose classification results are target categories meets a preset condition, determining that the classification result of the target video is the target category.
[0133] Among the multiple image frames of the target video, after the above processing, for the image frames whose first classification results are not the target category, the first classification results are adopted, and for the image frames whose first classification results are the target category, the second classification results are adopted. Classification statistics are performed on the categories of the multiple image frames to determine the number of image frames whose classification results are the target category. If the number of image frames whose classification results are the target category meets the preset conditions, it is determined that the classification result of the target video is also the target category.
[0134] The preset conditions may include multiple conditions, for example, the number of image frames of the target category exceeds the preset number, or the proportion of the number of image frames of the target category in the total number of multiple image frames exceeds the preset proportion, or any other applicable conditions, which are not limited in the embodiment of the present invention.
[0135] For example, if the proportion of risk-classified image frames in multiple image frames exceeds a preset value, the target video is determined to be risky. Due to the detection accuracy of the second classification result, the classification accuracy of the image frame category is improved, and then the classification accuracy of the video category is improved.
[0136] In order to enable those skilled in the art to better understand the present application, an implementation method of the present application is described below through specific examples.
[0137] like Figure 6 Schematic diagram of the video classification process shown.
[0138] Step 01, video input.
[0139] Step 02, perform frame extraction at 1fps.
[0140] Step 03, perform data preprocessing on the extracted image frames, first perform random Gaussian blur processing.
[0141] Step 04, perform operations such as horizontal inversion and vertical inversion, so as to obtain multiple frames of images with different shapes, angles, clarity, etc. from one frame of image, thereby achieving the effect of data augmentation.
[0142] Step 05: Reduce or enlarge the image frame obtained above to an image of 224×224 size
[0143] Step 06, reducing or enlarging the image frame obtained above to an image of 299×299 size.
[0144] Step 07, input the 224×224 size image into the first classification model using the DPN network architecture for feature extraction.
[0145] Step 08, output the first feature data of 832 dimensions.
[0146] Step 09, input the 299×299 size image into the second classification model using the SE-ResNeXt network architecture with Attention, and extract features.
[0147] Step 10, output the second feature data of 2048 dimensions.
[0148] Step 11, the 832-dimensional first feature data of the first classification model passes through the full connection and activation function in the first classification model.
[0149] Step 12: output the first classification result.
[0150] Step 13: The first feature data and the second feature data are subjected to feature fusion.
[0151] Step 14: output the fused 2080-dimensional features, i.e., the fourth feature data.
[0152] Step 15, for the fused 2080-dimensional fourth feature data, pass it through the full connection and activation function in the second classification model.
[0153] Step 16: output the second classification result.
[0154] Step 17, for the image frames (indicated by frame numbers) for which the first classification result is pornographic, a second filtering is performed using the second classification model.
[0155] Step 18, for multiple image frames in the video, the first classification result and the second classification result are merged, that is, some image frames use the first classification result, and some image frames use the second classification result. For image frames (indicated by frame numbers) that are non-pornographic according to the first classification result, the image frames use the first classification result, and for image frames (indicated by frame numbers) that are pornographic according to the first classification result, the image frames use the second classification result.
[0156] Step 19: Count the classification results after fusion.
[0157] Step 20, obtaining the classification result of the video according to the proportion of each category, and determining the classification of the video.
[0158] Step 21, output the classification result of the video.
[0159] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0160] Reference Figure 7 , shows a structural block diagram of an embodiment of an image classification device of the present invention, which may specifically include the following modules:
[0161] A first classification module 301 is used to detect the categories of multiple image frames of the target video using a first classification model, and obtain a first classification result of each of the image frames and first feature data generated in the feature extraction process;
[0162] A feature extraction module 302 is used to extract features from the image frame whose first classification result is a target category by using a feature extraction network of a second classification model to obtain second feature data;
[0163] The second classification module 303 is used to detect the category of the image frame whose first classification result is the target category by the classification network of the second classification model according to the first feature data and the second feature data, so as to obtain the second classification result that replaces the first classification result of the image frame.
[0164] Optionally, the feature extraction module includes:
[0165] an intermediate feature extraction submodule, configured to extract features of image frames whose first classification results are the target category by using the feature extraction network of the second classification model, so as to obtain intermediate feature data;
[0166] The weighting submodule is used to weight the intermediate feature data according to the channel importance and the space importance to obtain the second feature data.
[0167] Optionally, the device further comprises:
[0168] A category detection module, used to detect the categories of multiple image frames of the target video respectively using the first classification model to obtain the first classification results of each of the image frames, and before the first feature data is generated in the feature extraction process, use the first classification model to detect the categories of multiple image frame samples to obtain the initial classification results of each of the image frame samples; wherein the image frame samples are marked with sample categories;
[0169] A comparison module, used for comparing the initial classification result with the corresponding sample category for each of the image frame samples to see whether they are consistent;
[0170] An adding module is used to add the target image frame samples whose initial classification results are inconsistent with the corresponding sample categories to the training sample data set of the second classification model, wherein the proportion of the target image frame samples in the training sample data set exceeds a set proportion.
[0171] Optionally, the device further comprises: a first extraction module, configured to input the image frame samples in the training sample data set into the feature extraction network of the first classification model to perform feature extraction, and obtain first feature data of the samples;
[0172] A second extraction module, used for inputting the image frame samples in the training sample data set into the feature extraction network of the second classification model to perform feature extraction, so as to obtain second feature data of the samples;
[0173] An output module, used for outputting the sample first feature data and the sample second feature data to the classification network of the second classification model;
[0174] A training module is used to train the feature extraction network and the classification network of the second classification model using the sample categories marked for the image frame samples in the training sample data set, until the loss value of the second classification model is less than a set threshold or reaches a maximum number of iterations, thereby obtaining a trained second classification model.
[0175] Optionally, the first classification module includes:
[0176] A first extraction submodule, configured to extract features from the image frame using at least one first convolutional layer of the feature extraction network of the first classification model to obtain first feature data;
[0177] A second extraction submodule, configured to extract features from the first feature data using at least one second convolutional layer of the feature extraction network of the first classification model to obtain third feature data;
[0178] A detection submodule is used to detect the category of the image frame according to the third feature data by the classification network of the first classification model to obtain the first classification result.
[0179] Optionally, the device further comprises: a first conversion module, configured to convert the sizes of the multiple image frames into a first preset size before respectively detecting the categories of the multiple image frames of the target video using the first classification model to obtain the first classification results of the respective image frames and before generating the first feature data in the feature extraction process;
[0180] A first input module, used for inputting a plurality of image frames of the first preset size into the first classification model;
[0181] The device further includes: a second conversion module, configured to convert the size of the image frame into a second preset size before extracting features from the image frame whose first classification result is the target category using a feature extraction network of a second classification model to obtain second feature data;
[0182] A second input module, adapted to input a plurality of image frames of the second preset size into the second classification model;
[0183] Wherein, the second preset size is greater than or equal to the first preset size.
[0184] Optionally, the second classification module includes: a fusion submodule, configured to perform a fusion operation on the first feature data and the second feature data to obtain fourth feature data; wherein the fusion operation includes concatenation and / or addition;
[0185] The second classification submodule is used to detect the category of the image frame according to the fourth feature data by the classification network of the second classification model to obtain the second classification result.
[0186] Optionally, the device further comprises: a quantity determination module, configured to determine, according to the classification results of the plurality of image frames, the number of image frames whose classification results are the target category;
[0187] The video classification module is used to determine that the classification result of the target video is the target category if the number of image frames whose classification results are the target category meets a preset condition.
[0188] The image classification method provided by the embodiment of the present invention adopts a first classification model to respectively detect the categories of multiple image frames of the target video, obtain the first classification results of each of the image frames, and the first feature data generated in the feature extraction process, for the image frame whose first classification result is the target category, the feature extraction network of the second classification model is used to extract features from the image frame to obtain the second feature data; for the image frame whose first classification result is the target category, the classification network of the second classification model detects the category of the image frame according to the first feature data and the second feature data, and obtains the second classification result that replaces the first classification result of the image frame. In the classification stage, the classification results of the first classification model and the second classification model are cascaded, and the second classification result is used for filtering on the basis that the first classification result is the target category. Since the first feature data and the second feature data are directly used for fusion, feature complementarity is achieved at a lower cost, and the image frame is characterized more comprehensively and accurately, thereby improving the detection accuracy of the second classification result, and then improving the classification accuracy of the category of the image frame.
[0189] The embodiment of the present invention further provides an electronic device, such as Figure 8 As shown, it includes a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.
[0190] Memory 903, used for storing computer programs;
[0191] The processor 901 is used to implement the steps described in any of the above method embodiments when executing the program stored in the memory 903.
[0192] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0193] The communication interface is used for communication between the above terminal and other devices.
[0194] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0195] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0196] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer executes the image classification method described in any one of the above embodiments.
[0197] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the image classification method described in any one of the above embodiments.
[0198] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.
[0199] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0200] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0201] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims
1. An image classification method, characterized in that: include: Using a first classification model, detecting the categories of a plurality of image frame samples, and obtaining initial classification results of each of the image frame samples; wherein the image frame samples are correspondingly marked with sample categories; For each of the image frame samples, comparing whether the initial classification result is consistent with the corresponding sample category; Adding the target image frame samples whose initial classification results are inconsistent with the corresponding sample categories to the training sample data set of the second classification model, wherein the proportion of the target image frame samples in the training sample data set exceeds the set proportion; Using the first classification model, respectively detecting the categories of a plurality of image frames of the target video, obtaining a first classification result of each of the image frames, and first feature data generated in a feature extraction process; For the image frame whose first classification result is the target category, using the feature extraction network of the second classification model to extract features from the image frame to obtain second feature data; For image frames whose first classification result is a target category, the classification network of the second classification model detects the category of the image frame based on feature data obtained by fusing the first feature data and the second feature data, and obtains a second classification result that replaces the first classification result of the image frame. The feature extraction network of the first classification model includes multiple convolutional layers, and the first feature data is extracted by at least one first convolutional layer of the feature extraction network of the first classification model.
2. The method according to claim 1, characterized in that The first classification result is an image frame of the target category, and the feature extraction network of the second classification model is used to extract features from the image frame to obtain second feature data, including: Using the feature extraction network of the second classification model, extracting features from the image frames whose first classification results are the target category, to obtain intermediate feature data; The intermediate feature data is weighted according to channel importance and space importance to obtain the second feature data.
3. The method according to claim 1, characterized in that The method further comprises: Inputting the image frame samples in the training sample data set into the feature extraction network of the first classification model to extract features, thereby obtaining first feature data of the samples; Inputting the image frame samples in the training sample data set into the feature extraction network of the second classification model to extract features, thereby obtaining second feature data of the samples; Outputting the sample first feature data and the sample second feature data to the classification network of the second classification model; The feature extraction network and the classification network of the second classification model are trained using the sample categories marked for the image frame samples in the training sample data set until the loss value of the second classification model is less than a set threshold or the maximum number of iterations is reached, thereby obtaining a trained second classification model.
4. The method according to claim 1, characterized in that: The first classification model is used to detect the categories of multiple image frames of the target video respectively, and the first classification results of each of the image frames are obtained, and the first feature data generated in the feature extraction process include: Using at least one first convolutional layer of the feature extraction network of the first classification model to extract features from the image frame to obtain first feature data; Using at least one second convolutional layer of the feature extraction network of the first classification model, perform feature extraction on the first feature data to obtain third feature data; According to the third feature data, the classification network of the first classification model detects the category of the image frame to obtain the first classification result.
5. The method according to claim 1, characterized in that Before the first classification model is used to detect the categories of multiple image frames of the target video respectively, and the first classification results of each of the image frames and the first feature data generated in the feature extraction process are obtained, the method further includes: converting the sizes of the multiple image frames into a first preset size; Inputting a plurality of image frames of the first preset size into the first classification model; Before extracting features from the image frame whose first classification result is the target category using a feature extraction network of a second classification model to obtain second feature data, the method further includes: converting the size of the image frame into a second preset size; Inputting a plurality of image frames of the second preset size into the second classification model; Wherein, the second preset size is greater than or equal to the first preset size.
6. The method according to claim 1, characterized in that For the image frame whose first classification result is a target category, detecting the category of the image frame by a classification network of the second classification model according to the first feature data and the second feature data, and obtaining a second classification result that replaces the first classification result of the image frame includes: Performing a fusion operation on the first feature data and the second feature data to obtain fourth feature data; wherein the fusion operation includes concatenation and / or addition; According to the fourth feature data, the classification network of the second classification model detects the category of the image frame to obtain the second classification result.
7. The method according to claim 1, characterized in that The method further comprises: Determining, according to the classification results of the multiple image frames, the number of image frames whose classification results are the target category; If the number of image frames whose classification results are the target category meets a preset condition, it is determined that the classification result of the target video is the target category.
8. An image classification device, characterized in that: include: A first classification module, used to detect the categories of multiple image frames of the target video respectively using a first classification model, and obtain a first classification result of each of the image frames, and first feature data generated in the feature extraction process; A feature extraction module, configured to extract features of the image frame whose first classification result is a target category by using a feature extraction network of a second classification model to obtain second feature data; a second classification module, configured to detect the category of the image frame for which the first classification result is a target category, using a classification network of the second classification model to obtain a second classification result that replaces the first classification result of the image frame according to feature data obtained by fusing the first feature data and the second feature data, for the image frame for which the first classification result is a target category, wherein the feature extraction network of the first classification model includes a plurality of convolutional layers, and the first feature data is extracted by at least one first convolutional layer of the feature extraction network of the first classification model; The device also includes: A category detection module, used to detect the categories of multiple image frames of the target video respectively using the first classification model to obtain the first classification results of each of the image frames, and before the first feature data is generated in the feature extraction process, use the first classification model to detect the categories of multiple image frame samples to obtain the initial classification results of each of the image frame samples; wherein the image frame samples are marked with sample categories; A comparison module, used for comparing the initial classification result with the corresponding sample category for each of the image frame samples to see whether they are consistent; An adding module is used to add the target image frame samples whose initial classification results are inconsistent with the corresponding sample categories to the training sample data set of the second classification model, wherein the proportion of the target image frame samples in the training sample data set exceeds a set proportion.
9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 8 when executing a program stored in a memory.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.
