Image classification method, feature extraction network training method and device

Abstract

The present invention provides an image classification method, a training method and device for a feature extraction network, wherein a target image is input into a pre-trained feature extraction network, and the image features of the target image are output; based on the image features, the category of the target image is determined; wherein the feature extraction network includes a plurality of layers of feature extraction layers connected in series; each layer of feature extraction layer is used to output the hierarchical features corresponding to the current layer; and the image features are obtained by fusing the hierarchical features corresponding to at least two layers. In this method, the image features include at least two layers of hierarchical features, so the feature hierarchies contained in the image features are richer, and image recognition in complex scenes such as live broadcast can be handled, so that sensitive images can be accurately and effectively identified, and the missed detection rate and false detection rate of sensitive images are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image classification method, a training method and a device for a feature extraction network. Background Art

[0002] In order to conduct content supervision on videos circulated on the Internet, it is necessary to identify sensitive images from the videos, such as vulgar images, pornographic images, violent images, horror images, etc. In related technologies, these sensitive images can be identified through deep learning methods. Specifically, the high-level semantic features of the image are first extracted through a pre-trained detection model, and then the image is classified based on the high-level semantic features. However, in the live broadcast scenario, the source and structure of the video image are more complex, and the above method is difficult to accurately and effectively identify sensitive images, resulting in a high missed detection rate and false detection rate of sensitive images. Summary of the invention

[0003] In view of this, an object of the present invention is to provide an image classification method, a training method and a device for a feature extraction network to improve the accuracy of identifying sensitive images.

[0004] In a first aspect, an embodiment of the present invention provides an image classification method, which includes: inputting a target image into a pre-trained feature extraction network, and outputting image features of the target image; based on the image features, determining the category of the target image; wherein the feature extraction network includes a plurality of hierarchical feature extraction layers connected in series; each hierarchical feature extraction layer is used to output hierarchical features corresponding to the current hierarchical level; and the image features are obtained by fusing hierarchical features corresponding to at least two hierarchical levels.

[0005] Furthermore, the feature extraction network also includes at least two feature processing modules and a feature fusion module; each feature processing module is connected to a hierarchical feature extraction layer; the feature extraction layers connected to any two feature processing modules are different; the step of inputting the target image into the pre-trained feature extraction network and outputting the image features of the target image includes: through each feature processing module, based on the attention mechanism, processing the hierarchical features output by the feature extraction layer connected to the feature processing module, and outputting the intermediate features; through the feature fusion module, fusing the intermediate features output by each feature processing module to obtain the image features.

[0006] Furthermore, at least two feature processing modules include: a first feature processing module, a second feature processing module and a third feature processing module; wherein the first feature processing module is connected to the lowest level feature extraction layer; the lowest level feature extraction layer is used to input the target image; the second feature processing module is connected to the feature extraction layer specified in the middle level; the third feature processing module is connected to the highest level feature extraction layer.

[0007] Furthermore, the feature processing module includes a pooling layer, a first fully connected layer and a feature multiplication layer; the hierarchical features output by the feature extraction layer connected to the feature processing module are processed based on the attention mechanism, and the step of outputting intermediate features includes: performing a first pooling process on the input hierarchical features through the pooling layer, and outputting a first pooling result; performing a first fully connected process on the first pooling result through the first fully connected layer, and outputting a first fully connected result; multiplying the input hierarchical features and the first fully connected result through the feature multiplication layer to obtain a multiplication result; and outputting intermediate features based on the multiplication result.

[0008] Furthermore, the feature processing module also includes a spatial pyramid pooling layer; the spatial pyramid pooling layer is connected to the feature multiplication layer; the step of outputting the intermediate features based on the multiplication results includes: performing a second pooling process on the multiplication results through the spatial pyramid pooling layer, and outputting the intermediate features of the specified dimension.

[0009] Furthermore, the feature fusion module includes a feature splicing layer, a second fully connected layer and a third fully connected layer; the intermediate features output by each feature processing module are fused through the feature fusion module to obtain the step of image features, including: performing splicing processing on the intermediate features output by each feature processing module through the feature splicing layer, and outputting splicing features; performing second fully connected processing on the splicing features through the second fully connected layer, and outputting a second fully connected result; performing third fully connected processing on the second fully connected result through the third fully connected layer, and outputting image features.

[0010] Furthermore, based on the image features, the step of determining the category of the target image includes: inputting the image features into a preset normalized exponential function and outputting a probability distribution vector; wherein the probability distribution vector includes multiple categories and probability values ​​corresponding to each category; and determining the category corresponding to the largest probability value in the probability distribution vector as the category of the target image.

[0011] In a second aspect, an embodiment of the present invention provides a method for training a feature extraction network, the method comprising: determining training samples based on a preset sample set; wherein the training samples include sample images and category labels of the sample images; inputting the sample images into the feature extraction network to obtain sample features of the sample images; wherein the feature extraction network comprises a plurality of layers of feature extraction layers connected in series; each layer of feature extraction layer is used to output the layer features corresponding to the current layer; the sample features are obtained by fusing the layer features corresponding to at least two layers; based on the sample features, determining the category recognition result of the sample image; determining the loss value based on a preset classification loss function, the category label and the category recognition result; updating the network parameters of the feature extraction network based on the loss value; continuing to execute the step of determining the training samples based on the preset sample set until the loss value converges to obtain the trained feature extraction network.

[0012] In a third aspect, an embodiment of the present invention provides an image classification device, which includes: an output module, used to input a target image into a pre-trained feature extraction network, and output image features of the target image; a classification module, used to determine the category of the target image based on the image features; wherein the feature extraction network includes a plurality of layers of feature extraction layers connected in series; each feature extraction layer is used to output the hierarchical features corresponding to the current level; and the image features are obtained by fusing the hierarchical features corresponding to at least two levels.

[0013] In a fourth aspect, an embodiment of the present invention provides a training device for a feature extraction network, the device comprising: a sample determination module, used to determine training samples based on a preset sample set; wherein the training samples include sample images and category labels of the sample images; an image input module, used to input the sample images into the feature extraction network to obtain sample features of the sample images; wherein the feature extraction network comprises a plurality of layers of feature extraction layers connected in series; each layer of feature extraction layer is used to output the layer features corresponding to the current layer; the sample features are obtained by fusing the layer features corresponding to at least two layers; a parameter updating module, used to determine the category recognition result of the sample image based on the sample features; determine the loss value based on a preset classification loss function, the category label and the category recognition result; update the network parameters of the feature extraction network based on the loss value; a network determination module, used to continue to execute the step of determining the training samples based on the preset sample set until the loss value converges to obtain the trained feature extraction network.

[0014] In a fifth aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the image classification method of the first aspect, or the feature extraction network training method of the second aspect.

[0015] In a sixth aspect, an embodiment of the present invention provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the image classification method of the first aspect, or the training method of the feature extraction network of the second aspect.

[0016] The embodiments of the present invention bring the following beneficial effects:

[0017] An image classification method, a training method and device for a feature extraction network provided by an embodiment of the present invention input a target image into a pre-trained feature extraction network, output the image features of the target image; based on the image features, determine the category of the target image; wherein the feature extraction network includes a plurality of layers of feature extraction layers connected in series; each layer of feature extraction layer is used to output the hierarchical features corresponding to the current layer; the image features are obtained by fusing the hierarchical features corresponding to at least two layers. In this method, the image features include at least two layers of hierarchical features, so the image features contain richer feature hierarchies, which can cope with image recognition in complex scenes such as live broadcasts, thereby accurately and effectively identifying sensitive images and reducing the missed detection rate and false detection rate of sensitive images.

[0018] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A flowchart of an image classification method provided by an embodiment of the present invention;

[0022] Figure 2 A flowchart of another image classification method provided by an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of the structure of a feature extraction network provided by an embodiment of the present invention;

[0024] Figure 4 A flowchart of a method for training a feature extraction network provided by an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of the structure of an image classification device provided by an embodiment of the present invention;

[0026] Figure 6A schematic diagram of the structure of a training device for a feature extraction network provided by an embodiment of the present invention;

[0027] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0029] With the development of network technology and intelligent mobile platforms, live broadcast and mobile live broadcast have become deeply integrated into people's daily lives. If the videos circulated on the Internet are not regulated, they can easily become a means of spreading obscene, pornographic and violent content, causing harm to the majority of netizens. In order to regulate the content of videos circulated on the Internet, it is necessary to identify sensitive images from the videos. However, due to the huge number of live broadcast platforms, human supervision is often time-consuming and laborious, and consumes a lot of costs. In traditional methods, sensitive images can be identified through feature matching algorithms, but the live broadcast environment is diverse, the lighting changes strongly, the resolution is low, and the human body postures differ significantly, so it is impossible to achieve accurate classification with a simple feature matching algorithm. In addition, the number of training samples is too small and the training method is too simple to truly identify sensitive images with complex and diverse content.

[0030] In the related technology, these sensitive images can also be identified through deep learning methods, such as convolutional neural networks. This method has achieved good results in the field of image recognition. Specifically, the high-level semantic features of the image are first extracted through a pre-trained detection model, and then the image is classified based on the high-level semantic features. However, in the live broadcast scenario, the video images come from various sources and have complex structures. It is difficult for the above method to accurately and effectively identify sensitive images, resulting in high missed detection and false detection rates of sensitive images. Based on this, an image classification method, a training method and device for a feature extraction network provided in an embodiment of the present invention can be applied to mobile phones, computers and other devices, and in particular, can be applied to devices with online live broadcast or online video playback functions.

[0031] To facilitate understanding of this embodiment, an image classification method disclosed in an embodiment of the present invention is first described in detail. Figure 1 As shown, the method comprises the following steps:

[0032] Step S102, inputting the target image into a pre-trained feature extraction network, and outputting image features of the target image;

[0033] The target image may be a video image transmitted on the Internet or a video image in a live broadcast platform, for example, a live broadcast scene image is usually an image containing a person; the pre-trained feature extraction network may be a network model such as CNN (Convolutional Neural Networks), RNN (Recurrent Neural Network), DNN (Deep Neural Network), etc.; the network may usually include a multi-layer convolutional network and may also include multiple activation functions. The image features may usually include multi-level features of the target image, for example, may include one or more of the bottom layer (color, texture, etc.) features, middle layer (shape, etc.) features, or high layer (semantics, etc.) features of the target image. The image features may be feature vectors.

[0034] Specifically, the above target image can be expressed as X∈R H×W×3 , where H represents the height of the target image, W represents the width of the target image, and 3 represents that the target image is a three-channel image; the target image X∈R H×W×3 The feature extraction network is input into the above-mentioned feature extraction network, and the hierarchical features corresponding to at least two levels are extracted through the multi-level feature extraction layers therein. The hierarchical features are at least two of the bottom-level (color, texture, etc.) features, middle-level (shape, motion, etc.) features, or high-level (semantic, etc.) features of the target image. Then, the at least two hierarchical features are fused to obtain the image features. Specifically, the feature fusion can be performed by splicing or the like.

[0035] Step S104, based on the image features, determine the category of the target image; wherein the feature extraction network includes multiple levels of feature extraction layers connected in series; each level of feature extraction layer is used to output the level features corresponding to the current level; the image features are obtained by fusing the level features corresponding to at least two levels.

[0036] The above-mentioned target image may include multiple categories, which may be normal images and sensitive images; or normal images, vulgar images, pornographic images and violent images, among which vulgar images, pornographic images and violent images are sensitive images; in actual implementation, since the image features contain multi-level features of the target image, such as the color of the background and characters in the target image, the shape of the object, the semantics of the text and other features, the probability of each category of the target image can be determined according to the image features by calculating the probability, and finally the category of the target image can be determined according to the calculated probability of each category. The category of the target image can also be obtained by a classifier according to the image features.

[0037] The above-mentioned multi-level feature extraction layers include at least two, and may also include three, four or more. Generally, the more feature extraction layers there are, the richer the hierarchical features of the target image finally extracted, and the better the performance. Similarly, the time for extracting features will also increase, and the speed will be slower; therefore, the number of feature extraction layers can be set according to the actual classification speed and accuracy requirements. Each level of feature extraction layer mentioned above may include multiple convolutional networks (also called convolutional layers) and multiple activation functions; the function of the convolutional layer is convolution operation, and the purpose of convolution operation is to extract different features of the target image. The first convolutional layer can usually only extract some low-level image features such as edges, lines and corners. More layers of convolutional layers can iteratively extract more complex image features from low-level features. Therefore, for multiple layers of feature extraction, the image information of the hierarchical features output by each layer of feature extraction is different; for example, the hierarchical features output by the low-level feature extraction layer include relatively simple background color, texture and other features of the target image; the hierarchical features output by the middle-level feature extraction layer include the shape, character action, character skin color and area of ​​each object in the target image; the hierarchical features output by the higher-level feature extraction layer include the semantics of the text in the target image; the hierarchical features corresponding to different levels are fused to obtain the image features of the target image. This method can extract multiple hierarchical features to make the final fused image features richer.

[0038] An image classification method provided by an embodiment of the present invention inputs a target image into a pre-trained feature extraction network, outputs the image features of the target image; based on the image features, determines the category of the target image; wherein the feature extraction network includes multiple levels of feature extraction layers connected in series; each level of feature extraction layer is used to output the level features corresponding to the current level; the image features are obtained by fusing the level features corresponding to at least two levels. In this method, the image features include at least two levels of level features, so the image features contain richer feature levels, which can cope with image recognition in complex scenes such as live broadcasts, thereby accurately and effectively identifying sensitive images and reducing the missed detection rate and false detection rate of sensitive images.

[0039] This embodiment also provides another image classification method, which is implemented on the basis of the above embodiment. This embodiment focuses on the specific implementation process of the step of inputting the target image into the pre-trained feature extraction network and outputting the image features of the target image (implemented by steps S202-S204), and the specific implementation process of the step of determining the category of the target image based on the image features (implemented by steps S206-S208);

[0040] In this embodiment, the feature extraction network also includes at least two feature processing modules and a feature fusion module; each feature processing module is connected to a feature extraction layer of a hierarchy; the feature extraction layers connected to any two feature processing modules are different; wherein the feature processing module is used to further process the hierarchical features output by the corresponding feature extraction layer to obtain more accurate and discriminative features; the feature fusion module is used to fuse the features output by each feature processing module. The number of the feature processing modules is usually greater than or equal to the number of feature extraction layers.

[0041] like Figure 2 As shown, the method comprises the following steps:

[0042] Step S202, through each feature processing module, based on the attention mechanism, the hierarchical features output by the feature extraction layer connected to the feature processing module are processed to output intermediate features;

[0043] In order to make the hierarchical features output by the feature extraction layer more accurate and the feature information more obvious, the hierarchical features can be processed by the feature processing module. The above attention mechanism is similar to the different parts of the human retina having different degrees of information processing capabilities, that is, by scanning the hierarchical features output by the feature extraction layer connected to the feature processing module, the target feature that needs to be focused on is obtained, and then more attention resources are invested in this feature to obtain more detailed information related to the target feature, while ignoring other irrelevant information. Through this mechanism, limited attention resources can be used to quickly filter out high-value features from a large amount of information on hierarchical features, and then output intermediate features.

[0044] The above-mentioned at least two feature processing modules include: a first feature processing module, a second feature processing module and a third feature processing module; wherein the first feature processing module is connected to the lowest level feature extraction layer; the lowest level feature extraction layer is used to input the target image; the second feature processing module is connected to the feature extraction layer specified in the middle level; the third feature processing module is connected to the highest level feature extraction layer.

[0045] See also Figure 3The structural diagram of the feature extraction network shown in the figure takes the feature extraction network including five feature extraction layers connected in series as an example for explanation, namely feature extraction layer 1, feature extraction layer 2, feature extraction layer 3, feature extraction layer 4, and feature extraction layer 5; wherein feature extraction layer 1 corresponds to the feature extraction layer of the lowest level mentioned above; feature extraction layer 2, feature extraction layer 3, and feature extraction layer 4 correspond to the feature extraction layers in the middle level mentioned above; and feature extraction layer 5 corresponds to the feature extraction layer of the highest level mentioned above. In addition, the feature extraction network includes three feature processing modules, a first feature processing module, a second feature processing module, and a third feature processing module; which are connected to feature extraction layer 1, feature extraction layer 3, and feature extraction layer 5, respectively.

[0046] In addition, see above Figure 3 As shown, the feature processing module also includes a pooling layer, a first fully connected layer and a feature multiplication layer;

[0047] The above-mentioned pooling layer can also be called a pooling layer, which is mainly used to downsample the input features and reduce the number of parameters; the above-mentioned first fully connected layer (FC) plays the role of a "classifier" in the entire convolutional neural network, which can perform a weighted sum on the features output by the previous layer and map the feature space to the sample label space through a linear transformation; the above-mentioned feature multiplication layer (multiply) is mainly used to multiply the hierarchical features with the features output by the first fully connected layer.

[0048] A possible implementation:

[0049] The input hierarchical features are subjected to a first pooling process through the pooling layer, and a first pooling result is output; the first pooling result is subjected to a first fully connected process through the first fully connected layer, and a first fully connected result is output; the input hierarchical features and the first fully connected result are multiplied by the feature multiplication layer to obtain a multiplication result; and the intermediate features are output based on the multiplication result.

[0050] For details, see Figure 3 The data flow in the figure shown in the figure is explained by taking the feature processing module as the first feature processing module as an example. First, the target image X∈R H×W×3 Input into the lowest level feature extraction layer in the feature extraction network to obtain the hierarchical features, that is, the feature matrix f1∈R h1*w1*c1 , where h1 represents the height of the feature matrix, w1 represents the width of the feature matrix, and c1 represents the number of channels of the feature matrix. h1*w1*c1 Input to the pooling layer in the first feature processing module for the first pooling process, and output the first pooling result, that is, the feature vector f1′∈R c1 ; The first pooling result f1′∈Rc1 Input to the first fully connected layer, perform the first fully connected processing on the first pooling result, and output the first fully connected result f1″∈R c1 ; The hierarchical feature f1∈R h1*w1*c1 And the first full connection result f1″∈R c1 Multiply them and get the multiplication result f1″′∈R h1*w1*c1 ; Finally, the intermediate features are output based on the multiplication results; the intermediate features can represent the underlying features of the target image.

[0051] See also Figure 3 As shown, the feature processing module further includes a spatial pyramid pooling layer; the spatial pyramid pooling layer is connected to the feature multiplication layer; the spatial pyramid pooling layer (Spatial Pyramid Pooling, SPP) mainly processes features of different levels to obtain features of the same dimension.

[0052] A possible implementation of the above step of outputting the intermediate features based on the multiplication result is: performing a second pooling process on the multiplication result through a spatial pyramid pooling layer to output the intermediate features of a specified dimension, wherein the specified dimension can be set according to the actual application.

[0053] The multiplication result f1″′∈R is obtained by the above method h1*w1*c1 After that, the multiplication result f1″′∈R h1*w1*c1 Input to the spatial pyramid pooling layer, perform the second pooling process on the multiplication result, and output the intermediate features of the specified dimension, that is, the feature vector f1″″∈R c .

[0054] Similarly, taking the feature processing module as the second feature processing module as an example, the hierarchical feature f3∈R output by the designated feature extraction layer 3 of the intermediate level is also h3*w3*c3 , input to the second feature processing module, and obtain the second intermediate feature f3″″∈R c Taking the feature processing module as the third feature processing module as an example, the hierarchical feature f5∈R outputted by the highest level feature extraction layer 5 is also h5*w5*c5 , input to the third feature processing module, and obtain the third intermediate feature f5″″∈R c The spatial pyramid pooling layers in the first feature processing module, the second feature processing module, and the third feature processing module will output intermediate features of the same dimension respectively.

[0055] Step S204, fusing the intermediate features output by each feature processing module through a feature fusion module to obtain image features;

[0056] Specifically, the intermediate features output by each feature processing module, namely the first feature processing module, the second feature processing module and the third feature processing module, are input into the feature fusion module, and each intermediate feature is fused through feature splicing and other methods to obtain a multi-level fusion feature, namely the above-mentioned image feature.

[0057] See also Figure 3 The structural diagram of the feature extraction module shown in FIG. 1 is a schematic diagram of the feature extraction module, wherein the feature fusion module includes a feature concatenation layer, a second fully connected layer and a third fully connected layer; wherein the feature concatenation layer (concatenate) is mainly used to concatenate each intermediate feature f1″″∈R c , f3″″∈R c , f5″″∈R c The second fully connected layer and the third fully connected layer have the same function as the first fully connected layer.

[0058] A possible implementation of the above step of fusing the intermediate features output by each feature processing module through the feature fusion module to obtain image features is as follows:

[0059] The intermediate features output by each feature processing module are spliced ​​through the feature splicing layer to output the spliced ​​features; the spliced ​​features are subjected to a second fully connected processing through the second fully connected layer to output the second fully connected result; the second fully connected result is subjected to a third fully connected processing through the third fully connected layer to output the image features.

[0060] Specifically, the intermediate feature f1″″∈R of the target image X c , f3″″∈R c , f5″″∈R c Perform splicing processing and output splicing feature f = R 3c ; Set the splicing feature f = R 3c Input the second fully connected layer to perform the second fully connected processing on the splicing feature, and output the second fully connected result; input the second fully connected result to the third fully connected layer, perform the third fully connected processing on the second fully connected result, and output the output vector of the network, that is, the image feature z = R 3 .

[0061] Step S206, inputting the image features into a preset normalized exponential function, and outputting a probability distribution vector; wherein the probability distribution vector includes a plurality of categories and a probability value corresponding to each category;

[0062] The above normalized exponential function may be a softmax function; the above probability distribution vector may be expressed as p; the probability distribution vector may be specifically calculated by the following formula:

[0063]

[0064] Among them, p represents the probability distribution vector; z represents the image feature; m represents the mth feature processing template; p i and z i represent the i-th element of p and z respectively; e represents a natural constant;

[0065] Step S208: Determine the category corresponding to the maximum probability value in the probability distribution vector as the category of the target image.

[0066] Specifically, we can use the formula k = argmax i (p i ) Determine the coordinates corresponding to the maximum probability value in the probability distribution vector, that is, the category of the icon image; taking three preset categories as an example, k=1 represents that the target image is a normal image, k=2 represents that the target image is a vulgar image, and k=3 represents that the target image is a pornographic image.

[0067] In this method, through the multiple feature extraction layers included in the feature extraction network, and the first feature processing module, the second feature processing module, and the third feature processing module connected to different feature extraction layers; multiple hierarchical features in the target image can be extracted, and the hierarchical features are processed by the feature processing module to obtain intermediate features, which increases the discriminative power of the hierarchical features and makes the image information contained in the intermediate features more accurate and rich; each intermediate feature is fused by the feature fusion module to obtain image features, and the category of the target image is determined based on the image features. This method does not require manual design of the features of the target image, and automatically extracts features that are effective for image classification through a convolutional neural network. The algorithm has strong generalization ability and high robustness; by fusing the intermediate features of the bottom, middle and high layers of the image output by the feature extraction module, sensitive images of the live scene are recognized through image features, which can effectively identify sensitive images and improve the recognition accuracy of the feature extraction network; reduce the missed detection rate and false detection rate of sensitive images; and improve the accuracy of identifying sensitive images.

[0068] In live broadcast scenarios, the live broadcast images can be classified in the above manner to identify sensitive images in the images, so as to achieve the purpose of intelligent monitoring of the online live broadcast room while reducing labor costs.

[0069] This embodiment also provides a method for training a feature extraction network. Figure 4 As shown, the method comprises the following steps:

[0070] Step S402, determining a training sample based on a preset sample set; wherein the training sample includes a sample image and a category label of the sample image;

[0071] Specifically, a detailed image classification standard can be designed offline, which may include normal, vulgar, pornographic, and violent images (such as scenery is a normal image, exposed genitals is a pornographic image, kissing is a vulgar image, and knife-wounding is a violent image, etc.). According to this standard, a large number of live broadcast images can be manually annotated to obtain a data set D. According to a certain ratio, a part of the data set can be used as training samples D train , and the rest is used as test sample D test Specifically, the data set D can be divided into training samples D according to a ratio of 10:1. train and test sample D test ; The training samples include sample images and the category labels of the sample images may include four, that is, the annotation results can be expressed as y∈{1,2,3,4}, 1 represents normal, 2 represents vulgarity, 3 represents pornography, and 4 represents violence; the image classification is limited to the categories described in this application, and may also include other categories, such as leaking state secrets, breaking the law, etc.

[0072] Step S404, inputting the sample image into the feature extraction network to obtain sample features of the sample image; wherein the feature extraction network includes multiple levels of feature extraction layers connected in series; each level of feature extraction layer is used to output the level features corresponding to the current level; the sample features are obtained by fusing the level features corresponding to at least two levels;

[0073] Step S406, determining the category recognition result of the sample image based on the sample features; determining the loss value based on the preset classification loss function, the category label and the category recognition result; and updating the network parameters of the feature extraction network based on the loss value;

[0074] Step S408, continue to execute the step of determining training samples based on the preset sample set until the loss value converges to obtain a trained feature extraction network.

[0075] Specifically, the category recognition result z of the sample image can be 3 , input to the softmax function, through the formula Calculate the probability distribution vector p; then use the formula L = -log(p y ) calculates the loss value, where y is the label of the training sample image. Finally, the derivative of the loss value with respect to all network parameters W in the feature extraction network is calculated by the back propagation algorithm. Through the stochastic gradient descent algorithm, the network parameters of the feature extraction network are updated based on the calculated loss value, namely:

[0076]

[0077] Among them, α represents the learning rate (a pre-set hyperparameter, usually with a value of 0.01 or 0.001); the parameters of the feature extraction network are updated iteratively until the loss value converges, and the trained feature extraction network is obtained.

[0078] In addition, after training is completed, the test sample D test , test the trained feature extraction network, select multiple test images from the test samples, input them into the trained feature extraction network, and obtain the output vector, i.e., the image feature; based on the image features, determine the category of the test image and compare it with the labeled image. If the preset conditions are met, the trained feature extraction network is obtained.

[0079] The present embodiment provides a method for training a feature extraction network, which determines training samples based on a preset sample set; wherein the training samples include sample images and category labels of the sample images; the sample images are input into the feature extraction network to obtain sample features of the sample images; wherein the feature extraction network includes multiple levels of feature extraction layers connected in series; each level of feature extraction layer is used to output level features corresponding to the current level; the sample features are obtained by fusing level features corresponding to at least two levels; based on the sample features, the category recognition result of the sample image is determined; the loss value is determined based on a preset classification loss function, category labels and category recognition results; the network parameters of the feature extraction network are updated based on the loss value; and the step of determining the training samples based on the preset sample set is continued until the loss value converges to obtain a trained feature extraction network. In this method, the feature extraction network includes multiple layers of feature extraction layers, which can extract at least two levels of features in the target image, fuse the at least two levels of features to obtain image features, and determine the category of the target image based on the image features. Since the image features include at least two levels of hierarchical features, the image features contain richer feature levels and can cope with image recognition in complex scenarios such as live broadcasts, thereby accurately and effectively identifying sensitive images and reducing the missed detection rate and false detection rate of sensitive images.

[0080] In live broadcast scenarios, by collecting and annotating massive amounts of data as training samples and proposing detailed label classification standards to strongly annotate the training samples, the resulting feature extraction network is more in line with the regulatory requirements of live broadcast scenarios.

[0081] Corresponding to the above method embodiment, this embodiment also provides an image classification device, such as Figure 5 As shown, the device comprises:

[0082] An output module 51 is used to input the target image into a pre-trained feature extraction network and output image features of the target image;

[0083] The classification module 52 is used to determine the category of the target image based on the image features; wherein the feature extraction network includes multiple levels of feature extraction layers connected in series; each level of feature extraction layer is used to output the level features corresponding to the current level; the image features are obtained by fusing the level features corresponding to at least two levels.

[0084] An image classification device provided by an embodiment of the present invention inputs a target image into a pre-trained feature extraction network, outputs the image features of the target image; based on the image features, determines the category of the target image; wherein the feature extraction network includes multiple levels of feature extraction layers connected in series; each level of feature extraction layer is used to output the level features corresponding to the current level; the image features are obtained by fusing the level features corresponding to at least two levels. In this method, the image features include at least two levels of level features, so the image features contain richer feature levels, which can cope with image recognition in complex scenes such as live broadcasts, thereby accurately and effectively identifying sensitive images, and reducing the missed detection rate and false detection rate of sensitive images.

[0085] Furthermore, the feature extraction network also includes at least two feature processing modules and a feature fusion module; each feature processing module is connected to a feature extraction layer of a hierarchy; the feature extraction layers connected to any two feature processing modules are different; the output module is also used to: through each feature processing module, process the hierarchical features output by the feature extraction layer connected to the feature processing module based on the attention mechanism, and output intermediate features; through the feature fusion module, fuse the intermediate features output by each feature processing module to obtain image features.

[0086] Furthermore, the above-mentioned at least two feature processing modules include: a first feature processing module, a second feature processing module and a third feature processing module; wherein the first feature processing module is connected to the lowest-level feature extraction layer; the lowest-level feature extraction layer is used to input the target image; the second feature processing module is connected to the feature extraction layer specified in the middle level; and the third feature processing module is connected to the highest-level feature extraction layer.

[0087] Furthermore, the feature processing module includes a pooling layer, a first fully connected layer and a feature multiplication layer; the output module is also used to: perform a first pooling process on the input hierarchical features through the pooling layer, and output a first pooling result; perform a first fully connected process on the first pooling result through the first fully connected layer, and output a first fully connected result; multiply the input hierarchical features and the first fully connected result through the feature multiplication layer to obtain a multiplication result; and output intermediate features based on the multiplication result.

[0088] Furthermore, the feature processing module further includes a spatial pyramid pooling layer; the spatial pyramid pooling layer is connected to the feature multiplication layer; the output module is further used to: perform a second pooling process on the multiplication result through the spatial pyramid pooling layer, and output the intermediate features of the specified dimension.

[0089] Furthermore, the feature fusion module includes a feature splicing layer, a second fully connected layer and a third fully connected layer; the above-mentioned output module is also used to: perform splicing processing on the intermediate features output by each feature processing module through the feature splicing layer, and output the splicing features; perform second fully connected processing on the splicing features through the second fully connected layer, and output a second fully connected result; perform third fully connected processing on the second fully connected result through the third fully connected layer, and output image features.

[0090] Furthermore, the above-mentioned classification module is also used to: input image features into a preset normalized exponential function and output a probability distribution vector; wherein the probability distribution vector includes multiple categories and probability values ​​corresponding to each category; and determine the category corresponding to the largest probability value in the probability distribution vector as the category of the target image.

[0091] The image classification device provided in the embodiment of the present invention has the same technical features as the image classification method provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0092] Corresponding to the above method embodiment, this embodiment also provides a training device for a feature extraction network, such as Figure 6 As shown, the device comprises:

[0093] The sample determination module 61 is used to determine the training sample based on the preset sample set; wherein the training sample includes the sample image and the category label of the sample image;

[0094] The image input module 62 is used to input the sample image into the feature extraction network to obtain the sample features of the sample image; wherein the feature extraction network includes multiple levels of feature extraction layers connected in series; each level of feature extraction layer is used to output the level features corresponding to the current level; the sample features are obtained by fusing the level features corresponding to at least two levels;

[0095] The parameter updating module 63 is used to determine the category recognition result of the sample image based on the sample features; determine the loss value based on the preset classification loss function, the category label and the category recognition result; and update the network parameters of the feature extraction network based on the loss value;

[0096] The network determination module 64 is used to continue to execute the step of determining the training samples based on the preset sample set until the loss value converges to obtain the trained feature extraction network.

[0097] The feature extraction network training device provided in the embodiment of the present invention has the same technical features as the feature extraction network training method provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0098] This embodiment also provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned image classification method, or the training method of the feature extraction network.

[0099] See also Figure 7 As shown, the electronic device includes a processor 100 and a memory 101, wherein the memory 101 stores machine executable instructions that can be executed by the processor 100, and the processor 100 executes the machine executable instructions to implement the above-mentioned image classification method, or the training method of the feature extraction network.

[0100] Further, Figure 7 The electronic device shown further includes a bus 102 and a communication interface 103 , and the processor 100 , the communication interface 103 and the memory 101 are connected via the bus 102 .

[0101] The memory 101 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0102] The processor 100 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 100. The above processor 100 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 101, and the processor 100 reads the information in the memory 101 and completes the steps of the method of the above embodiment in combination with its hardware.

[0103] This embodiment also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned image classification method or feature extraction network training method.

[0104] The computer program product of the image classification method, feature extraction network training method and device provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0105] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0106] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0107] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0108] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0109] Finally, it should be noted that the above embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An image classification method, characterized in that: The method comprises: Inputting a target image into a pre-trained feature extraction network, and outputting image features of the target image; and determining the category of the target image based on the image features; The feature extraction network includes multiple levels of feature extraction layers connected in series; each level of the feature extraction layer is used to output the level features corresponding to the current level; the image features are obtained by fusing the level features corresponding to at least two levels; The feature extraction network also includes at least two feature processing modules and a feature fusion module; each feature processing module is connected to a hierarchical feature extraction layer; the feature extraction layers connected to any two feature processing modules are different; the step of inputting the target image into the pre-trained feature extraction network and outputting the image features of the target image includes: processing the hierarchical features output by the feature extraction layer connected to the feature processing module based on the attention mechanism through each feature processing module, and outputting the intermediate features; fusing the intermediate features output by each feature processing module through the feature fusion module to obtain the image features; The at least two feature processing modules include: a first feature processing module, a second feature processing module and a third feature processing module; wherein the first feature processing module is connected to the lowest level feature extraction layer; the lowest level feature extraction layer is used to input the target image; the second feature processing module is connected to a feature extraction layer specified in multiple intermediate levels; the third feature processing module is connected to the highest level feature extraction layer.

2. The method according to claim 1, characterized in that The feature processing module includes a pooling layer, a first fully connected layer and a feature multiplication layer; The step of processing the hierarchical features output by the feature extraction layer connected to the feature processing module based on the attention mechanism and outputting the intermediate features comprises: Performing a first pooling process on the input hierarchical features through the pooling layer, and outputting a first pooling result; performing a first full connection process on the first pooling result through the first fully connected layer, and outputting a first fully connected result; The input hierarchical feature and the first fully connected result are multiplied by the feature multiplication layer to obtain a multiplication result; and the intermediate feature is output based on the multiplication result.

3. The method according to claim 2, characterized in that The feature processing module further includes a spatial pyramid pooling layer; the spatial pyramid pooling layer is connected to the feature multiplication layer; The step of outputting the intermediate feature based on the multiplication result comprises: The multiplication result is subjected to a second pooling process by the spatial pyramid pooling layer, and the intermediate feature of a specified dimension is output.

4. The method according to claim 1, characterized in that: The feature fusion module includes a feature concatenation layer, a second fully connected layer and a third fully connected layer; The step of fusing the intermediate features output by each of the feature processing modules through the feature fusion module to obtain the image features comprises: The feature splicing layer performs splicing processing on the intermediate features output by each feature processing module, and outputs a spliced ​​feature; The second fully connected layer performs a second fully connected processing on the splicing feature, and outputs a second fully connected result; the third fully connected layer performs a third fully connected processing on the second fully connected result, and outputs the image feature.

5. The method according to claim 1, characterized in that The step of determining the category of the target image based on the image features comprises: Input the image features into a preset normalized exponential function and output a probability distribution vector; wherein the probability distribution vector includes multiple categories and a probability value corresponding to each category; The category corresponding to the maximum probability value in the probability distribution vector is determined as the category of the target image.

6. A method for training a feature extraction network, characterized in that: The method comprises: Determine a training sample based on a preset sample set; wherein the training sample includes a sample image and a category label of the sample image; Input the sample image into a feature extraction network to obtain sample features of the sample image; wherein the feature extraction network includes multiple levels of feature extraction layers connected in series; each level of the feature extraction layer is used to output level features corresponding to the current level; the sample features are obtained by fusing level features corresponding to at least two levels; Based on the sample features, determining the category recognition result of the sample image; determining a loss value based on a preset classification loss function, the category label and the category recognition result; and updating the network parameters of the feature extraction network based on the loss value; Continue to perform the step of determining training samples based on the preset sample set until the loss value converges to obtain the trained feature extraction network; The feature extraction network further includes at least two feature processing modules and a feature fusion module; each feature processing module is connected to a feature extraction layer of a hierarchy; the feature extraction layers connected to any two feature processing modules are different; through each feature processing module, the hierarchical features output by the feature extraction layer connected to the feature processing module are processed based on the attention mechanism to output intermediate features; through the feature fusion module, the intermediate features output by each feature processing module are fused to obtain image features; The at least two feature processing modules include: a first feature processing module, a second feature processing module and a third feature processing module; wherein the first feature processing module is connected to the lowest level feature extraction layer; the lowest level feature extraction layer is used to input the target image; the second feature processing module is connected to a feature extraction layer specified in multiple intermediate levels; the third feature processing module is connected to the highest level feature extraction layer.

7. An image classification device, characterized in that: The device comprises: An output module, used to input a target image into a pre-trained feature extraction network and output image features of the target image; A classification module, used for determining the category of the target image based on the image features; The feature extraction network includes multiple levels of feature extraction layers connected in series; each level of the feature extraction layer is used to output the level features corresponding to the current level; the image features are obtained by fusing the level features corresponding to at least two levels; The feature extraction network also includes at least two feature processing modules and a feature fusion module; each feature processing module is connected to a hierarchical feature extraction layer; the feature extraction layers connected to any two feature processing modules are different; the output module is used to process the hierarchical features output by the feature extraction layer connected to the feature processing module through each feature processing module based on the attention mechanism, and output intermediate features; the intermediate features output by each feature processing module are fused through the feature fusion module to obtain the image features; The at least two feature processing modules include: a first feature processing module, a second feature processing module and a third feature processing module; wherein the first feature processing module is connected to the lowest level feature extraction layer; the lowest level feature extraction layer is used to input the target image; the second feature processing module is connected to a feature extraction layer specified in multiple intermediate levels; the third feature processing module is connected to the highest level feature extraction layer.

8. A training device for a feature extraction network, characterized in that: The device comprises: A sample determination module, used to determine a training sample based on a preset sample set; wherein the training sample includes a sample image and a category label of the sample image; An image input module, used for inputting the sample image into a feature extraction network to obtain sample features of the sample image; wherein the feature extraction network comprises a plurality of feature extraction layers connected in series; each feature extraction layer is used for outputting the hierarchical features corresponding to the current level; the sample features are obtained by fusing the hierarchical features corresponding to at least two levels; A parameter updating module, used to determine the category recognition result of the sample image based on the sample feature; determine the loss value based on a preset classification loss function, the category label and the category recognition result; and update the network parameters of the feature extraction network based on the loss value; A network determination module, used to continue to perform the step of determining training samples based on a preset sample set until the loss value converges to obtain the trained feature extraction network; The feature extraction network further includes at least two feature processing modules and a feature fusion module; each feature processing module is connected to a feature extraction layer of a hierarchy; the feature extraction layers connected to any two feature processing modules are different; through each feature processing module, the hierarchical features output by the feature extraction layer connected to the feature processing module are processed based on the attention mechanism to output intermediate features; through the feature fusion module, the intermediate features output by each feature processing module are fused to obtain image features; The at least two feature processing modules include: a first feature processing module, a second feature processing module and a third feature processing module; wherein the first feature processing module is connected to the lowest level feature extraction layer; the lowest level feature extraction layer is used to input the target image; the second feature processing module is connected to a feature extraction layer specified in multiple intermediate levels; the third feature processing module is connected to the highest level feature extraction layer.

9. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the image classification method described in any one of claims 1 to 5, or the training method of the feature extraction network described in claim 6.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by the processor, the machine-executable instructions prompt the processor to implement the image classification method described in any one of claims 1 to 5, or the feature extraction network training method described in claim 6.

Images