An image data classification method, apparatus and electronic device

Abstract

Embodiments of the present application provide an image data classification method and device and electronic equipment, which are applied to the technical field of intelligent auditing. The method comprises: extracting image features of image data to be classified; determining a first pre-classification result of the image data based on the image features and using a first classification layer in an image classification model; wherein the image classification model further comprises at least one second classification layer; when the first pre-classification result indicates that the image data belongs to a specified image category, determining a second pre-classification result of the image data based on the image features and using the at least one second classification layer, and determining the image category of the image data based on the first pre-classification result and the second pre-classification result; and when the first pre-classification result indicates that the image data does not belong to the specified image category, taking the image category indicated by the first pre-classification result as the image category of the image data. Through the present solution, the accuracy and the false detection rate can be improved, and the efficiency of image data classification can be improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent review technology, and in particular to an image data classification method, apparatus, and electronic device. Background Technology

[0002] Intelligent review refers to using classification models to categorize image data to determine whether the image data belongs to the normal image category that meets the requirements or the abnormal image category that does not meet the requirements.

[0003] To improve the accuracy of classifying image data of abnormal image categories as abnormal images and reduce the false detection rate of classifying image data of normal image categories as abnormal images, related technologies require the use of multiple classification models to classify image data, and then combining the classification results of multiple classification models to determine the final classification result of the image data.

[0004] Because multiple classification models are required to classify image data, the intelligent review process needs to classify each image data multiple times, resulting in low efficiency in image data classification during intelligent review. Summary of the Invention

[0005] The purpose of this invention is to provide an image data classification method, apparatus, and electronic device to improve the efficiency of image data classification while increasing accuracy and reducing false detection rate. The specific technical solution is as follows:

[0006] In a first aspect of this invention, an image data classification method is provided, the method comprising:

[0007] Extract image features from the image data to be classified;

[0008] Based on the image features, a first pre-classification result of the image data is determined using a first classification layer in an image classification model, and is used as the first pre-classification result; wherein, the image classification model further includes at least a second classification layer;

[0009] When the first pre-classification result indicates that the image data belongs to a specified image category, based on the image features, the second pre-classification result of the image data is determined using the at least one second classification layer, and based on the first pre-classification result and the second pre-classification result, the image category of the image data is determined;

[0010] When the first pre-classification result indicates that the image data does not belong to the specified image category, the image category indicated by the first pre-classification result is taken as the image category of the image data.

[0011] Optionally, determining the image category of the image data based on the first pre-classification result and the second pre-classification result includes:

[0012] If the image category indicated by the second pre-classification result belongs to the specified image category, then the image category of the image data is determined to be the specified image category;

[0013] If the image category indicated by the second pre-classification result does not belong to the specified image category, then an image category is determined from each image category to be filtered as the image category of the image data; wherein, each image category to be filtered includes image categories other than the specified image category among the image categories that can be predicted by the first classification layer.

[0014] Optionally, an attention mechanism layer corresponding to each classification layer may be included before each classification layer;

[0015] Before the step of determining the pre-classification result of the image data using any classification layer based on the image features, the method further includes:

[0016] The image features are enhanced by using the corresponding attention mechanism layer of the classification layer to obtain the enhanced image features;

[0017] The step of determining the pre-classification result of the image data based on the image features using any classification layer includes:

[0018] Based on the enhanced image features, the pre-classification result of the image data is determined using this classification layer.

[0019] Optionally, the image features include: an image feature matrix output from at least one channel;

[0020] Each attention mechanism layer includes a channel attention mechanism layer and a spatial attention mechanism layer;

[0021] The step of using the attention mechanism layer corresponding to the classification layer to enhance the image features and obtain the enhanced image features includes:

[0022] Using the channel attention mechanism layer corresponding to the classification layer, for each channel, the image feature matrix of the channel is weighted according to the weights obtained by training for that channel in the channel attention mechanism layer of the classification layer, so as to obtain the preprocessed feature matrix of that channel.

[0023] Using the spatial attention mechanism layer corresponding to the classification layer, the preprocessed feature matrix of the at least one channel is weighted according to the weights trained by the spatial attention mechanism layer of the classification layer for each position in the feature matrix, to obtain the processed feature matrix of the at least one channel, which is used as the enhanced image feature.

[0024] Optionally, the image classification model is trained using the following methods:

[0025] Obtain a neural network model to be trained; wherein the neural network model to be trained includes a feature extraction layer, a first classification layer to be trained, and at least one second classification layer to be trained; the first classification layer to be trained and each second classification layer to be trained are respectively connected to the feature extraction layer;

[0026] The feature extraction layer and the first classification layer to be trained are trained using the first training sample set, and the trained first classification layer to be trained is used as the first classification layer.

[0027] After the first classification layer is obtained through training, for each of the at least one second classification layer to be trained, the second training sample set is used to train the second classification layer to be trained, and the trained second classification layer to be trained is taken as a second classification layer.

[0028] The neural network model to be trained after the training is completed is used as the image classification model.

[0029] Optionally, training the feature extraction layer and the first classification layer to be trained using the first training sample set includes:

[0030] Obtain sample images from the first training sample set;

[0031] The feature extraction layer is used to extract image features from the acquired sample images;

[0032] Based on the extracted image features, the pre-classification result of the acquired sample image is determined using the first classification layer to be trained;

[0033] Based on the pre-labeled image categories of the acquired sample images and the determined pre-classification results, the loss function value is determined;

[0034] The parameters in the feature extraction layer and the first training classification layer are adjusted using the determined loss function value.

[0035] Return to the step of obtaining sample images from the first training sample set until the number of training iterations for each sample image in the first training sample set is greater than the first specified number of iterations; wherein, the number of training iterations for each sample image is the number of times that sample image is used for training.

[0036] Optionally, training the second classification layer using the second training sample set includes:

[0037] Obtain sample images from the second training sample set;

[0038] The feature extraction layer is used to extract image features from the acquired sample images;

[0039] Based on the extracted image features, the pre-classification result of the acquired sample image is determined using the second classification layer to be trained;

[0040] The loss function value is determined based on the pre-labeled image categories of the acquired sample images and the determined pre-classification results;

[0041] The parameters in the second classification layer to be trained are adjusted using the determined loss function value.

[0042] Return to the step of obtaining sample images from the second training sample set until the number of training iterations for each sample image in the second training sample set is greater than the second specified number of iterations; wherein, the number of training iterations for each sample image is the number of times the sample image is used for training.

[0043] Optionally, the classification result determined by each classification layer includes: the confidence level of each image category that the classification layer can predict, wherein the confidence level of each image category represents the probability that the image data belongs to that image category as predicted by the classification layer;

[0044] Each classification result indicates the image category: the image category with the highest confidence among all image categories included in that classification result.

[0045] In a second aspect of the invention, an image data classification apparatus is also provided, the apparatus comprising:

[0046] The feature extraction module is used to extract image features from the image data to be classified.

[0047] The data classification module is used to determine a first pre-classification result of the image data based on the image features and using a first classification layer in the image classification model, as the first pre-classification result; wherein, the image classification model further includes at least a second classification layer;

[0048] The first determining module is configured to, when the first pre-classification result indicates that the image data belongs to a specified image category, determine a second pre-classification result of the image data based on the image features and using the at least one second classification layer, and determine the image category of the image data based on the first pre-classification result and the second pre-classification result;

[0049] The second determining module is used to take the image category indicated by the first pre-classification result as the image category of the image data when the first pre-classification result indicates that the image data does not belong to the specified image category.

[0050] Optionally, the first determining module includes:

[0051] The category determination submodule is used to determine the image category of the image data as the specified image category if the image category indicated by the second pre-classification result belongs to the specified image category; and to determine an image category from each image category to be filtered as the image category of the image data if the image category indicated by the second pre-classification result does not belong to the specified image category. The image categories to be filtered include image categories other than the specified image category among the image categories that can be predicted by the first classification layer.

[0052] Optionally, an attention mechanism layer corresponding to each classification layer may be included before each classification layer;

[0053] The first determining module is further configured to, before the step of determining the pre-classification result of the image data based on the image features using any classification layer, perform feature enhancement on the image features using the attention mechanism layer corresponding to the classification layer to obtain enhanced image features;

[0054] The first determining module is specifically used to determine the pre-classification result of the image data based on the enhanced image features and using the classification layer.

[0055] Optionally, the image features include: an image feature matrix output from at least one channel;

[0056] Each attention mechanism layer includes a channel attention mechanism layer and a spatial attention mechanism layer;

[0057] The first determining module includes:

[0058] The attention processing submodule is used to utilize the channel attention mechanism layer corresponding to the classification layer to perform weighted processing on the image feature matrix of each channel according to the weights trained for that channel in the channel attention mechanism layer of the classification layer, thereby obtaining the preprocessed feature matrix of that channel; and to utilize the spatial attention mechanism layer corresponding to the classification layer to perform weighted processing on the preprocessed feature matrix of the at least one channel according to the weights trained for each position in the feature matrix in the spatial attention mechanism layer of the classification layer, thereby obtaining the processed feature matrix of the at least one channel, which serves as the enhanced image feature.

[0059] Optionally, the image classification model is trained using the following modules, wherein:

[0060] A model acquisition module is used to acquire a neural network model to be trained; wherein the neural network model to be trained includes a feature extraction layer, a first classification layer to be trained, and at least one second classification layer to be trained; the first classification layer to be trained and each of the second classification layers to be trained are respectively connected to the feature extraction layer;

[0061] The first training module is used to train the feature extraction layer and the first classification layer to be trained using the first training sample set, and to use the trained first classification layer to be trained as the first classification layer.

[0062] The second training module is used to train each of the at least one second training classification layers after the first classification layer is trained, using the second training sample set, and to take the trained second training classification layer as a second classification layer.

[0063] The model determination module is used to use the neural network model to be trained after training as the image classification model.

[0064] Optionally, the first training module is specifically used to acquire sample images from the first training sample set;

[0065] The feature extraction layer is used to extract image features from the acquired sample images; based on the extracted image features, the first training classification layer is used to determine the pre-classification result of the acquired sample images; based on the pre-labeled image categories of the acquired sample images and the determined pre-classification result, a loss function value is determined; the determined loss function value is used to adjust the parameters in the first training classification layer; the process returns to the step of acquiring sample images in the first training sample set until the number of training iterations for each sample image in the first training sample set is greater than a first specified number of iterations; wherein, the number of training iterations for each sample image is the number of times the sample image is used for training.

[0066] Optionally, the second training module is specifically used to acquire sample images from the second training sample set; extract image features from the acquired sample images using the feature extraction layer; determine the pre-classification result of the acquired sample images using the second classification layer to be trained based on the extracted image features; determine the loss function value based on the pre-labeled image categories of the acquired sample images and the determined pre-classification result; adjust the parameters in the second classification layer to be trained using the determined loss function value; and return to the step of acquiring sample images from the second training sample set until the number of training iterations for each sample image in the second training sample set is greater than the second specified number of iterations; wherein, the number of training iterations for each sample image is the number of times training is performed using that sample image.

[0067] Optionally, the classification result determined by each classification layer includes: the confidence level of each image category that the classification layer can predict, wherein the confidence level of each image category represents the probability that the image data belongs to that image category as predicted by the classification layer;

[0068] Each classification result indicates the image category: the image category with the highest confidence among all image categories included in that classification result.

[0069] In another aspect of the present invention, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement the steps of any of the above-described image data classification methods when executing the program stored in the memory.

[0070] In another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements any of the image data classification methods described above.

[0071] In another aspect of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the image data classification methods described above.

[0072] This invention provides an image data classification method, apparatus, and electronic device that can extract image features from image data to be classified, and then determine a first pre-classification result of the image data based on the image features using a first classification layer in an image classification model. The image classification model further includes at least one second classification layer. When the first pre-classification result indicates that the image data belongs to a specified image category, a second pre-classification result of the image data is determined based on the image features using at least one second classification layer, and the image category of the image data is determined based on the first and second pre-classification results. When the first pre-classification result indicates that the image data does not belong to a specified image category, the image category indicated by the first pre-classification result is taken as the image category of the image data. Since a first pre-classification result can be obtained using the first classification layer, and then when the first pre-classification result indicates that the image data belongs to a specified image category, a second pre-classification result can be obtained using the second classification layer. The first and second pre-classification results are then combined to obtain the final classification result of the image data. When the first pre-classification result indicates that the image data does not belong to a specified image category, the image category indicated by the first pre-classification result is directly used as the classification result of the image data. It can be seen that only one classification is needed for each image data, instead of multiple classifications. This improves accuracy and reduces false detection rate while increasing classification efficiency. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0074] Figure 1 A flowchart of the image data classification method provided in the embodiments of the present invention;

[0075] Figure 2 This is another flowchart of the image data classification method provided in the embodiments of the present invention;

[0076] Figure 3 This is another flowchart of the image data classification method provided in an embodiment of the present invention;

[0077] Figure 4 This is a schematic diagram of an image classification model processing procedure provided in an embodiment of the present invention;

[0078] Figure 5 This is a schematic diagram of an image classification model processing procedure combined with an application scenario provided in an embodiment of the present invention;

[0079] Figure 6 This is a schematic diagram of the image data classification device provided in an embodiment of the present invention;

[0080] Figure 7This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0081] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0082] In order to improve the efficiency of image data classification while increasing accuracy and reducing false positive rate, embodiments of the present invention provide an image data classification method, apparatus and electronic device.

[0083] It should be noted that, in specific applications, the embodiments of the present invention can be applied to various electronic devices, such as personal computers, servers, mobile phones, and other devices with data processing capabilities. Furthermore, the image data classification method provided by the embodiments of the present invention can be implemented through software, hardware, or a combination of both.

[0084] The image data classification method provided in this embodiment of the invention may include:

[0085] Extract image features from the image data to be classified;

[0086] Based on image features, the first pre-classification result of the image data is determined using the first classification layer in the image classification model; wherein, the image classification model also includes at least one second classification layer;

[0087] When the first pre-classification result indicates that the image data belongs to a specified image category, a second pre-classification result of the image data is determined based on the image features using at least one second classification layer, and the image category of the image data is determined based on the first pre-classification result and the second pre-classification result.

[0088] When the first pre-classification result indicates that the image data does not belong to the specified image category, the image category indicated by the first pre-classification result is taken as the image category of the image data.

[0089] In the above-described scheme of this invention, since a first pre-classification result can be obtained using a first classification layer, and then when the first pre-classification result indicates that the image data belongs to a specified image category, a second pre-classification result can be obtained using a second classification layer. The first and second pre-classification results are then combined to obtain the final classification result of the image data. When the first pre-classification result indicates that the image data does not belong to a specified image category, the image category indicated by the first pre-classification result is directly used as the classification result of the image data. It can be seen that for each image data, only one classification is required, without multiple classifications. This improves accuracy and reduces false detection rate while increasing classification efficiency.

[0090] The image data classification method provided by the embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0091] like Figure 1 As shown, this embodiment of the invention provides an image data classification method, including steps S101-S104, wherein:

[0092] S101, Extract image features from the image data to be classified;

[0093] The image data to be classified can be any image data that needs to be reviewed, such as user-uploaded images and videos. Optionally, image data that needs to be reviewed can be read from a specified database as the image data to be classified. This specified database can be a database used to store images and videos that need to be reviewed.

[0094] After acquiring the image data to be classified, image features can be extracted. For example, the image data can be processed using a convolution kernel of a specified size, and the data obtained from the convolution process can be used as the image features.

[0095] In one implementation, a pre-trained feature extraction layer can be used to extract image features from the image data. This feature extraction layer may include a convolutional feature extraction network layer, which then performs convolution processing on the image data to obtain its features. These image features can be single-channel or multi-channel. For example, the image features could be 7x7x2048 high-dimensional feature data, meaning it contains 2048 channels, with each channel outputting a 7x7 image feature matrix. Alternatively, the image features could be 10×10×2048 high-dimensional feature data, meaning it contains 2048 channels, with each channel outputting a 10×10 image feature matrix.

[0096] Optionally, the image classification model described above can be an image classification model trained on a MobileNetV3 (third-generation mobile network) model. The feature extraction layer described above may include convolutional layers, depthwise separable convolutional layers, and projection layers, etc., for extracting image features from image data.

[0097] S102, Based on image features, determine the first pre-classification result of the image data using the first classification layer in the image classification model; wherein, the image classification model also includes at least one second classification layer;

[0098] After extracting the image features from the image data, the first classification layer in the image classification model can be used to process the extracted image features to obtain the first pre-classification result of the image data.

[0099] Optionally, in one implementation, the feature extraction layer used to extract image features from the image data in step S101 can be a feature extraction layer in an image classification model, which can be an image classification model trained on a MobileNetV3 (third-generation mobile network) model. In this case, the feature extraction layer can include convolutional layers, depthwise separable convolutional layers, and projection layers, etc., used to extract image features from the image data. Optionally, the first classification layer can be a classification layer included in the MobileNetV3 model, and the second classification layer can be a filtering branch added to the MobileNetV3 model. This filtering branch can include a classification layer; for ease of distinction, in this application, the classification layer included in the added filtering branch is referred to as the second classification layer.

[0100] Optionally, in order to improve the efficiency of image feature utilization, an attention mechanism layer corresponding to each classification layer may be included before each classification layer. That is, a corresponding attention mechanism layer may be included before the first classification layer and / or each second classification layer. The attention mechanism layer may include a channel attention mechanism layer and / or a spatial attention mechanism layer.

[0101] After obtaining the image features, these features can be input into the first classification layer so that the first classification layer can obtain the first pre-classification result of the image data based on the image features.

[0102] Optionally, the classification result determined by each classification layer includes: the confidence level of each image category that the classification layer can predict, wherein the confidence level of each image category represents the probability that the image data belongs to that image category as predicted by the classification layer; and the image category indicated by each classification result is: the image category with the highest confidence level among all image categories included in the classification result. The classification layer can be the first classification layer or any of the second classification layers described above.

[0103] For example, the classification results determined by the first classification layer include (image category 1, 50%), (image category 2, 30%), and (image category 3, 20%), which indicate that the confidence level of image data belonging to image category 1 is 50%, the confidence level of image data belonging to image category 2 is 30%, and the confidence level of image data belonging to image category 3 is 20%. The image category indicated by this classification result is image category 1, which has the highest confidence level.

[0104] Image categories refer to the results of classifying image data. For example, in scenarios where video images are classified, image categories could include normal, violation, or abnormal categories. Similarly, in scenarios where images are classified as objects, image categories could include people, cats, dogs, cars, etc. Optionally, in intelligent review scenarios, the aforementioned image categories could include sensitive or normal categories, which is also acceptable.

[0105] S103, when the first pre-classification result indicates that the image data belongs to a specified image category, based on the image features, at least one second classification layer is used to determine the second pre-classification result of the image data, and based on the first pre-classification result and the second pre-classification result, the image category of the image data is determined;

[0106] The specified image category can be determined based on needs and experience. For example, if the image categories include normal, sexy, and pornographic categories, the specified image category can be the pornographic category. This allows the second classification layer to correct image data that was originally not classified as pornographic by the first classification layer, thereby reducing the false detection rate of intelligent review.

[0107] If the first pre-classification result indicates that the image data belongs to a specified image category, the aforementioned image features will be input into at least one second classification layer, so that each second classification layer obtains a second pre-classification result for the image data based on the image features. Then, the image category of the image data can be determined based on the first and second pre-classification results.

[0108] Optionally, if the image category indicated by the second pre-classification result belongs to the specified image category, then the image category of the image data is determined to be the specified image category. If the image category indicated by the second pre-classification result does not belong to the specified image category, then an image category is determined from each of the image categories to be filtered as the image category of the image data. Each image category to be filtered includes image categories other than the specified image category that can be predicted by the first classification layer. Optionally, an image category can be randomly selected from each of the image categories to be filtered as the image category of the image data. Alternatively, to improve classification accuracy, if the classification result determined by each classification layer includes the confidence level of each image category that can be predicted by that classification layer, the image category with the highest confidence level in the first pre-classification result can be selected from each of the image categories to be filtered as the image category of the image data. In this case, if there are multiple image categories with the highest confidence level in the first pre-classification result, an image category can be randomly selected from these multiple image categories with the highest confidence level. Alternatively, to ensure recall, if the first pre-classification result indicates that the image data belongs to a specified image category, and the second classification result does not belong to the image category with the largest difference from the specified image category, then the first pre-classification result can be used as the image category of the image data. If the second classification result belongs to the image category with the largest difference from the specified image category, then the second classification result can be output as the image category of the image data. For example, if the image classification results are normal, sexy, and pornographic, and the specified image category is pornographic, then the image category with the largest difference from the specified image category is normal; similarly, if the specified image category is normal, then the image category with the largest difference from the specified image category is pornographic.

[0109] For example, such as Figure 2 As shown in the illustration, this embodiment of the invention provides a schematic diagram of image data classification. The categories to be classified include normal category, other category, and abnormal category. The specified image category is the abnormal category. In this case, when the image category output by the first classification layer is the abnormal category, a second classification layer is needed to further determine the category of the image data. If the image category determined by the second classification layer is still the abnormal category, then the category of the image data is determined to be the abnormal category. Otherwise, the sub-category determined by the first classification layer is taken as the final image category. For example, it can be determined whether the sub-category of the first classification layer is the normal category. If it is, then the image category of the image data is determined to be the normal category. Otherwise, the image category of the input image data is the other category.

[0110] S104, when the first pre-classification result indicates that the image data does not belong to the specified image category, the image category indicated by the first pre-classification result is taken as the image category of the image data.

[0111] When the first pre-classification result indicates that the image data does not belong to the specified image category, the image category indicated by the first pre-classification result can be directly used as the image category of the image data.

[0112] For example, still using Figure 2 The image data classification diagram shown below illustrates the process. The categories to be classified include normal, other, and abnormal. If the image category is specified as abnormal, then when the image category output by the first classification layer is normal or another category, the normal or other category will be directly used as the image category of the image data.

[0113] In one business scenario, specifically for intelligent image data review, the aforementioned "other categories" can be classified as "sexy" and the aforementioned "abnormal categories" as "pornographic." Therefore, if the image category output by the first classification layer is either "normal" or "sexy," it is directly output as either category. However, if the model outputs "pornographic," a second classification layer is needed for further classification. If the second classification layer classifies it as "normal," the output image data is classified as "normal"; otherwise, it is classified as "pornographic." Both of these are possible.

[0114] In the above-described scheme of this invention, since a first pre-classification result can be obtained using a first classification layer, and then when the first pre-classification result indicates that the image data belongs to a specified image category, a second pre-classification result can be obtained using a second classification layer. The first and second pre-classification results are then combined to obtain the final classification result of the image data. When the first pre-classification result indicates that the image data does not belong to a specified image category, the image category indicated by the first pre-classification result is directly used as the classification result of the image data. It can be seen that for each image data, only one classification is required, without multiple classifications. This improves accuracy and reduces false detection rate while increasing classification efficiency.

[0115] In one embodiment, to improve the accuracy of classification, an attention mechanism layer corresponding to each classification layer is included before each classification layer, wherein each classification layer includes a first classification layer and / or any second classification layer.

[0116] In this case, in order to improve the accuracy of classification, before determining the pre-classification result of the image data based on image features using any classification layer, for any classification layer, the image features can be enhanced using the attention mechanism layer corresponding to that classification layer to obtain enhanced image features. Then, based on the enhanced image features, the pre-classification result of the image data can be determined using that classification layer.

[0117] Optionally, the aforementioned image features may include an image feature matrix output by at least one channel. Each attention mechanism layer includes a channel attention mechanism layer and a spatial attention mechanism layer. In this case, for each classification layer, the channel attention mechanism layer corresponding to that classification layer can be used to weight the image feature matrix of that channel according to the weights trained for that channel in the channel attention mechanism layer of that classification layer, to obtain the preprocessed feature matrix of that channel. The spatial attention mechanism layer corresponding to that classification layer can be used to weight the preprocessed feature matrix of at least one channel according to the weights trained for each position in the feature matrix in the spatial attention mechanism layer of that classification layer, to obtain the processed feature matrix of at least one channel, which serves as the enhanced image features.

[0118] The main function of the attention mechanism layer is to re-encode the high-dimensional features (e.g., 7x7x2048-dimensional features) output by the feature extraction network layer, so that the model can perform weighted processing on both channel-dimensional and spatial-dimensional features during training and inference, thereby obtaining more effective feature representations. Effective feature representation mainly manifests in two aspects: 1) Effective features at the channel dimension: Each channel of the feature represents a certain feature of the original image. Channel dimension attention refers to weighting the channel dimension so that the model can learn which features are meaningful. Specifically, the high-dimensional feature 7x7x2048 is treated with spatial global average pooling and global max pooling to retain the channel dimension. At this point, the feature dimension is 1x1x2048. Then, the weight coefficients are obtained by using the Sigmoid activation function (a non-linear activation function). Finally, the weight 1x1x2048 is multiplied by the original feature 7x7x2048 to obtain the weighted feature. This feature can effectively represent the key information of the channel dimension, enabling the model to distinguish which features need attention and which features are noise that need to be avoided. 2) Effective features at the spatial dimension: This part of the weighting is mainly to enable the model to identify where the key information is in space and where the invalid noise is. Specifically, the high-dimensional feature 7x7x2048 is treated with channel dimension average pooling and max pooling to retain the spatial dimension. The features obtained from the two pooling methods are summed to obtain a 7x7x2 value. Then, a convolutional layer with a 7x7 kernel is passed through to obtain a spatial dimension weight of 7x7x1. Finally, 7x7x2048 is multiplied by the weight 7x7x1 to obtain the spatial dimension-weighted feature representation. This two-step weighting process enables the model to learn the differences in features, thereby distinguishing which features and where features are important, thus improving the accuracy of the recognition results.

[0119] In the above-described scheme of this invention, classification efficiency can be improved while simultaneously increasing accuracy and reducing false positive rate. Furthermore, the accuracy of classification can be further improved through an attention mechanism layer.

[0120] based on Figure 1 Examples, such as Figure 3 As shown, another embodiment of the image data classification method provided by the present invention trains an image classification model in the following manner, which may include steps S301-S304:

[0121] S301, Obtain the neural network model to be trained; wherein the neural network model to be trained includes a feature extraction layer, a first classification layer to be trained, and at least one second classification layer to be trained; the first classification layer to be trained and each second classification layer to be trained are respectively connected to the feature extraction layer;

[0122] Optionally, the aforementioned feature extraction layer can also be the feature extraction layer to be trained. The aforementioned neural network model to be trained can be a MobileNetV3 model, and the output layer of the MobileNetV3 model is replaced with a filtering branch, which can contain a classification layer. The aforementioned feature extraction layer is the feature extraction layer of the MobileNetV3 model, the first classification layer to be trained is the classification layer of the MobileNetV3 model, and the second classification layer to be trained is the classification layer contained in the newly added filtering branch.

[0123] In one implementation, before each classification layer in the neural network model to be trained, there may be an attention mechanism layer to be trained corresponding to that classification layer. This attention mechanism layer may include a channel attention mechanism layer and a spatial attention mechanism layer.

[0124] S302, using the first training sample set, train the first classification layer to be trained, and use the trained first classification layer to be trained as the first classification layer;

[0125] In one implementation, when the feature extraction layer in the neural network model to be trained is the feature extraction layer to be trained, the training feature extraction layer can be trained at the same time as the first classification layer to be trained, that is, the feature extraction layer to be trained and the first classification layer to be trained can be trained simultaneously.

[0126] The aforementioned first training sample set can be allocated according to the re-recall method based on the proportion of categories in the training set, completing the training of a large-scale re-recall first-level model, thereby ensuring the model's recall rate. The first-level model is the aforementioned first classification layer to be trained, and may also include the aforementioned feature extraction layer to be trained.

[0127] The above-mentioned training of the first classification layer using the first training sample set may include steps A1-A5:

[0128] Step A1: Obtain sample images from the first training sample set;

[0129] Step A2: Use the feature extraction layer to extract the image features of the acquired sample images;

[0130] Step A3: Based on the extracted image features, determine the pre-classification result of the acquired sample image using the first classification layer to be trained;

[0131] Step A4: Based on the pre-labeled image categories of the acquired sample images and the determined pre-classification results, determine the loss function value;

[0132] Step A5: Adjust the parameters in the first classification layer to be trained using the determined loss function value; return to step A1 until the number of training times for each sample image in the first training sample set is greater than the first specified number of times; wherein, the number of training times for each sample image is the number of times the sample image is used for training.

[0133] The first designated test can be set according to the requirements, such as 10 times. In this case, the first training sample set needs to be used to train the first classification layer for 10 rounds.

[0134] When the feature extraction layer in the neural network model to be trained is the feature extraction layer to be trained, the parameters in the feature extraction layer to be trained can be adjusted using the determined loss function value. After the first classification layer to be trained is trained, the model parameters in the trained feature extraction layer can be frozen, and the trained feature extraction layer can be used as the feature extraction layer, and the trained first classification layer to be trained can be used as the first classification layer.

[0135] S303, after training to obtain the first classification layer, for each of the at least one second classification layer to be trained, the second training sample set is used to train the second classification layer to be trained, and the trained second classification layer to be trained is taken as a second classification layer.

[0136] After freezing the model parameters of the trained feature extraction layer and the first classification layer, the second classification layer can be trained. It should be noted that during the training of the second classification layer, only the model parameters within the second classification layer need to be adjusted; the frozen model parameters of the trained feature extraction layer and the first classification layer will not be adjusted.

[0137] After completing the first-level model training and achieving the required accuracy, the entire backbone network is frozen. This involves freezing the model parameters of the first classification layer and feature extraction layer, and then replacing the last layer with a binary classification filtering branch. An attention mechanism is added to this filtering branch, effectively creating a second training classification layer. The second training set is then used to train this second classification layer. Compared to the first training set, the class distribution in the second training set can be optimized to reduce false positives, increasing the proportion of normal class data.

[0138] The above-mentioned training of the second classification layer using the second training sample set may include steps B1-B5:

[0139] Step B1: Obtain sample images from the second training sample set;

[0140] Step B2: Use the feature extraction layer to extract the image features of the acquired sample images;

[0141] Step B3: Based on the extracted image features, the pre-classification result of the acquired sample image is determined using the second classification layer to be trained;

[0142] In one implementation, if an attention mechanism layer and a classification sub-layer to be trained are included before the second classification layer to be trained, the attention mechanism layer can be used to enhance the extracted image features to obtain enhanced image features. Then, based on the enhanced image features, the classification sub-layer to be trained is used to determine the pre-classification result of the acquired sample image.

[0143] Step B4: Determine the loss function value based on the pre-labeled image categories of the acquired sample images and the determined pre-classification results;

[0144] Step B5: Adjust the parameters in the second classification layer to be trained using the determined loss function value; return to step B1 until the number of training iterations for each sample image in the second training sample set is greater than the second specified number of iterations; wherein, the number of training iterations for each sample image is the number of times the sample image is used for training.

[0145] The first and second specified number of times can be determined according to requirements. The first and second specified number of times can be the same or different, which is also acceptable.

[0146] S304 uses the neural network model to be trained after training as an image classification model.

[0147] After training to obtain the first and second classification layers, the image feature extraction layer, the first classification layer, and the second classification layer can be fused together to obtain a complete network structure. An update module is added to perform secondary filtering on images identified as belonging to a specified image category by the first classification layer, thereby reducing false positives. During inference, the first classification layer first yields a preliminary inference result. Then, based on the initial pre-classification result of the first classification layer, different logics are used in the update module for secondary filtering to obtain the final inference result.

[0148] The above-described solution in this invention can improve classification efficiency while simultaneously increasing accuracy and reducing false detection rate. It also provides a training method for an image classification model, laying the foundation for improving classification efficiency while increasing accuracy and reducing false detection rate.

[0149] In one embodiment, such as Figure 4 As shown in the diagram, this embodiment of the invention provides a schematic diagram of an image classification model processing procedure. After image data is input into the image classification model, the feature extraction layer first extracts image features from the image data. These features are then input into a first classification layer to obtain a first pre-classification result. When the first pre-classification result indicates that the image data belongs to a specified image category, it is first processed by a channel attention mechanism layer, then by a spatial attention mechanism layer, and finally input into a second classification layer to obtain a second pre-classification result. Finally, through a fusion strategy, the first and second pre-classification results are combined to obtain the final result of the image data and output it.

[0150] In one embodiment, such as Figure 5 As shown in the diagram, this embodiment of the invention provides a schematic diagram of an image classification model processing procedure combined with an application scenario. In the intelligent review business scenario for image data, the image classification model can classify images into categories including normal, sexy, and pornographic. The specified image category is pornographic, and the category with the largest difference from the specified image category is the normal category. At this time, the image features extracted from the image data are input into the first classification layer to obtain a first pre-classification result. If the first pre-classification result is a normal or sexy category, then the normal or sexy category is output. If the first pre-classification result is pornographic, then the image features of the image data are input into the second classification layer to obtain a second pre-classification result. If the second pre-classification result does not belong to the normal category, then the pornographic category is output; otherwise, if the second pre-classification result belongs to the normal category, then it is determined whether the sub-category in the first pre-classification result is a normal category. If yes, then the normal category is output; otherwise, the sexy category is output.

[0151] Corresponding to the image data classification method provided in the above embodiments of the present invention, such as Figure 6 As shown, this embodiment of the invention also provides an image data classification device, the device comprising:

[0152] Feature extraction module 601 is used to extract image features from the image data to be classified;

[0153] The data classification module 602 is used to determine a first pre-classification result of the image data based on the image features and using a first classification layer in the image classification model, as the first pre-classification result; wherein, the image classification model further includes at least a second classification layer;

[0154] The first determining module 603 is configured to, when the first pre-classification result indicates that the image data belongs to a specified image category, determine a second pre-classification result of the image data based on the image features and using the at least one second classification layer, and determine the image category of the image data based on the first pre-classification result and the second pre-classification result;

[0155] The second determining module 604 is used to take the image category indicated by the first pre-classification result as the image category of the image data when the first pre-classification result indicates that the image data does not belong to the specified image category.

[0156] Optionally, the first determining module includes:

[0157] The category determination submodule is used to determine the image category of the image data as the specified image category if the image category indicated by the second pre-classification result belongs to the specified image category; and to determine an image category from each image category to be filtered as the image category of the image data if the image category indicated by the second pre-classification result does not belong to the specified image category. The image categories to be filtered include image categories other than the specified image category among the image categories that can be predicted by the first classification layer.

[0158] Optionally, an attention mechanism layer corresponding to each classification layer may be included before each classification layer;

[0159] The first determining module is further configured to, before the step of determining the pre-classification result of the image data based on the image features using any classification layer, perform feature enhancement on the image features using the attention mechanism layer corresponding to the classification layer to obtain enhanced image features;

[0160] The first determining module is specifically used to determine the pre-classification result of the image data based on the enhanced image features and using the classification layer.

[0161] Optionally, the image features include: an image feature matrix output from at least one channel;

[0162] Each attention mechanism layer includes a channel attention mechanism layer and a spatial attention mechanism layer;

[0163] The first determining module includes:

[0164] The attention processing submodule is used to utilize the channel attention mechanism layer corresponding to the classification layer to perform weighted processing on the image feature matrix of each channel according to the weights trained for that channel in the channel attention mechanism layer of the classification layer, thereby obtaining the preprocessed feature matrix of that channel; and to utilize the spatial attention mechanism layer corresponding to the classification layer to perform weighted processing on the preprocessed feature matrix of the at least one channel according to the weights trained for each position in the feature matrix in the spatial attention mechanism layer of the classification layer, thereby obtaining the processed feature matrix of the at least one channel, which serves as the enhanced image feature.

[0165] Optionally, the image classification model is trained using the following modules, wherein:

[0166] A model acquisition module is used to acquire a neural network model to be trained; wherein the neural network model to be trained includes a feature extraction layer, a first classification layer to be trained, and at least one second classification layer to be trained; the first classification layer to be trained and each of the second classification layers to be trained are respectively connected to the feature extraction layer;

[0167] The first training module is used to train the feature extraction layer and the first classification layer to be trained using the first training sample set, and to use the trained first classification layer to be trained as the first classification layer.

[0168] The second training module is used to train each of the at least one second training classification layers after the first classification layer is trained, using the second training sample set, and to take the trained second training classification layer as a second classification layer.

[0169] The model determination module is used to use the neural network model to be trained after training as the image classification model.

[0170] Optionally, the first training module is specifically used to acquire sample images from a first training sample set; extract image features from the acquired sample images using the feature extraction layer; determine the pre-classification result of the acquired sample images using the first training classification layer based on the extracted image features; determine a loss function value based on the pre-labeled image categories of the acquired sample images and the determined pre-classification result; adjust the parameters in the feature extraction layer and the first training classification layer using the determined loss function value; and return to the step of acquiring sample images from the first training sample set until the number of training iterations for each sample image in the first training sample set is greater than a first specified number of iterations; wherein, the number of training iterations for each sample image is the number of times training is performed using that sample image.

[0171] Optionally, the second training module is specifically used to acquire sample images from the second training sample set; extract image features from the acquired sample images using the feature extraction layer; determine the pre-classification result of the acquired sample images using the second classification layer to be trained based on the extracted image features; determine the loss function value based on the pre-labeled image categories of the acquired sample images and the determined pre-classification result; adjust the parameters in the second classification layer to be trained using the determined loss function value; and return to the step of acquiring sample images from the second training sample set until the number of training iterations for each sample image in the second training sample set is greater than the second specified number of iterations; wherein, the number of training iterations for each sample image is the number of times training is performed using that sample image.

[0172] Optionally, the classification result determined by each classification layer includes: the confidence level of each image category that the classification layer can predict, wherein the confidence level of each image category represents the probability that the image data belongs to that image category as predicted by the classification layer;

[0173] Each classification result indicates the image category: the image category with the highest confidence among all image categories included in that classification result.

[0174] In the above-described scheme of this invention, since a first pre-classification result can be obtained using a first classification layer, and then when the first pre-classification result indicates that the image data belongs to a specified image category, a second pre-classification result can be obtained using a second classification layer. The first and second pre-classification results are then combined to obtain the final classification result of the image data. When the first pre-classification result indicates that the image data does not belong to a specified image category, the image category indicated by the first pre-classification result is directly used as the classification result of the image data. It can be seen that for each image data, only one classification is required, without multiple classifications. This improves accuracy and reduces false detection rate while increasing classification efficiency.

[0175] This invention also provides an electronic device, such as... Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703, and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0176] Memory 703 is used to store computer programs;

[0177] When processor 701 executes a program stored in memory 703, it performs the following steps:

[0178] Extract image features from the image data to be classified;

[0179] Based on the image features, a first pre-classification result of the image data is determined using a first classification layer in an image classification model; wherein, the image classification model further includes at least a second classification layer;

[0180] When the first pre-classification result indicates that the image data belongs to a specified image category, based on the image features, the second pre-classification result of the image data is determined using the at least one second classification layer, and based on the first pre-classification result and the second pre-classification result, the image category of the image data is determined;

[0181] When the first pre-classification result indicates that the image data does not belong to the specified image category, the image category indicated by the first pre-classification result is taken as the image category of the image data.

[0182] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0183] The communication interface is used for communication between the aforementioned terminal and other devices.

[0184] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0185] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0186] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements any of the image data classification methods described in the above embodiments.

[0187] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the image data classification methods described in the above embodiments.

[0188] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0189] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0190] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0191] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. An image data classification method characterized by, The method comprises: extracting image features of image data to be classified; determining a first pre-classification result of the image data by using a first classification layer in an image classification model based on the image features, wherein the image classification model further comprises at least one second classification layer; when the first pre-classification result indicates that the image data belongs to a specified image category, determining a second pre-classification result of the image data by using the at least one second classification layer based on the image features, and if the image category indicated by the second pre-classification result belongs to the specified image category, determining the image category of the image data as the specified image category; if the image category indicated by the second pre-classification result does not belong to the specified image category, determining an image category from each image category to be screened as the image category of the image data, wherein the each image category to be screened includes image categories other than the specified image category in each image category that can be predicted by the first classification layer; when the first pre-classification result indicates that the image data does not belong to the specified image category, determining the image category of the image data as the image category indicated by the first pre-classification result.

2. The method of claim 1, wherein, Each classification layer further comprises a corresponding attention mechanism layer; Before the step of determining a pre-classification result of the image data by using any classification layer based on the image features, the method further comprises: performing feature enhancement on the image features by using the corresponding attention mechanism layer of the classification layer to obtain enhanced image features; the step of determining a pre-classification result of the image data by using any classification layer based on the image features comprises: determining a pre-classification result of the image data by using the classification layer based on the enhanced image features.

3. The method of claim 2, wherein, The image features comprise image feature matrices output by at least one channel; Each attention mechanism layer comprises a channel attention mechanism layer and a spatial attention mechanism layer; the step of performing feature enhancement on the image features by using the corresponding attention mechanism layer of the classification layer to obtain enhanced image features comprises: for each channel, performing weighted processing on the image feature matrix of the channel by using the corresponding channel attention mechanism layer of the classification layer according to the weight trained for the channel in the channel attention mechanism layer of the classification layer to obtain a pre-processed feature matrix of the channel; performing weighted processing on the pre-processed feature matrix of the at least one channel by using the corresponding spatial attention mechanism layer of the classification layer according to the weight trained for each position in the feature matrix in the spatial attention mechanism layer of the classification layer to obtain a processed feature matrix of the at least one channel as the enhanced image features.

4. The method of claim 1, wherein, The image classification model is trained in the following manner, comprising: obtaining a neural network model to be trained; wherein the neural network model to be trained comprises a feature extraction layer, a first classification layer to be trained, and at least one second classification layer to be trained; the first classification layer to be trained and each second classification layer to be trained are connected with the feature extraction layer respectively; training the first to-be-trained classification layer by using the first training sample set, and taking the trained first to-be-trained classification layer as the first classification layer; after the first classification layer is trained, for each to-be-trained classification layer in the at least one second to-be-trained classification layer, training the second to-be-trained classification layer by using the second training sample set, and taking the trained second to-be-trained classification layer as a second classification layer; taking the trained neural network model as the image classification model.

5. The method of claim 4, wherein, The training of the first to-be-trained classification layer by using the first training sample set comprises: obtaining a sample image in the first training sample set; extracting image features of the obtained sample image by using the feature extraction layer; determining a pre-classification result of the obtained sample image by using the first to-be-trained classification layer based on the extracted image features; determining a loss function value based on a labeled image category pre-labeled by the obtained sample image and the determined pre-classification result; adjusting parameters in the first to-be-trained classification layer by using the determined loss function value; returning to execute the step of obtaining a sample image in the first training sample set until the training times of each sample image in the first training sample set are greater than a first specified number; wherein the training times of each sample image is the number of times of training by using the sample image.

6. The method of claim 4, wherein, The training of the second to-be-trained classification layer by using the second training sample set comprises: obtaining a sample image in the second training sample set; extracting image features of the obtained sample image by using the feature extraction layer; determining a pre-classification result of the obtained sample image by using the second to-be-trained classification layer based on the extracted image features; determining a loss function value based on a labeled image category pre-labeled by the obtained sample image and the determined pre-classification result; adjusting parameters in the second to-be-trained classification layer by using the determined loss function value; returning to execute the step of obtaining a sample image in the second training sample set until the training times of each sample image in the second training sample set are greater than a second specified number; wherein the training times of each sample image is the number of times of training by using the sample image.

7. The method of claim 1, wherein, Each classification result comprises: a confidence of each image category that can be predicted by the classification layer, wherein the confidence of each image category represents the probability of the image data belonging to the image category predicted by the classification layer; The image category indicated by each classification result is: the image category with the maximum confidence among the image categories contained in the classification result.

8. An image data classifying apparatus characterized by comprising: The device comprises: a feature extraction module configured to extract image features of image data to be classified; a data classification module configured to determine a first pre-classification result of the image data by using a first classification layer in an image classification model based on the image features, as a first pre-classification result; wherein the image classification model further comprises at least one second classification layer. The first determining module is configured to, when the first pre-classification result indicates that the image data belongs to a specified image category, determine a second pre-classification result of the image data based on the image features and the at least one second classification layer, and determine the image category of the image data based on the first pre-classification result and the second pre-classification result. The second determining module is configured to, when the first pre-classification result indicates that the image data does not belong to the specified image category, determine the image category indicated by the first pre-classification result as the image category of the image data. The first determining module comprises: The category determining submodule is configured to, if the image category indicated by the second pre-classification result belongs to the specified image category, determine the image category of the image data as the specified image category; if the image category indicated by the second pre-classification result does not belong to the specified image category, determine an image category from each image category to be screened, as the image category of the image data; wherein the each image category to be screened comprises an image category other than the specified image category in each image category that can be predicted by the first classification layer.

9. An electronic device, comprising: The apparatus comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus. The memory is configured to store a computer program. The processor is configured to execute the program stored in the memory to implement the method steps in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer program stored in the computer readable storage medium is executed by the processor to implement the method steps in any one of claims 1-7.

Images