Image classification method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to the technical field of image processing, and particularly provides an image classification method and device, electronic equipment, a storage medium and a program product. The image classification method comprises the following steps: transforming an original image to be classified to obtain at least one transformed image; performing feature extraction on the original image by adopting a first feature extraction network in the image classification model to obtain original image features; adopting a second feature extraction network in the image classification model to perform feature extraction on the at least one transformed image to obtain transformed image features corresponding to the at least one transformed image; and adopting a fusion classification module in the image classification model to obtain the image category of the original image according to the original image feature and the at least one transformed image feature. Therefore, image classification is carried out by combining the original image features and the transformed image features, and the accuracy of image classification is improved while the cost is considered.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, electronic device, storage medium and program product for image classification. Background Art

[0002] In the image review scenario, the platform usually uses image classification to determine whether the images uploaded by users contain negative information, so as to prohibit the spread of images containing negative information. For example, in the homework grading system, it is usually necessary to determine whether the images to be published contain violent and terrifying images.

[0003] In the prior art, image features of an image are usually extracted, and based on the image features, it is determined whether the image contains bad information.

[0004] However, using this method, the misclassification rate of images is usually high. Under the premise of taking cost into consideration, it is usually difficult to meet the scenarios with high requirements for image classification accuracy. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method, device, electronic device, storage medium and program product for image classification, so as to improve the accuracy of image classification while taking into account the cost.

[0006] On the one hand, an embodiment of the present application provides a method for image classification, including:

[0007] Transforming the original image to be classified to obtain at least one transformed image;

[0008] Using the first feature extraction network in the image classification model, extract features from the original image to obtain the original image features;

[0009] The second feature extraction network in the image classification model is used to extract features from at least one transformed image to obtain transformed image features corresponding to the at least one transformed image; the first feature extraction network and the second feature extraction network respectively use different image feature extraction methods to extract image features;

[0010] The image category of the original image is obtained according to the original image feature and at least one transformed image feature by using the fusion classification module in the image classification model.

[0011] In one implementation, a fusion classification module in an image classification model is used to obtain an image category of an original image according to the original image feature and at least one transformed image feature, including:

[0012] For each transformed image feature, the following steps are performed: determining an initial category according to the original image features and the transformed image features through a fusion classification module;

[0013] If it is determined that the initial category corresponding to at least one transformed image feature meets the set category condition, the image category is determined to be the target category; otherwise, it is determined that the image category is not the target category.

[0014] In one implementation, setting the category condition includes any one of the following:

[0015] At least one of the transformed image features has initial categories corresponding to each of the target categories;

[0016] The number of target categories included in the initial category corresponding to at least one transformed image feature is not less than a set number;

[0017] The proportion of the number of target categories included in the initial categories corresponding to at least one transformed image feature is not less than a set proportion.

[0018] In one embodiment, the transformed image includes at least one of the following:

[0019] Grayscale image of the original image; Green channel image of the original image; Red channel image of the original image; Blue channel image of the original image.

[0020] In one embodiment, the method further comprises:

[0021] If the image category is not the target category, it is determined that the image review has passed; otherwise, it is determined that the image review has failed.

[0022] In one embodiment, the method further comprises:

[0023] If the image category is not the target category, the image is deemed to have passed the review;

[0024] If it is determined that the image category is a target category, it is determined whether there is a target category image matching the original image in the target category image set. If so, it is determined that the image review has failed. Otherwise, it is determined that the image review has passed. The target category image set contains multiple target category images.

[0025] In one implementation, the target category image set further includes target category image features corresponding to each target category image;

[0026] Determine whether there is a target category image matching the original image in the target category image set, including:

[0027] If it is determined that there are target category image features matching the original image features in the target category image set, then it is determined that there are target category images matching the original image; otherwise, it is determined that there are no target category images matching the original image.

[0028] In one implementation, the target category image set further includes target category image features corresponding to each target category image;

[0029] Determine whether there is a target category image matching the original image in the target category image set, including:

[0030] The third feature extraction network is used to extract features from the original image to obtain the retrieval features of the original image;

[0031] If it is determined that there are target category image features matching the original image retrieval features in the target category image set, then it is determined that there are target category images matching the original image; otherwise, it is determined that there are no target category images matching the original image.

[0032] In one embodiment, the method further comprises:

[0033] The following steps are used to train the image classification model:

[0034] Obtain training data; the training data includes multiple image samples and their corresponding category labels;

[0035] Transform each image sample respectively to obtain at least one transformed image corresponding to each image sample;

[0036] For each image sample and each corresponding transformed image, the following steps are performed respectively until the set training conditions are met: using the initial image classification model, based on the image sample and its corresponding transformed image, the corresponding initial category is determined;

[0037] Determining a classification loss based on a category label corresponding to the image sample and an initial category corresponding to each of the at least one transformed image features;

[0038] According to the classification loss, the model parameters are adjusted.

[0039] On the one hand, an embodiment of the present application provides an image classification device, including:

[0040] A transformation unit, used for transforming the original image to be classified to obtain at least one transformed image;

[0041] A first extraction unit is used to extract features from the original image using a first feature extraction network in the image classification model to obtain features of the original image;

[0042] The second extraction unit is used to use the second feature extraction network in the image classification model to perform feature extraction on at least one transformed image respectively, and obtain the transformed image features corresponding to each of the at least one transformed image; the first feature extraction network and the second feature extraction network respectively use different image feature extraction methods to extract image features;

[0043] The classification unit is used to obtain the image category of the original image according to the original image feature and at least one transformed image feature by using the fusion classification module in the image classification model.

[0044] In one embodiment, the classification unit is used to:

[0045] For each transformed image feature, the following steps are performed: determining an initial category according to the original image features and the transformed image features through a fusion classification module;

[0046] If it is determined that the initial category corresponding to at least one transformed image feature meets the set category condition, the image category is determined to be the target category; otherwise, it is determined that the image category is not the target category.

[0047] In one implementation, setting the category condition includes any one of the following:

[0048] At least one of the transformed image features has initial categories corresponding to each of the target categories;

[0049] The number of target categories included in the initial category corresponding to at least one transformed image feature is not less than a set number;

[0050] The proportion of the number of target categories included in the initial categories corresponding to at least one transformed image feature is not less than a set proportion.

[0051] In one embodiment, the transformed image includes at least one of the following:

[0052] Grayscale image of the original image; Green channel image of the original image; Red channel image of the original image; Blue channel image of the original image.

[0053] In one embodiment, the classification unit is also used to:

[0054] If the image category is not the target category, it is determined that the image review has passed; otherwise, it is determined that the image review has failed.

[0055] In one embodiment, the classification unit is also used to:

[0056] If the image category is not the target category, the image is deemed to have passed the review;

[0057] If it is determined that the image category is a target category, it is determined whether there is a target category image matching the original image in the target category image set. If so, it is determined that the image review has failed. Otherwise, it is determined that the image review has passed. The target category image set contains multiple target category images.

[0058] In one implementation, the target category image set further includes target category image features corresponding to each target category image;

[0059] The classification unit is also used to: if it is determined that there are target category image features matching the original image features in the target category image set, then determine that there are target category images matching the original image; otherwise, determine that there are no target category images matching the original image.

[0060] In one implementation, the target category image set further includes target category image features corresponding to each target category image;

[0061] Taxa are also used for:

[0062] The third feature extraction network is used to extract features from the original image to obtain the retrieval features of the original image;

[0063] If it is determined that there are target category image features matching the original image retrieval features in the target category image set, then it is determined that there are target category images matching the original image; otherwise, it is determined that there are no target category images matching the original image.

[0064] In one embodiment, the classification unit is also used to:

[0065] The following steps are used to train the image classification model:

[0066] Obtain training data; the training data includes multiple image samples and their corresponding category labels;

[0067] Transform each image sample respectively to obtain at least one transformed image corresponding to each image sample;

[0068] For each image sample and each corresponding transformed image, the following steps are performed respectively until the set training conditions are met: using the initial image classification model, based on the image sample and its corresponding transformed image, the corresponding initial category is determined;

[0069] Determining a classification loss based on a category label corresponding to the image sample and an initial category corresponding to each of the at least one transformed image features;

[0070] According to the classification loss, the model parameters are adjusted.

[0071] On the one hand, an embodiment of the present application provides an electronic device, including:

[0072] Processor; and

[0073] A memory stores computer instructions, wherein the computer instructions are used to enable a processor to execute the steps of the method provided in any of the various optional implementations of any of the above-mentioned image classifications.

[0074] On the one hand, an embodiment of the present application provides a computer-readable storage medium storing computer instructions, which are used to enable a computer to execute the steps of the method provided in any of the various optional implementations of any of the above-mentioned image classifications.

[0075] On the one hand, a computer program product is provided in an embodiment of the present application, including a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the steps of the method provided in any of the various optional implementations of any of the above-mentioned image classifications.

[0076] The image classification method in the embodiment of the present application includes transforming the original image to be classified to obtain at least one transformed image; using the first feature extraction network in the image classification model to extract features from the original image to obtain original image features; using the second feature extraction network in the image classification model to extract features from at least one transformed image to obtain transformed image features corresponding to at least one transformed image; the first feature extraction network and the second feature extraction network respectively use different image feature extraction methods to extract image features; using the fusion classification module in the image classification model to obtain the image category of the original image according to the original image features and at least one transformed image feature. In this way, the image classification is performed by combining the original image features and the transformed image features, which improves the accuracy of image classification while taking into account the cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0078] Figure 1 It is a flow chart of an image classification method in an embodiment of the present application.

[0079] Figure 2 It is a structural diagram of an initial classification module in an embodiment of the present application.

[0080] Figure 3 It is an architecture diagram of an image review system in an embodiment of the present application.

[0081] Figure 4 It is a structural block diagram of an image classification device in an embodiment of the present application.

[0082] Figure 5 It is a structural schematic diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0083] The technical solution of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described implementation methods are part of the implementation methods of the present application, rather than all of the implementation methods. Based on the implementation methods in the present application, all other implementation methods obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In addition, the technical features involved in the different implementation methods of the present application described below can be combined with each other as long as they do not conflict with each other.

[0084] In the image review scenario, the platform usually determines whether the images uploaded by the user contain negative information, such as violent and terrorist images, by classification. Violent and terrorist images are images that contain negative information such as weapons (such as guns, bombs, and knives), and violent behaviors (such as beatings, stabbings, and explosions). For example, the homework grading system usually classifies the images to be published to determine whether the images to be published are violent and terrorist images. Whether a violent and terrorist image is misjudged as a non-violent and terrorist image and allowed to be published, or a non-violent and terrorist image is misjudged as a violent and terrorist image and prohibited from being published, it is a very bad experience for users.

[0085] In traditional methods, the misclassification rate of image classification is relatively high, and the accuracy of image classification is usually improved by classification plus detection or segmentation. However, the computational complexity of this method is usually very large and the cost is high.

[0086] Based on the defects of the above-mentioned related technologies, the embodiments of the present application provide a method, device, electronic device, storage medium and program product for image classification, aiming to improve the accuracy of image classification while taking into account the cost.

[0087] An image classification method is provided in an embodiment of the present application. The method can be applied to electronic devices. The present application does not limit the type of electronic device. It can be any type of device suitable for implementation, such as a terminal device and a server, etc. The present application will not go into details about this.

[0088] See also Figure 1 As shown, it is a flowchart of a method for image classification in an embodiment of the present application. Figure 1The method is described below. The specific implementation process of the method is as follows:

[0089] Step 101: transforming an original image to be classified to obtain at least one transformed image.

[0090] The transformed image may include at least one of the following: a grayscale image of the original image; a green channel image of the original image; a red channel image of the original image; or a blue channel image of the original image.

[0091] In one implementation, the original image is transformed in different ways to obtain a grayscale image, a green channel image, a red channel image, and a blue channel image of the original image.

[0092] Optionally, the transformed image can be an image derived from the original image, or a transformation matrix image calculated by a transformation function on the original image. The transformation function can be an ordinary transformation function or a simulated transformation function trained by a neural network, and there is no limitation here.

[0093] Furthermore, the implementation process of step 101 may also include preprocessing the original image to be classified and each transformed image respectively.

[0094] In the embodiments of the present application, the preprocessing may include at least one of the following: scaling to a set size (ie, a fixed size), and image data standardization. In practical applications, any other method may be used to preprocess the image, which is not limited here.

[0095] In this way, multiple transformed images of the original image can be obtained through grayscale transformation and separate transformation of the three primary color channels.

[0096] Step 102: Using the first feature extraction network in the image classification model, extract features from the original image to obtain original image features.

[0097] The image classification model includes an initial classification module and a classification fusion network. The initial classification module includes a first feature extraction network, a second feature extraction network, and an initial classification network. The initial classification network and the classification fusion network constitute a fusion classification module. There can be one or more initial classification modules.

[0098] Combine the following Figure 2 The initial classification module is described. Figure 2 This is a structural diagram of an initial classification module. Figure 2The initial classification module includes a first feature extraction network, a second feature extraction network, and an initial classification network. The initial classification module includes an upper and lower part. The upper branch is the first feature extraction network for inputting the original image, and the lower branch is the second feature extraction network for inputting a certain transformed image. The first feature extraction network and the second feature extraction network are connected to the initial classification network to fuse the image features extracted by the two networks and perform initial classification through the initial classification network.

[0099] The first feature extraction network and the second feature extraction network can respectively use different image feature extraction methods to extract image features. For example, the first feature extraction network can be built based on the moblienetv3 samll model, and the second feature extraction network can be built based on the EfficientVit model.

[0100] Step 103: using the second feature extraction network in the image classification model, respectively extracting features from at least one transformed image to obtain transformed image features corresponding to each of the at least one transformed image.

[0101] Among them, the first feature extraction network and the second feature extraction network respectively use different image feature extraction methods to extract image features.

[0102] In this way, based on the first feature extraction network and the second feature extraction network, feature extraction can be performed on the original image and its corresponding transformed image respectively to obtain multiple feature data of the original image.

[0103] When executing step 103, feature extraction may be performed for each transformed image using any of the following methods:

[0104] Method 1: Input the transformed image into the second feature extraction network to obtain the transformed image features.

[0105] Method 1: Copy the transformed image three times to obtain multiple identical transformed images, and input each transformed image into the second feature extraction network to obtain the transformed image features of the transformed image.

[0106] For example, a grayscale image is copied three times to obtain four identical grayscale images, and the four grayscale images are input into a second feature extraction network to obtain transformed image features of the grayscale images.

[0107] This is because after copying the transformed image three times, the number of channels can be kept consistent with the original image to facilitate subsequent feature fusion and classification.

[0108] Step 104: using a fusion classification module in the image classification model to obtain an image category of the original image according to the original image feature and at least one transformed image feature.

[0109] In one embodiment, for each transformed image feature, the following steps are performed: an initial category is determined based on the original image feature and the transformed image feature through a fusion classification module. If it is determined that the initial category corresponding to at least one transformed image feature meets the set category condition, the image category is determined to be the target category, otherwise, the image category is determined not to be the target category.

[0110] Optionally, the set category condition may include at least one of the following items: the initial categories corresponding to at least one transformed image feature are all target categories; the number of target categories included in the initial category corresponding to at least one transformed image feature is not less than a set number; the number of target categories included in the initial category corresponding to at least one transformed image feature is not less than a set proportion. For example, the target category may be a violent terrorist category, the set number may be 1, and the set proportion may be 3 / 4.

[0111] As an example, the transformed image includes the grayscale image, green channel image, red channel image, and blue channel image of the original image. If the image classification model only includes one initial classification module, the original image and each transformed image can be combined and input into the same initial classification module in sequence to obtain multiple initial classification results. Figure 2 The first feature extraction network in , while the grayscale image is input Figure 2 The second feature extraction network in obtains the first initial category. Input the original image into Figure 2 The first feature extraction network in , while the green channel map is input into Figure 2 The second feature extraction network in obtains the second initial category. Input the original image into Figure 2 The first feature extraction network in , while inputting the red channel map into Figure 2 The second feature extraction network in obtains the third initial category. Input the original image into Figure 2 The first feature extraction network in , while inputting the blue channel map into Figure 2 The second feature extraction network in is used to obtain the fourth initial category. If the first initial category, the second initial category, the third initial category and the fourth initial category are all violent and terrorist categories, the image category is determined to be a violent and terrorist category. Otherwise, the image category is determined to be a non-violent and terrorist category, that is, a normal image.

[0112] In the embodiment of the present application, the image classification model draws on the idea of ​​the loop-constrained network and ensures the accuracy of image classification through a multiple confirmation mechanism. The image classification model is different from the loop-constrained network in that the input data sources of the two are different. The input data of the image classification model is the original image and its corresponding transformed image, while the input data of the loop-constrained network is the original image and the limiting factor. In addition, the image classification model and the loop-constrained network output different image categories. The image classification model generally outputs two categories of categories, and confirms the target category, that is, the target category or the non-target category. The loop-constrained network outputs multiple categories, and the output value is dynamically changed each time it is confirmed, and its output value is determined by the limiting factor.

[0113] In this way, the idea of ​​the loop-restricted network is used to construct an image classification model, and multiple features are extracted based on the image classification model, so that the classification result has a higher accuracy. In addition, the first feature extraction network and the second feature extraction network can use heterogeneous networks to improve the feature extraction ability of the network, so that more abundant image features can be extracted, further improving the classification ability of the image classification model.

[0114] Furthermore, it is also possible to determine whether the image has been approved based on the image category.

[0115] In one implementation, if the image category is not the target category, it is determined that the image review has passed; otherwise, it is determined that the image review has failed.

[0116] Furthermore, in order to reduce misjudgment of images, further discrimination can be performed in combination with image retrieval results.

[0117] In one implementation, if the image category is not the target category, it is determined that the image has passed the review; if it is determined that the image category is the target category, it is determined whether there is a target category image matching the original image in the target category image set, and if so, it is determined that the image has failed the review, otherwise, it is determined that the image has passed the review; the target category image set contains multiple target category images.

[0118] Among them, when judging whether there is a target category image matching the original image in the target category image set, any of the following methods can be used:

[0119] Method 1: The target category image set also includes target category image features corresponding to each target category image; if it is determined that there are target category image features matching the original image features in the target category image set, then it is determined that there are target category images matching the original image; otherwise, it is determined that there are no target category images matching the original image.

[0120] Optionally, whether the image features match can be determined by the similarity between the features.

[0121] In one implementation, the similarities between the original image features and each target category image feature are determined, and the maximum similarity among the similarities is determined. If the maximum similarity is higher than a set similarity threshold, it is determined that there are matching target category image features, that is, there are matching target category images. Otherwise, it is determined that there are no target category images that match the original image.

[0122] In this way, the original image features extracted by the first feature extraction network can be directly used for retrieval.

[0123] Method 2: The target category image set also includes target category image features corresponding to each target category image; a third feature extraction network is used to extract features of the original image to obtain original image retrieval features; if it is determined that there are target category image features in the target category image set that match the original image retrieval features, then it is determined that there is a target category image that matches the original image; otherwise, it is determined that there is no target category image that matches the original image.

[0124] In one implementation, after preprocessing the original image, the original image is input into a third feature extraction network to perform forward reasoning and output the original image retrieval features of a fixed dimension.

[0125] The third feature extraction network may be the same as or different from the first feature extraction network, that is, the original image retrieval features may be the same as or different from the original image features.

[0126] For example, the feature extraction network is the same as the first feature extraction network, and the first feature extraction network is built based on the mobilenetv3 samll model. Then the 576-dimensional feature vector output by the last convolutional layer of the mobilenetv3 samll model is the original image retrieval feature.

[0127] In this way, the same feature extraction network can be used for image classification and image retrieval, so that the model parameters can be shared and the amount of calculation can be reduced. The extraction of original image retrieval features and image classification can be calculated simultaneously, which greatly reduces the amount of calculation and saves computer resources.

[0128] Furthermore, the following steps can be used to train and obtain an image classification model:

[0129] Acquire training data; the training data includes a plurality of image samples and their corresponding category labels; transform each image sample respectively to obtain at least one transformed image corresponding to each image sample;

[0130] For each image sample and each corresponding transformed image, the following steps are performed respectively until the set training conditions are met: using the initial image classification model, based on the image sample and its corresponding transformed image, the corresponding initial category is determined;

[0131] A classification loss is determined according to a category label corresponding to an image sample and an initial category corresponding to at least one transformed image feature; and a model parameter is adjusted according to the classification loss.

[0132] The training condition may be set such that the classification loss is lower than a loss threshold. In practical applications, the training condition and the loss threshold may be set according to the actual application scenario and are not limited here.

[0133] In one implementation, the classification loss between the class label and each initial class can be determined, and the model parameters can be adjusted according to each classification loss, for example, the parameters can be adjusted according to the weighted sum of the classification losses. The following steps can also be performed in a loop: for each transformed image, the classification loss between the class label and the corresponding initial class can be determined, and the parameters can be adjusted according to the classification loss.

[0134] In this way, a set of annotated data (i.e., training data) can be shared, thereby reducing the data annotation and maintenance costs during model training. Furthermore, the image classification model can be used to guide the third feature extraction network, so that the third feature extraction network can extract richer and more accurate features without the need to train the third feature extraction network separately. In addition, a multiple confirmation mechanism can be adopted to simultaneously optimize the classification effect and the feature vector extraction effect through multiple rounds of confirmation calculations when training on the same image.

[0135] When the traditional image classification model is trained together with the feature extraction network (i.e., training the image classification model including the feature extraction network), each training can only provide one guidance to the feature extraction network (i.e., for one training image, only one classification result can be output for model parameter optimization during each training). However, for each training image, the image classification model can provide multiple rounds of guidance for the feature extraction network through multiple rounds of confirmation mechanisms (i.e., whether multiple initial categories are classified correctly) during each training. This training method makes the trained feature extraction network have better feature extraction effect than the traditional feature extraction network.

[0136] Combine the following Figure 3 , the architecture of the image review system is explained. Figure 3 The following is an architectural diagram of an image review system. Figure 3 It includes input module, image classification module, image retrieval module, result fusion module and output module.

[0137] Among them, the input module is used to input the original image, the image classification module is used to determine the image category of the original image, and the image retrieval module is used to obtain the retrieval result of whether there is a target category image matching the original image. The result fusion module is used to obtain the audit result of whether the audit is passed based on the image category and the retrieval result. The output module is used to output the audit result. Among them, the specific implementation process of each module in the image audit system can be referred to the above steps 101-step 104, which will not be repeated here.

[0138] In the embodiment of the present application, the idea of ​​the circular limited network is used to construct an image classification model, and based on the image classification model, multiple feature extraction is performed to improve the accuracy of the image classification result. In addition, the first feature extraction network and the second feature extraction network adopt heterogeneous networks, which improve the feature extraction ability of the network, can extract richer image features, and further enhance the classification ability of the image classification model. And, the image retrieval is integrated for audit judgment, which further improves the accuracy of the audit, reduces the misjudgment rate, and does not require a large amount of computing resources and other costs, and does not affect the system performance. And, in image classification and image retrieval, the same feature extraction network can be used so that model parameters can be shared and the amount of calculation can be reduced. The extraction of the original image retrieval feature and the image classification can be calculated simultaneously, thereby greatly reducing the amount of calculation and saving computer resources. It is also possible to share a copy of the annotation data (i.e., training data), so that the annotation cost and maintenance cost of the data can be reduced during model training. Furthermore, the image classification model can be used to guide the third feature extraction network so that the third feature extraction network can extract richer and more accurate features without the need for additional training of the third feature extraction network. In addition, a multiple confirmation mechanism can be adopted to simultaneously optimize the classification effect and the feature vector extraction effect through multiple rounds of confirmation calculations when training on the same image.

[0139] Based on the same inventive concept, an image classification device is also provided in the embodiment of the present application. Since the principle of solving the problem by the above-mentioned device and equipment is similar to that of an image classification method, the implementation of the above-mentioned device can refer to the implementation of the method, and the repeated parts will not be repeated. The device can be applied to electronic devices. The present application does not limit the type of electronic devices. It can be any type of device suitable for implementation, such as terminal devices and servers, etc., and the present application will not repeat them.

[0140] See also Figure 4 FIG. 1 is a block diagram of a device for image classification in an embodiment of the present application. In some embodiments, the device for image classification in the present application example includes:

[0141] A transformation unit 401, configured to transform an original image to be classified to obtain at least one transformed image;

[0142] The first extraction unit 402 is used to extract features from the original image using the first feature extraction network in the image classification model to obtain features of the original image;

[0143] The second extraction unit 403 is used to use the second feature extraction network in the image classification model to perform feature extraction on at least one transformed image respectively to obtain transformed image features corresponding to each of the at least one transformed image; the first feature extraction network and the second feature extraction network respectively use different image feature extraction methods to extract image features;

[0144] The classification unit 404 is used to obtain the image category of the original image according to the original image feature and at least one transformed image feature by using the fusion classification module in the image classification model.

[0145] In one implementation, the classification unit 404 is used to:

[0146] For each transformed image feature, the following steps are performed: determining an initial category according to the original image features and the transformed image features through a fusion classification module;

[0147] If it is determined that the initial category corresponding to at least one transformed image feature meets the set category condition, the image category is determined to be the target category; otherwise, it is determined that the image category is not the target category.

[0148] In one implementation, setting the category condition includes any one of the following:

[0149] At least one of the transformed image features has initial categories corresponding to each of the target categories;

[0150] The number of target categories included in the initial category corresponding to at least one transformed image feature is not less than a set number;

[0151] The proportion of the number of target categories included in the initial categories corresponding to at least one transformed image feature is not less than a set proportion.

[0152] In one embodiment, the transformed image includes at least one of the following:

[0153] Grayscale image of the original image; Green channel image of the original image; Red channel image of the original image; Blue channel image of the original image.

[0154] In one implementation, the classification unit 404 is further configured to:

[0155] If the image category is not the target category, it is determined that the image review has passed; otherwise, it is determined that the image review has failed.

[0156] In one implementation, the classification unit 404 is further configured to:

[0157] If the image category is not the target category, the image is deemed to have passed the review;

[0158] If it is determined that the image category is a target category, it is determined whether there is a target category image matching the original image in the target category image set. If so, it is determined that the image review has failed. Otherwise, it is determined that the image review has passed. The target category image set contains multiple target category images.

[0159] In one implementation, the target category image set further includes target category image features corresponding to each target category image;

[0160] The classification unit 404 is further configured to: if it is determined that there are target category image features matching the original image features in the target category image set, then determine that there are target category images matching the original image; otherwise, determine that there are no target category images matching the original image.

[0161] In one implementation, the target category image set further includes target category image features corresponding to each target category image;

[0162] The classification unit 404 is also used to:

[0163] The third feature extraction network is used to extract features from the original image to obtain the retrieval features of the original image;

[0164] If it is determined that there are target category image features matching the original image retrieval features in the target category image set, then it is determined that there are target category images matching the original image; otherwise, it is determined that there are no target category images matching the original image.

[0165] In one implementation, the classification unit 404 is further configured to:

[0166] The following steps are used to train the image classification model:

[0167] Obtain training data; the training data includes multiple image samples and their corresponding category labels;

[0168] Transform each image sample respectively to obtain at least one transformed image corresponding to each image sample;

[0169] For each image sample and each corresponding transformed image, the following steps are performed respectively until the set training conditions are met: using the initial image classification model, based on the image sample and its corresponding transformed image, the corresponding initial category is determined;

[0170] Determining a classification loss based on a category label corresponding to the image sample and an initial category corresponding to each of the at least one transformed image features;

[0171] According to the classification loss, the model parameters are adjusted.

[0172] The image classification method in the embodiment of the present application includes transforming the original image to be classified to obtain at least one transformed image; using the first feature extraction network in the image classification model to extract features from the original image to obtain original image features; using the second feature extraction network in the image classification model to extract features from at least one transformed image to obtain transformed image features corresponding to at least one transformed image; the first feature extraction network and the second feature extraction network respectively use different image feature extraction methods to extract image features; using the fusion classification module in the image classification model to obtain the image category of the original image according to the original image features and at least one transformed image feature. In this way, the image classification is performed by combining the original image features and the transformed image features, which improves the accuracy of image classification while taking into account the cost.

[0173] In an embodiment of the present application, an electronic device is provided, including:

[0174] Processor; and

[0175] The memory stores computer instructions, where the computer instructions are used to enable the processor to execute the method of any of the above embodiments.

[0176] In an embodiment of the present application, a computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a computer to execute a method in any of the above-mentioned embodiments.

[0177] An embodiment of the present application also provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device implements any of the above-mentioned implementation methods.

[0178] Figure 5 FIG. 5 shows a schematic diagram of the structure of an electronic device 5000. Figure 5 As shown, the electronic device 5000 includes: a processor 5010 and a memory 5020 , and optionally, may also include a power supply 5030 , a display unit 5040 , and an input unit 5050 .

[0179] The processor 5010 is the control center of the electronic device 5000. It connects various components using various interfaces and lines, and performs various functions of the electronic device 5000 by running or executing software programs and / or data stored in the memory 5020, thereby monitoring the electronic device 5000 as a whole.

[0180] In the embodiment of the present application, the processor 5010 executes the various steps in the above embodiment when calling the computer program stored in the memory 5020.

[0181] Optionally, the processor 5010 may include one or more processing units; preferably, the processor 5010 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and applications, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 5010. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips.

[0182] The memory 5020 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, various applications, etc.; the data storage area may store data created according to the use of the electronic device 5000, etc. In addition, the memory 5020 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices, etc.

[0183] The electronic device 5000 also includes a power source 5030 (such as a battery) for supplying power to various components. The power source can be logically connected to the processor 5010 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0184] The display unit 5040 may be used to display information input by the user or information provided to the user and various menus of the electronic device 5000, and in the embodiment of the present application, is mainly used to display the display interface of each application in the electronic device 5000 and objects such as text and images displayed in the display interface. The display unit 5040 may include a display panel 5041. The display panel 5041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0185] The input unit 5050 may be used to receive information such as numbers or characters input by the user. The input unit 5050 may include a touch panel 5051 and other input devices 5052. The touch panel 5051, also known as a touch screen, may collect touch operations of the user on or near it (such as operations of the user using a finger, a stylus, or any other suitable object or accessory on or near the touch panel 5051).

[0186] Specifically, the touch panel 5051 can detect the user's touch operation, detect the signal brought by the touch operation, convert these signals into touch point coordinates, send them to the processor 5010, and receive and execute the command sent by the processor 5010. In addition, the touch panel 5051 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. Other input devices 5052 can include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, power on / off keys, etc.), a trackball, a mouse, a joystick, etc.

[0187] Of course, the touch panel 5051 can cover the display panel 5041. When the touch panel 5051 detects a touch operation on or near it, it is transmitted to the processor 5010 to determine the type of touch event. Then the processor 5010 provides corresponding visual output on the display panel 5041 according to the type of touch event. Figure 5 In the embodiment, the touch panel 5051 and the display panel 5041 are used as two independent components to realize the input and output functions of the electronic device 5000, but in some embodiments, the touch panel 5051 and the display panel 5041 can be integrated to realize the input and output functions of the electronic device 5000.

[0188] The electronic device 5000 may also include one or more sensors, such as a pressure sensor, a gravity acceleration sensor, a proximity light sensor, etc. Of course, according to the needs of specific applications, the electronic device 5000 may also include other components such as a camera. Since these components are not the key components used in the embodiments of the present application, Figure 5 It is not shown and will not be described in detail.

[0189] Those skilled in the art will understand that Figure 5 The electronic device is merely an example and does not limit the electronic device, and may include more or less components than those shown in the figure, or may combine certain components, or may include different components.

[0190] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

Claims

1. A method for image classification, characterized in that: The method comprises: Transforming the original image to be classified to obtain at least one transformed image; Using the first feature extraction network in the image classification model to extract features from the original image to obtain original image features; Using the second feature extraction network in the image classification model, respectively extract features from the at least one transformed image to obtain transformed image features corresponding to each of the at least one transformed image; the first feature extraction network and the second feature extraction network respectively use different image feature extraction methods to extract image features; The image category of the original image is obtained according to the original image feature and at least one transformed image feature by using the fusion classification module in the image classification model.

2. The method according to claim 1, characterized in that The adopting the fusion classification module in the image classification model to obtain the image category of the original image according to the original image feature and at least one transformed image feature includes: For each transformed image feature, the following steps are performed: determining an initial category according to the original image feature and the transformed image feature through the fusion classification module; If it is determined that the initial category corresponding to each of the at least one transformed image feature meets the set category condition, then the image category is determined to be the target category; otherwise, it is determined that the image category is not the target category.

3. The method according to claim 2, characterized in that The setting category condition includes any one of the following: The initial categories corresponding to each of the at least one transformed image feature are the target categories; The number of the target categories included in the initial categories corresponding to each of the at least one transformed image features is not less than a set number; The proportion of the target categories contained in the initial categories corresponding to each of the at least one transformed image features is not less than a set proportion.

4. The method according to any one of claims 1 to 3, characterized in that: The at least one transformed image comprises at least one of the following: The grayscale image of the original image; the green channel image of the original image; the red channel image of the original image; and the blue channel image of the original image.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: If the image category is not the target category, it is determined that the image review has passed; otherwise, it is determined that the image review has failed.

6. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: If the image category is not the target category, then the image is determined to have passed the review; If it is determined that the image category is a target category, it is determined whether there is a target category image matching the original image in the target category image set. If so, it is determined that the image review has failed. Otherwise, it is determined that the image review has passed. The target category image set contains multiple target category images.

7. The method according to claim 6, characterized in that The target category image set also includes target category image features corresponding to each target category image; The step of determining whether there is a target category image matching the original image in the target category image set comprises: If it is determined that there are target category image features matching the original image features in the target category image set, then it is determined that there are target category images matching the original image; otherwise, it is determined that there are no target category images matching the original image.

8. The method according to claim 6, characterized in that The target category image set also includes target category image features corresponding to each target category image; The step of determining whether there is a target category image matching the original image in the target category image set comprises: Using a third feature extraction network to extract features from the original image to obtain original image retrieval features; If it is determined that the target category image features matching the original image retrieval features exist in the target category image set, it is determined that there are target category images matching the original image; otherwise, it is determined that there are no target category images matching the original image.

9. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: The image classification model is trained by adopting the following steps: Acquire training data; the training data includes a plurality of image samples and their corresponding category labels; Transform each image sample respectively to obtain at least one transformed image corresponding to each image sample; For each image sample and each corresponding transformed image, the following steps are performed respectively until the set training conditions are met: using an initial image classification model, based on the image sample and the corresponding transformed image, a corresponding initial category is determined; Determining a classification loss according to a category label corresponding to the image sample and an initial category corresponding to each of at least one transformed image features; According to the classification loss, model parameters are adjusted.

10. An image classification device, characterized in that: The device comprises: A transformation unit, used for transforming the original image to be classified to obtain at least one transformed image; A first extraction unit is used to extract features from the original image using a first feature extraction network in the image classification model to obtain features of the original image; A second extraction unit is used to use a second feature extraction network in the image classification model to perform feature extraction on the at least one transformed image respectively to obtain transformed image features corresponding to each of the at least one transformed image; the first feature extraction network and the second feature extraction network respectively use different image feature extraction methods to extract image features; The classification unit is used to obtain the image category of the original image according to the original image feature and at least one transformed image feature by using the fusion classification module in the image classification model.

11. An electronic device, characterized in that: include: processor; as well as A memory storing computer instructions, wherein the computer instructions are used to enable the processor to execute the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: Computer instructions are stored, and the computer instructions are used to make a computer execute the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The invention comprises a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code is executed in a processor of an electronic device, the processor in the electronic device performs the method according to any one of claims 1 to 9.

Images