Image classification method, device, electronic device and storage medium

Abstract

The present disclosure relates to an image classification method, apparatus, electronic device, storage medium, and computer program product. The method comprises: obtaining an image to be processed; inputting the image to be processed into a trained image classification model to obtain image category information for the image to be processed; the trained image classification model is obtained by training the image classification model using at least two sample images of different image styles and a target classification result for each sample image, wherein the target classification result for each sample image is obtained by processing the model according to a different image classification model; and the image category information is determined as the image classification result for the image to be processed. This method can improve the model performance of the trained image classification model, thereby improving the accuracy of the image classification model when processing image classification.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to an image classification method, apparatus, electronic device, storage medium, and computer program product. Background Art

[0002] With the widespread application of deep learning in the field of computer vision, network models are often used to handle image classification tasks in scenarios such as medical imaging, security monitoring, and unmanned driving. For example, in unmanned driving scenarios, people and vehicles in driving images are predicted and classified.

[0003] Traditionally, mutual learning methods are used to improve the learning capabilities of two models by using the output of one model as supervision for another. However, this mutual learning method simply allows the two models to exchange their predictions and classification results, without fully exploiting the effective information in the data. This results in low accuracy of the trained models when handling image classification tasks. Summary of the Invention

[0004] The present disclosure provides an image classification method, apparatus, electronic device, storage medium, and computer program product to at least address the problem of low accuracy of models in related technologies when processing image classification tasks. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image classification method, comprising:

[0006] Get the image to be processed;

[0007] Inputting the image to be processed into a trained image classification model to obtain image category information of the image to be processed; the trained image classification model is obtained by training the image classification model using at least two sample images of different image styles and a target classification result for each sample image, wherein the target classification result for each sample image is obtained by processing according to a different image classification model;

[0008] The image category information is determined as the image classification result of the image to be processed.

[0009] In an exemplary embodiment, the trained image classification model is trained in the following manner:

[0010] Acquire sample images; the sample images include at least a first sample image and a second sample image of different image styles;

[0011] Obtaining a first target classification result of the first image classification model for the first sample image, and a second target classification result of the second image classification model for the second sample image;

[0012] inputting the first sample image and the second sample image into the image classification model to be trained respectively, to obtain a first predicted classification result corresponding to the first sample image and a second predicted classification result corresponding to the second sample image;

[0013] According to the difference between the first predicted classification result and the first target classification result, and the difference between the second predicted classification result and the second target classification result, the image classification model to be trained is trained to obtain the trained image classification model.

[0014] In an exemplary embodiment, training the image classification model to be trained based on the difference between the first predicted classification result and the first target classification result, and the difference between the second predicted classification result and the second target classification result to obtain the trained image classification model includes:

[0015] Obtaining a first loss value based on a difference between the first predicted classification result and the first target classification result, and obtaining a second loss value based on a difference between the second predicted classification result and the second target classification result;

[0016] Obtaining a target loss value according to the first loss value and the second loss value;

[0017] Training the image classification model to be trained according to the target loss value to obtain a trained image classification model;

[0018] The trained image classification model is trained again until the trained image classification model is obtained.

[0019] In an exemplary embodiment, obtaining a target loss value according to the first loss value and the second loss value includes:

[0020] Obtaining a first weight corresponding to the first loss value and a second weight corresponding to the second loss value;

[0021] The first loss value and the second loss value are weightedly summed according to the first weight and the second weight respectively to obtain the target loss value.

[0022] In an exemplary embodiment, retraining the trained image classification model includes:

[0023] Obtaining an updated first target classification result of the trained first image classification model for the first sample image, and an updated second target classification result of the trained second image classification model for the second sample image;

[0024] inputting the first sample image and the second sample image into the trained image classification model respectively to obtain an updated first prediction classification result corresponding to the first sample image and an updated second prediction classification result corresponding to the second sample image;

[0025] The trained image classification model is trained based on the difference between the updated first predicted classification result and the updated first target classification result, and the difference between the updated second predicted classification result and the updated second target classification result.

[0026] In an exemplary embodiment, the trained first image classification model is obtained by training in the following manner:

[0027] performing image style conversion on the second sample image according to the first sample image to obtain a processed second sample image;

[0028] Inputting the processed second sample image into the first image classification model to obtain a predicted classification result of the processed second sample image;

[0029] The first image classification model is trained according to the difference between the predicted classification result of the processed second sample image and the second target classification result of the second sample image to obtain the trained first image classification model.

[0030] In an exemplary embodiment, the trained second image classification model is obtained by training in the following manner:

[0031] performing image style conversion on the first sample image according to the second sample image to obtain a processed first sample image;

[0032] Inputting the processed first sample image into the second image classification model to obtain a predicted classification result of the processed first sample image;

[0033] The second image classification model is trained according to the difference between the predicted classification result of the processed first sample image and the first target classification result of the first sample image to obtain the trained second image classification model.

[0034] In an exemplary embodiment, the sample image further includes at least one third sample image, and the image style of the third sample image is different from that of the first sample image and the second sample image;

[0035] Before training the image classification model to be trained based on the difference between the first predicted classification result and the first target classification result, and the difference between the second predicted classification result and the second target classification result, to obtain the trained image classification model, the method further includes: obtaining a third target classification result of a third image classification model for the third sample image; inputting the third sample image into the image classification model to be trained, and obtaining a third predicted classification result corresponding to the third sample image;

[0036] According to the difference between the first predicted classification result and the first target classification result, and the difference between the second predicted classification result and the second target classification result, the image classification model to be trained is trained to obtain the trained image classification model, including: according to the difference between the first predicted classification result and the first target classification result, the difference between the second predicted classification result and the second target classification result, and the difference between the third predicted classification result and the third target classification result, the image classification model to be trained is trained to obtain the trained image classification model.

[0037] According to a second aspect of an embodiment of the present disclosure, there is provided an image classification model training method, comprising:

[0038] Acquire sample images; the sample images include at least a first sample image and a second sample image of different image styles;

[0039] Obtaining a first target classification result of the first image classification model for the first sample image, and a second target classification result of the second image classification model for the second sample image;

[0040] inputting the first sample image and the second sample image into an image classification model to be trained respectively, to obtain a first predicted classification result corresponding to the first sample image and a second predicted classification result corresponding to the second sample image;

[0041] Obtaining a first loss value based on a difference between the first predicted classification result and the first target classification result, and obtaining a second loss value based on a difference between the second predicted classification result and the second target classification result;

[0042] The image classification model to be trained is trained according to the first loss value and the second loss value to obtain a trained image classification model.

[0043] According to a third aspect of an embodiment of the present disclosure, there is provided an image classification apparatus, comprising:

[0044] A processing image acquisition unit, configured to acquire an image to be processed;

[0045] an image category acquisition unit configured to input the image to be processed into a trained image classification model to obtain image category information of the image to be processed; the trained image classification model is obtained by training the image classification model using at least two sample images of different image styles and a target classification result for each sample image, wherein the target classification result for each sample image is obtained by processing the image classification model according to a different image classification model;

[0046] The classification result determining unit is configured to determine the image category information as the image classification result of the image to be processed.

[0047] In an exemplary embodiment, the image classification device further includes: a classification model training unit, configured to execute acquisition of sample images; the sample images include at least a first sample image and a second sample image of different image styles; acquire a first target classification result of the first image classification model for the first sample image, and a second target classification result of the second image classification model for the second sample image; input the first sample image and the second sample image into the image classification model to be trained respectively, and obtain a first predicted classification result corresponding to the first sample image and a second predicted classification result corresponding to the second sample image; train the image classification model to be trained according to the difference between the first predicted classification result and the first target classification result, and the difference between the second predicted classification result and the second target classification result, to obtain the trained image classification model.

[0048] In an exemplary embodiment, the image classification device further includes: a classification model acquisition unit, configured to obtain a first loss value based on the difference between the first predicted classification result and the first target classification result, and obtain a second loss value based on the difference between the second predicted classification result and the second target classification result; obtain a target loss value based on the first loss value and the second loss value; train the image classification model to be trained based on the target loss value to obtain a trained image classification model; and train the trained image classification model again until the trained image classification model is obtained.

[0049] In an exemplary embodiment, the image classification device further includes: a target loss acquisition unit, configured to execute acquisition of a first weight corresponding to the first loss value and a second weight corresponding to the second loss value; and perform weighted summation of the first loss value and the second loss value according to the first weight and the second weight, respectively, to obtain the target loss value.

[0050] In an exemplary embodiment, the image classification device further includes: a model retraining unit, configured to execute acquisition of an updated first target classification result of the trained first image classification model for the first sample image, and an updated second target classification result of the trained second image classification model for the second sample image; inputting the first sample image and the second sample image into the trained image classification model respectively to obtain an updated first predicted classification result corresponding to the first sample image and an updated second predicted classification result corresponding to the second sample image; and training the trained image classification model according to the difference between the updated first predicted classification result and the updated first target classification result, and the difference between the updated second predicted classification result and the updated second target classification result.

[0051] In an exemplary embodiment, the image classification device also includes: a first model training unit, configured to perform image style conversion on the second sample image based on the first sample image to obtain a processed second sample image; input the processed second sample image into the first image classification model to obtain a predicted classification result of the processed second sample image; and train the first image classification model based on the difference between the predicted classification result of the processed second sample image and the second target classification result of the second sample image to obtain the trained first image classification model.

[0052] In an exemplary embodiment, the image classification device further includes: a second model training unit, configured to perform image style conversion on the first sample image based on the second sample image to obtain a processed first sample image; input the processed first sample image into the second image classification model to obtain a predicted classification result of the processed first sample image; and train the second image classification model based on the difference between the predicted classification result of the processed first sample image and the first target classification result of the first sample image to obtain the trained second image classification model.

[0053] In an exemplary embodiment, the sample image also includes at least one third sample image, and the image style of the third sample image is different from that of the first sample image and the second sample image; the image classification device also includes: a classification result acquisition unit, configured to execute acquisition of a third target classification result of a third image classification model for the third sample image; inputting the third sample image into the image classification model to be trained to obtain a third predicted classification result corresponding to the third sample image; the classification model training unit is further configured to execute training of the image classification model to be trained based on the difference between the first predicted classification result and the first target classification result, the difference between the second predicted classification result and the second target classification result, and the difference between the third predicted classification result and the third target classification result to obtain the trained image classification model.

[0054] According to a fourth aspect of an embodiment of the present disclosure, there is provided an image classification model training apparatus, comprising:

[0055] A sample image acquisition unit is configured to acquire sample images; the sample images include at least a first sample image and a second sample image of different image styles;

[0056] a target result acquisition unit configured to acquire a first target classification result of the first image classification model for the first sample image and a second target classification result of the second image classification model for the second sample image;

[0057] a prediction result acquisition unit configured to input the first sample image and the second sample image into an image classification model to be trained, respectively, to obtain a first prediction classification result corresponding to the first sample image and a second prediction classification result corresponding to the second sample image;

[0058] a loss function determining unit configured to obtain a first loss value based on a difference between the first predicted classification result and the first target classification result, and to obtain a second loss value based on a difference between the second predicted classification result and the second target classification result;

[0059] The target model acquisition unit is configured to train the image classification model to be trained based on the first loss value and the second loss value to obtain a trained image classification model.

[0060] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the image classification method as described in any one of the embodiments of the first aspect, or the image classification model training method as described in any one of the embodiments of the second aspect.

[0061] According to the sixth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the image classification method as described in any one of the embodiments in the first aspect, or the image classification model training method as described in any one of the embodiments in the second aspect.

[0062] According to the seventh aspect of the embodiments of the present disclosure, a computer program product is provided, which includes instructions. When the instructions are executed by a processor of an electronic device, the electronic device is capable of performing the image classification method as described in any one of the embodiments of the first aspect, or the image classification model training method as described in any one of the embodiments of the second aspect.

[0063] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0064] Obtain an image to be processed; input the image to be processed into a trained image classification model to obtain image category information in the image to be processed; the trained image classification model is obtained by training the image classification model to be trained using at least two sample images of different image styles and a target classification result for each sample image, and the target classification result for each sample image is obtained by processing different image classification models; the image category information is determined as the image classification result of the image to be processed. The present disclosure trains the image classification model to be trained using at least two sample images of different styles, so that the image classification model can mine more feature information in the sample images, and further introduces different image classification models to perform model training on the image classification model, thereby improving the model performance of the trained image classification model, and further improving the accuracy of the image classification model when processing image classification.

[0065] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0067] Figure 1The figure is a flowchart of an image classification method according to an exemplary embodiment.

[0068] Figure 2 The figure is a flowchart of a training image classification model according to an exemplary embodiment.

[0069] Figure 3 The figure is a flowchart of retraining a trained image classification model according to an exemplary embodiment.

[0070] Figure 4 The figure is a flowchart of obtaining a trained first image classification model according to an exemplary embodiment.

[0071] Figure 5 The figure is a flowchart of obtaining a trained second image classification model according to an exemplary embodiment.

[0072] Figure 6 The figure is a flowchart of a method for training an image classification model according to an exemplary embodiment.

[0073] Figure 7 The figure is a flowchart of another image classification model training method according to an exemplary embodiment.

[0074] Figure 8 The figure is a block diagram of an image classification device according to an exemplary embodiment.

[0075] Figure 9 The figure is a block diagram of an image classification model training device according to an exemplary embodiment.

[0076] Figure 10 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0077] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0078] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0079] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0080] Figure 1 FIG. 1 is a flow chart showing an image classification method according to an exemplary embodiment. Figure 1 As shown, the image classification method is used in a terminal and includes the following steps.

[0081] In step S101, an image to be processed is obtained.

[0082] The image to be processed refers to the original image that needs to be classified.

[0083] Specifically, when an image needs to be classified, the user can trigger a processing request for performing the image classification to the terminal, and the terminal can respond to the processing request to obtain the image to be processed that needs to be classified.

[0084] In step S102, the image to be processed is input into the trained image classification model to obtain image category information in the image to be processed; the trained image classification model is obtained by training the image classification model to be trained using at least two sample images with different image styles and the target classification result of each sample image, and the target classification result of each sample image is obtained by processing according to different image classification models.

[0085] Among them, the image classification model refers to a model used to classify images, which can identify and classify information such as people, objects, and background in the image; the image classification model to be trained has the same model structure and the number of parameters in each layer of the model as different image classification models, but the values ​​of the parameters in each layer of the model are different.

[0086] Here, sample images of different image styles refer to sample images with different styles such as image texture, brightness or color, but the same people or objects in the images. For example, two sample images, one with a daytime style and the other with a nighttime style, have the same position and size of the people, objects and background in the two sample images. Due to the different light brightness of the daytime style and the nighttime style, the people, objects and background in the two sample images appear different in color and texture. The daytime style sample images and the nighttime style sample images are referred to as sample images of different image styles. It should be noted that image style refers to the image style, which belongs to the style of texture, color, visual pattern, etc. in a narrow sense, and does not belong to artistic styles or painting styles such as abstractionism, hyperrealism, symbolism, classicism, etc.

[0087] Specifically, the trained image classification model is trained by using at least two sample images with different image styles and target classification results obtained by processing the two sample images with different image styles by different image classification models respectively; after the image classification model is trained, the terminal can further input the image to be processed that needs to be classified into the trained image classification model, and the trained image classification model predicts and classifies the image category of each image information in the image to be processed to obtain the image category information in the image to be processed.

[0088] For example, the image to be processed may be an unmanned driving image, and the image classification model can predict the category of each area in the unmanned driving image, such as whether a certain area belongs to the road category, vehicle category, sidewalk category, tree category, or sky category.

[0089] In step S103, the image category information is determined as the image classification result of the image to be processed.

[0090] Specifically, the terminal outputs the image category information of the trained image classification model as the image classification result of image classification processing for the image to be processed.

[0091] In the above-mentioned image classification method, an image to be processed is obtained; the image to be processed is input into a trained image classification model to obtain image category information in the image to be processed; the trained image classification model is obtained by training the image classification model to be trained using at least two sample images of different image styles and the target classification result of each sample image, and the target classification result of each sample image is obtained by processing it according to different image classification models; the image category information is determined as the image classification result of the image to be processed. The present disclosure trains the image classification model to be trained by using at least two sample images of different styles, so that the image classification model can mine more feature information in the sample images, and further introduces different image classification models to perform model training on the image classification model, thereby improving the model performance of the trained image classification model, and further improving the accuracy of the image classification model when processing image classification.

[0092] In an exemplary embodiment, Figure 2 As shown in the figure, the trained image classification model is trained in the following way:

[0093] In step S201 , sample images are acquired; the sample images include at least a first sample image and a second sample image of different image styles.

[0094] The first sample image may be a single sample image or a sample image set consisting of multiple images, and the same applies to the second sample image. The first sample image and the second sample image have different image styles, which means that the image style of most images in the first sample image is different from the image style of most images in the second sample image.

[0095] For example, all images in the first sample image are of daytime style, and all images in the second sample image are of nighttime style, or the proportion of images in the first sample image are of daytime style is 80%, and the proportion of images in the nighttime style is 20%, while the proportion of images in the second sample image are of daytime style is 30%, and the proportion of images in the nighttime style is 70%, that is, the image style of the first sample image is daytime, and the image style of the second sample image is nighttime.

[0096] Specifically, the terminal may capture an image in the scene as the original sample image, or may use a stored image as the original sample image. The terminal performs sampling processing on the original sample image to obtain a first sample image and a second sample image, and uses the first sample image and the second sample image as the sample image. The number of the first sample image and the second sample image is much smaller than the number of the original sample images.

[0097] In step S202, a first target classification result of the first image classification model for the first sample image and a second target classification result of the second image classification model for the second sample image are obtained.

[0098] The first target classification result refers to the classification result of the first image classification model performing image classification processing on the first sample image. The first target classification result is used to supervise the learning of the first sample image by other image classification models besides the first image classification model. For example, it supervises the learning of the first sample image by the image classification model to be trained and the second image classification model. Of course, it can also supervise the learning of the first sample image by the third image classification model, the fourth image classification model, and other models. The first image classification model refers to the teacher model that transfers the learned information to the image classification model to be trained in order to train the image classification model.

[0099] The second target classification result refers to the classification result of the second image classification model performing image classification processing on the second sample image. The second target classification result is used to supervise the learning of the second sample image by other image classification models besides the second image classification model. For example, it can supervise the learning of the second sample image by the image classification model to be trained and the first image classification model. Of course, it can also supervise the learning of the second sample image by the third image classification model, the fourth image classification model, and other models. The second image classification model refers to the teacher model that transfers the learned information to the image classification model to be trained in order to train the image classification model.

[0100] Specifically, the terminal performs image style conversion on the second sample image to obtain a processed second sample image, and performs image style conversion on the first sample image to obtain a processed first sample image; after performing image style conversion on the sample image, the terminal inputs the first sample image and the processed second sample image into the first image classification model to obtain the first target classification result corresponding to the first sample image output by the first image classification model; at the same time, the second sample image and the processed first sample image are input into the second image classification model to obtain the second target classification result corresponding to the second sample image output by the second image classification model.

[0101] For example, the processed second sample image is obtained by converting the image style of the second sample image to the image style of the first sample image, that is, the image style of the processed second sample image is the same as the image style of the first sample image; the first sample image and the processed second sample image are input into a first image classification model, so that the first image classification model learns a large number of images with the image style of the first sample image, and thus the first image classification model has a high prediction accuracy for the first sample image, that is, the accuracy of the first target classification result is high, and therefore, the first target classification result is used to supervise the image classification model to be trained. The processed first sample image is obtained by converting the image style of the first sample image to the image style of the second sample image, that is, the image style of the processed first sample image is the same as the image style of the second sample image; the second sample image and the processed first sample image are input into a second image classification model, so that the second image classification model learns a large number of images with the image style of the second sample image, and thus the second image classification model has a high prediction accuracy for the second sample image, that is, the accuracy of the second target classification result is high, and therefore, the second target classification result is used to supervise the image classification model to be trained.

[0102] In step S203 , the first sample image and the second sample image are respectively input into the image classification model to be trained to obtain a first predicted classification result corresponding to the first sample image and a second predicted classification result corresponding to the second sample image.

[0103] The first predicted classification result refers to the predicted category image output by the image classification model to be trained, corresponding to the first input sample image. The second predicted classification result refers to the predicted category image output by the image classification model to be trained, corresponding to the second input sample image.

[0104] Specifically, the terminal can input the first sample image and the second sample image sampled in step S201 into the image classification model to be trained for image classification processing, and the model outputs the predicted image category information corresponding to the first sample image, that is, the first predicted classification result, and the predicted image category information corresponding to the second sample image, that is, the second predicted classification result.

[0105] In step S204, the image classification model to be trained is trained according to the difference between the first predicted classification result and the first target classification result, and the difference between the second predicted classification result and the second target classification result to obtain a trained image classification model.

[0106] Specifically, the terminal can further calculate the classification difference between the first predicted classification result output by the image classification model to be trained and the first target classification result output by the first image classification model, and calculate the classification difference between the second predicted classification result output by the image classification model to be trained and the second target classification result output by the second image classification model, so as to use the above differences to perform model training on the image classification model to be trained, thereby obtaining a trained image classification model.

[0107] In this embodiment, the terminal can collect a first sample image and a second sample image of different image styles, as well as a first target classification result for the first sample image and a second target classification result for the second sample image, and can obtain the corresponding first prediction classification result and second prediction classification result based on the above-mentioned first sample image and second sample image, and then use the first prediction classification result, the first target classification result, the second prediction classification result and the second target classification result to train the image classification model, thereby improving the prediction accuracy of the trained image classification model.

[0108] In an exemplary embodiment, the above-mentioned step S204 may further include: obtaining a first loss value based on the difference between the first predicted classification result and the first target classification result, and obtaining a second loss value based on the difference between the second predicted classification result and the second target classification result; obtaining a target loss value based on the first loss value and the second loss value; training the image classification model to be trained based on the target loss value to obtain a trained image classification model; and retraining the trained image classification model until a trained image classification model is obtained.

[0109] Among them, the first loss value refers to the loss value calculated by the first predicted classification result and the first target classification result through the pre-constructed first loss function; the second loss value refers to the loss value calculated by the second predicted classification result and the second target classification result through the pre-constructed second loss function. The first loss function and the second loss function can have the same expression form or different expression forms.

[0110] Specifically, after the terminal obtains the first predicted classification result, the first target classification result, the second predicted classification result, and the second target classification result, the first loss value corresponding to the image classification model to be trained can be obtained based on the image classification difference between the first predicted classification result and the first target classification result, and the second loss value corresponding to the image classification model to be trained can be obtained based on the image classification difference between the second predicted classification result and the second target classification result. Thus, the image classification model can be trained based on the first loss value and the second loss value to update the model parameters of the image classification model and obtain a trained image classification model; the trained image classification model is retrained, and if within a preset number of training rounds, the performance improvement of the trained image classification model is detected to be higher than a preset threshold, the terminal retrains the trained image classification model; if within the preset number of training rounds, the performance improvement of the trained image classification model is detected to be lower than or equal to the preset threshold, the model training is terminated to obtain a trained image classification model. The preset number of training rounds can be 1, 2, 3, 4, 5, 6 rounds, etc., which are not specifically limited here.

[0111] In this embodiment, the terminal can obtain a first loss value based on the first predicted classification result and the first target classification result, and obtain a second loss value based on the second predicted classification result and the second target classification result, so that the first loss value and the second loss value can be used to train the image classification model to obtain the trained image classification model, and the trained image classification model can be trained again, so as to further improve the accuracy of the trained image classification model.

[0112] In an exemplary embodiment, a target loss value is obtained based on the first loss value and the second loss value, specifically including the following contents: obtaining a first weight corresponding to the first loss value and a second weight corresponding to the second loss value; performing weighted summation of the first loss value and the second loss value according to the first weight and the second weight, respectively, to obtain the target loss value.

[0113] The first weight refers to the weighted weight set for the first loss value, and the second weight refers to the weighted weight set for the second loss value. The first weight and the second weight can both be set to 1 so that the image processing model is trained equally with the first loss value and the second loss value during model training. Of course, the first weight and the second weight can also be set to other values, depending on the actual situation.

[0114] Specifically, the user can specify a first weight corresponding to the first loss value and a second weight corresponding to the second loss value, and send the first weight and the second weight to the terminal. Furthermore, the terminal can also obtain the first weight and the second weight by itself according to the weight instruction; the terminal performs weighted summation on the first loss value and the second loss value according to the first weight and the second weight, respectively, and then obtains the target loss value corresponding to the first loss value and the second loss value.

[0115] Furthermore, after obtaining the first loss value and the second loss value, the terminal uses the first loss value and the second loss value to train the image classification model to be trained. Even if the image classification model to be trained is trained with the first image classification model and the second image classification model as targets, during the training process, the parameters of each layer in the image classification model to be trained will continue to approach the parameters of each layer in the first image classification model, and the parameters of each layer in the image classification model to be trained will also continue to approach the parameters of each layer in the second image classification model at the same time. After multiple trainings, the parameters of each layer in the trained image classification model are infinitely close to the parameters of each layer in the trained first image classification model and the parameters of each layer in the trained second image classification model.

[0116] In this embodiment, a first weight corresponding to the first loss value and a second weight corresponding to the second loss value are obtained, and the first loss value and the second loss value are fused according to the first weight and the second weight to obtain a target loss value, and then the image classification model can be trained based on the target loss value, thereby improving the prediction accuracy of the trained image classification model.

[0117] In an exemplary embodiment, Figure 3 As shown, retraining the trained image classification model may further include:

[0118] In step S301, an updated first target classification result of the trained first image classification model for the first sample image and an updated second target classification result of the trained second image classification model for the second sample image are obtained.

[0119] The trained first image classification model refers to a model obtained by further training the first image classification model; and the trained second image classification model refers to a model obtained by further training the second image classification model.

[0120] Specifically, the terminal inputs the first sample image into the trained first image classification model, obtains the updated first target classification result for the first sample image output by the trained first image classification model, and inputs the second sample image into the trained second image classification model, obtains the updated second target classification result for the second sample image output by the trained second image classification model.

[0121] In step S302 , the first sample image and the second sample image are respectively input into the trained image classification model to obtain an updated first prediction classification result corresponding to the first sample image and an updated second prediction classification result corresponding to the second sample image.

[0122] The updated first predicted classification result refers to the predicted category image output by the trained image classification model corresponding to the first input sample image. The updated second predicted classification result refers to the predicted category image output by the trained image classification model corresponding to the second input sample image.

[0123] Specifically, the terminal inputs the first sample image and the second sample image sampled in step S201 into the trained image classification model, so that the trained image classification model performs image classification processing on the first sample image and the second sample image again, and the trained image classification model outputs the new predicted image category information corresponding to the first sample image as the updated first predicted classification result, and the new predicted image category information corresponding to the second sample image as the updated second predicted classification result.

[0124] In step S303, the trained image classification model is trained according to the difference between the updated first predicted classification result and the updated first target classification result, and the difference between the updated second predicted classification result and the updated second target classification result.

[0125] Specifically, the terminal can further calculate the classification difference between the updated first predicted classification result output by the trained image classification model and the updated first target classification result output by the trained first image classification model, and calculate the classification difference between the updated second predicted classification result output by the trained image classification model and the updated second target classification result output by the trained second image classification model, thereby using the above differences to perform model training on the trained image classification model.

[0126] In this embodiment, the terminal inputs the first sample image and the second sample image into the trained image classification model to obtain the corresponding updated first prediction classification result and the updated second prediction classification result, and uses the difference between the updated first prediction classification result and the updated first target classification result, as well as the difference between the updated second prediction classification result and the updated second target classification result to train the trained image classification model, thereby further improving the prediction accuracy of the trained image classification model.

[0127] In an exemplary embodiment, Figure 4 As shown, the trained first image classification model is obtained by training in the following way:

[0128] In step S401 , image style conversion is performed on the second sample image based on the first sample image to obtain a processed second sample image.

[0129] The processed second sample image refers to the second sample image whose image style has been transformed to the style of the first sample image. Image style transfer refers to changing the texture, color, and other styles of an image while preserving the primary content of the image. For example, in a narrow sense, it changes the brightness of the image, rather than changing the artistic or painterly style of the image, such as transforming the image into an abstract image, or into an ink painting style or a comic book style.

[0130] For example, the first sample image may be an image style of a cat in the daytime, and the second sample image may be an image style of a cat in the nighttime. Due to the different image styles, the colors and textures of the cats in the first sample image and the second sample image are different. For example, the color of a cat in the daytime is brighter and the fur texture is clearer, while the color of a cat at night is darker and the fur texture is blurry. The processed second sample image can be understood as transforming a cat with a night style into a cat with a daytime style, that is, changing the color, texture and other image styles of the original image, while the image categories in the original image remain unchanged.

[0131] Specifically, the terminal calculates the mean and variance of the first sample image, and performs image style conversion on the second sample image based on the mean and variance of the first sample image to obtain a processed second sample image. Further, performing image style conversion on the second sample image based on the mean and variance of the first sample image to obtain a processed second sample image may include: the terminal subtracting the mean of the first sample image from the second sample image to obtain a subtracted second sample image, and then dividing the subtracted second sample image by the variance of the first sample image to obtain the processed second sample image.

[0132] In step S402, the processed second sample image is input into the first image classification model to obtain a predicted classification result of the processed second sample image.

[0133] Specifically, after obtaining the processed second sample image in step S401, the terminal may further input the processed second sample image into the first image classification model, and the first image classification model performs image classification processing to obtain an output prediction classification result for the processed second sample image.

[0134] In step S403, the first image classification model is trained according to the difference between the predicted classification result of the processed second sample image and the second target classification result of the second sample image to obtain a trained first image classification model.

[0135] Specifically, the terminal can further calculate the classification difference between the predicted classification result of the processed second sample image output by the first image classification model in step S402 and the second target classification result output by the second image classification model, thereby using the above difference to train the first image classification model to obtain the trained first image classification model.

[0136] In practical applications, since the image style of the processed second sample image is the same as that of the first sample image, the first image classification model learns a large amount of image styles of the first sample image, resulting in a low prediction accuracy of the first image classification model under the image style of the second sample image, while the second image classification model has a high prediction accuracy under the image style of the second sample image. Therefore, the second target classification result output by the second image classification model is used to supervise the training of the first image classification model.

[0137] Furthermore, the first image classification model is trained with the second image classification model as the target. During the training process, the parameters of each layer in the first image classification model will approach the parameters of each layer in the second image classification model. After multiple trainings, the parameters of each layer in the trained first image classification model will be infinitely close to the parameters of each layer in the trained second image classification model.

[0138] In this embodiment, image style conversion is performed on the second sample image to obtain a processed second sample image, a predicted classification result of the processed second sample image is obtained through the first image score model, and the predicted classification result of the second sample image and the second target classification result of the second sample image are used to train the first image classification model to obtain a trained first image classification model, so that the trained first image classification model learns second sample images of multiple styles, thereby improving the prediction accuracy of the trained first image classification model.

[0139] In an exemplary embodiment, Figure 5 As shown, the trained second image classification model is trained in the following way:

[0140] In step S501 , image style conversion is performed on the first sample image according to the second sample image to obtain a processed first sample image.

[0141] The processed first sample image refers to the first sample image whose image style is transformed into the image style of the second sample image.

[0142] For example, the first sample image may be an image of a cat in the daytime, and the second sample image may be an image of a cat in the nighttime. The processed first sample image is a cat transformed from a daytime cat to a nighttime cat.

[0143] Specifically, the terminal calculates the mean and variance of the second sample image, and performs image style conversion on the first sample image based on the mean and variance of the second sample image to obtain a processed first sample image. Further, performing image style conversion on the first sample image based on the mean and variance of the second sample image to obtain a processed first sample image may include: the terminal subtracting the mean of the second sample image from the first sample image to obtain a subtracted first sample image, and then dividing the subtracted first sample image by the variance of the second sample image to obtain the processed first sample image.

[0144] In step S502, the processed first sample image is input into the second image classification model to obtain a predicted classification result of the processed first sample image.

[0145] Specifically, after obtaining the processed first sample image in step S501, the terminal may further input the processed first sample image into the second image classification model, and the second image classification model performs image classification processing to obtain an output prediction classification result for the processed first sample image.

[0146] In step S503, the second image classification model is trained according to the difference between the predicted classification result of the processed first sample image and the first target classification result of the first sample image to obtain a trained second image classification model.

[0147] Specifically, the terminal can further calculate the classification difference between the predicted classification result of the processed first sample image output by the second image classification model in step S502 and the first target classification result output by the first image classification model, thereby using the above difference to train the second image classification model to obtain a trained second image classification model.

[0148] Furthermore, the second image classification model is trained with the first image classification model as the target. During the training process, the parameters of each layer in the second image classification model will approach the parameters of each layer in the second image classification model. After multiple trainings, the parameters of each layer in the trained second image classification model are infinitely close to the parameters of each layer in the trained first image classification model.

[0149] In this embodiment, image style conversion is performed on the first sample image to obtain a processed first sample image, a predicted classification result of the processed first sample image is obtained through the second image score model, and the predicted classification result of the first sample image and the first target classification result of the first sample image are used to train the second image classification model to obtain a trained second image classification model, so that the trained second image classification model learns first sample images of multiple styles, thereby improving the prediction accuracy of the trained second image classification model.

[0150] In an exemplary embodiment, the sample image also includes at least one third sample image, and the image style of the third sample image is different from that of the first sample image and the second sample image; before the above-mentioned step S204, it also includes: obtaining a third target classification result of the third image classification model for the third sample image; inputting the third sample image into the image classification model to be trained to obtain a third predicted classification result corresponding to the third sample image.

[0151] Among them, the third sample image can be a sample image or a sample image set composed of multiple images; the image style of the third sample image is different from the first sample image and the second sample image, which means that the image style of most images in the first sample image, the image style of most images in the second sample image, and the image style of most images in the third sample image are all different.

[0152] For example, the image style of all images in the first sample image is daytime, the image style of all images in the second sample image is nighttime, and the image style of all images in the third sample image is dusk.

[0153] For another example, in the first sample image, the proportion of images with daytime style is 90%, the proportion of images with night style is 4%, and the proportion of images with dusk style is 6%, so the image style of the first sample image is daytime; in the second sample image, the proportion of images with daytime style is 10%, the proportion of images with night style is 85%, and the proportion of images with dusk style is 5%, so the image style of the second sample image is night; and in the third sample image, the proportion of images with daytime style is 10%, the proportion of images with night style is 10%, and the proportion of images with dusk style is 80%, so the image style of the third sample image is dusk.

[0154] Among them, the third target classification result refers to the classification result of the third image classification model performing image classification processing on the third sample image, which is used to supervise the learning of the third sample image by other image classification models, that is, used to supervise the learning of the third sample image by the image classification model to be trained, the first image classification model and the second image classification model.

[0155] The third predicted classification result refers to the predicted category image corresponding to the input third sample image output by the image classification model to be trained. The third image classification model refers to another teacher model used to train the image classification model.

[0156] Specifically, after the terminal samples the first sample image, the second sample image and the third sample image from the original sample image, it obtains the first target classification result of the first image classification model for the first sample image, and the second target classification result of the second image classification model for the second sample image, and inputs the third sample image into the third image classification model to obtain the third target classification result for the third sample image output by the third image classification model, and inputs the third sample image, the first sample image and the second sample image into the trained image classification model for image classification processing, and the image classification model outputs the predicted image category information corresponding to the first sample image, that is, the first predicted classification result, the predicted image category information corresponding to the second sample image, that is, the second predicted classification result, and the predicted image category information corresponding to the third sample image, that is, the third predicted classification result.

[0157] Furthermore, the sample images may also include a fourth sample image, a fifth sample image, a sixth sample image, and so on, wherein the image styles of each sample image are different. When the number of sample images increases, the number of image classification models required for the training model will also increase accordingly; wherein, the number of image classification models required for the training model is equal to the number of sample image sets extracted plus one. For example, if four sample images with different image styles are extracted from the original sample images, the number of image classification models required for the training model is five. Of these five models, four are used as teacher models, and the remaining one is used as a student model. Finally, the trained student model is used as the trained image classification model.

[0158] The above-mentioned step S204 may further include: training the image classification model to be trained according to the difference between the first predicted classification result and the first target classification result, the difference between the second predicted classification result and the second target classification result, and the difference between the third predicted classification result and the third target classification result to obtain a trained image classification model.

[0159] Specifically, the terminal can further calculate the classification difference between the first predicted classification result output by the image classification model to be trained and the first target classification result output by the first image classification model, calculate the classification difference between the second predicted classification result output by the image classification model to be trained and the second target classification result output by the second image classification model, and calculate the classification difference between the third predicted classification result output by the image classification model to be trained and the third target classification result output by the third image classification model, so as to use the above differences to perform model training on the image classification model to be trained, thereby obtaining a trained image classification model.

[0160] In actual applications, the terminal first randomly selects three subsets from the original dataset, namely the first sample image (labeled as A), the second sample image (labeled as B), and the third sample image (labeled as C). Then, based on the image style of A, image style conversion is performed on B and C, and A and B and C with the image style of A are used as the first set of datasets, that is, the image style of A is more prevalent in the first set of datasets; based on the image style of B, image style conversion is performed on A and C, and B and A and C with the image style of B are used as the first set of datasets, that is, the image style of B is more prevalent in the second set of datasets; based on the image style of C, image style conversion is performed on A and B, and C and A and B with the image style of C are used as the third set of datasets, that is, the image style of C is more prevalent in the third set of datasets.

[0161] The terminal obtains four networks NetA, NetB, NetC and NetD, and uses the first set of data sets with more image style A to train NetA, and obtains NetA's first target classification result for A and NetA's prediction classification results for B and C of image style A respectively; uses the second set of data sets with more image style B to train NetB, and obtains NetB's second target classification result for B and NetB's prediction classification results for A and C of image style B respectively; uses the third set of data sets with more image style C to train NetC, and obtains NetC's third target classification result for C and NetC's prediction classification results for A and B of image style C respectively; uses A, B and C to train NetD, and obtains NetD's first prediction classification result for A, NetD's second prediction classification result for B and NetD's third prediction classification result for C.

[0162] Then, the NetD is trained according to the difference between the first predicted classification result and the first target classification result, the difference between the second predicted classification result and the second target classification result, and the difference between the third predicted classification result and the third target classification result to obtain a trained NetD.

[0163] In this embodiment, the terminal obtains the third target classification result of the third image classification model for the third sample image, obtains the third predicted classification result corresponding to the third sample image through the image classification model to be trained, and then trains the image classification model according to the first predicted classification result, the difference between the first target classification result, the second predicted classification result, the second target classification result, the third predicted classification result and the third target classification result to obtain a trained image classification model, thereby realizing the training of sample images of different image styles through sample images of multiple image styles, and using the target classification results learned by each model to supervise the image classification model to be trained. Through the above method, the image classification models learn from each other, so that the prediction accuracy of the trained image classification model is further improved.

[0164] Figure 6 FIG. 1 is a flowchart of an image classification model training method according to an exemplary embodiment. Figure 6 As shown, the image classification model training method is used in the terminal and includes the following steps.

[0165] In step S601 , sample images are acquired; the sample images include at least a first sample image and a second sample image of different image styles.

[0166] The first sample image may be a single sample image or a sample image set consisting of multiple images, and the same applies to the second sample image. The first sample image and the second sample image have different image styles, which means that the image style of most images in the first sample image is different from the image style of most images in the second sample image.

[0167] Specifically, the terminal can collect images in the scene as original sample images, or use stored images as original sample images. The terminal performs sampling processing on the original sample images to obtain first sample images and second sample images whose number is much smaller than the original sample images, and uses the first sample images and second sample images as sample images.

[0168] In step S602, a first target classification result of the first image classification model for the first sample image and a second target classification result of the second image classification model for the second sample image are obtained.

[0169] The first target classification result refers to the classification result of the first image classification model performing image classification processing on the first sample image. It is used to supervise the learning of the first sample image by other image classification models, that is, to supervise the learning of the first sample image by the image classification model to be trained and the second image classification model. The first image classification model refers to a teacher model that transfers learned information to the image classification model to be trained in order to train the image classification model.

[0170] The second target classification result refers to the classification result of the second image classification model performing image classification processing on the second sample image. It is used to supervise the learning of the second sample image by other image classification models, that is, to supervise the learning of the second sample image by the image classification model to be trained and the first image classification model. The second image classification model refers to another teacher model that transfers learned information to the image classification model to be trained in order to train the image classification model.

[0171] Specifically, the terminal performs image style conversion on the second sample image to obtain a processed second sample image, and performs image style conversion on the first sample image to obtain a processed first sample image; after performing image style conversion on the sample image, the terminal inputs the first sample image and the processed second sample image into the first image classification model to obtain the first target classification result corresponding to the first sample image output by the first image classification model; at the same time, the second sample image and the processed first sample image are input into the second image classification model to obtain the second target classification result corresponding to the second sample image output by the second image classification model.

[0172] In step S603 , the first sample image and the second sample image are respectively input into the image classification model to be trained to obtain a first predicted classification result corresponding to the first sample image and a second predicted classification result corresponding to the second sample image.

[0173] The first predicted classification result refers to the predicted category image output by the image classification model to be trained, corresponding to the first input sample image. The second predicted classification result refers to the predicted category image output by the image classification model to be trained, corresponding to the second input sample image.

[0174] Specifically, the terminal can input the first sample image and the second sample image sampled in step S601 into the image classification model to be trained for image classification processing, and the model outputs the predicted image category information corresponding to the first sample image, that is, the first predicted classification result, and the predicted image category information corresponding to the second sample image, that is, the second predicted classification result.

[0175] In step S604, a first loss value is obtained according to the difference between the first predicted classification result and the first target classification result, and a second loss value is obtained according to the difference between the second predicted classification result and the second target classification result.

[0176] Among them, the first loss value refers to the loss value calculated by the first predicted classification result and the first target classification result through the pre-constructed first loss function; the second loss value refers to the loss value calculated by the second predicted classification result and the second target classification result through the pre-constructed second loss function. The first loss function and the second loss function can have the same expression form or different expression forms.

[0177] Specifically, after the terminal obtains the first predicted classification result, the first target classification result, the second predicted classification result and the second target classification result, the first loss value corresponding to the image classification model to be trained can be obtained based on the image classification difference between the first predicted classification result and the first target classification result, and the second loss value corresponding to the image classification model to be trained can be obtained based on the image classification difference between the second predicted classification result and the second target classification result.

[0178] In step S605, the image classification model to be trained is trained according to the first loss value and the second loss value to obtain a trained image classification model.

[0179] Specifically, the terminal trains the image classification model to be trained according to the first loss value and the second loss value to update the model parameters of the image classification model and obtain a trained image classification model; when the trained image classification model meets the preset training completion conditions, the trained image classification model is used as the trained image classification model.

[0180] The above-mentioned image classification model training method obtains sample images; the sample images include at least a first sample image and a second sample image of different image styles; obtains a first target classification result of the first image classification model for the first sample image, and a second target classification result of the second image classification model for the second sample image; respectively inputs the first sample image and the second sample image into the image classification model to be trained to obtain a first predicted classification result corresponding to the first sample image and a second predicted classification result corresponding to the second sample image; obtains a first loss value based on the difference between the first predicted classification result and the first target classification result, and obtains a second loss value based on the difference between the second predicted classification result and the second target classification result; trains the image classification model to be trained based on the first loss value and the second loss value to obtain a trained image classification model. When training the image classification model, the present disclosure not only utilizes multiple sample images of different image styles, but also supervises the model training through the first target classification result and the second target classification result, thereby greatly improving the model performance of the trained image classification model, and further improving the accuracy of the image classification model when processing image classification.

[0181] In an exemplary embodiment, another image classification model training method is provided. The specific process of the solution can be as follows: Figure 7 As shown, the following steps are included:

[0182] First, the terminal obtains the original sample image, the first image classification model, the second image classification model and the image classification model to be trained, marks the first image classification model as NetC, marks the second image classification model as NetE, and marks the image classification model to be trained as NetD; among them, the first image classification model, the second image classification model and the image classification model to be trained can be online models.

[0183] The terminal randomly samples the original sample images to obtain the first sample image, which is marked as set A, and the second sample image, which is marked as set B; the terminal calculates the variance and mean of set A, calculates the variance and mean of set B, subtracts the mean of set B from the image in set A, and then divides it by the variance of set B to obtain set A of the image style of B, which is marked as A B , subtract the mean of set A from the images in set B, and then divide it by the variance of set A to get set B of image style A, marked as B A .

[0184] Then set A B And set B are input into the first image classification model NetC, the first target classification result corresponding to set B is obtained and marked as feat(B), and set A B The corresponding predicted classification results are marked as feat(A B ); Set A and Set B A Input into the second image classification model NetE, obtain the second target classification result corresponding to set A and mark it as feat(A), and set B A The corresponding predicted classification results are marked as feat(B A ); input set A and set B into the image classification model to be trained NetD, obtain the first predicted classification result corresponding to set B and mark it as Feat(B), and the second predicted classification result corresponding to set A and mark it as Feat(A).

[0185] Then, the terminal determines the first student loss value according to the first target classification result feat(B) and the first predicted classification result Feat(B), determines the second student loss value according to the second target classification result feat(A) and the second predicted classification result Feat(A), sums the first student loss value and the second student loss value to obtain the total student loss value; takes the first image classification model NetC and the second image classification model NetE as the target, trains the image classification model NetD to be trained according to the total student loss value, and obtains the trained image classification model; according to the set A B The corresponding predicted classification results feat(A B ) and the second target classification result feat(A), determine the second teacher loss value, and train the first image classification model according to the second teacher loss value to obtain the trained first image classification model; according to the set B A The corresponding predicted classification results feat(B A ) and the first target classification result feat(B), determine the first teacher loss value, train the second image classification model according to the first teacher loss value, and obtain the trained second image classification model.

[0186] Finally, the trained image classification model, the trained first image classification model, and the trained second image classification model are retrained. If, within a preset number of training rounds, the performance improvement of the trained image classification model is detected to be higher than a preset threshold, the terminal retrains the trained image classification model, the trained first image classification model, and the trained second image classification model. If, within a preset number of training rounds, the performance improvement of the trained image classification model is detected to be lower than or equal to the preset threshold, the model training is terminated to obtain a trained image classification model. The preset number of training rounds can be 1, 2, 3, 4, 5, 6, etc., and is not specifically limited here.

[0187] This embodiment proposes a method for training an image classification model. First, by processing sample images through image style conversion, more feature information in the sample images can be mined, thereby improving the training effect of the image classification model, and further improving the accuracy of the image classification model when processing image classification; second, the target classification results can be used to supervise the re-training of multiple prediction image classification models, so that the performance of the trained first image classification model and the trained second image classification model is continuously improved, and the first target classification result of the trained first image classification model and the second target classification result of the trained second image classification model are used to supervise the re-training of the trained image classification model, so that the information learned by the trained first image classification model and the trained second image classification model is integrated into the trained image classification model. Unlike traditional technologies that require performance testing of each model separately and select the model with the best performance as the final image classification model, the present disclosure not only improves the training efficiency of the model, but also improves the performance of the trained image classification model, thereby further improving the accuracy of the image classification model when processing image classification.

[0188] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0189] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can be referred to each other, and each embodiment focuses on the differences from other embodiments. For related parts, please refer to the description of other method embodiments.

[0190] Based on the same inventive concept, an embodiment of the present disclosure further provides an image classification device for implementing the above-mentioned image classification method.

[0191] Figure 8 FIG. 1 is a block diagram of an image classification device according to an exemplary embodiment. Figure 8 The device includes a processed image acquisition unit 801, an image category acquisition unit 802 and a classification result determination unit 803.

[0192] The processed image acquisition unit 801 is configured to acquire an image to be processed.

[0193] The image category acquisition unit 802 is configured to input the image to be processed into a trained image classification model to obtain image category information in the image to be processed; the trained image classification model is obtained by training the image classification model to be trained using at least two sample images of different image styles and the target classification result of each sample image, and the target classification result of each sample image is obtained by processing according to different image classification models.

[0194] The classification result determining unit 803 is configured to determine the image category information as the image classification result of the image to be processed.

[0195] In an exemplary embodiment, the image classification device also includes: a classification model training unit, configured to execute acquisition of sample images; the sample images include at least a first sample image and a second sample image with different image styles; acquire a first target classification result of the first image classification model for the first sample image, and a second target classification result of the second image classification model for the second sample image; input the first sample image and the second sample image into the image classification model to be trained respectively, and obtain a first predicted classification result corresponding to the first sample image and a second predicted classification result corresponding to the second sample image; train the image classification model to be trained according to the difference between the first predicted classification result and the first target classification result, and the difference between the second predicted classification result and the second target classification result, to obtain a trained image classification model.

[0196] In an exemplary embodiment, the image classification device further includes: a classification model acquisition unit, configured to obtain a first loss value based on the difference between the first predicted classification result and the first target classification result, and to obtain a second loss value based on the difference between the second predicted classification result and the second target classification result; obtain a target loss value based on the first loss value and the second loss value; train the image classification model to be trained based on the target loss value to obtain a trained image classification model; and train the trained image classification model again until a trained image classification model is obtained.

[0197] In an exemplary embodiment, the image classification device further includes: a target loss acquisition unit, configured to execute acquisition of a first weight corresponding to the first loss value and a second weight corresponding to the second loss value; and perform weighted summation of the first loss value and the second loss value according to the first weight and the second weight, respectively, to obtain a target loss value.

[0198] In an exemplary embodiment, the image classification device also includes: a model retraining unit, configured to execute obtaining an updated first target classification result of the trained first image classification model for the first sample image, and an updated second target classification result of the trained second image classification model for the second sample image; inputting the first sample image and the second sample image into the trained image classification model respectively to obtain an updated first predicted classification result corresponding to the first sample image and an updated second predicted classification result corresponding to the second sample image; and training the trained image classification model according to the difference between the updated first predicted classification result and the updated first target classification result, and the difference between the updated second predicted classification result and the updated second target classification result.

[0199] In an exemplary embodiment, the image classification device also includes: a first model training unit, configured to perform image style conversion on the second sample image based on the first sample image to obtain a processed second sample image; input the processed second sample image into the first image classification model to obtain a predicted classification result of the processed second sample image; and train the first image classification model based on the difference between the predicted classification result of the processed second sample image and the second target classification result of the second sample image to obtain a trained first image classification model.

[0200] In an exemplary embodiment, the image classification device also includes: a second model training unit, configured to perform image style conversion on the first sample image based on the second sample image to obtain a processed first sample image; input the processed first sample image into the second image classification model to obtain a predicted classification result of the processed first sample image; and train the second image classification model based on the difference between the predicted classification result of the processed first sample image and the first target classification result of the first sample image to obtain a trained second image classification model.

[0201] In an exemplary embodiment, the sample image also includes at least one third sample image, and the image style of the third sample image is different from that of the first sample image and the second sample image; the image classification device also includes: a classification result acquisition unit, configured to execute acquisition of a third target classification result of a third image classification model for the third sample image; inputting the third sample image into the image classification model to be trained to obtain a third predicted classification result corresponding to the third sample image; the classification model training unit, further configured to execute training of the image classification model to be trained based on the difference between the first predicted classification result and the first target classification result, the difference between the second predicted classification result and the second target classification result, and the difference between the third predicted classification result and the third target classification result to obtain a trained image classification model.

[0202] Figure 9 FIG. 1 is a block diagram of an image classification model training device according to an exemplary embodiment. Figure 9 The device includes a sample image acquisition unit 901, a target result acquisition unit 902, a prediction result acquisition unit 903, a loss function determination unit 904 and a target model acquisition unit 905.

[0203] The sample image acquisition unit 901 is configured to acquire sample images; the sample images include at least a first sample image and a second sample image of different image styles.

[0204] The target result acquisition unit 902 is configured to acquire a first target classification result of the first image classification model for the first sample image and a second target classification result of the second image classification model for the second sample image.

[0205] The prediction result acquisition unit 903 is configured to input the first sample image and the second sample image into the image classification model to be trained respectively to obtain a first prediction classification result corresponding to the first sample image and a second prediction classification result corresponding to the second sample image.

[0206] The loss function determination unit 904 is configured to obtain a first loss value based on the difference between the first predicted classification result and the first target classification result, and to obtain a second loss value based on the difference between the second predicted classification result and the second target classification result.

[0207] The target model acquisition unit 905 is configured to train the image classification model to be trained according to the first loss value and the second loss value to obtain a trained image classification model.

[0208] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0209] Each unit in the above-mentioned image classification device and image classification model training device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned units can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0210] Figure 10 The block diagram of an electronic device 1000 for implementing an image classification method or an image classification model training method according to an exemplary embodiment is shown. For example, the electronic device 1000 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0211] Reference Figure 10 , the electronic device 1000 may include one or more of the following components: a processing component 1002 , a memory 1004 , a power component 1006 , a multimedia component 1008 , an audio component 1010 , an input / output (I / O) interface 1012 , a sensor component 1014 , and a communication component 1016 .

[0212] The processing component 1002 generally controls the overall operation of the electronic device 1000, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 1002 may include one or more processors 1020 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 1002 may include one or more modules to facilitate interaction between the processing component 1002 and other components. For example, the processing component 1002 may include a multimedia module to facilitate interaction between the multimedia component 1008 and the processing component 1002.

[0213] The memory 1004 is configured to store various types of data to support operations on the electronic device 1000. Examples of such data include instructions for any application or method operating on the electronic device 1000, contact data, phone book data, messages, pictures, videos, etc. The memory 1004 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, optical disk, or graphene memory.

[0214] The power supply assembly 1006 provides power to the various components of the electronic device 1000. The power supply assembly 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1000.

[0215] The multimedia component 1008 includes a screen that provides an output interface between the electronic device 1000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1008 includes a front camera and / or a rear camera. When the electronic device 1000 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0216] The audio component 1010 is configured to output and / or input audio signals. For example, the audio component 1010 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 1000 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 1004 or transmitted via the communication component 1016. In some embodiments, the audio component 1010 also includes a speaker for outputting audio signals.

[0217] I / O interface 1012 provides an interface between processing component 1002 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0218] The sensor assembly 1014 includes one or more sensors for providing various aspects of status assessment for the electronic device 1000. For example, the sensor assembly 1014 can detect the open / closed state of the electronic device 1000, the relative positioning of components, such as the display and keypad of the electronic device 1000. The sensor assembly 1014 can also detect changes in the position of the electronic device 1000 or components of the electronic device 1000, the presence or absence of user contact with the electronic device 1000, the orientation or acceleration / deceleration of the device 1000, and temperature changes of the electronic device 1000. The sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 1014 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1014 may also include an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0219] The communication component 1016 is configured to facilitate wired or wireless communication between the electronic device 1000 and other devices. The electronic device 1000 can access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 1016 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1016 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0220] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described methods.

[0221] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, and the instructions can be executed by the processor 1020 of the electronic device 1000 to perform the above method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0222] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions, and the instructions can be executed by the processor 1020 of the electronic device 1000 to implement the above method.

[0223] It should be noted that the above-mentioned devices, electronic devices, computer-readable storage media, computer program products, etc. can also include other implementation methods according to the description of the method embodiments. The specific implementation methods can refer to the description of the relevant method embodiments and will not be described one by one here.

[0224] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

[0225] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image classification method, characterized in that: include: Get the image to be processed; Inputting the image to be processed into a trained image classification model to obtain image category information of the image to be processed; the trained image classification model is obtained by training the image classification model to be trained using a difference between a first predicted classification result and a first target classification result of a first sample image of at least two different image styles, and a difference between a second predicted classification result and a second target classification result of a second sample image, wherein the first predicted classification result and the second predicted classification result are obtained by respectively inputting the first sample image and the second sample image into the image classification model to be trained; The first target classification result is obtained by a first image classification model based on the first sample image and the processed second sample image; The second target classification result is obtained by a second image classification model based on the second sample image and the processed first sample image; The processed second sample image is obtained by converting the image style of the second sample image into the image style of the first sample image according to the mean and variance of the first sample image; The processed first sample image is obtained by converting the image style of the first sample image into the image style of the second sample image according to the mean and variance of the second sample image; The image category information is determined as the image classification result of the image to be processed.

2. The method according to claim 1, characterized in that The trained image classification model is obtained by training in the following manner: Acquire sample images; the sample images include at least a first sample image and a second sample image of different image styles; Obtaining a first target classification result of the first image classification model for the first sample image, and a second target classification result of the second image classification model for the second sample image; inputting the first sample image and the second sample image into the image classification model to be trained respectively, to obtain a first predicted classification result corresponding to the first sample image and a second predicted classification result corresponding to the second sample image; According to the difference between the first predicted classification result and the first target classification result, and the difference between the second predicted classification result and the second target classification result, the image classification model to be trained is trained to obtain the trained image classification model.

3. The method according to claim 2, characterized in that The method of training the image classification model to be trained according to the difference between the first predicted classification result and the first target classification result, and the difference between the second predicted classification result and the second target classification result, to obtain the trained image classification model, includes: Obtaining a first loss value based on a difference between the first predicted classification result and the first target classification result, and obtaining a second loss value based on a difference between the second predicted classification result and the second target classification result; Obtaining a target loss value according to the first loss value and the second loss value; Training the image classification model to be trained according to the target loss value to obtain a trained image classification model; The trained image classification model is trained again until the trained image classification model is obtained.

4. The method according to claim 3, characterized in that Obtaining a target loss value according to the first loss value and the second loss value includes: Obtaining a first weight corresponding to the first loss value and a second weight corresponding to the second loss value; The first loss value and the second loss value are weightedly summed according to the first weight and the second weight respectively to obtain the target loss value.

5. The method according to claim 3, characterized in that The retraining of the trained image classification model comprises: Obtaining an updated first target classification result of the trained first image classification model for the first sample image, and an updated second target classification result of the trained second image classification model for the second sample image; inputting the first sample image and the second sample image into the trained image classification model respectively to obtain an updated first prediction classification result corresponding to the first sample image and an updated second prediction classification result corresponding to the second sample image; The trained image classification model is trained based on the difference between the updated first predicted classification result and the updated first target classification result, and the difference between the updated second predicted classification result and the updated second target classification result.

6. The method according to claim 5, characterized in that The trained first image classification model is obtained by training in the following manner: performing image style conversion on the second sample image according to the first sample image to obtain a processed second sample image; Inputting the processed second sample image into the first image classification model to obtain a predicted classification result of the processed second sample image; The first image classification model is trained according to the difference between the predicted classification result of the processed second sample image and the second target classification result of the second sample image to obtain the trained first image classification model.

7. The method according to claim 5, characterized in that The trained second image classification model is obtained by training in the following manner: performing image style conversion on the first sample image according to the second sample image to obtain a processed first sample image; Inputting the processed first sample image into the second image classification model to obtain a predicted classification result of the processed first sample image; The second image classification model is trained according to the difference between the predicted classification result of the processed first sample image and the first target classification result of the first sample image to obtain the trained second image classification model.

8. The method according to claim 2, characterized in that The sample images further include at least one third sample image, and the image style of the third sample image is different from that of the first sample image and the second sample image; Before training the image classification model to be trained based on the difference between the first predicted classification result and the first target classification result, and the difference between the second predicted classification result and the second target classification result, to obtain the trained image classification model, the method further includes: Obtaining a third target classification result of a third image classification model for the third sample image; Inputting the third sample image into the image classification model to be trained to obtain a third predicted classification result corresponding to the third sample image; The method of training the image classification model to be trained according to the difference between the first predicted classification result and the first target classification result, and the difference between the second predicted classification result and the second target classification result, to obtain the trained image classification model, includes: According to the difference between the first predicted classification result and the first target classification result, the difference between the second predicted classification result and the second target classification result, and the difference between the third predicted classification result and the third target classification result, the image classification model to be trained is trained to obtain the trained image classification model.

9. A method for training an image classification model, characterized in that: include: Acquire sample images; the sample images include at least a first sample image and a second sample image of different image styles; converting the image style of the first sample image into the image style of the second sample image according to the mean and variance of the second sample image to obtain a processed first sample image; and converting the image style of the second sample image into the image style of the first sample image according to the mean and variance of the first sample image to obtain a processed second sample image; Inputting the first sample image and the processed second sample image into a first image classification model to obtain a first target classification result of the first sample image by the first image classification model, and inputting the second sample image and the processed first sample image into a second image classification model to obtain a second target classification result of the second image classification model for the second sample image; inputting the first sample image and the second sample image into an image classification model to be trained respectively, to obtain a first predicted classification result corresponding to the first sample image and a second predicted classification result corresponding to the second sample image; Obtaining a first loss value based on a difference between the first predicted classification result and the first target classification result, and obtaining a second loss value based on a difference between the second predicted classification result and the second target classification result; The image classification model to be trained is trained according to the first loss value and the second loss value to obtain a trained image classification model.

10. An image classification device, characterized in that: include: A processed image acquisition unit, configured to acquire an image to be processed; an image category acquisition unit configured to input the image to be processed into a trained image classification model to obtain image category information of the image to be processed; the trained image classification model is obtained by training the image classification model to be trained using a difference between a first predicted classification result and a first target classification result of a first sample image of at least two different image styles, and a difference between a second predicted classification result and a second target classification result of a second sample image, wherein the first predicted classification result and the second predicted classification result are obtained by respectively inputting the first sample image and the second sample image into the image classification model to be trained; The first target classification result is obtained by a first image classification model based on the first sample image and the processed second sample image; The second target classification result is obtained by a second image classification model based on the second sample image and the processed first sample image; The processed second sample image is obtained by converting the image style of the second sample image into the image style of the first sample image according to the mean and variance of the first sample image; The processed first sample image is obtained by converting the image style of the first sample image into the image style of the second sample image according to the mean and variance of the second sample image; The classification result determining unit is configured to determine the image category information as the image classification result of the image to be processed.

11. The device according to claim 10, characterized in that The apparatus further includes a classification model training unit configured to acquire sample images, wherein the sample images include at least a first sample image and a second sample image of different image styles; acquire a first target classification result of the first image classification model for the first sample image, and a second target classification result of the second image classification model for the second sample image; The first sample image and the second sample image are respectively input into the image classification model to be trained to obtain a first predicted classification result corresponding to the first sample image and a second predicted classification result corresponding to the second sample image; according to the difference between the first predicted classification result and the first target classification result, and the difference between the second predicted classification result and the second target classification result, the image classification model to be trained is trained to obtain the trained image classification model.

12. The device according to claim 11, characterized in that The device also includes a classification model acquisition unit, which is configured to obtain a first loss value based on the difference between the first predicted classification result and the first target classification result, and obtain a second loss value based on the difference between the second predicted classification result and the second target classification result; obtain a target loss value based on the first loss value and the second loss value; train the image classification model to be trained based on the target loss value to obtain a trained image classification model; and retrain the trained image classification model until the trained image classification model is obtained.

13. The device according to claim 12, characterized in that The device also includes a target loss acquisition unit, which is configured to execute acquisition of a first weight corresponding to the first loss value and a second weight corresponding to the second loss value; and perform weighted summation of the first loss value and the second loss value according to the first weight and the second weight, respectively, to obtain the target loss value.

14. The device according to claim 12, characterized in that The apparatus further includes a model retraining unit configured to execute acquiring an updated first target classification result of the trained first image classification model for the first sample image, and an updated second target classification result of the trained second image classification model for the second sample image; inputting the first sample image and the second sample image into the trained image classification model respectively to obtain an updated first prediction classification result corresponding to the first sample image and an updated second prediction classification result corresponding to the second sample image; The trained image classification model is trained based on the difference between the updated first predicted classification result and the updated first target classification result, and the difference between the updated second predicted classification result and the updated second target classification result.

15. The device according to claim 14, characterized in that The device also includes a first model training unit, which is configured to perform image style conversion on the second sample image based on the first sample image to obtain a processed second sample image; input the processed second sample image into the first image classification model to obtain a predicted classification result of the processed second sample image; and train the first image classification model based on the difference between the predicted classification result of the processed second sample image and the second target classification result of the second sample image to obtain the trained first image classification model.

16. The device according to claim 14, characterized in that The device also includes a second model training unit, which is configured to perform image style conversion on the first sample image based on the second sample image to obtain a processed first sample image; input the processed first sample image into the second image classification model to obtain a predicted classification result of the processed first sample image; and train the second image classification model based on the difference between the predicted classification result of the processed first sample image and the first target classification result of the first sample image to obtain the trained second image classification model.

17. The device according to claim 11, characterized in that The sample images further include at least one third sample image, wherein the image style of the third sample image is different from that of the first sample image and the second sample image; the apparatus further includes a classification result acquisition unit configured to execute acquisition of a third target classification result of a third image classification model for the third sample image; input the third sample image into the image classification model to be trained, and obtain a third predicted classification result corresponding to the third sample image; The classification model training unit is further configured to train the image classification model to be trained based on the difference between the first predicted classification result and the first target classification result, the difference between the second predicted classification result and the second target classification result, and the difference between the third predicted classification result and the third target classification result to obtain the trained image classification model.

18. An image classification model training device, characterized in that: include: A sample image acquiring unit is configured to acquire a sample image; The sample images include at least a first sample image and a second sample image of different image styles; a target result acquisition unit configured to convert the image style of the first sample image into the image style of the second sample image based on the mean and variance of the second sample image to obtain a processed first sample image, and convert the image style of the second sample image into the image style of the first sample image based on the mean and variance of the first sample image to obtain a processed second sample image; Inputting the first sample image and the processed second sample image into a first image classification model to obtain a first target classification result of the first sample image by the first image classification model, and inputting the second sample image and the processed first sample image into a second image classification model to obtain a second target classification result of the second image classification model for the second sample image; a prediction result acquisition unit configured to input the first sample image and the second sample image into an image classification model to be trained, respectively, to obtain a first prediction classification result corresponding to the first sample image and a second prediction classification result corresponding to the second sample image; a loss function determining unit configured to obtain a first loss value based on a difference between the first predicted classification result and the first target classification result, and to obtain a second loss value based on a difference between the second predicted classification result and the second target classification result; The target model acquisition unit is configured to train the image classification model to be trained based on the first loss value and the second loss value to obtain a trained image classification model.

19. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image classification method according to any one of claims 1 to 8, or the image classification model training method according to claim 9.

20. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image classification method as described in any one of claims 1 to 8, or the image classification model training method as described in claim 9.

21. A computer program product comprising instructions, characterized in that: When the instruction is executed by the processor of the electronic device, the electronic device is enabled to execute the image classification method as described in any one of claims 1 to 8, or the image classification model training method as described in claim 9.

Images