Training method of image classification model, image processing method and device

Abstract

The application discloses a training method of an image classification model, an image processing method and device, which are used for training an image classification model with limited labeled images to obtain higher prediction accuracy and accurately predicting the image category. The training method comprises the following steps: generating a cross-modal image based on a description text of a labeled image and inputting the cross-modal image into a first classifier of an image classification model to output a predicted classification result of the cross-modal image, wherein the description text is used for describing the image content of the labeled image; inputting the labeled image into a second classifier of the image classification model to output a predicted classification result of the labeled image; inputting an unlabeled image into the first classifier to output a first predicted classification result, and inputting the unlabeled image into the second classifier to output a second predicted classification result; determining a total prediction loss based on the category label of the labeled image and the predicted classification results of the images; and adjusting the network parameters of the first classifier and the second classifier based on the total prediction loss.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a training method of an image classification model, and an image processing method and device. BACKGROUND

[0002] Semi-supervised learning (SSL) is a key problem in the field of pattern recognition and machine learning, and is a learning method combining supervised learning and unsupervised learning. SSL uses a large number of unlabeled samples and a limited number of labeled samples to perform pattern recognition and machine learning work.

[0003] In the field of computer vision, the guidance knowledge provided by the sample images used to train the image classification model is limited, and this phenomenon is further aggravated when facing a semi-supervised learning task due to the limited number of labeled images, resulting in an unsatisfactory model training effect and affecting the prediction accuracy of the image classification model.

[0004] Therefore, how to train an image classification model with higher prediction accuracy using limited labeled images is a problem that needs to be solved at present. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a training method of an image classification model, an image processing method and device, for training an image classification model with higher prediction accuracy using limited labeled images and accurately predicting the image category.

[0006] In order to achieve the above purpose, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, the embodiments of the present application provide a training method of an image classification model, comprising:

[0008] generating a cross-modal image based on a description text of a labeled image, and inputting the cross-modal image into a first classifier of an image classification model to output a predicted classification result of the cross-modal image, wherein the description text is used to describe the image content of the labeled image;

[0009] inputting the labeled image into a second classifier of the image classification model to output a predicted classification result of the labeled image;

[0010] inputting an unlabeled image into the first classifier to output a first predicted classification result of the unlabeled image, and inputting the unlabeled image into the second classifier to output a second predicted classification result of the unlabeled image;

[0011] determine a total prediction loss of the image classification model based on the class label of the labeled image and the prediction classification result of each image;

[0012] adjust the network parameters of the first classifier and the second classifier respectively based on the total prediction loss.

[0013] It can be seen that in the embodiments of the present application, by introducing the description text describing the image content of the labeled image, the cross-modal image is generated based on the description text of the labeled image, so that the cross-modal image can express the semantic information of the labeled image; the cross-modal image and the unlabeled image are taken as the input of the first classifier, and the first classifier is used for semi-supervised learning, and the labeled image and the unlabeled image are taken as the input of the second classifier, and the second classifier is used for semi-supervised learning, the prediction classification result output by the first classifier and the second classifier is used to determine the total prediction loss, and then the network parameters of the first classifier and the second classifier are adjusted based on the total prediction loss, which is equivalent to training the two classifiers from the semantic and visual modalities, which is beneficial to the image classification model to learn more knowledge, thereby enhancing the semi-supervised learning effect, and realizing the training of the image classification model with limited labeled images to obtain higher prediction accuracy.

[0014] In a second aspect, the embodiments of the present application provide an image processing method, comprising:

[0015] inputting a to-be-processed image into an image classification model to output a prediction classification result of the to-be-processed image, the prediction classification result of the to-be-processed image comprising a first prediction classification result and / or a second prediction classification result, wherein the image classification model comprises a first classifier and a second classifier, the first classifier is used for classification prediction of the to-be-processed image to obtain the first prediction classification result, the second classifier is used for classification prediction of the to-be-processed image to obtain the second prediction classification result, and the image classification model is an image classification model trained based on the method of the first aspect;

[0016] determining a class to which the to-be-processed image belongs based on the prediction classification result of the to-be-processed image.

[0017] It can be seen that, in the embodiment of the present application, by inputting the to-be-processed image into the trained image classification model, the prediction classification result of the to-be-processed image can be obtained, and then based on the obtained prediction classification result, the category shown by the to-be-processed image can be determined, which is simple, fast and efficient. In addition, in the training process of the image classification model, by introducing the description text describing the image content of the labeled image, the cross-modal image is generated based on the description text of the labeled image, so that the cross-modal image can express the semantic information of the labeled image. The cross-modal image and the unlabeled image are input into the first classifier for semi-supervised learning, and the labeled image and the unlabeled image are input into the second classifier for semi-supervised learning. The prediction classification results output by the first classifier and the second classifier are used to determine the total prediction loss, and then the network parameters of the first classifier and the second classifier are adjusted based on the total prediction loss. This is equivalent to training the two classifiers from the semantic and visual modalities, which helps the image classification model to learn more knowledge, thereby enhancing the semi-supervised learning effect and improving the prediction accuracy of the image classification model. Thus, the trained image classification model can accurately classify and predict images.

[0018] In a third aspect, an embodiment of the present application provides a device for training an image classification model, comprising:

[0019] a first prediction module configured to generate a cross-modal image based on a description text of a labeled image, and input the cross-modal image into a first classifier of an image classification model, and output a prediction classification result of the cross-modal image, wherein the description text is used to describe image content of the labeled image;

[0020] a second prediction module configured to input the labeled image into a second classifier of the image classification model, and output a prediction classification result of the labeled image;

[0021] a third prediction module configured to input an unlabeled image into the first classifier, and output a first prediction classification result of the unlabeled image, and input the unlabeled image into the second classifier, and output a second prediction classification result of the unlabeled image;

[0022] a loss determination module configured to determine a total prediction loss of the image classification model based on a category label of the labeled image and the prediction classification results of the images;

[0023] an adjustment module configured to adjust network parameters of the first classifier and the second classifier based on the total prediction loss.

[0024] In a fourth aspect, an embodiment of the present application provides an image processing device, comprising:

[0025] a fourth prediction module, configured to input a to-be-processed image into an image classification model, and output a predicted classification result of the to-be-processed image, the predicted classification result of the to-be-processed image comprising a first predicted classification result and / or a second predicted classification result, wherein the image classification model comprises a first classifier and a second classifier, the first classifier is configured to perform classification prediction on the to-be-processed image to obtain the first predicted classification result, and the second classifier is configured to perform classification prediction on the to-be-processed image to obtain the second predicted classification result, the image classification model being an image classification model trained based on the method of the first aspect;

[0026] a category determination module, configured to determine a category to which the to-be-processed image belongs based on the predicted classification result of the to-be-processed image.

[0027] In a fifth aspect, an electronic device comprises:

[0028] a processor;

[0029] a memory for storing instructions executable by the processor;

[0030] The processor is configured to execute the instructions to implement the method of the first aspect.

[0031] In a sixth aspect, a computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method of the first aspect.

[0032] In a seventh aspect, a processor;

[0033] a memory for storing instructions executable by the processor;

[0034] The processor is configured to execute the instructions to implement the method of the second aspect.

[0035] In an eighth aspect, a computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method of the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of this application and illustrate exemplary embodiments of the present application and the description thereof serve to explain the present application and do not limit the present application in any manner. In the drawings:

[0037] Figure 1 a flowchart of a training method of an image classification model according to an embodiment of the present application;

[0038] Figure 2A flowchart illustrating a training method for an image classification model provided in another embodiment of this application;

[0039] Figure 3 A schematic flowchart of an image processing method provided for one embodiment of this application;

[0040] Figure 4 A schematic diagram of the structure of a training device for an image classification model provided in one embodiment of this application;

[0041] Figure 5 A schematic diagram of the structure of an image processing apparatus provided in one embodiment of this application;

[0042] Figure 6 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] The terms "first," "second," etc., used in this specification and claims are for distinguishing similar objects and are not intended to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in this specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0045] To train a higher-accuracy image classification model using a limited number of labeled images, this application proposes a training method for an image classification model based on semi-supervised learning. By introducing descriptive text describing the content of labeled images, two classifiers are trained collaboratively from both semantic and visual modal perspectives, thereby enhancing the semi-supervised learning gain. This allows the image classification model to learn more knowledge and improve its prediction accuracy. Furthermore, the trained image classification model can accurately classify and predict images.

[0046] It should be understood that the training method and image processing method of the image classification model provided in the embodiments of this application can be executed by an electronic device or software installed in an electronic device, specifically by a terminal device or a server device.

[0047] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0048] Please refer to Figure 1 The following is a flowchart illustrating a training method for an image classification model, provided as an embodiment of this application. The method may include the following steps:

[0049] S102, Generate cross-modal images based on descriptive text of labeled images.

[0050] In this context, labeled images refer to images with category labels. An image's category label indicates the category to which the image belongs, specifically the category of the content presented in the image. For example, an image's category label could be people, animals, or scenery; or it could be a subcategory within a broader category, such as images of people expressing feelings like sadness, happiness, or anger.

[0051] The descriptive text of an image is used to describe the image content. Specifically, the descriptive text includes at least one of the following: words, sentences, and other textual descriptions that describe the image content. For example, the descriptive text for an image of a bird could describe the bird's color, shape, and other characteristics, specifically, "A bird with a whitebreast and a black crown and large wings." In this embodiment, the descriptive text for a tagged image is used to describe the image content of the tagged image.

[0052] It should be noted that in practical applications, the descriptive text of labeled images can be obtained by any appropriate technical means commonly used in the field. This application does not limit this, and will not be described in detail here.

[0053] Images themselves only provide visual modal information for image classification models to learn from. In semi-supervised learning, the supervisory information they provide is relatively weak, especially when the number of labeled images is limited. This makes image classification models prone to underfitting, preventing them from fully utilizing unlabeled images and thus affecting prediction accuracy. To address this, we can leverage the fact that descriptive text for labeled images can express their semantic information. By converting the descriptive text into corresponding cross-modal images, we can obtain images that also express the semantic information of the labeled images. Introducing these cross-modal images into the semi-supervised learning task effectively adds the semantic information of the labeled images. This allows labeled images to provide richer guidance and stronger supervisory information for the image classification model, preventing underfitting and improving prediction accuracy.

[0054] In this embodiment, the cross-modal image corresponding to the labeled image can be obtained by converting the descriptive text of the labeled image in any appropriate way. The specific conversion method can be selected according to actual needs, and this embodiment does not limit it.

[0055] To quickly convert the descriptive text of labeled images into corresponding cross-modal images and ensure that the cross-modal images can accurately express the semantic information of the labeled images, in one optional implementation, the above S102 can be specifically implemented as follows: inputting the descriptive text of labeled images into an adversarial generative network and outputting cross-modal images, wherein the adversarial network is trained based on sample images carrying category labels and their descriptive text.

[0056] More specifically, the sample image carrying the category label can be encoded to obtain the image code of the sample image (hereinafter referred to as the "first image code"), and the descriptive text of the sample image can be encoded to obtain the text code of the sample image; then, the text code of the sample image is input into the generator of the generative adversarial network, and the generator maps the text code to the image space, i.e., G. t =(φ T (t i ))≈φ I (x i ), to obtain the corresponding image code (hereinafter referred to as "second image code"), where G t Represents the generator, (φ T (t i )) represents the second image encoding, t i The descriptive text representing the sample image, φ T (t i ) represents the text encoding of the sample image, x iRepresents the sample image, φ I (x i ) represents the first image encoding;

[0057] Furthermore, the first and second image codes are input together into the discriminator of the Generative Adversarial Network (GAN). The discriminator distinguishes the sources of the two image codes and outputs a discrimination result. Next, based on the discrimination result output by the discriminator and a preset loss function, the prediction loss of the GAN is determined. Then, the network parameters of both the generator and the discriminator are adjusted based on the prediction loss of the GAN. The preset loss function can be set according to actual needs; for example, the preset loss function can be as follows:

[0058]

[0059] in, G represents the prediction loss of the adversarial generative network. t Let x denote the generator of the adversarial generative network, D denote the discriminator of the adversarial generative network, and x i Represents the sample image, Φ I (x i ) represents the first image encoding, D(Φ I (x i )) represents the first discrimination result obtained by the discriminator in judging the authenticity of the first image code, t i The descriptive text representing the sample image, Φ T (t i ) represents the text encoding of the sample image, z represents the probability density distribution function, and G t (Φ T (t i D(G), z) represents the second image encoding output by the generator based on the text encoding of the sample image, where D(G) represents the second image encoding. t (Φ T (t i (),z)) represents the second discrimination result obtained by the discriminator in judging the authenticity of the second image code. This represents the expectation of the first discrimination result corresponding to the first image code of the sample images in the sample set. This represents the expectation of the second discrimination result corresponding to the second image encoding of the sample images in the sample set.

[0060] It is worth noting that the above process is only one iteration of the Generative Adversarial Network (GAN). In practical applications, multiple iterations may be required, and therefore, the iterations can be repeated until a preset training stopping condition is met, thereby obtaining the final GAN. The preset training stopping condition could be that the predicted loss of the GAN is less than a preset loss threshold, or it could be that the number of iterations reaches a preset number, etc. This embodiment does not limit the specific conditions.

[0061] This application embodiment illustrates one specific implementation of S102 described above. Of course, it should be understood that S102 can also be implemented in other ways, and this application embodiment does not limit this implementation.

[0062] S104 inputs the cross-modal image into the first classifier of the image classification model and outputs the predicted classification result of the cross-modal image.

[0063] Since cross-modal images express the semantic information of labeled images, the prediction classification results of cross-modal images can represent the prediction classification results obtained by the first classifier after understanding the image content of the labeled images at the semantic level.

[0064] S106: Input the labeled image into the second classifier of the image classification model, and output the predicted classification result of the labeled image.

[0065] Since labeled images can reflect visual modal information, the predicted classification result of labeled images can represent the predicted classification result obtained by the second classifier after understanding the image content of the labeled image at the visual level.

[0066] S108, input the unlabeled image into the first classifier and output the first predicted classification result of the unlabeled image, and input the unlabeled image into the second classifier and output the second predicted classification result of the unlabeled image.

[0067] To train an image classification model with higher prediction accuracy using a limited number of labeled images, the image classification model in this embodiment can employ a semi-supervised learning model, the relevant structure of which is as follows: Figure 2 As shown, this image classification model includes two classifiers: a first classifier and a second classifier. The first and second classifiers are used to learn different modal information of the image. In practical applications, the first and second classifiers can have different network structures, or they can have the same network structure. Preferably, to ensure that the predicted classification results output by the first and second classifiers for the same image are consistent, the first and second classifiers have the same network structure.

[0068] Specifically, the cross-modal image corresponding to the labeled image and the unlabeled image can be used as input to the first classifier. This allows the cross-modal image to provide the first classifier with semantic information from the labeled image, guiding the first classifier to fully utilize the semantic information from the unlabeled image for classification learning. In other words, the first classifier can learn the semantic information of the image, understand and predict the image at the semantic level, and obtain the corresponding prediction result. In this embodiment, inputting the cross-modal image corresponding to the labeled image into the first classifier yields the predicted classification result of the cross-modal image, and inputting the unlabeled image into the second classifier yields the first predicted classification result of the unlabeled image.

[0069] Furthermore, labeled and unlabeled images are used as inputs to the second classifier. This allows the labeled images to provide the first classifier with visual modal information about the labeled images, guiding the second classifier to fully utilize the visual modal information of the unlabeled images for classification learning. In other words, the second classifier can learn the visual modal information of the images, understand and predict them at the visual level, and obtain the corresponding prediction results. In this embodiment, inputting a labeled image into the second classifier yields the predicted classification result for the labeled image, and inputting an unlabeled image into the second classifier yields the predicted classification result for the unlabeled image.

[0070] To enable classifiers in an image classification model to fully understand and learn from the input image and improve the model's expressive power, data augmentation can be performed on the image before it is input into the corresponding classifier, thus adding perturbation. Specifically, for unlabeled images, various data augmentation processes can be applied to obtain multiple augmented images. These augmented images are then input into a first classifier, which outputs the first predicted classification result for each augmented image. Similarly, these augmented images are input into a second classification model, which outputs the second predicted classification result for each augmented image. It is understood that different data augmentation processes introduce different levels of perturbation. Ideally, the same image, after undergoing different levels of perturbation, will output the same predicted classification result when input into the same classifier. Therefore, using the predicted classification results output by the classifier under different perturbations can serve as a supervisory function for the classifier, improving its expressive power and ultimately its prediction accuracy.

[0071] More specifically, various data augmentation processes are applied to unlabeled images. Specifically, this can be achieved by performing weak augmentation on the unlabeled image to obtain a weakly augmented image, and by performing strong augmentation on the unlabeled image to obtain a strongly augmented image. Weak augmentation can include, but is not limited to, translation and flipping, while strong augmentation can include, but is not limited to, occlusion and color transformation. It is understood that because weak augmentation introduces less perturbation and does not distort the unlabeled image, the likelihood of the classifier making incorrect predictions is lower. However, using only weakly augmented images may lead to overfitting in the image classification model, preventing the extraction of essential features. Strongly augmented images, on the other hand, introduce greater perturbation and may distort the unlabeled image, but they still retain enough features to identify the category. The prediction results from weakly augmented images can be used to enhance the prediction results from strongly augmented images, thereby further improving the expressive power of the classifier.

[0072] To further improve the expressive power of each classifier in the image classification model, optionally, before S104 above, that is, before inputting the cross-modal image corresponding to the labeled image into the first classifier, the training method of the image classification model provided in this application embodiment may further include: performing weak enhancement processing on the cross-modal image. Accordingly, S104 above can be specifically implemented as: inputting the weakly enhanced cross-modal image into the first classifier to obtain the predicted classification result of the cross-modal image.

[0073] Optionally, before S106 above, that is, before inputting the labeled image into the second classifier, the training method of the image classification model provided in this application embodiment may further include: performing weak enhancement processing on the labeled image. Accordingly, S106 above can be specifically implemented as: inputting the weakly enhanced labeled image into the second classifier to obtain the predicted classification result of the labeled image.

[0074] S110, Based on the category labels of the labeled images and the predicted classification results of each image, determine the total prediction loss of the image classification model.

[0075] In this embodiment, the total prediction loss of the image classification model is used to represent the deviation between the predicted classification result output by the image classification model when classifying the input image and the actual category to which the input image belongs.

[0076] In one optional implementation, for each classifier, there is a certain deviation between the predicted classification result output by the classifier for the input image and the actual category to which the input image belongs. Furthermore, considering that the first and second classifiers understand and predict from the semantic and visual modal perspectives of the image respectively, there may also be a certain prediction deviation between the first and second classifiers. To improve the consistency of the predicted classification results output by the first and second classifiers for the same image, and thus improve the prediction accuracy of the image classification model, the total prediction loss of the image classification model can be determined based on the prediction losses generated by the first and second classifiers respectively, as well as the prediction deviation between the first and second classifiers. Specifically, S110 above may include the following steps:

[0077] S1101, based on the category labels of labeled images, the predicted classification results of cross-modal images, and the first predicted classification results of unlabeled images, determine the first prediction loss.

[0078] Here, the first prediction loss is used to represent the prediction loss generated by the first classifier.

[0079] Considering that the first classifier performs a semi-supervised learning task based on the input image, which combines supervised learning based on cross-modal images and corresponding labeled images with category labels, and unsupervised learning based on unlabeled images, a certain prediction loss will be generated under each learning task. Therefore, more specifically, the first prediction loss includes a first supervised loss and a first unsupervised loss, wherein the first supervised loss is used to represent the prediction loss generated by the first classifier performing supervised learning, and the first unsupervised loss is used to represent the prediction loss generated by the first classifier performing unsupervised learning.

[0080] To more accurately determine the prediction loss generated by the first classifier, in one optional implementation, the unlabeled image is subjected to multiple data augmentations before being input into the first classifier. Therefore, the first predicted classification result of the unlabeled image includes the first predicted classification results of multiple augmented images. Accordingly, the first supervised loss can be determined based on the predicted classification results of the cross-modal image and the class labels of the corresponding labeled images; the first unsupervised loss can be determined based on the first predicted classification results of multiple augmented images.

[0081] More specifically, the aforementioned multiple enhanced images include a weakly enhanced image obtained by weakly enhancing the unlabeled image and a strongly enhanced image obtained by strongly enhancing the unlabeled image. Accordingly, S1101 can be specifically implemented as follows:

[0082] Step A1: Based on the first prediction classification result of the weakly enhanced image, determine the pseudo-label of the unlabeled image.

[0083] It is understandable that since unlabeled images do not have category labels, the first pseudo-label of an unlabeled image is determined based on the first predicted classification result output by the first classifier for the weakly enhanced image. This is equivalent to giving the unlabeled image an artificial label to indicate the category predicted by the first classifier for the unlabeled image.

[0084] Specifically, the predicted classification results output by the first classifier for the input image include the classification probabilities of the input image in multiple categories. Based on this, as an optional approach, the category with the highest classification probability can be selected from the first predicted classification results of the weakly enhanced image and determined as the first pseudo-label of the unlabeled image.

[0085] As another preferred option, considering that the weakly enhanced image introduces a certain amount of perturbation compared to the original unlabeled image, which may cause some errors in the predicted classification results output by the first classifier, in order to ensure that the first pseudo-label can accurately represent the category to which the unlabeled image belongs, the category corresponding to the maximum classification probability exceeding the first preset probability threshold can be selected from the first predicted classification results of the weakly enhanced image and determined as the first pseudo-label of the unlabeled image.

[0086] Step A2: Based on the category labels of the labeled images and the predicted classification results of the cross-modal images, determine the first supervised loss.

[0087] Specifically, as shown in Equation (1) below, the first supervised loss can be determined using the cross-entropy loss function, the class labels of the labeled images, and the predicted classification results of the cross-modal images:

[0088]

[0089] in, This indicates that there is a supervised loss, y ξ The category label, v, represents the category label of a labeled image. ξ v represents the predicted classification result for a cross-modal image. ξ =p1(y∣α(G) T (t ξ p1 represents the first classifier, y represents the output of the first classifier, and t ξ G represents the descriptive text for a tagged image. T G represents a generator. T (t ξ ) represents a cross-modal image, α(·) represents weak enhancement processing, α(G T (t ξ )) represents the weakly enhanced cross-modal image, max(v ξ() represents the category corresponding to the highest classification probability in the predicted classification results of cross-modal images, and B represents the number of labeled images. This represents the cross-entropy loss function.

[0090] Step A3: Determine the first unsupervised loss based on the first pseudo-label of the unlabeled image and the first predicted classification result of the strongly enhanced image.

[0091] Specifically, as shown in Equation (2) below, the first unsupervised loss can be determined using the cross-entropy loss function, the first pseudo-label of the unlabeled image, and the first predicted classification result of the strongly enhanced image:

[0092]

[0093] in, Indicates the first unsupervised loss, u ξ This represents an unlabeled image. This indicates strong enhancement processing. Let y represent the strongly enhanced image, y represent the output of the first classifier, and p1 represent the first classifier. This represents the first predicted classification result for the strongly enhanced image. Represents the cross-entropy loss function. The first pseudo-label represents the unlabeled image. q ξ The first predicted classification result for the weakly enhanced image is represented by max(q). ξ q represents the maximum classification probability in the first predicted classification result of the weakly enhanced image. ξ =p1(y∣α(u) ξ )), α(·) represents weak enhancement treatment, α(u ξ ) represents a weakly enhanced image, γ represents the first preset probability threshold, and μB represents the number of unlabeled images.

[0094] Understandably, the first classifier performs a semi-supervised learning task based on the input image. It combines supervised learning based on class labels of cross-modal images and labeled images with unsupervised learning based on unlabeled images. Each learning task will produce a certain prediction loss. Therefore, based on the predicted classification results of the first classifier for cross-modal images and the class labels of labeled images, a first supervised loss is determined, ensuring that the obtained first supervised loss accurately reflects the prediction loss generated by the first classifier during supervised learning tasks. Utilizing the different perturbation sizes introduced by weak and strong enhancement processing, and the theoretically similar prediction results obtained when the same image is input into the same classifier after different perturbations, a first pseudo-label is assigned to the unlabeled image based on the first predicted classification result of the weakly enhanced image. Then, using the first pseudo-label of the unlabeled image and the first predicted classification result of the strongly enhanced image, a first unsupervised loss is determined. This ensures that the obtained first unsupervised loss not only accurately reflects the prediction loss generated by the first classifier during unsupervised learning tasks, but also facilitates the first classifier in using the first predicted classification result of the weakly enhanced image to supervise and enhance the first predicted classification result of the strongly enhanced image during unsupervised learning, thereby improving the prediction accuracy of the first classifier.

[0095] This application illustrates one specific implementation for determining the first prediction loss. It should be understood that the first prediction loss can also be determined in other ways, and this application does not limit this approach.

[0096] S1102, based on the category label of the labeled image, the predicted classification result of the labeled image, and the second predicted classification result of the unlabeled image, determine the second prediction loss.

[0097] The second prediction loss is used to represent the prediction loss generated by the second classifier.

[0098] Considering that the second classifier performs a semi-supervised learning task based on the input image, which combines supervised learning based on labeled images and their class labels with unsupervised learning based on unlabeled images, a certain prediction loss will be generated under each learning task. Therefore, more specifically, the second prediction loss includes a second supervised loss and a second unsupervised loss, wherein the second supervised loss is used to represent the prediction loss generated by the second classifier in supervised learning, and the second unsupervised loss is used to represent the prediction loss generated by the second classifier in unsupervised learning.

[0099] To more accurately determine the prediction loss generated by the second classifier, in one optional implementation, the unlabeled image is subjected to multiple data augmentations before being input into the second classifier. Therefore, the resulting second predicted classification result for the unlabeled image includes the second predicted classification results of multiple augmented images. Accordingly, the second supervised loss can be determined based on the predicted classification results of the labeled image and the class labels of the labeled image; the first unsupervised loss can be determined based on the second predicted classification results of the multiple augmented images.

[0100] More specifically, the aforementioned multiple enhanced images include a weakly enhanced image obtained by weakly enhancing the unlabeled image and a strongly enhanced image obtained by strongly enhancing the unlabeled image. Accordingly, S1102 can be specifically implemented as follows:

[0101] Step B1: Based on the second prediction classification result of the weakly enhanced image, determine the second pseudo-label of the unlabeled image.

[0102] It is understandable that since unlabeled images do not have category labels, determining the second pseudo-label of an unlabeled image based on the second prediction classification result output by the second classifier for the weakly enhanced image is equivalent to giving the unlabeled image an artificial label to indicate the category predicted by the second classifier for the unlabeled image.

[0103] Specifically, the predicted classification results output by the second classifier for the input image include the classification probabilities of the input image in multiple categories. Based on this, as an optional approach, the category with the highest classification probability can be selected from the second predicted classification results of the weakly enhanced image and determined as the second pseudo-label for the unlabeled image.

[0104] As another preferred option, considering that the weakly enhanced image introduces a certain amount of perturbation compared to the original unlabeled image, which may cause some errors in the predicted classification results output by the second classifier, in order to ensure that the second pseudo-label can accurately represent the category to which the unlabeled image belongs, the category corresponding to the maximum classification probability exceeding the second preset probability threshold can be selected from the second predicted classification results of the weakly enhanced image and determined as the second pseudo-label of the unlabeled image.

[0105] Step B2: Based on the category labels of the labeled images and the predicted classification results of the labeled images, determine the second supervised loss.

[0106] Specifically, as shown in Equation (3) below, the second supervised loss can be determined using the cross-entropy loss function, the class labels of the labeled images, and the predicted classification results of the labeled images:

[0107]

[0108] in, This indicates that there is a second supervised loss, y ξ Indicates a labeled image x ξ Category labels, p2(y|α(x) ξ )) represents a labeled image x ξ The predicted classification result, p2 represents the second classifier, y represents the output of the second classifier, α(·) represents weak enhancement processing, α(x ξ ) represents the weakly enhanced labeled image, and B represents the labeled image x. ξ Quantity, This represents the cross-entropy loss function.

[0109] Step B3: Determine the second unsupervised loss based on the second pseudo-label of the unlabeled image and the second predicted classification result of the strongly enhanced image.

[0110] Specifically, as shown in Equation (4) below, the second unsupervised loss can be determined using the cross-entropy loss function, the second pseudo-label of the unlabeled image, and the second predicted classification result of the strongly enhanced image:

[0111]

[0112] in, This represents the second unsupervised loss, u ξ This represents an unlabeled image. The second pseudo-label represents the unlabeled image. q′ ξ q′ represents the second predicted classification result for the weakly enhanced image. ξ =p2(y∣α(u) ξ p2 represents the second classifier, y represents the output of the second classifier, α(·) represents weak enhancement processing, and α(u ξ ) represents a weakly enhanced image, max(q′) ξ ) represents the maximum classification probability in the second predicted classification result of the weakly enhanced image, and γ represents the second preset probability threshold. This indicates strong enhancement processing. Indicates a strongly enhanced image. This represents the second predicted classification result for the strongly enhanced image, where μB represents the number of unlabeled images. This represents the cross-entropy loss function.

[0113] It should be noted that in practical applications, the first preset probability threshold and the second preset probability threshold can be set according to actual needs. They can be set to the same value or different values. This application embodiment does not limit this.

[0114] Understandably, the second classifier performs a semi-supervised learning task based on the input image, which combines supervised learning based on labeled images and their class labels with unsupervised learning based on unlabeled images. Each learning task will generate a certain prediction loss. Therefore, based on the predicted classification results of the second classifier for labeled images and the category labels of the labeled images, a second supervised loss is determined, ensuring that the obtained second supervised loss accurately reflects the prediction loss generated by the second classifier during supervised learning tasks. Utilizing the different perturbation sizes introduced by weak and strong enhancement processing, and the theoretically similar prediction results obtained when the same image is input into the same classifier after different perturbations, a second pseudo-label is assigned to the unlabeled image based on the second predicted classification result of the weakly enhanced image. Then, using the second pseudo-label of the unlabeled image and the second predicted classification result of the strongly enhanced image, a second unsupervised loss is determined. This ensures that the obtained second unsupervised loss not only accurately reflects the prediction loss generated by the second classifier during unsupervised learning tasks, but also helps the second classifier to supervise and enhance the second predicted classification result of the strongly enhanced image during unsupervised learning, thereby improving the prediction accuracy of the second classifier.

[0115] This application illustrates one specific implementation for determining the second prediction loss. It should be understood that the second prediction loss can also be determined in other ways, and this application does not limit this approach.

[0116] S1103, Based on the first and second prediction classification results of the unlabeled image, determine the collaborative prediction loss.

[0117] The collaborative prediction loss represents the prediction bias between the first classifier and the second classifier. To more accurately determine the prediction loss generated by the first classifier, in one optional implementation, the first prediction classification result of the unlabeled image includes the first prediction classification results of both the weakly enhanced image and the strongly enhanced image, and the second prediction classification result of the unlabeled image includes the second prediction classification results of both the weakly enhanced image and the strongly enhanced image; correspondingly, S1103 can be specifically implemented as follows:

[0118] Step C1: Based on the first prediction classification result of the weakly enhanced image, determine the first pseudo-label of the unlabeled image.

[0119] Step C1 is similar to the specific implementation of step A1 above. For details, please refer to the specific implementation of step A1 above. It will not be repeated here.

[0120] Step C2: Based on the second prediction classification result of the weakly enhanced image, determine the second pseudo-label of the unlabeled image.

[0121] Step 2 is similar to the specific implementation of step B1 above. For details, please refer to the specific implementation of step B1 above. It will not be repeated here.

[0122] Step C3: Based on the first predicted classification result and the second pseudo-label of the strongly enhanced image, determine the first collaborative prediction sub-loss.

[0123] The first collaborative prediction sub-loss is used to represent the difference between the first predicted classification result and the second pseudo-label of the strongly enhanced image.

[0124] Specifically, as shown in the following formula (5), the first collaborative prediction sub-loss can be determined using the cross-entropy loss function, the first predicted classification result of the strongly enhanced image, and the second pseudo-label:

[0125]

[0126] in, Let q′ represent the loss of the first collaborative predictor. ξ q′ represents the second predicted classification result for the weakly enhanced image. ξ =p2(y∣α(u) ξ )), u ξ This represents an unlabeled image, α(·) represents weak enhancement processing, and α(u ξ ) represents the weakly enhanced image, p2 represents the second classifier, and max(q′) represents the second classifier. ξ ) represents the maximum classification probability in the second predicted classification result of the weakly enhanced image, and γ represents the second preset probability threshold. This indicates strong enhancement processing. p1 represents the strongly enhanced image, and p1 represents the first classifier. This represents the first predicted classification result for the strongly enhanced image. The second pseudo-label, q′, represents the unlabeled image. ξ =p2(y∣α(u) ξ )), max(q′ ξ p2 represents the maximum classification probability in the second predicted classification result of the weakly enhanced image, p2 represents the second classifier, and μB represents the number of unlabeled images. This represents the cross-entropy loss function.

[0127] Step C4: Based on the second predicted classification result of the weakly enhanced image and the first pseudo-label, determine the second collaborative prediction sub-loss.

[0128] The second collaborative prediction sub-loss is used to represent the difference between the first predicted classification result and the second pseudo-label of the strongly enhanced image.

[0129] Specifically, as shown in the following formula (6), the first collaborative prediction sub-loss can be determined using the cross-entropy loss function, the second predicted classification result of the strongly enhanced image, and the first pseudo-label:

[0130]

[0131] in, Let q represent the loss of the second collaborative predictor. ξ q represents the first predicted classification result for the weakly enhanced image. ξ =p1(y∣α(u) ξ )), u ξ This represents an unlabeled image, α(·) represents weak enhancement processing, and α(u ξ ) represents the weakly enhanced image, p1 represents the first classifier, and max(q) represents the first classifier. ξ ) represents the maximum classification probability in the first predicted classification result of the weakly enhanced image, and γ represents the first preset probability threshold. This indicates strong enhancement processing. Indicates a strongly enhanced image. This represents the second predicted classification result for the strongly enhanced image. The first pseudo-label represents the unlabeled image. μB represents the number of unlabeled images, and p2 represents the second classifier. This represents the cross-entropy loss function.

[0132] Step C5: Determine the collaborative prediction loss based on the first collaborative prediction sub-loss and the second collaborative prediction sub-loss.

[0133] Specifically, the sum of the first collaborative predictor loss and the second collaborative predictor loss can be determined as the collaborative prediction loss, that is...

[0134] It is understandable that, based on the principle that the predicted classification results of the same image input to different classifiers are theoretically the same, the perturbation introduced by weak enhancement processing and strong enhancement processing are different in magnitude, and the predicted results of the same image input to the same classifier after different perturbations are theoretically the same, a pseudo-label is assigned to the unlabeled image based on the predicted classification result of one classifier for a weakly enhanced image. Then, using this pseudo-label and the predicted classification result of another classifier for a strong image, a corresponding collaborative prediction sub-loss is determined. This ensures that the obtained collaborative prediction sub-loss not only accurately reflects the prediction loss generated by each classifier when performing unsupervised learning tasks, but also helps each classifier to supervise and enhance the prediction classification results of other classifiers for strongly enhanced images by using its own prediction classification results for weakly enhanced images, thereby improving the prediction accuracy of each classifier.

[0135] This application illustrates one specific implementation for determining the collaborative prediction loss. It should be understood that the collaborative prediction loss can also be determined in other ways, and this application does not limit this approach.

[0136] S1104, Based on the first prediction loss, the second preset loss, and the collaborative prediction loss, determine the total prediction loss of the image classification model.

[0137] Specifically, the first prediction loss, the second prediction loss, and the collaborative prediction loss can be weighted and summed to determine the total prediction loss of the image classification model.

[0138] For example, the total prediction loss of an image classification model can be determined by the following formula (7):

[0139]

[0140] in, This represents the total prediction loss of the image classification model. This indicates that there is a loss due to oversight. This indicates that there is a second supervisory loss. This indicates the first unsupervised loss. This indicates the second unsupervised loss. λ1 and λ2 represent the collaborative prediction loss, and are hyperparameters that adjust the weights of the unsupervised loss and the collaborative training loss, which can be set according to actual needs. This application does not limit this setting.

[0141] It is understandable that, since the first classifier processes and predicts the input image from a semantic level, while the second classifier processes and predicts the input image from a visual modality level, there will be a certain deviation between the prediction results output by each classifier and the true category to which the input image belongs. Therefore, the prediction deviations generated by each classifier will affect the prediction accuracy of the image classification model. Furthermore, in an ideal state, the same image input to different classifiers will produce the same prediction results, and thus the prediction deviations generated by each classifier will be the same or similar. By determining the prediction losses of the first and second classifiers respectively, as well as the collaborative prediction loss between the first and second classifiers, and then determining the total prediction loss of the image classification model based on the prediction losses of the first and second classifiers and the collaborative prediction loss, the total prediction loss of the image classification model can be determined. This total prediction loss can more accurately reflect the prediction deviation of the image classification model. Then, by using this total prediction loss to adjust the network parameters of each classifier of the image classification model, it is beneficial to improve the prediction accuracy of the image classification model.

[0142] This application illustrates one specific implementation for determining the total prediction loss. It should be understood that the total prediction loss can also be determined in other ways, and this application does not limit this approach.

[0143] S112, Based on the total prediction loss, adjust the network parameters of the first classifier and the second classifier respectively.

[0144] For each classifier, its network parameters may include, but are not limited to, the number of neurons in each network layer, the connection relationships and edge weights between neurons in different network layers, and the biases of neurons in each network layer.

[0145] Since the total prediction loss of an image classification model can reflect the difference between the predicted classification result output by the image classification model and the actual category to which the input image belongs, a backpropagation algorithm can be used to adjust the network parameters of the first and second classifiers based on the total prediction loss of the image classification model in order to obtain a high-accuracy image classification model.

[0146] More specifically, when adjusting the network parameters of the first and second classifiers using the backpropagation algorithm, the prediction loss caused by each network layer of the first and second classifiers can be determined based on the total prediction loss of the image classification model, the current network parameters of the first classifier, and the current network parameters of the second classifier. Then, with the goal of reducing the total prediction loss of the image classification model, the relevant parameters of each network layer in the first classifier and the relevant parameters of each network layer in the second classifier are adjusted layer by layer.

[0147] This application embodiment illustrates one specific implementation of the above-described S112. Of course, it should be understood that S112 can also be implemented in other ways, and this application embodiment does not limit this implementation.

[0148] It should be noted that the above process is only a single adjustment process. In practical applications, multiple adjustments may be required. Therefore, steps S104 to S112 can be repeated multiple times until a preset training stopping condition is met, thereby obtaining the final image classification model. The preset training stopping condition may be that the total prediction loss of the image classification model is less than a preset loss threshold, or it may be that the number of adjustments reaches a preset number, etc. This application embodiment does not limit this.

[0149] The image classification model training method provided in this application introduces descriptive text describing the content of labeled images, and generates cross-modal images based on the descriptive text of labeled images, so that the cross-modal images can express the semantic information of labeled images. The cross-modal images and unlabeled images are used as inputs to a first classifier, which performs semi-supervised learning. The labeled images and unlabeled images are used as inputs to a second classifier, which performs semi-supervised learning. The total prediction loss is determined using the prediction classification results output by the first and second classifiers, and then the network parameters of the first and second classifiers are adjusted using the total prediction loss. This is equivalent to training the two classifiers collaboratively from the perspectives of semantics and vision, which helps the image classification model learn more knowledge, thereby enhancing the semi-supervised learning effect and achieving an image classification model with higher prediction accuracy by training with a limited number of labeled images.

[0150] The above embodiments describe a training method for an image classification model. This method can train image classification models for different application scenarios. The unlabeled images, labeled images, and descriptive text for labeled images used in model training can be selected according to the application scenario. The application scenarios used by the training method provided in this application embodiment can include, but are not limited to, facial expression classification, natural animal classification, and handwritten digit recognition. The following detailed explanation uses facial expression classification as an example to illustrate the training method for the image classification model provided in this application embodiment.

[0151] In this scenario, labeled images can be facial images with category labels. The category labels of facial images are used to identify the expressions presented by the facial images, such as happy, angry, sad, etc. The descriptive text of these facial images is used to describe the shape of the facial features presented by the facial images, such as almond eyes, hooked nose, etc. Unlabeled images can be facial images without category labels.

[0152] After obtaining unlabeled face images, labeled face images, and descriptive text for the labeled face images, a cross-modal face image can be generated based on the descriptive text of the labeled face images. This cross-modal face image is then input into the first classifier of the image classification model, outputting the expression prediction result for the cross-modal image. Furthermore, the labeled face image is input into the second classifier of the image classification model, outputting the label prediction result for the labeled face image. The unlabeled face image is input into the first classifier, outputting the first expression prediction result for the unlabeled face image, and the unlabeled face image is input into the second classifier, outputting the second expression prediction result for the unlabeled face image. Next, based on the category labels of the labeled face images and the prediction classification results for each face image, the total prediction loss of the image classification model is determined, and based on the total prediction loss, the network parameters of the first and second classifiers are adjusted. Thus, the trained image classification model can recognize the expressions presented in face images.

[0153] Based on the training method of the image classification model shown in the above embodiments of this application, the trained image classification model can be applied to any scenario that requires image classification and prediction. The application process based on the image classification model is described in detail below.

[0154] This application also provides an image processing method that can be based on... Figure 1 The image classification model trained by the method shown is used to classify and predict the image to be processed.

[0155] Please refer to Figure 3 The following is a flowchart illustrating an image processing method according to an embodiment of this application. The method may include the following steps:

[0156] S302, input the image to be processed into the image classification model, and output the predicted classification result of the image to be processed.

[0157] The predicted classification result of the image to be processed includes a first predicted classification result and / or a second predicted classification result. The image classification model includes a first classifier and a second classifier. The first classifier is used to perform classification prediction on the image to be processed to obtain a first predicted classification result of the image to be processed; the second classifier is used to perform classification prediction on the image to be processed to obtain a second predicted classification result of the image to be processed.

[0158] S304, Based on the predicted classification results of the image to be processed, determine the category to which the image to be processed belongs.

[0159] Optionally, the category to which the image to be processed belongs can be determined based on the first predicted classification result of the image to be processed. For example, the category corresponding to the highest classification probability in the first predicted classification result of the image to be processed can be determined as the category to which the image to be processed belongs.

[0160] Alternatively, the category to which the image to be processed belongs can be determined based on the second predicted classification result of the image to be processed. For example, the category corresponding to the highest classification probability in the second predicted classification result of the image to be processed can be determined as the category to which the image to be processed belongs.

[0161] Optionally, the category to which the image to be processed belongs can be determined by combining the first and second predicted classification results of the image to be processed. For example, if the category corresponding to the highest classification probability in the first predicted classification result of the image to be processed is consistent with the category corresponding to the highest classification probability in the second predicted classification result of the image to be processed, then that category can be determined as the category to which the image to be processed belongs; or, for example, the category to which the image to be processed belongs can be determined from the intersection between the first target category set in the first predicted classification result of the image to be processed and the second target category set in the second predicted classification result of the image to be processed, wherein the first target category set may include categories whose corresponding classification probabilities exceed a corresponding preset probability threshold, the second target category set may include categories whose corresponding classification probabilities exceed a corresponding preset probability threshold, and so on.

[0162] The image processing method provided in this application provides a method that, by inputting the image to be processed into a trained image classification model, obtains a predicted classification result for the image to be processed. Based on this predicted classification result, the category of the image to be processed can be determined, achieving simplicity, speed, and high efficiency. Furthermore, during the training process of the image classification model, descriptive text describing the content of labeled images is introduced, and cross-modal images are generated based on this descriptive text, enabling these cross-modal images to express the semantic information of the labeled images. The cross-modal images and unlabeled images are used as input to a first classifier, which performs semi-supervised learning. Similarly, labeled and unlabeled images are used as input to a second classifier, which also performs semi-supervised learning. The total prediction loss is determined using the predicted classification results output by the first and second classifiers, and then the network parameters of the first and second classifiers are adjusted using this total prediction loss. This is equivalent to collaboratively training two classifiers from both semantic and visual modal perspectives, which helps the image classification model learn more knowledge, thereby enhancing the semi-supervised learning effect, improving the prediction accuracy of the image classification model, and ultimately enabling the trained image classification model to accurately classify and predict images.

[0163] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0164] In addition, with the above Figure 1 Corresponding to the training method of the image classification model shown, this application also provides a training apparatus for an image classification model. Please refer to... Figure 4 The diagram below illustrates the structure of a training apparatus 400 for an image classification model, provided in one embodiment of this application. The apparatus includes:

[0165] The first prediction module 410 is used to generate a cross-modal image based on the descriptive text of the labeled image, input the cross-modal image into the first classifier of the image classification model, and output the prediction classification result of the cross-modal image. The descriptive text is used to describe the image content of the labeled image.

[0166] The second prediction module 420 is used to input the labeled image into the second classifier of the image classification model and output the predicted classification result of the labeled image.

[0167] The third prediction module 430 is used to input the unlabeled image into the first classifier and output the first prediction classification result of the unlabeled image, and to input the unlabeled image into the second classifier and output the second prediction classification result of the unlabeled image.

[0168] The loss determination module 440 is used to determine the total prediction loss of the image classification model based on the category labels of the labeled images and the prediction classification results of each image.

[0169] The adjustment module 450 is used to adjust the network parameters of the first classifier and the second classifier respectively based on the total prediction loss.

[0170] The image classification model training device provided in this application introduces descriptive text describing the image content of labeled images, and generates cross-modal images based on the descriptive text of labeled images, so that the cross-modal images can express the semantic information of labeled images. The cross-modal images and unlabeled images are used as inputs to a first classifier, which performs semi-supervised learning. The labeled images and unlabeled images are used as inputs to a second classifier, which performs semi-supervised learning. The total prediction loss is determined using the prediction classification results output by the first and second classifiers, and then the network parameters of the first and second classifiers are adjusted using the total prediction loss. This is equivalent to training the two classifiers collaboratively from the perspectives of semantics and vision, which helps the image classification model learn more knowledge, thereby enhancing the semi-supervised learning effect and realizing an image classification model with higher prediction accuracy by training with a limited number of labeled images.

[0171] Optionally, the loss determination module includes:

[0172] The first loss determination submodule is used to determine a first prediction loss based on the category label of the labeled image, the predicted classification result of the cross-modal image, and the first predicted classification result of the unlabeled image. The first prediction loss is used to represent the prediction loss generated by the first classifier.

[0173] The second loss determination submodule is used to determine a second prediction loss based on the category label of the labeled image, the predicted classification result of the labeled image, and the second predicted classification result of the unlabeled image. The second prediction loss is used to represent the prediction loss generated by the second classifier.

[0174] The collaborative loss determination submodule is used to determine the collaborative prediction loss based on the first and second prediction classification results of the unlabeled image. The collaborative prediction loss is used to represent the prediction deviation between the first classifier and the second classifier.

[0175] The total loss determination submodule is used to determine the total prediction loss of the image classification model based on the first prediction loss, the second prediction loss, and the collaborative prediction loss.

[0176] Optionally, the device 400 further includes:

[0177] An enhancement processing module is used to perform various data enhancement processes on the unlabeled image before the third prediction module inputs the unlabeled image into the first classifier and the second classifier to obtain multiple enhanced images.

[0178] The third prediction module includes:

[0179] The first input submodule is used to input the plurality of enhanced images into the first classifier and output a first predicted classification result for each of the plurality of enhanced images, and to input the plurality of enhanced images into the second classifier and output a second predicted classification result for each of the plurality of enhanced images.

[0180] Optionally, the enhancement processing module includes:

[0181] The weak enhancement processing submodule is used to perform weak enhancement processing on the unlabeled image to obtain a weakly enhanced image;

[0182] The enhanced image processing submodule is used to perform enhanced image processing on the unlabeled image to obtain an enhanced image.

[0183] Optionally, the collaborative loss determination submodule is used for:

[0184] Based on the first prediction classification result of the weakly enhanced image, a first pseudo-label of the unlabeled image is determined, and based on the second prediction classification result of the weakly enhanced image, a second pseudo-label of the unlabeled image is determined.

[0185] Based on the first predicted classification result of the strongly enhanced image and the second pseudo-label, the first collaborative prediction sub-loss is determined;

[0186] Based on the second prediction classification result of the weakly enhanced image and the first pseudo-label, the second collaborative prediction sub-loss is determined;

[0187] The collaborative prediction loss is determined based on the first collaborative prediction sub-loss and the second collaborative prediction sub-loss.

[0188] Optionally, the first prediction loss includes a first supervised loss and a first unsupervised loss, and the first loss determination submodule is used for:

[0189] Based on the first prediction classification result of the weakly enhanced image, determine the first pseudo label of the unlabeled image;

[0190] Based on the category labels of the labeled images and the predicted classification results of the cross-modal images, the first supervised loss is determined;

[0191] The first unsupervised loss is determined based on the first pseudo-label of the unlabeled image and the first predicted classification result of the strongly enhanced image.

[0192] Optionally, the second prediction loss includes a second supervised loss and a second unsupervised loss, and the second loss determination submodule is used for:

[0193] Based on the second prediction classification result of the weakly enhanced image, determine the second pseudo-label of the unlabeled image;

[0194] The second supervised loss is determined based on the category label of the labeled image and the predicted classification result of the labeled image;

[0195] The second unsupervised loss is determined based on the second pseudo-label of the unlabeled image and the second predicted classification result of the strongly enhanced image.

[0196] Optionally, the first pseudo-label of the unlabeled image is determined in the following way:

[0197] From the first predicted classification results of the weakly enhanced image, the category corresponding to the maximum classification probability exceeding the first preset probability threshold is selected and determined as the first pseudo-label of the unlabeled image.

[0198] Optionally, the second pseudo-label of the unlabeled image is determined in the following manner:

[0199] From the second predicted classification results of the weakly enhanced image, select the category corresponding to the maximum classification probability exceeding the second preset probability threshold, and determine it as the second pseudo-label of the unlabeled image.

[0200] Optionally, the first prediction module includes:

[0201] The image generation submodule is used to input the descriptive text of the labeled image into the adversarial generative network and output the cross-modal image. The adversarial network is trained based on sample images carrying category labels and their descriptive text.

[0202] Optionally, the first classifier and the second classifier have the same network structure.

[0203] Obviously, the training device for the image classification model provided in this application embodiment can be used as described above. Figure 1 The main body executing the training method of the image classification model shown is thus able to realize the training device of the image classification model in Figure 1 The functions implemented are the same, so they will not be explained again here.

[0204] In addition, with Figure 3 Corresponding to the image processing method shown, this application also provides an image processing apparatus. Please refer to... Figure 5 The image processing apparatus 500 provided in one embodiment of this application is shown in the following schematic diagram. The apparatus includes:

[0205] The fourth prediction module 510 is used to input the image to be processed into an image classification model and output the predicted classification result of the image to be processed. The predicted classification result of the image to be processed includes a first predicted classification result and / or a second predicted classification result. The image classification model includes a first classifier and a second classifier. The first classifier is used to perform classification prediction on the image to be processed to obtain a first predicted classification result. The second classifier is used to perform classification prediction on the image to be processed to obtain a second predicted classification result. The image classification model is an image classification model trained based on the training method of the image classification model described in the embodiments of this application.

[0206] The category determination module 520 is used to determine the category to which the image to be processed belongs based on the predicted classification result of the image to be processed.

[0207] The image processing apparatus provided in this application provides a method to obtain a predicted classification result of the image to be processed by inputting it into a trained image classification model. Based on this predicted classification result, the category of the image to be processed can be determined, achieving a simple, fast, and efficient process. Furthermore, during the training of the image classification model, descriptive text describing the content of labeled images is introduced, and cross-modal images are generated based on this text. These cross-modal images can express the semantic information of the labeled images. The cross-modal images and unlabeled images are used as input to a first classifier for semi-supervised learning, and labeled and unlabeled images are used as input to a second classifier for semi-supervised learning. The total prediction loss is determined using the predicted classification results output by the first and second classifiers, and then the network parameters of the first and second classifiers are adjusted using this total prediction loss. This is equivalent to collaboratively training the two classifiers from both semantic and visual modal perspectives, which helps the image classification model learn more knowledge, thereby enhancing the semi-supervised learning effect, improving the prediction accuracy of the image classification model, and ultimately enabling accurate image classification prediction using the trained image classification model.

[0208] Obviously, the image processing apparatus provided in the embodiments of this application can be used as described above. Figure 3 The image processing method shown is the execution subject, and therefore can realize the image processing method in Figure 3 The functions implemented are the same, so they will not be explained again here.

[0209] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 6At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0210] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0211] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0212] The processor reads the corresponding computer program from non-volatile memory into main memory and then runs it, forming a training device for the image classification model at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0213] Based on the descriptive text of the labeled image, a cross-modal image is generated, and the cross-modal image is input into the first classifier of the image classification model to output the predicted classification result of the cross-modal image. The descriptive text is used to describe the image content of the labeled image.

[0214] The labeled image is input into the second classifier of the image classification model, and the predicted classification result of the labeled image is output.

[0215] The unlabeled image is input into the first classifier, and the first predicted classification result of the unlabeled image is output; the unlabeled image is input into the second classifier, and the second predicted classification result of the unlabeled image is output.

[0216] Based on the category labels of the labeled images and the predicted classification results of each image, the total prediction loss of the image classification model is determined.

[0217] Based on the total prediction loss, the network parameters of the first classifier and the second classifier are adjusted respectively.

[0218] Alternatively, the processor reads the corresponding computer program from non-volatile memory into memory and runs it, forming an image processing device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0219] The image to be processed is input into an image classification model, which outputs a predicted classification result of the image to be processed. The predicted classification result of the image to be processed includes a first predicted classification result and / or a second predicted classification result. The image classification model includes a first classifier and a second classifier. The first classifier is used to perform classification prediction on the image to be processed to obtain a first predicted classification result. The second classifier is used to perform classification prediction on the image to be processed to obtain a second predicted classification result. The image classification model is an image classification model trained based on the image classification model training method described in the embodiments of this application.

[0220] Based on the predicted classification result of the image to be processed, the category to which the image to be processed belongs is determined.

[0221] The above is as stated in this application. Figure 1 The illustrated embodiment discloses a method for training an image classification model, or as described in this application. Figure 3The image processing apparatus disclosed in the illustrated embodiments can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0222] The electronic device can also perform Figure 1 The method, and the implementation of the training device for the image classification model in Figure 1 Alternatively, the electronic device may also perform the method described in embodiment 3 and enable the image processing apparatus to... Figure 3 The functions of the embodiments shown are not described again in this application.

[0223] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0224] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by a portable electronic device including multiple applications, enable the portable electronic device to perform... Figure 1The method of the illustrated embodiment is specifically used to perform the following operations:

[0225] Based on the descriptive text of the labeled image, a cross-modal image is generated, and the cross-modal image is input into the first classifier of the image classification model to output the predicted classification result of the cross-modal image. The descriptive text is used to describe the image content of the labeled image.

[0226] The labeled image is input into the second classifier of the image classification model, and the predicted classification result of the labeled image is output.

[0227] The unlabeled image is input into the first classifier, and the first predicted classification result of the unlabeled image is output; the unlabeled image is input into the second classifier, and the second predicted classification result of the unlabeled image is output.

[0228] Based on the category labels of the labeled images and the predicted classification results of each image, the total prediction loss of the image classification model is determined.

[0229] Based on the total prediction loss, the network parameters of the first classifier and the second classifier are adjusted respectively.

[0230] Alternatively, when executed by a portable electronic device that includes multiple applications, the instruction can enable the portable electronic device to perform... Figure 3 The method of the illustrated embodiment is specifically used to perform the following operations:

[0231] The image to be processed is input into an image classification model, which outputs a predicted classification result of the image to be processed. The predicted classification result of the image to be processed includes a first predicted classification result and / or a second predicted classification result. The image classification model includes a first classifier and a second classifier. The first classifier is used to perform classification prediction on the image to be processed to obtain a first predicted classification result. The second classifier is used to perform classification prediction on the image to be processed to obtain a second predicted classification result. The image classification model is an image classification model trained based on the image classification model training method described in the embodiments of this application.

[0232] Based on the predicted classification result of the image to be processed, the category to which the image to be processed belongs is determined.

[0233] In summary, the above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

[0234] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0235] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0236] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0237] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A method for training an image classification model, the method comprising: The method comprises: generating a cross-modal image based on a description text of a labeled image, inputting the cross-modal image into a first classifier of an image classification model, and outputting a predicted classification result of the cross-modal image, wherein the description text is used to describe image content of the labeled image; inputting the labeled image into a second classifier of the image classification model, and outputting a predicted classification result of the labeled image; inputting an unlabeled image into the first classifier to output a first predicted classification result of the unlabeled image, and inputting the unlabeled image into the second classifier to output a second predicted classification result of the unlabeled image; determining a total predicted loss of the image classification model based on a category label of the labeled image and the predicted classification results of the images; and adjusting network parameters of the first classifier and the second classifier based on the total predicted loss.

2. The method of claim 1, wherein, The method further comprises: determining a first predicted loss based on the category label of the labeled image, the predicted classification result of the cross-modal image, and the first predicted classification result of the unlabeled image, wherein the first predicted loss is used to represent a predicted loss generated by the first classifier; determining a second predicted loss based on the category label of the labeled image, the predicted classification result of the labeled image, and the second predicted classification result of the unlabeled image, wherein the second predicted loss is used to represent a predicted loss generated by the second classifier; determining a collaborative predicted loss based on the first predicted classification result and the second predicted classification result of the unlabeled image, wherein the collaborative predicted loss is used to represent a prediction deviation between the first classifier and the second classifier; and determining the total predicted loss of the image classification model based on the first predicted loss, the second predicted loss, and the collaborative predicted loss.

3. The method of claim 2, wherein, Before inputting the unlabeled image into the first classifier and the second classifier, the method further comprises: performing a plurality of data augmentation processes on the unlabeled image to obtain a plurality of augmented images; and the method further comprises: inputting the plurality of augmented images into the first classifier to output a first predicted classification result of each of the plurality of augmented images, and inputting the plurality of augmented images into the second classifier to output a second predicted classification result of each of the plurality of augmented images.

4. The method of claim 3, wherein, The method further comprises: performing a weak augmentation process on the unlabeled image to obtain a weak augmented image, and performing a strong augmentation process on the unlabeled image to obtain a strong augmented image.

5. The method of claim 4, wherein, The method further comprises: determining the collaborative predicted loss based on the first predicted classification result and the second predicted classification result of the unlabeled image, wherein the collaborative predicted loss is used to represent a prediction deviation between the first classifier and the second classifier. determine a first pseudo label of the unlabeled image based on the first prediction classification result of the weak augmented image, and determine a second pseudo label of the unlabeled image based on the second prediction classification result of the weak augmented image; determine a first collaborative prediction sub-loss based on the first prediction classification result of the strong augmented image and the second pseudo label; determine a second collaborative prediction sub-loss based on the second prediction classification result of the weak augmented image and the first pseudo label; determine the collaborative prediction loss based on the first collaborative prediction sub-loss and the second collaborative prediction sub-loss.

6. The method of claim 4, wherein, The first prediction loss includes a first supervised loss and a first unsupervised loss. The first prediction loss is determined based on the category label of the labeled image, the prediction classification result of the cross-modal image, and the first prediction classification result of the unlabeled image, including: determine a first pseudo label of the unlabeled image based on the first prediction classification result of the weak augmented image; determine the first supervised loss based on the category label of the labeled image and the prediction classification result of the cross-modal image; determine the first unsupervised loss based on the first pseudo label of the unlabeled image and the first prediction classification result of the strong augmented image.

7. The method of claim 4, wherein, The second prediction loss includes a second supervised loss and a second unsupervised loss. The second prediction loss is determined based on the category label of the labeled image, the prediction classification result of the labeled image, and the second prediction classification result of the unlabeled image, including: determine a second pseudo label of the unlabeled image based on the second prediction classification result of the weak augmented image; determine the second supervised loss based on the category label of the labeled image and the prediction classification result of the labeled image; determine the second unsupervised loss based on the second pseudo label of the unlabeled image and the second prediction classification result of the strong augmented image.

8. The method of claim 1, wherein, The cross-modal image is generated based on the description text of the labeled image, including: input the description text of the labeled image into the generative adversarial network to output the cross-modal image, and the generative adversarial network is trained based on the sample image carrying the category label and its description text.

9. An image processing method characterized by, including: input the to-be-processed image into the image classification model to output the prediction classification result of the to-be-processed image, and the prediction classification result of the to-be-processed image includes a first prediction classification result and / or a second prediction classification result, wherein the image classification model includes a first classifier and a second classifier, the first classifier is used for classification prediction of the to-be-processed image to obtain the first prediction classification result, and the second classifier is used for classification prediction of the to-be-processed image to obtain the second prediction classification result; determine the category to which the to-be-processed image belongs based on the prediction classification result of the to-be-processed image.

10. A training device for an image classification model, characterized in that, including: a first prediction module, configured to generate a cross-modal image based on a description text of a labeled image, and input the cross-modal image into a first classifier of an image classification model to output a prediction classification result of the cross-modal image, and the description text is used to describe the image content of the labeled image; a second prediction module configured to input the labeled image into a second classifier of the image classification model, and output a predicted classification result of the labeled image; a third prediction module configured to input an unlabeled image into the first classifier, and output a first predicted classification result of the unlabeled image, and input the unlabeled image into the second classifier, and output a second predicted classification result of the unlabeled image; a loss determination module configured to determine a total predicted loss of the image classification model based on the class label of the labeled image and the predicted classification result of each image; an adjustment module configured to adjust network parameters of the first classifier and the second classifier respectively based on the total predicted loss.

11. An electronic device, comprising: comprise: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method of any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method of any one of claims 1-9.

Images