Training method of image classification model, image processing method and device

Abstract

The application discloses a training method of an image classification model, an image processing method and device, and is used for training an image classification model with limited labeled images to obtain higher prediction accuracy and accurately predict the image category. The training method comprises the following steps: inputting a sample set into a first classifier of an image classification model to output a predicted category embedding vector of an image in the sample set, wherein the sample set comprises unlabeled images and labeled images; inputting the sample set into a second classifier of the image classification model to output a predicted category code of the image in the sample set; determining a total prediction loss of the image classification model based on the predicted category embedding vector and the predicted category code of the image in the sample set and label information of the sample set, wherein the label information comprises a category label and a label embedding vector of the labeled image, and the category label is the encoding of the real category information of the labeled image; and adjusting the network parameters of the first classifier and the second classifier based on the total prediction loss.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to an image classification model training method and an image processing method and device. BACKGROUND

[0002] Semi-supervised learning (SSL) is a key problem in the field of pattern recognition and machine learning, and is a learning method combining supervised learning and unsupervised learning. SSL uses a large number of unlabeled samples and a limited number of labeled samples to perform pattern recognition and machine learning.

[0003] At present, the class labels of the labeled samples used for training the image classification model are usually in an encoding format, that is, the class to which the input image belongs is uniquely determined, and the number of labeled samples is limited, which can easily lead to a large amount of useful information being lost during model training, especially the similarities and differences between images of different classes, for example, the visual features of husky and wolf are similar, and the visual features of wolf and sofa are quite different, which can further cause the image classification model to fall into an incorrect training direction and fail to fully utilize the unlabeled samples, thereby resulting in an unsatisfactory model training effect and affecting the prediction accuracy of the image classification model.

[0004] Therefore, how to train an image classification model with higher prediction accuracy using limited labeled images is a problem to be solved at present. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide an image classification model training method, an image processing method and device, which are used to train an image classification model with higher prediction accuracy using limited labeled images and accurately predict the class of an image.

[0006] To achieve the above purpose, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, the embodiments of the present application provide an image classification model training method, comprising:

[0008] inputting a sample set into a first classifier of an image classification model to output a predicted class embedding vector of an image in the sample set, the predicted class embedding vector being an embedding vector of predicted class information of the corresponding image, and the sample set comprising unlabeled images and labeled images;

[0009] inputting the sample set into a second classifier of the image classification model to output a predicted class code of the image in the sample set, the predicted class code being a code of the predicted class information of the corresponding image;

[0010] Based on the predicted class embedding vectors and predicted class codes of the images in the sample set, as well as the label information of the sample set, the total prediction loss of the image classification model is determined. The label information includes the class labels and label embedding vectors of the labeled images. The class labels are the codes of the true class information of the labeled images, and the label embedding vectors are the embedding vectors of the true class information of the labeled images.

[0011] Based on the total prediction loss, the network parameters of the first classifier and the second classifier are adjusted respectively.

[0012] As can be seen, in this embodiment, by introducing category information embedding processing, the category information of the image is converted into a category embedding vector, which allows the category embedding vector to retain more potential information in the real category information, thereby reflecting the intrinsic relationship with other category information, such as differences and similarities; by introducing category information encoding, the category information of the image is converted into a corresponding code, which allows the category code to clearly represent the category of the image; by adopting an image classification model containing a first classifier and a second classifier, the first classifier performs semi-supervised learning based on labeled images and their label embedding vectors and unlabeled images, that is, using the information provided by labeled images and their label embedding vectors to the first classifier to extract useful information from unlabeled images. Information is used for self-training, thereby better learning and understanding the intrinsic relationships between different categories of images and avoiding the image classification model from falling into the wrong learning direction. At the same time, the second classifier performs semi-supervised learning based on labeled images and their category labels, as well as unlabeled images. That is, it uses the information provided by labeled images and their category labels to extract useful information from unlabeled images for its own training, and then outputs well-defined predicted category information. Thus, it is equivalent to enabling the first and second classifiers to understand and learn from the input images from different directions, which is conducive to the image classification model learning more knowledge, thereby enhancing the semi-supervised learning effect and realizing an image classification model with higher prediction accuracy by training with a limited number of labeled images.

[0013] Secondly, embodiments of this application provide an image processing method, including:

[0014] The image to be processed is input into an image classification model, which outputs a prediction result set of the image to be processed. The prediction result set includes a predicted category embedding vector and / or a predicted category code. The image classification model includes a first classifier and a second classifier. The first classifier is used to perform classification prediction on the image to be processed and to embed the obtained predicted category information to obtain a predicted category embedding vector of the image to be processed. The second classifier is used to perform classification prediction on the image to be processed and to encode the obtained predicted category information to obtain a predicted category code of the image to be processed. The image classification model is an image classification model trained based on the method described in the first aspect.

[0015] Based on the prediction result set of the image to be processed, the category to which the image to be processed belongs is determined.

[0016] As can be seen, in this embodiment, by inputting the image to be processed into the trained image classification model, a prediction result set of the image to be processed can be obtained. Then, based on the obtained prediction result set, the category to which the image to be processed belongs can be determined, which is simple, fast, and efficient. In addition, during the training process of the image classification model, by introducing category information embedding processing, the category information of the image is converted into a category embedding vector, so that the category embedding vector can retain more potential information in the real category information, thereby reflecting the intrinsic relationship between other category information, such as differences and similarities. By introducing category information encoding, the category information of the image is converted into a corresponding code, so that the category code can clearly represent the category of the image. By adopting an image classification model containing a first classifier and a second classifier, the first classifier is based on the labeled image and its label embedding vector and the unlabeled image. Semi-supervised learning involves using labeled images and their label embedding vectors to provide information for the first classifier, extracting useful information from unlabeled images for its own training. This allows the classifier to better learn and understand the intrinsic relationships between different image categories, preventing the image classification model from falling into incorrect learning directions. Simultaneously, a second classifier performs semi-supervised learning based on labeled images and their category labels, as well as unlabeled images. This means it uses the information provided by labeled images and their category labels to extract useful information from unlabeled images for its own training, outputting clearly defined predicted category information. Thus, the first and second classifiers understand and learn from the input images from different perspectives, enabling the image classification model to learn more knowledge and enhancing the effectiveness of semi-supervised learning. This allows for the achievement of an image classification model with higher prediction accuracy using a limited number of labeled images for training.

[0017] Thirdly, embodiments of this application provide a training apparatus for an image classification model, comprising:

[0018] The first prediction module is used to input the sample set into the first classifier of the image classification model and output the predicted category embedding vector of the image in the sample set. The predicted category embedding vector is the embedding vector of the predicted category information of the corresponding image. The sample set includes unlabeled images and labeled images.

[0019] The second prediction module is used to input the sample set into the second classifier of the image classification model and output the predicted category code of the image in the sample set. The predicted category code is the code of the predicted category information of the corresponding image.

[0020] The loss determination module is used to determine the total prediction loss of the image classification model based on the predicted category embedding vector and predicted category encoding of the images in the sample set and the label information of the sample set. The label information includes the category label and label embedding vector of the labeled image. The category label is the encoding of the true category information of the labeled image, and the label embedding vector is the embedding vector of the true category information of the labeled image.

[0021] An adjustment module is used to adjust the network parameters of the first classifier and the second classifier respectively based on the total prediction loss.

[0022] Fourthly, embodiments of this application provide an image processing apparatus, comprising:

[0023] The third prediction module is used to input the image to be processed into the image classification model and output a prediction result set of the image to be processed. The prediction result set includes a prediction category embedding vector and / or a prediction category code. The image classification model includes a first classifier and a second classifier. The first classifier is used to perform classification prediction on the image to be processed and to embed the obtained prediction category information to obtain a prediction category embedding vector of the image to be processed. The second classifier is used to perform classification prediction on the image to be processed and to encode the obtained prediction category information to obtain a prediction category code of the image to be processed. The image classification model is an image classification model trained based on the method described in the first aspect.

[0024] The category determination module is used to determine the category to which the image to be processed belongs based on the prediction result set of the image to be processed.

[0025] Fifthly, electronic devices, including:

[0026] processor;

[0027] Memory used to store the processor's executable instructions;

[0028] The processor is configured to execute the instructions to implement the method as described in the first aspect.

[0029] A sixth aspect is a computer-readable storage medium that, when executed by a processor of an electronic device, enables the electronic device to perform the method described in the first aspect.

[0030] The seventh aspect is the processor;

[0031] Memory used to store the processor's executable instructions;

[0032] The processor is configured to execute the instructions to implement the method as described in the second aspect.

[0033] Eighthly, a computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the method as described in the second aspect. Attached Figure Description

[0034] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0035] Figure 1 A flowchart illustrating a training method for an image classification model provided in one embodiment of this application;

[0036] Figure 2 A flowchart illustrating a training method for an image classification model provided in another embodiment of this application;

[0037] Figure 3 A schematic flowchart of an image processing method provided for one embodiment of this application;

[0038] Figure 4 A schematic diagram of the structure of a training device for an image classification model provided in one embodiment of this application;

[0039] Figure 5 A schematic diagram of the structure of an image processing apparatus provided in one embodiment of this application;

[0040] Figure 6 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] The terms "first," "second," etc., used in this specification and claims are for distinguishing similar objects and are not intended to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in this specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0043] To train a more accurate image classification model using a limited number of labeled images, this application proposes a training method for an image classification model based on semi-supervised learning. This method represents the true category information of labeled images in an embedding space to better reflect the differences and similarities between images of different categories. Then, using the representation of the image category information in the embedding space and its encoded form, the first and second classifiers of the image classification model are trained collaboratively. This enhances the semi-supervised learning gain, enabling the image classification model to learn and understand the similarities and differences between images of different categories, thereby improving the prediction accuracy of the image classification model. This application also proposes an image processing method that uses the trained image classification model to accurately classify and predict images.

[0044] It should be understood that the training method and image processing method of the image classification model provided in the embodiments of this application can be executed by an electronic device or software installed in an electronic device, specifically by a terminal device or a server device.

[0045] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0046] Please refer to Figure 1 The following is a flowchart illustrating a training method for an image classification model, provided as an embodiment of this application. The method may include the following steps:

[0047] S102 takes the sample set as input to the first classifier of the image classification model and outputs the predicted class embedding vector of the images in the sample set.

[0048] In this embodiment, the sample set includes labeled images and unlabeled images, where labeled images refer to images with label information. In practical applications, to further improve the prediction accuracy of the image classification model, the sample set may include multiple labeled images and multiple unlabeled images, and the multiple labeled images may belong to different categories.

[0049] The label information of a labeled image is used to represent the true category information of the labeled image, specifically the true category to which the content presented in the image belongs. For example, the category to which the image belongs can be people, animals, landscapes, etc.; or the category to which the image belongs can be a subcategory under a major category, such as for the major category of people, the category to which the image belongs could be depressed, happy, angry, etc.

[0050] The labeling information of a labeled image can include the category label and the label embedding vector. The category label is the encoding of the true category information of the labeled image. In practical applications, the category label can be obtained by encoding the true category information of the labeled image, for example, by performing one-hot encoding. Thus, the category label uniquely identifies the true category to which the labeled image belongs.

[0051] The label embedding vector of a labeled image is an embedding vector of the true category information of the labeled image. In practical applications, the label embedding vector of a labeled image can be obtained by embedding the true category information of the labeled image. Therefore, converting the true category information of a labeled image into a fixed-length vector representation helps to retain more potential information in the true category information, thereby facilitating the identification of differences and similarities with other category information.

[0052] Specifically, as an optional implementation, the true category label information of the labeled image can be input into the embedding model, and the label embedding vector of the labeled image can be output. The embedding model is trained based on the true category information of the sample image and the label embedding vector of the sample image.

[0053] It should be noted that, in practical applications, the embedding model can be any suitable model with embedding processing capabilities. For example, the embedding model can be a word embedding model such as GloVe or word2vec. The type of embedding model can be selected according to actual needs, and this application embodiment does not limit this. In addition, the embedding model can be trained using any suitable training method. For example, it can be trained using the real category information of the sample image as the training sample and the label embedding vector of the sample image as the label corresponding to the training sample. The specific training method can be selected according to actual needs, and this application embodiment does not limit this either.

[0054] To enable the image classification model to fully learn the differences and similarities between different image categories and output well-defined prediction results, even with a limited number of labeled images, thereby improving the prediction accuracy of the image classification model, the image classification model in this embodiment can adopt a semi-supervised learning model, the relevant structure of which is as follows: Figure 2 As shown, this image classification model includes two classifiers: a first classifier and a second classifier. The first and second classifiers are used to interpret and predict the input image from different directions, and then output the corresponding prediction results. In practical applications, the first and second classifiers can have different network structures.

[0055] Specifically, the first classifier performs classification prediction on the input image, and then embeds the obtained predicted category information to obtain the predicted category embedding vector of the image's predicted category information. In other words, the predicted category embedding vector is the embedding vector of the corresponding image's predicted category information. In S102 above, the sample set is input into the first classifier, which performs classification prediction on both labeled and unlabeled images in the sample set, and then embeds the obtained predicted category information to obtain the predicted category embedding vectors for labeled and unlabeled images.

[0056] S104 takes the sample set as input to the second classifier of the image classification model and outputs the predicted class codes of the images in the sample set.

[0057] The second classifier is used to classify and predict the input image and then encode the predicted category information, such as one-hot encoding, to obtain the predicted category code of the image. In other words, the predicted category code is the encoding of the predicted category information of the corresponding image.

[0058] In S104 above, the sample set is input into the second classifier. The second classifier performs classification prediction on the labeled and unlabeled images in the sample set respectively. Then, the obtained predicted category information is encoded to obtain the predicted category code of the labeled image and the predicted category code of the unlabeled image.

[0059] Optionally, in order to enable each classifier in the image classification model to fully understand and learn from the input image and improve the expressive power of the image classification model, data augmentation processing can be performed on the images in the sample set before inputting the sample set into the corresponding classifier, thereby adding perturbation to the image.

[0060] Specifically, for unlabeled images, various data augmentation processes can be applied to obtain multiple augmented images. These augmented images are then input into a first classifier, which outputs the predicted class embedding vectors for each augmented image. Similarly, these augmented images are input into a second classification model, which outputs the predicted class codes for each augmented image. It can be understood that different data augmentation processes introduce different levels of perturbation. Ideally, the same image, after undergoing different perturbations, will output the same prediction result when input into the same classifier. Therefore, the predicted class embedding vectors output by the first classifier under different perturbations can supervise the first classifier, improving its performance. Likewise, the predicted class codes output by the second classifier under different perturbations can supervise the second classifier, improving its performance and ultimately increasing the prediction accuracy of the image classification model.

[0061] More specifically, various data augmentation processes are applied to unlabeled images. Specifically, this can be achieved by performing weak augmentation on the unlabeled image to obtain a weakly augmented image, and by performing strong augmentation on the unlabeled image to obtain a strongly augmented image. Weak augmentation can include, but is not limited to, translation and flipping, while strong augmentation can include, but is not limited to, occlusion and color transformation. It is understood that weak augmentation introduces less perturbation and is less likely to distort the unlabeled image, thus reducing the likelihood of the classifier making incorrect predictions. However, using only weakly augmented images may lead to overfitting and prevent the classifier from extracting essential features. Strongly augmented images introduce greater perturbation and may distort the unlabeled image, but they still retain enough features to identify the category. The prediction results from weakly augmented images can guide the prediction results from strongly augmented images, thereby further improving the classifier's expressive power.

[0062] For labeled images, weak enhancement processing can be performed on the labeled images. Accordingly, in S102 above, the weakly enhanced labeled image is input into the first classifier, and the predicted category embedding vector of the labeled image is output; in S104 above, the weakly enhanced labeled image is input into the second classifier, and the predicted category code of the labeled image is output.

[0063] S106. Based on the predicted class embedding vectors and predicted class codes of the images in the sample set, as well as the label information of the sample set, determine the total prediction loss of the image classification model.

[0064] The label information of the sample set includes the category labels and label embedding vectors of the labeled images in the sample set.

[0065] In this embodiment, the total prediction loss of the image classification model is used to represent the deviation between the prediction result output by the image classification model when classifying the input image and the true category to which the input image belongs.

[0066] Considering that the first classifier and the second classifier process and predict the input image from different directions, both the first classifier and the second classifier will produce certain prediction biases. Therefore, in an optional implementation, to more accurately determine the total prediction loss of the image classification model and further improve the prediction accuracy of the image classification model, the above-mentioned S106 may include:

[0067] S161, determine the first prediction loss based on the predicted class embedding vector of the images in the sample set and the label embedding vector of the labeled images.

[0068] The first prediction loss represents the prediction loss generated by the first classifier, which reflects the difference between the prediction result obtained by the first classifier in classifying the input image and the true category of the input image.

[0069] Considering that the first classifier performs a semi-supervised learning task based on the input image, which combines supervised learning based on labeled images and their label embedding vectors with unsupervised learning based on unlabeled images, a certain prediction loss will be generated under each learning task. Therefore, more specifically, the first prediction loss includes a first supervised loss and a first unsupervised loss, wherein the first supervised loss is used to represent the prediction loss generated by the first classifier performing supervised learning, and the first unsupervised loss is used to represent the prediction loss generated by the first classifier performing unsupervised learning.

[0070] To more accurately determine the prediction loss generated by the first classifier, in one optional implementation, the unlabeled image is subjected to multiple data augmentations before being input into the first classifier. Therefore, the predicted class embedding vector of the unlabeled image includes the predicted class embedding vectors of each of the augmented images. Accordingly, the first supervised loss can be determined based on the predicted class embedding vector of the labeled image and the labeled embedding vector of the labeled image; the first unsupervised loss can be determined based on the predicted class embedding vectors of each of the augmented images.

[0071] More specifically, the aforementioned multiple enhanced images include a weakly enhanced image obtained by weakly enhancing the unlabeled image and a strongly enhanced image obtained by strongly enhancing the unlabeled image. Accordingly, S161 can be specifically implemented as follows:

[0072] Step A1: Determine the first supervised loss based on the predicted class embedding vector of the labeled image and the label embedding vector of the labeled image.

[0073] Specifically, as shown in Equation (1) below, the first supervised loss can be determined using the cosine loss function, the predicted class embedding vector of the labeled image, and the label embedding vector of the labeled image:

[0074]

[0075] in, Let C(·) denote the first supervised loss, C(·) denote the cosine loss function, and B denote the labeled image x. ξ The quantity, α(·) represents weak enhancement treatment, α(x ξ ) represents the weakly augmented labeled image, p1 represents the first classifier, p1(y|α(x) ξ )) represents the predicted class embedding vector of a labeled image, x ξ W represents the output of the first classifier. T y ξ Indicates a labeled image x ξ The label embedding vector, y ξ Indicates a labeled image x ξ Category tags, W T This represents the transpose of the embedding matrix W, which contains label embedding vectors for various labeled images.

[0076] Step A2: Based on the predicted category embedding vector of the weakly enhanced image and the embedding vector of the labeled image, determine the pseudo-label embedding vector of the unlabeled image.

[0077] It is understandable that since unlabeled images do not have category information, the pseudo-label embedding vector of the unlabeled image is determined based on the predicted category embedding vector output by the first classifier for the weakly enhanced image. This is equivalent to giving the unlabeled image an "artificial label" to represent the category predicted by the first classifier for the unlabeled image.

[0078] Specifically, the pseudo-label embedding vector of an unlabeled image can be determined as follows: First, based on the similarity between the predicted class embedding vector of the weakly enhanced image and the label embedding vectors of each labeled image in the sample set, the classification probabilities of the weakly enhanced image in various categories are determined, where the sample set includes labeled images in various categories; then, based on the classification probabilities of the weakly enhanced image in various categories, the pseudo-class information of the unlabeled image is determined; further, the pseudo-class information of the weakly enhanced image is embedded, and the resulting embedding vector is determined as the pseudo-label embedding vector of the unlabeled image.

[0079] More specifically, determining the classification probabilities of a weakly enhanced image in multiple categories can be achieved as follows: Based on the label embedding vectors of each labeled image in the sample set, an embedding matrix W is generated, where each row vector of the embedding matrix is ​​the label embedding vector of the corresponding labeled image; further, the predicted category embedding vector q of the weakly enhanced image can be calculated. ξ The cosine similarity between the embedded vectors and the row vectors of the embedding matrix is ​​used to transform the predicted class embedding vector of the weakly enhanced image into the classification probabilities of the weakly enhanced image in multiple categories. in, q represents the classification probability of a weakly enhanced image across multiple categories. ξ Let W represent the predicted category embedding vector of the weakly enhanced image, W represent the embedding matrix containing the label embedding vectors of various labeled images, and Cos Sim(·) represent the cosine similarity. Then, the classification probabilities of the weakly enhanced image in various categories are sharpened as shown in the following formula (2):

[0080]

[0081] in, This represents the classification probability of the weakly enhanced image after sharpening in various categories. This represents the classification probability of the weakly enhanced image corresponding to the r-th category. Let Ω represent the classification probability of the weakly enhanced image in the i-th category, Ω represent the weight adjustment hyperparameter, and K represent the total number of categories of labeled images in the sample set.

[0082] The method described above, which determines the pseudo-label information of an unlabeled image based on the classification probabilities of a weakly enhanced image across multiple categories, can be implemented as follows: From the classification probabilities of the weakly enhanced image across various categories, select the highest classification probability exceeding a first preset probability threshold, and determine the category information corresponding to this selected probability as the pseudo-category information of the unlabeled image. It is understood that since the weakly enhanced image introduces a certain amount of perturbation compared to the original unlabeled image, it may cause the category represented by the predicted category embedding vector output by the first classifier to be incorrect. By selecting the category corresponding to the highest classification probability exceeding the first preset probability threshold as the pseudo-category information, the deviation between the pseudo-category information and the true category information of the unlabeled image can be reduced, thereby improving the prediction accuracy of the first classifier.

[0083] Step A3: Determine the first unsupervised loss based on the pseudo-label embedding vector of the unlabeled image and the predicted class embedding vector of the strongly enhanced image.

[0084] Specifically, as shown in Equation (3) below, the first unsupervised loss can be determined using the cosine loss function, the pseudo-label embedding vector of the unlabeled image, and the predicted class embedding vector of the strongly enhanced image:

[0085]

[0086] in, Indicates the first unsupervised loss, u ξ Let C(·) represent the unlabeled image, C(·) represent the cosine loss function, and μB represent the unlabeled image u. ξ Quantity, This indicates strong enhancement processing. Let p1 represent the strongly enhanced image, p1 represent the first classifier, and y represent the output of the first classifier. This represents the predicted class embedding vector of the strongly enhanced image. Represents an unlabeled image u ξ pseudo-label embedding vector, This represents the classification probability of the weakly enhanced image after sharpening in various categories. W represents the maximum classification probability of a weakly enhanced image among the classification probabilities corresponding to each category. T γ represents the transpose of the embedding matrix W, which contains label embedding vectors of various labeled images, and γ represents the first preset probability threshold.

[0087] Understandably, the first classifier performs a semi-supervised learning task based on the input image, which combines supervised learning based on labeled images and their label embedding vectors with unsupervised learning based on unlabeled images. Each learning task will generate a certain prediction loss. Therefore, based on the predicted class embedding vector of the first classifier for labeled images and the label embedding vector of the labeled images, a first supervised loss is determined, ensuring that the obtained first supervised loss accurately reflects the prediction loss generated by the first classifier during supervised learning tasks. Utilizing the different perturbation magnitudes introduced by weak and strong enhancement processing, and the theoretically similar prediction results obtained when the same image is input into the same classifier after different perturbations, a pseudo-label embedding vector is assigned to the unlabeled image based on the predicted class embedding vector of the weakly enhanced image and the label embedding vector of the labeled image. Then, using the pseudo-label embedding vector of the unlabeled image and the predicted class embedding vector of the strongly enhanced image, a first unsupervised loss is determined. This ensures that the obtained first unsupervised loss not only accurately reflects the prediction loss generated by the first classifier during unsupervised learning tasks, but also facilitates the first classifier's use of the predicted class embedding vector of the weakly enhanced image to supervise and enhance the predicted class embedding vector of the strongly enhanced image during unsupervised learning, thereby improving the prediction accuracy of the first classifier.

[0088] This application illustrates one specific implementation for determining the first prediction loss. It should be understood that the first prediction loss can also be determined in other ways, and this application does not limit this approach.

[0089] S162, based on the predicted class encoding of images in the sample set and the class labels of labeled images, determine the second prediction loss.

[0090] The second prediction loss represents the prediction loss generated by the second classifier, which reflects the difference between the prediction result obtained by the second classifier in classifying the input image and the true category of the input image.

[0091] Considering that the second classifier performs a semi-supervised learning task based on the input image, which combines supervised learning based on labeled images and their class labels with unsupervised learning based on unlabeled images, a certain prediction loss will be generated under each learning task. Therefore, more specifically, the second prediction loss includes a second supervised loss and a second unsupervised loss, wherein the second supervised loss is used to represent the prediction loss generated by the second classifier in supervised learning, and the second unsupervised loss is used to represent the prediction loss generated by the second classifier in unsupervised learning.

[0092] To more accurately determine the prediction loss generated by the second classifier, in one optional implementation, the unlabeled image is subjected to multiple data augmentations before being input into the second classifier. Thus, the predicted class code of the unlabeled image includes the predicted class codes of each of the augmented images. Accordingly, the second supervised loss can be determined based on the predicted class codes and class labels of the labeled images; the second unsupervised loss can be determined based on the predicted class codes of each of the augmented images.

[0093] More specifically, the aforementioned multiple enhanced images include a weakly enhanced image obtained by weakly enhancing the unlabeled image and a strongly enhanced image obtained by strongly enhancing the unlabeled image. Accordingly, S162 can be specifically implemented as follows:

[0094] Step B1: Determine the second supervised loss based on the predicted class encoding and class label of the labeled image.

[0095] Specifically, as shown in Equation (4) below, the second supervised loss can be determined using the cross-entropy loss function, the predicted class encoding of the labeled image, and the class label of the labeled image:

[0096]

[0097] in, This indicates that there is a second supervisory loss. Let x represent the cross-entropy loss function. ξ B represents the number of labeled images, α(·) represents weak enhancement processing, and α(x) represents the number of labeled images. ξ p1 represents the weakly augmented labeled image, p2 represents the second classifier, and p2(y|α(x)) represents the second classifier. ξ ) represents the predicted class code of the labeled image, and y represents the output of the second classifier. ξ This indicates the category label of the labeled image.

[0098] Step B2: Based on the predicted class encoding of the weakly enhanced image, determine the pseudo-class label of the unlabeled image.

[0099] It is understandable that since unlabeled images do not have category information, the pseudo-category label of the unlabeled image is determined based on the predicted category encoding of the second classifier's output for the weakly enhanced image. This is equivalent to giving the unlabeled image an "artificial label" to indicate the category predicted by the second classifier for the unlabeled image.

[0100] Specifically, the predicted class code output by the second classifier for the input image includes the predicted class codes of the input image corresponding to multiple categories and their classification probabilities. Based on this, as an optional approach, the predicted class code with the highest classification probability can be selected from the predicted class codes of the weakly enhanced image and determined as the pseudo-class label of the unlabeled image.

[0101] As another preferred option, considering that the weakly enhanced image introduces a certain amount of perturbation compared to the original unlabeled image, which may cause some errors in the predicted category code output by the second classifier, in order to ensure that the pseudo-category label can accurately represent the category to which the unlabeled image belongs, the predicted category code with the corresponding classification probability exceeding the second preset probability threshold can be selected from the predicted category code of the weakly enhanced image and determined as the pseudo-category label of the unlabeled image.

[0102] Step B3: Determine the second unsupervised loss based on the pseudo-class labels of the unlabeled image and the predicted class encoding of the strongly enhanced image.

[0103] Specifically, as shown in Equation (5) below, the second unsupervised loss can be determined using the cross-entropy loss function, the pseudo-class label of the unlabeled image, and the predicted class encoding of the strongly enhanced image:

[0104]

[0105] in, This indicates the second unsupervised loss. Let u represent the cross-entropy loss function. ξ μB represents an unlabeled image, and μB represents an unlabeled image u. ξ Quantity, p2 represents the strongly enhanced image, p2 represents the second classifier, and y represents the output of the second classifier. Indicates strongly enhanced image Predicted category coding, Represents an unlabeled image u ξ pseudo-category labels, q′ ξ The predicted class encoding of the weakly enhanced image, max(q′) ξ ) represents the maximum classification probability corresponding to the predicted class encoding of the weakly enhanced image, q′ ξ =p2(y|α(u) ξ )), α(u ξ ) represents a weakly enhanced image, and τ represents the second preset probability threshold.

[0106] This application illustrates one specific implementation for determining the second prediction loss. It should be understood that the second prediction loss can also be determined in other ways, and this application does not limit this approach.

[0107] Understandably, the second classifier performs a semi-supervised learning task based on the input image, combining supervised learning based on labeled images and their class labels with unsupervised learning based on unlabeled images. Each learning task incurs a certain prediction loss. Therefore, a second supervised loss is determined based on the predicted class code and class label of the labeled image output by the second classifier. This ensures that the obtained second supervised loss accurately reflects the prediction loss generated by the second classifier during supervised learning. Utilizing the different perturbation sizes introduced by weak and strong enhancement processing, and the theoretically similar prediction results obtained when the same image is input into the same classifier after different perturbations, a pseudo-label is assigned to the unlabeled image based on the predicted class code of the weakly enhanced image. Then, using the pseudo-label of the unlabeled image and the predicted class code of the strongly enhanced image, a second unsupervised loss is determined. This ensures that the obtained second unsupervised loss not only accurately reflects the prediction loss generated by the second classifier during unsupervised learning but also helps the second classifier supervise the enhancement of the predicted class code of the strongly enhanced image during unsupervised learning, thereby improving the prediction accuracy of the second classifier.

[0108] S163, based on the first prediction loss and the second prediction loss, determine the total prediction loss of the image classification model.

[0109] In one alternative implementation, to more accurately determine the total prediction loss of the image classification model, the first prediction loss and the second prediction loss are weighted and summed based on the weights of the first classifier and the second classifier, respectively, to obtain the total prediction loss of the image classification model.

[0110] Specifically, the total prediction loss of the image classification model can be determined by the following formula (6):

[0111]

[0112] in, This represents the total prediction loss of the image classification model. This indicates that there is a loss due to oversight. This indicates the first unsupervised loss. This indicates that there is a second supervisory loss. Let λ represent the second unsupervised loss, λ1 represent the weights corresponding to the first classifier, and the weights corresponding to the second classifier are 1.

[0113] It is understandable that, since the first classifier processes and predicts the input image from the perspective of the embedding vector of category information, while the second classifier processes and predicts the input image from the perspective of the encoding of category information, there will be a certain deviation between the prediction results output by each classifier and the true category to which the input image belongs. Therefore, the prediction deviations generated by each classifier will affect the prediction accuracy of the image classification model. Furthermore, in an ideal state, the same image input to different classifiers will produce the same prediction results after classification and prediction, and thus the prediction deviations generated by each classifier will be the same or close. By determining the prediction losses of the first classifier and the second classifier respectively, and then determining the total prediction loss of the image classification model based on the prediction losses of the first classifier and the second classifier respectively, the total prediction loss can more accurately reflect the prediction deviation of the image classification model. Then, the network parameters of each classifier of the image classification model can be adjusted using the total prediction loss, which is beneficial to improving the prediction accuracy of the image classification model.

[0114] This application illustrates one specific implementation for determining the total prediction loss. It should be understood that the total prediction loss can also be determined in other ways, and this application does not limit this approach.

[0115] S108, based on the total prediction loss, adjust the network parameters of the first classifier and the second classifier respectively.

[0116] For each classifier, its network parameters may include, but are not limited to, the number of neurons in each network layer, the connection relationships and edge weights between neurons in different network layers, and the biases of neurons in each network layer.

[0117] Since the total prediction loss of an image classification model can reflect the difference between the prediction result output by the image classification model and the true category to which the input image belongs, in order to obtain a high-accuracy image classification model, the backpropagation algorithm can be used to adjust the network parameters of the first classifier and the second classifier based on the total prediction loss of the image classification model.

[0118] More specifically, when adjusting the network parameters of the first and second classifiers using the backpropagation algorithm, the prediction loss caused by each network layer of the first and second classifiers can be determined based on the total prediction loss of the image classification model, the current network parameters of the first classifier, and the current network parameters of the second classifier. Then, with the goal of reducing the total prediction loss of the image classification model, the relevant parameters of each network layer in the first classifier and the relevant parameters of each network layer in the second classifier are adjusted layer by layer.

[0119] This application embodiment illustrates one specific implementation of S108 described above. Of course, it should be understood that S108 can also be implemented in other ways, and this application embodiment does not limit this implementation.

[0120] It should be noted that the above process is only a single adjustment process. In practical applications, multiple adjustments may be required. Therefore, steps S102 to S108 can be repeated multiple times until a preset training stopping condition is met, thereby obtaining the final image classification model. The preset training stopping condition may be that the total prediction loss of the image classification model is less than a preset loss threshold, or it may be that the number of adjustments reaches a preset number, etc. This application embodiment does not limit this.

[0121] Specifically, after S108 above, the training method of the image classification model provided in this application embodiment may further include: if the image classification model does not meet the preset training stopping condition, then reduce the weight corresponding to the first classifier, and repeat S102 to S108 above until the image classification model meets the preset training stopping condition.

[0122] Understandably, the first classifier outputs the predicted class embedding vector of the input image, which retains more latent information from the class information of the input image. The second classifier outputs the predicted class encoding of the input image, which can clearly define the predicted class of the input image. By gradually reducing the weight of the first classifier during training, the image classification model can not only fully learn the latent information in the label information of labeled images in the early stage of training, especially the differences and similarities between images of different classes, thus alleviating the problem of insufficient supervision information provided by limited labeled images in semi-supervised training, but also enable the image classification model to output clearly defined predicted classes in the later stage of training. This achieves the goal of more smoothly and scientifically conducting semi-supervised training of the image classification model, further improving the prediction accuracy of the image classification model.

[0123] The image classification model training method provided in this application introduces category information embedding processing to convert the image's category information into category embedding vectors. This allows the category embedding vectors to retain more potential information from the real category information, thereby reflecting the intrinsic relationship between the image and other category information, such as differences and similarities. Furthermore, by introducing category information encoding, the image's category information is converted into corresponding codes, enabling the category codes to clearly represent the image's category. The method employs an image classification model that includes a first classifier and a second classifier. The first classifier performs semi-supervised learning based on labeled images and their label embedding vectors, as well as unlabeled images. Specifically, it utilizes the information provided by labeled images and their label embedding vectors to the first classifier to classify unlabeled images. Useful information is extracted for its own training, thereby better learning and understanding the intrinsic relationships between different categories of images and avoiding the image classification model from falling into the wrong learning direction. At the same time, the second classifier performs semi-supervised learning based on labeled images and their category labels as well as unlabeled images. That is, it uses the information provided by labeled images and their category labels to extract useful information from unlabeled images for its own training, and then outputs well-defined predicted category information. Thus, it is equivalent to enabling the first and second classifiers to understand and learn from the input images from different directions, which is conducive to the image classification model learning more knowledge, thereby enhancing the semi-supervised learning effect and realizing an image classification model with higher prediction accuracy by training with a limited number of labeled images.

[0124] The above embodiments describe a training method for image classification models. This method allows for the training of image classification models for different application scenarios. The sample set and its label information used for model training can be selected according to the specific application scenario. The application scenarios used by the training method provided in this application embodiment can include, but are not limited to, facial expression classification, animal classification, and handwritten digit recognition. The following detailed explanation uses facial expression classification as an example to illustrate the training method for the image classification model provided in this application embodiment.

[0125] In this scenario, a labeled image can be a face image with tagged information. The label information represents the facial expression, such as happiness, anger, or sadness. Specifically, the label information includes a category label and a label embedding vector. The category label encodes the facial expression, and the label embedding vector is the embedding vector of that expression. An unlabeled image can be a face image without tagged information.

[0126] After obtaining the sample set and its label information, steps S102 to S108 described above can be executed. Thus, the trained image classification model can recognize the facial expressions presented in a face image.

[0127] Based on the training method of the image classification model shown in the above embodiments of this application, the trained image classification model can be applied to any scenario that requires image classification and prediction. The application process based on the image classification model is described in detail below.

[0128] This application also provides an image processing method that can be based on... Figure 1 The image classification model trained by the method shown is used to classify and predict the image to be processed.

[0129] Please refer to Figure 3 The following is a flowchart illustrating an image processing method according to an embodiment of this application. The method may include the following steps:

[0130] S302: Input the image to be processed into the image classification model and output the prediction result set of the image to be processed.

[0131] The prediction result set for the image to be processed includes the predicted class embedding vector and / or predicted class encoding of the image to be processed. The image classification model includes a first classifier and a second classifier. The first classifier is used to perform classification prediction on the image to be processed and embed the obtained predicted class information to obtain the predicted class embedding vector of the image to be processed. The second classifier is used to perform classification prediction on the image to be processed and encode the obtained predicted class information to obtain the predicted class encoding of the image to be processed. The image classification model is an image classification model trained based on the training method of the image classification model described in the embodiments of this application.

[0132] S304, Based on the prediction result set of the image to be processed, determine the category to which the image to be processed belongs.

[0133] Optionally, the category to which the image to be processed belongs can be determined based on the predicted category embedding vector of the image to be processed. For example, the predicted category embedding vector of the image to be processed can be converted into the category to which the image to be processed belongs.

[0134] Alternatively, the category to which the image to be processed belongs can be determined based on the predicted category code of the image to be processed. For example, the category indicated by the predicted category code of the image to be processed can be determined as the category to which the image to be processed belongs.

[0135] Optionally, the predicted category embedding vector and the predicted category code of the image to be processed can be combined to determine the category to which the image to be processed belongs. For example, if the predicted category determined based on the predicted category embedding vector of the image to be processed is consistent with the predicted category determined based on the predicted category code of the image to be processed, then that category can be determined as the category to which the image to be processed belongs, and so on.

[0136] The image processing method provided in this application provides a prediction result set for the image to be processed by inputting it into a trained image classification model. Based on this prediction result set, the category of the image to be processed can be determined, achieving simplicity, speed, and high efficiency. Furthermore, during the training process of the image classification model, by introducing category information embedding processing, the image's category information is converted into a category embedding vector. This allows the category embedding vector to retain more potential information from the true category information, thus reflecting the inherent relationship with other category information, such as differences and similarities. By introducing category information encoding, the image's category information is converted into a corresponding code, enabling the category code to clearly represent the image's category. The method employs an image classification model containing a first classifier and a second classifier, where the first classifier bases the classification on labeled images and their label embedding vectors, as well as unlabeled images. Semi-supervised learning of images involves using labeled images and their label embedding vectors to provide information for the first classifier, extracting useful information from unlabeled images for its own training. This allows the classifier to better learn and understand the intrinsic relationships between different image categories, preventing the image classification model from falling into incorrect learning directions. Simultaneously, a second classifier performs semi-supervised learning based on labeled images and their category labels, as well as unlabeled images. This involves using the information provided by labeled images and their category labels to extract useful information from unlabeled images for its own training, ultimately outputting clearly defined predicted category information. This is equivalent to enabling the first and second classifiers to understand and learn from the input images from different perspectives, allowing the image classification model to learn more knowledge and thus enhancing the effectiveness of semi-supervised learning. This enables the image classification model to achieve higher prediction accuracy using a limited number of labeled images for training.

[0137] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0138] In addition, with the above Figure 1Corresponding to the training method of the image classification model shown, this application also provides a training apparatus for an image classification model. Please refer to... Figure 4 The diagram below illustrates the structure of a training apparatus 400 for an image classification model, provided in one embodiment of this application. The apparatus includes:

[0139] The first prediction module 410 is used to input the sample set into the first classifier of the image classification model and output the predicted category embedding vector of the image in the sample set. The predicted category embedding vector is the embedding vector of the predicted category information of the corresponding image. The sample set includes unlabeled images and labeled images.

[0140] The second prediction module 420 is used to input the sample set into the second classifier of the image classification model and output the predicted category code of the image in the sample set, wherein the predicted category code is the code of the predicted category information of the corresponding image;

[0141] The loss determination module 430 is used to determine the total prediction loss of the image classification model based on the predicted category embedding vector and predicted category encoding of the images in the sample set and the label information of the sample set. The label information includes the category label and label embedding vector of the labeled image. The category label is the encoding of the true category information of the labeled image, and the label embedding vector is the embedding vector of the true category information of the labeled image.

[0142] The adjustment module 440 is used to adjust the network parameters of the first classifier and the second classifier respectively based on the total prediction loss.

[0143] The image classification model training apparatus provided in this application introduces category information embedding processing to convert the image's category information into category embedding vectors. This allows the category embedding vectors to retain more potential information from the real category information, thereby reflecting the intrinsic relationship between the image and other category information, such as differences and similarities. By introducing category information encoding, the image's category information is converted into corresponding codes, enabling the category codes to clearly represent the image's category. The image classification model employs an image classification model containing a first classifier and a second classifier. The first classifier performs semi-supervised learning based on labeled images and their label embedding vectors, as well as unlabeled images. Specifically, it utilizes the information provided by labeled images and their label embedding vectors to the first classifier to classify unlabeled images... Useful information is extracted for its own training, thereby better learning and understanding the intrinsic relationships between different categories of images and avoiding the image classification model from falling into the wrong learning direction. At the same time, the second classifier performs semi-supervised learning based on labeled images and their category labels as well as unlabeled images. That is, it uses the information provided by labeled images and their category labels to extract useful information from unlabeled images for its own training, and then outputs well-defined predicted category information. Thus, it is equivalent to enabling the first and second classifiers to understand and learn from the input images from different directions, which is conducive to the image classification model learning more knowledge, thereby enhancing the semi-supervised learning effect and realizing an image classification model with higher prediction accuracy by training with a limited number of labeled images.

[0144] Optionally, the loss determination module includes:

[0145] The first loss determination submodule is used to determine a first prediction loss based on the predicted class embedding vector of the image in the sample set and the label embedding vector of the labeled image. The first prediction loss is used to represent the prediction loss generated by the first classifier.

[0146] The second loss determination submodule is used to determine a second prediction loss based on the predicted class encoding of the images in the sample set and the class label of the labeled images. The second prediction loss is used to represent the prediction loss generated by the second classifier.

[0147] The total loss determination submodule is used to determine the total prediction loss of the image classification model based on the first prediction loss and the second prediction loss.

[0148] Optionally, the device 400 further includes:

[0149] An enhancement processing module is used to perform various enhancement processes on the unlabeled image before the first prediction module inputs the sample set into the first classifier of the image classification model and the second prediction module inputs the sample set into the second classifier of the image classification model, to obtain multiple enhanced images.

[0150] The first prediction module includes:

[0151] The first prediction submodule is used to input the labeled image and the plurality of augmented images into the first classifier to obtain the predicted category embedding vectors of the labeled image and the plurality of augmented images respectively;

[0152] The second prediction module includes:

[0153] The second prediction submodule is used to input the labeled image and the plurality of augmented images into the second classifier to obtain the predicted category codes of the labeled image and the plurality of augmented images respectively.

[0154] Optionally, the enhancement processing module includes:

[0155] A first enhancement processing submodule is used to perform weak enhancement processing on the unlabeled image to obtain a weakly enhanced image; and...

[0156] The second enhancement processing submodule is used to perform strong enhancement processing on the unlabeled image to obtain a strongly enhanced image.

[0157] Optionally, the first prediction loss includes a first supervised loss and a first unsupervised loss, and the first loss determination submodule is used for:

[0158] The first supervised loss is determined based on the predicted class embedding vector of the labeled image and the label embedding vector of the labeled image.

[0159] Based on the predicted category embedding vector of the weakly enhanced image and the label embedding vector of the labeled image, the pseudo-label embedding vector of the unlabeled image is determined.

[0160] The first unsupervised loss is determined based on the pseudo-label embedding vector of the unlabeled image and the predicted class embedding vector of the strongly enhanced image.

[0161] Optionally, the first loss determination submodule determines the pseudo-label embedding vector of the unlabeled image based on the predicted class embedding vector of the weakly enhanced image and the label embedding vector of the labeled image, including:

[0162] Based on the similarity between the predicted class embedding vector of the weakly enhanced image and the label embedding vector of each labeled image in the sample set, the classification probability of the weakly enhanced image in multiple categories is determined, and the sample set includes labeled images of multiple categories.

[0163] Based on the classification probabilities of the weakly enhanced image in various categories, the pseudo-class information of the unlabeled image is determined;

[0164] The pseudo-category information of the weakly enhanced image is embedded, and the resulting embedding vector is determined as the pseudo-label embedding vector of the unlabeled image.

[0165] Optionally, the first loss determination submodule determines the pseudo-class information of the unlabeled image based on the classification probabilities of the weakly enhanced image in various categories, including:

[0166] The maximum classification probability exceeding a first preset probability threshold is selected from the classification probabilities corresponding to various categories of the weakly enhanced image.

[0167] The category information corresponding to the selected classification probability is determined as the pseudo-category information of the unlabeled image.

[0168] Optionally, the second prediction loss includes a second supervised loss and a second unsupervised loss, and the second loss determination submodule is used for:

[0169] Based on the predicted class code and class label of the labeled image, the second supervised loss is determined;

[0170] Based on the predicted category encoding of the weakly enhanced image, the pseudo-category label of the unlabeled image is determined;

[0171] The second unsupervised loss is determined based on the pseudo-class label of the unlabeled image and the predicted class encoding of the strongly enhanced image.

[0172] Optionally, the second loss determination submodule determines the pseudo-class label of the unlabeled image based on the predicted class encoding of the weakly enhanced image, including:

[0173] From the predicted category codes of the weakly enhanced image, select the predicted category codes whose classification probability exceeds a second preset probability threshold, and determine them as the pseudo-category labels of the unlabeled image.

[0174] Optionally, the total loss determination submodule is used for:

[0175] Based on the weights of the first classifier and the second classifier, the first prediction loss and the second prediction loss are weighted and summed to obtain the total prediction loss of the image classification model.

[0176] After adjusting the network parameters of the first classifier and the second classifier based on the total prediction loss, the method further includes:

[0177] If the image classification model does not meet the preset training stopping condition, the weights corresponding to the first classifier are reduced, and the steps of inputting the sample set into the first classifier of the image classification model and adjusting the network parameters of the first classifier and the second classifier based on the total prediction loss are repeated until the image classification model meets the preset training stopping condition.

[0178] Optionally, the device 400 further includes:

[0179] The embedding module, before the loss determination module determines the total prediction loss of the image classification model based on the predicted class embedding vectors and predicted class codes of the images in the sample set and the label information of the sample set, inputs the true class information of the labeled images into the embedding model and outputs the label embedding vectors of the labeled images. The embedding model is trained based on the true class information of the sample images and the label embedding vectors of the sample images; and / or,

[0180] Before the loss determination module determines the total prediction loss of the image classification model based on the predicted category embedding vectors and predicted category codes of the images in the sample set and the label information of the sample set, the encoding module encodes the true category information of the labeled images to obtain the category labels of the labeled images.

[0181] Obviously, the training device for the image classification model provided in this application embodiment can be used as described above. Figure 1 The main body executing the training method of the image classification model shown is thus able to realize the training device of the image classification model in Figure 1 The functions implemented are the same, so they will not be explained again here.

[0182] In addition, with Figure 3 Corresponding to the image processing method shown, this application also provides an image processing apparatus. Please refer to... Figure 5 The image processing apparatus 500 provided in one embodiment of this application is shown in the following schematic diagram. The apparatus includes:

[0183] The third prediction module 510 is used to input the image to be processed into an image classification model and output a prediction result set of the image to be processed. The prediction result set includes a prediction category embedding vector and / or a prediction category code. The image classification model includes a first classifier and a second classifier. The first classifier is used to perform classification prediction on the image to be processed and to embed the obtained prediction category information to obtain a prediction category embedding vector of the image to be processed. The second classifier is used to perform classification prediction on the image to be processed and to encode the obtained prediction category information to obtain a prediction category code of the image to be processed. The image classification model is an image classification model trained based on the image classification model training method provided in the embodiments of this application.

[0184] The category determination module 520 is used to determine the category to which the image to be processed belongs based on the prediction result set of the image to be processed.

[0185] The image processing apparatus provided in this application provides a set of prediction results for the image to be processed by inputting it into a trained image classification model. Based on this set, the category of the image to be processed can be determined, achieving simplicity, speed, and high efficiency. Furthermore, during the training of the image classification model, embedding of category information is introduced to convert the image's category information into category embedding vectors. This allows the category embedding vectors to retain more potential information from the true category information, thus reflecting the intrinsic relationships with other category information, such as differences and similarities. Encoding of category information is also introduced to convert the image's category information into corresponding codes, enabling the category codes to clearly represent the image's category. By employing an image classification model that includes a first classifier and a second classifier, the first classifier determines the category of the image based on labeled images and their label embedding vectors, as well as unlabeled images. Semi-supervised learning of images involves using labeled images and their label embedding vectors to provide information for the first classifier, extracting useful information from unlabeled images for its own training. This allows the classifier to better learn and understand the intrinsic relationships between different image categories, preventing the image classification model from falling into incorrect learning directions. Simultaneously, a second classifier performs semi-supervised learning based on labeled images and their category labels, as well as unlabeled images. This involves using the information provided by labeled images and their category labels to extract useful information from unlabeled images for its own training, ultimately outputting clearly defined predicted category information. This is equivalent to enabling the first and second classifiers to understand and learn from the input images from different perspectives, allowing the image classification model to learn more knowledge and thus enhancing the effectiveness of semi-supervised learning. This enables the image classification model to achieve higher prediction accuracy using a limited number of labeled images for training.

[0186] Obviously, the image processing apparatus provided in the embodiments of this application can be used as described above.Figure 3 The image processing method shown is the execution subject, and therefore can realize the image processing method in Figure 3 The functions implemented are the same, so they will not be explained again here.

[0187] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 6 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0188] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0189] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0190] The processor reads the corresponding computer program from non-volatile memory into main memory and then runs it, forming a training device for the image classification model at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0191] The sample set is input into the first classifier of the image classification model, and the predicted class embedding vector of the image in the sample set is output. The predicted class embedding vector is the embedding vector of the predicted class information of the corresponding image. The sample set includes unlabeled images and labeled images.

[0192] The sample set is input into the second classifier of the image classification model, and the predicted class code of the image in the sample set is output. The predicted class code is the code of the predicted class information of the corresponding image.

[0193] Based on the predicted class embedding vectors and predicted class codes of the images in the sample set, as well as the label information of the sample set, the total prediction loss of the image classification model is determined. The label information includes the class labels and label embedding vectors of the labeled images. The class labels are the codes of the true class information of the labeled images, and the label embedding vectors are the embedding vectors of the true class information of the labeled images.

[0194] Based on the total prediction loss, the network parameters of the first classifier and the second classifier are adjusted respectively.

[0195] Alternatively, the processor reads the corresponding computer program from non-volatile memory into memory and runs it, forming an image processing device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0196] The image to be processed is input into an image classification model, which outputs a prediction result set of the image to be processed. The prediction result set includes a predicted category embedding vector and / or a predicted category code. The image classification model includes a first classifier and a second classifier. The first classifier is used to perform classification prediction on the image to be processed and to embed the obtained predicted category information to obtain a predicted category embedding vector of the image to be processed. The second classifier is used to perform classification prediction on the image to be processed and to encode the obtained predicted category information to obtain a predicted category code of the image to be processed. The image classification model is an image classification model trained based on the image classification model training method described in the embodiments of this application.

[0197] Based on the prediction result set of the image to be processed, the category to which the image to be processed belongs is determined.

[0198] The above is as stated in this application. Figure 1 The illustrated embodiment discloses a method for training an image classification model, or as described in this application. Figure 3The image processing apparatus disclosed in the illustrated embodiments can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0199] The electronic device can also perform Figure 1 The method, and the implementation of the training device for the image classification model in Figure 1 Alternatively, the electronic device may also perform the method described in embodiment 3 and enable the image processing apparatus to... Figure 3 The functions of the embodiments shown are not described in detail here.

[0200] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0201] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by a portable electronic device including multiple applications, enable the portable electronic device to perform... Figure 1The method of the illustrated embodiment is specifically used to perform the following operations:

[0202] The sample set is input into the first classifier of the image classification model, and the predicted class embedding vector of the image in the sample set is output. The predicted class embedding vector is the embedding vector of the predicted class information of the corresponding image. The sample set includes unlabeled images and labeled images.

[0203] The sample set is input into the second classifier of the image classification model, and the predicted class code of the image in the sample set is output. The predicted class code is the code of the predicted class information of the corresponding image.

[0204] Based on the predicted class embedding vectors and predicted class codes of the images in the sample set, as well as the label information of the sample set, the total prediction loss of the image classification model is determined. The label information includes the class labels and label embedding vectors of the labeled images. The class labels are the codes of the true class information of the labeled images, and the label embedding vectors are the embedding vectors of the true class information of the labeled images.

[0205] Based on the total prediction loss, the network parameters of the first classifier and the second classifier are adjusted respectively.

[0206] Alternatively, when executed by a portable electronic device that includes multiple applications, the instruction can enable the portable electronic device to perform... Figure 3 The method of the illustrated embodiment is specifically used to perform the following operations:

[0207] The image to be processed is input into an image classification model, which outputs a prediction result set of the image to be processed. The prediction result set includes a predicted category embedding vector and / or a predicted category code. The image classification model includes a first classifier and a second classifier. The first classifier is used to perform classification prediction on the image to be processed and to embed the obtained predicted category information to obtain a predicted category embedding vector of the image to be processed. The second classifier is used to perform classification prediction on the image to be processed and to encode the obtained predicted category information to obtain a predicted category code of the image to be processed. The image classification model is an image classification model trained based on the image classification model training method described in the embodiments of this application.

[0208] Based on the prediction result set of the image to be processed, the category to which the image to be processed belongs is determined.

[0209] In summary, the above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

[0210] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0211] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0212] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0213] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A training method for an image classification model, characterized in that, include: The sample set is input into the first classifier of the image classification model, and the predicted class embedding vector of the image in the sample set is output. The predicted class embedding vector is the embedding vector of the predicted class information of the corresponding image. The sample set includes unlabeled images and labeled images. The sample set is input into the second classifier of the image classification model, and the predicted class code of the image in the sample set is output. The predicted class code is the code of the predicted class information of the corresponding image. Based on the predicted class embedding vectors and predicted class codes of the images in the sample set, as well as the label information of the sample set, the total prediction loss of the image classification model is determined. The label information includes the class labels and label embedding vectors of the labeled images. The class labels are the codes of the true class information of the labeled images, and the label embedding vectors are the embedding vectors of the true class information of the labeled images. The total prediction loss includes the prediction loss generated by the first classifier, which is determined based on the predicted class embedding vectors of the images in the sample set and the label embedding vectors of the labeled images. Based on the total prediction loss, the network parameters of the first classifier and the second classifier are adjusted respectively.

2. The method according to claim 1, characterized in that, The determination of the total prediction loss of the image classification model based on the predicted class embedding vectors and predicted class codes of the images in the sample set, as well as the label information of the sample set, includes: Based on the predicted category embedding vectors of the images in the sample set and the label embedding vectors of the labeled images, a first prediction loss is determined, which represents the prediction loss generated by the first classifier. Based on the predicted class encoding of the images in the sample set and the class label of the labeled images, a second prediction loss is determined, which represents the prediction loss generated by the second classifier. Based on the first prediction loss and the second prediction loss, the total prediction loss of the image classification model is determined.

3. The method according to claim 2, characterized in that, Before inputting the sample set into the first classifier of the image classification model and the second classifier of the image classification model, the method further includes: Multiple enhancement processes are applied to the unlabeled image to obtain multiple enhanced images; The first classifier that inputs the sample set into the image classification model includes: The labeled image and the plurality of augmented images are input into the first classifier to obtain the predicted category embedding vectors of the labeled image and the plurality of augmented images respectively; The second classifier that inputs the sample set into the image classification model includes: The labeled image and the plurality of augmented images are input into the second classifier to obtain the predicted category codes for each of the labeled image and the plurality of augmented images.

4. The method according to claim 3, characterized in that, The unlabeled image undergoes various enhancement processes to obtain multiple enhanced images, including: The unlabeled image is subjected to weak enhancement processing to obtain a weakly enhanced image; and, The unlabeled image is subjected to strong enhancement processing to obtain a strongly enhanced image.

5. The method according to claim 4, characterized in that, The first prediction loss includes a first supervised loss and a first unsupervised loss; The determination of the first prediction loss based on the predicted class embedding vector of the images in the sample set and the label embedding vector of the labeled images includes: The first supervised loss is determined based on the predicted class embedding vector of the labeled image and the label embedding vector of the labeled image. Based on the predicted category embedding vector of the weakly enhanced image and the label embedding vector of the labeled image, the pseudo-label embedding vector of the unlabeled image is determined. The first unsupervised loss is determined based on the pseudo-label embedding vector of the unlabeled image and the predicted class embedding vector of the strongly enhanced image.

6. The method according to claim 5, characterized in that, The step of determining the pseudo-label embedding vector of the unlabeled image based on the predicted class embedding vector of the weakly enhanced image and the label embedding vector of the labeled image includes: Based on the similarity between the predicted class embedding vector of the weakly enhanced image and the label embedding vector of each labeled image in the sample set, the classification probability of the weakly enhanced image in multiple categories is determined, and the sample set includes labeled images of multiple categories. Based on the classification probabilities of the weakly enhanced image in various categories, the pseudo-class information of the unlabeled image is determined; The pseudo-category information of the weakly enhanced image is embedded, and the resulting embedding vector is determined as the pseudo-label embedding vector of the unlabeled image.

7. The method according to claim 6, characterized in that, The step of determining the pseudo-class information of the unlabeled image based on the classification probabilities of the weakly enhanced image in various categories includes: The maximum classification probability exceeding a first preset probability threshold is selected from the classification probabilities corresponding to various categories of the weakly enhanced image. The category information corresponding to the selected classification probability is determined as the pseudo-category information of the unlabeled image.

8. The method according to claim 4, characterized in that, The second prediction loss includes a second supervised loss and a second unsupervised loss; The determination of the second prediction loss based on the predicted class encoding of the images in the sample set and the class labels of the labeled images includes: Based on the predicted class code and class label of the labeled image, the second supervised loss is determined; Based on the predicted category encoding of the weakly enhanced image, the pseudo-category label of the unlabeled image is determined; The second unsupervised loss is determined based on the pseudo-class label of the unlabeled image and the predicted class encoding of the strongly enhanced image.

9. The method according to claim 2, characterized in that, The step of determining the total prediction loss of the image classification model based on the first prediction loss and the second prediction loss includes: Based on the weights of the first classifier and the second classifier, the first prediction loss and the second prediction loss are weighted and summed to obtain the total prediction loss of the image classification model. After adjusting the network parameters of the first classifier and the second classifier based on the total prediction loss, the method further includes: If the image classification model does not meet the preset training stopping condition, the weights corresponding to the first classifier are reduced, and the steps of inputting the sample set into the first classifier of the image classification model and adjusting the network parameters of the first classifier and the second classifier based on the total prediction loss are repeated until the image classification model meets the preset training stopping condition.

10. An image processing method, characterized in that, include: The image to be processed is input into an image classification model, which outputs a prediction result set of the image to be processed. The prediction result set includes a prediction category embedding vector and / or a prediction category code. The image classification model includes a first classifier and a second classifier. The first classifier is used to perform classification prediction on the image to be processed and to embed the obtained prediction category information to obtain a prediction category embedding vector of the image to be processed. The second classifier is used to perform classification prediction on the image to be processed and to encode the obtained prediction category information to obtain a prediction category code of the image to be processed. Based on the prediction result set of the image to be processed, the category to which the image to be processed belongs is determined.

11. A training device for an image classification model, characterized in that, include: The first prediction module is used to input the sample set into the first classifier of the image classification model and output the predicted category embedding vector of the image in the sample set. The predicted category embedding vector is the embedding vector of the predicted category information of the corresponding image. The sample set includes unlabeled images and labeled images. The second prediction module is used to input the sample set into the second classifier of the image classification model and output the predicted category code of the image in the sample set. The predicted category code is the code of the predicted category information of the corresponding image. The loss determination module is used to determine the total prediction loss of the image classification model based on the predicted class embedding vectors and predicted class codes of the images in the sample set and the label information of the sample set. The label information includes the class labels and label embedding vectors of the labeled images. The class labels are the codes of the true class information of the labeled images, and the label embedding vectors are the embedding vectors of the true class information of the labeled images. The total prediction loss includes the prediction loss generated by the first classifier, and the prediction loss is determined based on the predicted class embedding vectors of the images in the sample set and the label embedding vectors of the labeled images. An adjustment module is used to adjust the network parameters of the first classifier and the second classifier respectively based on the total prediction loss.

12. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1 to 10.

Images