An image classification method, device, storage medium and terminal
By combining self-supervised and semi-supervised learning, a pseudo-labeled dataset is generated using an unlabeled dataset, and the model is trained using a labeled dataset. This solves the problem of low data utilization in image classification by deep learning models and improves classification accuracy and feature extraction capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TERMINUSBEIJING TECH CO LTD
- Filing Date
- 2021-12-31
- Publication Date
- 2026-06-09
AI Technical Summary
In existing technologies, deep learning models suffer from low data utilization and local optima in image classification tasks due to a lack of labeled data, which affects classification accuracy.
We employ a combination of self-supervised learning, supervised learning, and semi-supervised learning. We use unlabeled datasets for self-supervised learning to generate pseudo-labeled datasets, and then combine them with labeled datasets for model training, thereby improving the model's feature extraction capabilities and classification accuracy.
By using self-supervised learning on massive unlabeled datasets and supervised learning on a small number of labeled datasets, the model's classification accuracy was improved, the utilization rate of unlabeled data was increased, and the model's global feature extraction capability was enhanced.
Smart Images
Figure CN114494718B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of machine learning technology, and in particular to an image classification method, apparatus, storage medium, and terminal. Background Technology
[0002] With advancements in machine learning, image classification models, once trained, can categorize images based on their content. The accuracy of image classification is typically related to the training level of the image classification model.
[0003] In current technologies, supervised deep learning has achieved successful progress on many tasks. It typically requires pre-training on large-scale labeled data, followed by targeted training on downstream tasks using a small amount of labeled data from specific scenarios. While this learning approach yields excellent performance on specific tasks, it demands labeled data from those scenarios for training. However, data labeling often requires significant human intervention, meaning that in practice only a small portion of the data can be labeled and used for training. A large amount of unlabeled data remains unused, leading to the model easily getting trapped in local optima rather than global optima. This also results in low data utilization, ultimately reducing the model's classification accuracy. Summary of the Invention
[0004] This application provides an image classification method, apparatus, storage medium, and terminal. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general description, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.
[0005] In a first aspect, embodiments of this application provide an image classification method, the method comprising:
[0006] Obtain the target image to be classified;
[0007] The target image is input into a pre-trained image classification model. The pre-trained image classification model is generated by training through self-supervised learning, supervised learning, and semi-supervised learning in sequence. Self-supervised learning is trained on an unlabeled dataset, supervised learning is trained on a labeled dataset, and semi-supervised learning is trained on both a pseudo-labeled dataset and a labeled dataset. The pseudo-labeled dataset is generated based on the unlabeled dataset.
[0008] Output the image category corresponding to the target image.
[0009] Optionally, a pre-trained image classification model is generated by following these steps:
[0010] Obtain the dataset; the dataset includes both unlabeled and labeled datasets;
[0011] Create a classification model, and build an encoder and a momentum encoder based on the model parameters of the classification model;
[0012] The trained encoder is obtained by performing self-supervised learning based on the unlabeled dataset, the encoder, and the momentum encoder.
[0013] The parameters of the trained encoder are loaded into the classification model, and the fully connected layers of the classification model are initialized to obtain the first classification model.
[0014] The labeled dataset is input into the first classification model for supervised learning, resulting in the trained first classification model.
[0015] The first classification model after preprocessing training is obtained;
[0016] The labeled dataset is input into the preprocessed first classification model for semi-supervised learning to obtain a pre-trained image classification model.
[0017] Optionally, self-supervised learning is performed based on the unlabeled dataset, the encoder, and the momentum encoder to obtain the trained encoder, including:
[0018] Initialize a queue of a preset size;
[0019] Divide the unlabeled dataset into multiple sub-datasets;
[0020] To identify a target subset of data from multiple subsets;
[0021] Image transformations are performed on the target subset to obtain first transformed data and second transformed data;
[0022] The first transformation data and the second transformation data are respectively input into the encoder and the momentum encoder, and the first embedded characterization result and the second embedded characterization result are output.
[0023] The first embedded representation result and the second embedded representation result are respectively subjected to dimensional expansion to obtain the first expansion result and the second expansion result;
[0024] Calculate the feature similarity of positive samples based on the first and second expansion results;
[0025] The features in the queue are permuted to obtain the permutation matrix, and the similarity of negative sample features is calculated based on the permutation matrix and the first embedded representation result.
[0026] The self-supervised learning loss value is calculated based on the similarity of positive sample features and the similarity of negative sample features. When the self-supervised learning loss value reaches the preset value, the trained encoder is obtained.
[0027] Optionally, when the self-supervised learning loss value reaches a preset value, the trained encoder is obtained, including:
[0028] When the self-supervised learning loss value does not reach the preset value, the encoder is backpropagated based on the self-supervised learning loss value to update the encoder parameters;
[0029] Continue executing the step of identifying a target subset from multiple subsets until the self-supervised learning loss value reaches the preset value.
[0030] Optionally, the pre-trained first classification model is preprocessed to obtain the pre-processed first classification model, including:
[0031] Determine the backbone network and the first fully connected layer of the first classification model after training;
[0032] Construct a second fully connected layer with the same structure as the first fully connected layer;
[0033] The second fully connected layer is connected to the last layer of the backbone network to obtain the second classification model;
[0034] The parameters of the backbone network and the first fully connected layer in the second classification model are fixed, and all Dropout layers in the second classification model are enabled to obtain the preprocessed first classification model.
[0035] Optionally, the labeled dataset can be input into the preprocessed first classification model for semi-supervised learning to obtain a pre-trained image classification model, including:
[0036] The labeled data is input into the preprocessed first classification model for multiple parallel calculations, and multiple first target prediction values are output.
[0037] Calculate the first mean and the first standard deviation for each first target prediction value, and calculate the semi-supervised learning loss value based on the first mean and the first standard deviation;
[0038] When the semi-supervised learning loss value reaches the preset value, a third classification model is obtained;
[0039] Enable the Dropout layer in the second fully connected layer of the third classification model;
[0040] Disabling the Dropout layers in all layers except the second fully connected layer in the third classification model yields the preprocessed third classification model.
[0041] The unlabeled data is input into the preprocessed third classification model for multiple parallel calculations, and multiple second target probability values and random uncertainty parameters are output.
[0042] The second mean and second standard deviation are calculated based on each second target probability value;
[0043] The pseudo-label dataset is obtained based on the random uncertainty parameter, the second mean, and the second standard deviation.
[0044] When the pseudo-labels meet multiple preset conditions, the pseudo-label dataset is added to the labeled dataset to obtain the target dataset;
[0045] The target dataset is input into the first classification model for supervised learning to obtain a pre-trained image classification model.
[0046] Optionally, when the semi-supervised learning loss value reaches a preset value, a third classification model is obtained, including:
[0047] If the semi-supervised learning loss value does not reach the preset value, continue to execute the step of inputting labeled data into the preprocessed first classification model for multiple parallel calculations.
[0048] Secondly, embodiments of this application provide an image classification apparatus, the apparatus comprising:
[0049] The image acquisition module is used to acquire the target image to be classified.
[0050] The image input module is used to input the target image into a pre-trained image classification model. The pre-trained image classification model is generated by training through self-supervised learning, supervised learning, and semi-supervised learning in sequence. Self-supervised learning is trained on an unlabeled dataset, supervised learning is trained on a labeled dataset, and semi-supervised learning is trained on both a pseudo-labeled dataset and a labeled dataset. The pseudo-labeled dataset is generated based on the unlabeled dataset.
[0051] The category output module is used to output the image category corresponding to the target image.
[0052] Thirdly, embodiments of this application provide a computer storage medium storing multiple instructions adapted for loading and execution of the above-described method steps by a processor.
[0053] Fourthly, embodiments of this application provide a terminal that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed by the above-described method steps.
[0054] The technical solutions provided in this application embodiment may include the following beneficial effects:
[0055] In this embodiment, the image classification device first acquires the target image to be classified, and then inputs the target image into a pre-trained image classification model. The pre-trained image classification model is generated through self-supervised learning, supervised learning, and semi-supervised learning in sequence. Self-supervised learning is trained on an unlabeled dataset, supervised learning is trained on a labeled dataset, and semi-supervised learning is trained on both a pseudo-labeled dataset and a labeled dataset. The pseudo-labeled dataset is generated based on the unlabeled dataset. Finally, the image category corresponding to the target image is output. Because this application first uses a massive unlabeled dataset in a specific scenario to perform self-supervised learning on the neural network model, allowing the model to "access" global data as much as possible to obtain more comprehensive feature extraction capabilities, and then uses a small amount of labeled data in a specific and similar scenario to perform supervised learning on the pre-trained model to further optimize the model, and finally generates a pseudo-labeled dataset from the unlabeled dataset and combines it with the labeled dataset to further train the model, the utilization rate of unlabeled data is improved, resulting in higher classification accuracy.
[0056] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description
[0057] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0058] Figure 1 This is a flowchart illustrating an image classification method provided in an embodiment of this application;
[0059] Figure 2 This is a flowchart illustrating a training method for an image classification network provided in an embodiment of this application;
[0060] Figure 3 This is a schematic block diagram illustrating the training process of an image classification network provided in an embodiment of this application;
[0061] Figure 4 This is a schematic diagram of the structure of an image classification device provided in an embodiment of this application;
[0062] Figure 5 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. Detailed Implementation
[0063] The following description and accompanying drawings fully illustrate specific embodiments of the invention to enable those skilled in the art to practice them.
[0064] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.
[0065] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.
[0066] In the description of this invention, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances. Furthermore, in the description of this invention, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0067] This application provides an image classification method, apparatus, storage medium, and terminal to address the problems existing in the aforementioned related technologies. In the technical solution provided by this application, the neural network model is first subjected to self-supervised learning using a massive unlabeled dataset in a specific scenario, allowing the model to "access" global data as much as possible to obtain more comprehensive feature extraction capabilities. Then, the pre-trained model is further optimized using a small amount of labeled data in a specific and similar scenario for supervised learning. Finally, a pseudo-labeled dataset is generated from the unlabeled dataset and combined with the labeled dataset to further train the model, thereby improving the utilization rate of unlabeled data and resulting in higher classification accuracy. Exemplary embodiments are described in detail below.
[0068] The following will be combined with the appendix Figure 1 - Appendix Figure 3 This application provides a detailed description of the image classification method provided in its embodiments. This method can be implemented using a computer program and can run on an image classification device based on the von Neumann architecture. The computer program can be integrated into an application or run as a standalone utility application.
[0069] Please see Figure 1 This is a flowchart illustrating an image classification method provided in an embodiment of this application. Figure 1As shown, the method in this application embodiment may include the following steps:
[0070] S101, Obtain the target image to be classified;
[0071] In one possible implementation, the target image to be classified can be of any type, format, and size; this application embodiment does not limit this. The user terminal stores at least one image, and can directly retrieve an image from its storage space and identify that image as the target image to be classified. The user terminal can also provide an image upload entry point, allowing the user to upload an image, which the user terminal then identifies as the target image to be classified. Of course, other methods can also be used to obtain the target image to be classified; this application embodiment does not limit this method.
[0072] S102, Input the target image into the pre-trained image classification model;
[0073] The pre-trained image classification model is generated through self-supervised learning, supervised learning, and semi-supervised learning in sequence. Self-supervised learning is trained on an unlabeled dataset, supervised learning is trained on a labeled dataset, and semi-supervised learning is trained on both a pseudo-labeled dataset and a labeled dataset. The pseudo-labeled dataset is generated based on the unlabeled dataset.
[0074] Generally, self-supervised learning is a model training method that mainly uses auxiliary tasks (pretext) to mine its own supervision information from large-scale unsupervised data. The network is trained with this constructed supervision information, so that it can learn representations that are valuable for downstream tasks. Supervised learning is a method of deriving prediction functions from labeled training data. Semi-supervised learning is a machine learning method that uses a small amount of labeled data and a large amount of unlabeled data for model training.
[0075] In this embodiment, when generating a pre-trained image classification model, a dataset is first obtained, which includes an unlabeled dataset and a labeled dataset. A classification model is then created, and an encoder and a momentum encoder are constructed based on the model parameters of the classification model. Self-supervised learning is then performed based on the unlabeled dataset, the encoder, and the momentum encoder to obtain a trained encoder. The parameters of the trained encoder are then loaded onto the classification model, and the fully connected layers of the classification model are initialized to obtain a first classification model. Next, the labeled dataset is input into the first classification model for supervised learning to obtain a trained first classification model. The trained first classification model is then preprocessed to obtain a preprocessed first classification model. Finally, the labeled dataset is input into the preprocessed first classification model for semi-supervised learning to obtain a pre-trained image classification model.
[0076] In one possible implementation, after obtaining the target image to be classified based on step S101, the user terminal extracts the target image classification model from its storage space, which is the pre-trained image classification model obtained by the user terminal. The training process of this image classification model is as follows: Figure 2 The embodiments shown are illustrated and will not be repeated here.
[0077] S103, output the image category corresponding to the target image.
[0078] In one possible implementation, after the target image is identified and processed based on the target image classification model, multiple reference categories corresponding to the target image and the probability of each reference category are obtained. The reference category with the highest probability value that meets the target requirements can be output as the final category.
[0079] In this embodiment, the image classification device first acquires the target image to be classified, and then inputs the target image into a pre-trained image classification model. The pre-trained image classification model is generated through self-supervised learning, supervised learning, and semi-supervised learning in sequence. Self-supervised learning is trained on an unlabeled dataset, supervised learning is trained on a labeled dataset, and semi-supervised learning is trained on both a pseudo-labeled dataset and a labeled dataset. The pseudo-labeled dataset is generated based on the unlabeled dataset. Finally, the image category corresponding to the target image is output. Because this application first uses a massive unlabeled dataset in a specific scenario to perform self-supervised learning on the neural network model, allowing the model to "access" global data as much as possible to obtain more comprehensive feature extraction capabilities, and then uses a small amount of labeled data in a specific and similar scenario to perform supervised learning on the pre-trained model to further optimize the model, and finally generates a pseudo-labeled dataset from the unlabeled dataset and combines it with the labeled dataset to further train the model, the utilization rate of unlabeled data is improved, resulting in higher classification accuracy.
[0080] Please see Figure 2 This is a flowchart illustrating an image classification model training method provided in this application. Figure 2 As shown, the method in this application embodiment may include the following steps:
[0081] S201, Obtain the dataset; the dataset includes unlabeled datasets and labeled datasets;
[0082] In one possible implementation, the dataset is a large-scale dataset specific to a particular scenario. ,in For unlabeled datasets, For labeled data, Number of samples Much larger Number of samples .
[0083] S202, Create a classification model and build an encoder and momentum encoder based on the model parameters of the classification model;
[0084] In one possible implementation, a classification model is first created. This classification model By backbone network and fully connected layer The system consists of a backbone network responsible for extracting sample features, and fully connected layers responsible for predicting class probabilities based on these features. Then, the two structures and the classification model are initialized. The models have the same backbone network parameters, one of which is the encoder. The other is a momentum encoder. Finally, initialize a queue of size K to store... The extracted embedded representation is represented as C is The final output channel dimension.
[0085] S203, self-supervised learning is performed based on the unlabeled dataset, the encoder, and the momentum encoder to obtain the trained encoder;
[0086] In this embodiment, during self-supervised learning, the unlabeled dataset is first divided into multiple subsets, and a target subset is determined from these subsets. Then, image transformation is performed on the target subset to obtain first transformed data and second transformed data. The first transformed data and second transformed data are then input into the encoder and momentum encoder, respectively, to output the first embedded representation result and the second embedded representation result. Next, the first embedded representation result and the second embedded representation result are dimensionally expanded to obtain the first expanded result and the second expanded result. The positive sample feature similarity is then calculated based on the first expanded result and the second expanded result. The features in the queue are then permuted to obtain the permutation matrix. The negative sample feature similarity is then calculated based on the permutation matrix and the first embedded representation result. Finally, the self-supervised learning loss value is calculated based on the positive sample feature similarity and the negative sample feature similarity. When the self-supervised learning loss value reaches a preset value, the trained encoder is obtained.
[0087] Furthermore, when the self-supervised learning loss value does not reach the preset value, the encoder is backpropagated based on the self-supervised learning loss value to update the encoder parameters, and the step of determining a target subset of the dataset from multiple subsets continues until the self-supervised learning loss value reaches the preset value.
[0088] In one possible implementation, the unlabeled dataset Randomly divide the dataset into multiple subsets of the same size. The model is trained using minimum batch processing, with each subset of data processed in batches. The model then updates the parameters once.
[0089] exist Before inputting the image into the model, it needs to be processed using any two different image enhancement techniques (such as horizontal flipping, vertical flipping, contrast adjustment, brightness adjustment, cropping, etc.). Perform image transformations to obtain the following results: and The formula can be expressed as , , and These represent two random and different image enhancement operations. The first one... Sample Compared with samples enhanced based on it and They are considered as positive samples, and the rest are considered as positive samples. and ( Each sample is considered a negative sample to the other.
[0090] Will and Enter them separately and Embedded representations were obtained respectively. and Where N is the number of samples in the subset, and C represents... and The final output channel dimension can be expressed by the formula as follows: , .
[0091] right and By expanding the dimensions, we obtain and The formula can be expressed as , , This refers to dimensional expansion. `input` is the tensor to be expanded, and `dim` is the position of the expanded dimension (starting from 0).
[0092] calculate Obtain the feature similarity of positive samples The formula can be expressed as ;calculate Obtain negative sample feature similarity The formula can be expressed as , This indicates that the feature matrix in the queue will be transposed.
[0093] based on and The self-supervised learning loss value is calculated using the InfoNCE loss function. The calculation formula is: , This is the temperature setpoint, which you can set yourself according to different situations; loss value. Only for Perform gradient backpropagation to update parameters, using the following formula: , This is the momentum control factor, and its typical setpoint is approximately 1. Then, the parameters are updated using the momentum update method, as shown in the formula: Finally, The oldest representation is deleted, and the current one is removed. Place it in; in the self-supervised learning loss value If the preset value is not reached, continue to select a subset of data. Continue training until the self-supervised learning loss value is reached. Once the preset value is reached, indicating that model training has converged, the trained encoder is used as the pre-trained backbone model. .
[0094] S204: Load the parameters of the trained encoder into the classification model and initialize the fully connected layer of the classification model to obtain the first classification model;
[0095] In one possible implementation, after obtaining the pre-trained backbone model... Afterwards, Load the parameters into the classification model backbone network Then initialize the classification model. The parameters of the fully connected layer are used to obtain the first classification model.
[0096] S205, the labeled dataset is input into the first classification model for supervised learning, and the trained first classification model is obtained;
[0097] In one possible implementation, after obtaining the first classification model, the labeled dataset can be input into the first classification model, and the supervised learning loss value is output. When the supervised learning loss value reaches a preset threshold, step S206 is entered; otherwise, the step of inputting the labeled dataset into the first classification model continues.
[0098] Specifically, using labeled data According to traditional supervised learning methods, the classification model The model is then tuned until it converges, resulting in the optimized classification model. .
[0099] S206, the first classification model after preprocessing training, is obtained as the preprocessed first classification model;
[0100] In this embodiment of the application, when preprocessing the first classification model after training, the backbone network and the first fully connected layer of the first classification model after training are first determined, then a second fully connected layer with the same structure as the first fully connected layer is constructed, and then the second fully connected layer is connected to the last layer of the backbone network to obtain the second classification model. Finally, the parameters of the backbone network and the first fully connected layer in the second classification model are fixed, and all Dropout layers in the second classification model are enabled to obtain the preprocessed first classification model.
[0101] In one possible implementation, in the classification model The last layer of the backbone network is connected to an additional fully connected layer. Fully connected layers with identical structure To obtain the classification model this In addition to predicting class probabilities, it also needs to output random uncertainty. ;right backbone network in and fully connected layer The parameters are fixed, and all Dropout layers are enabled.
[0102] S207. The labeled dataset is input into the preprocessed first classification model for semi-supervised learning to obtain a pre-trained image classification model.
[0103] In this embodiment, during semi-supervised learning, labeled data is first input into a preprocessed first classification model for multiple parallel computations, outputting multiple first target prediction values. Then, a first mean and a first standard deviation are calculated based on each first target prediction value, and a semi-supervised learning loss value is calculated based on the first mean and the first standard deviation. When the semi-supervised learning loss value reaches a preset value, a third classification model is obtained. The Dropout layer in the second fully connected layer of the third classification model is then enabled, and the Dropout layers in all layers of the third classification model except the second fully connected layer are disabled, resulting in a preprocessed third classification model. Unlabeled data is then input into the preprocessed third classification model for multiple parallel computations, outputting multiple second target probability values and random uncertainty parameters. A second mean and a second standard deviation are calculated based on each second target probability value, and a pseudo-label dataset is obtained based on the random uncertainty parameter, the second mean, and the second standard deviation. When the pseudo-labels meet multiple preset conditions, the pseudo-label dataset is added to the labeled dataset to obtain the target dataset. Finally, the target dataset is input into the first classification model for supervised learning to obtain a pre-trained image classification model.
[0104] Furthermore, when the semi-supervised learning loss value does not reach the preset value, the process continues to execute the step of inputting labeled data into the preprocessed first classification model for multiple parallel calculations.
[0105] In one possible implementation, there will be tagged data. Enter to middle, The algorithm performs T parallel calculations to obtain T predicted values, and then calculates the mean of the T predicted values. and standard deviation Next, the random uncertainty loss function is calculated. Finally, the fully connected layer is updated via gradient backpropagation. Among them, the loss function of random uncertainty for:
[0106]
[0107] in, This is the output when the network parameter is W. This is due to random uncertainty (equivalent to noise). The obedience of the model in the t-th prediction The Gaussian noise is a parameter inherent in the model itself; The number of predictions made by the model. and As can be seen from the loss function above, in learning random uncertainty... When label supervision is not required, it is equivalent to an adaptive weight that adjusts its size based on whether a sample is difficult to predict correctly.
[0108] Further, complete After training, only the fully connected layers are enabled. The Dropout layer in the middle is turned off, the rest of the Dropout layers are turned off, and then the unlabeled data is processed. Enter to Parallel computation T times. Due to the backbone network and fully connected layer Dropout is turned off, therefore the fully connected layer Output the predicted probabilities of T categories They are the same, but the fully connected layer Output the predicted probabilities of T categories They are not the same; next, we need to predict the probability based on the T categories. Calculate the mean and standard deviation In addition, fully connected layers It also outputs random uncertainty. Finally, based on all the outputs in this step, we obtain the pseudo-labels and their uncertainty (inversely proportional to their reliability). We can then use relevant rules to determine whether the pseudo-labels should be used in the next round of training. The rule is: if... and and and If the pseudo-label is found to be false, it will be included in the next round of training; otherwise, it will not be included. (The above rules apply.) It is about finding the category with the highest class probability. For category confidence threshold, It is an uncertainty threshold.
[0109] Specifically, in pseudo-label generation, Bayesian neural networks and the currently mainstream decisional neural networks differ significantly in principle: decisional neural networks optimize the parameters within the network, aiming to achieve optimal neural network model parameters; Bayesian neural networks optimize the parameter distribution, aiming to approximate the target parameter distribution of the neural network through continuous optimization. The optimization object is no longer a single parameter, but rather the parameters controlling the neural network's parameter distribution. For example, if a model uses a Gaussian distribution, then the optimization object is the mean of the Gaussian distribution. and standard deviation .
[0110] The definition of a Bayesian network (NN) is also based on the Bayesian inference formula. Assume the model parameters follow a Gaussian prior distribution. Meanwhile, given the observation dataset The corresponding label is The posterior distribution of model parameters can be calculated using Bayesian inference. This refers to the distribution of the model's target parameters. For classification tasks, typically... ,in This is the output of the model. Although Bayesian neural networks are easy to define, they present difficulties in inference, mainly due to the marginal distribution of real-world data. In the real world, it is almost impossible to calculate, so current solutions address the problems of Bayesian inference through approximation methods. Dropout variational inference is a commonly used approximation method, and experiments have shown that Dropout can make the parameter distribution follow a Bernoulli distribution. This invention will use this MC-Dropout (one type of Dropout variational inference method) to calculate cognitive uncertainty.
[0111] In addition, this invention addresses two types of uncertainty inherent in Bayesian neural networks: one is called accidental uncertainty, caused by inherent noise in the observed data, which cannot be eliminated; the other is called perceptual uncertainty, model-related, caused by incomplete training. Theoretically, this uncertainty can be eliminated by providing more training data to compensate for the existing model's knowledge gaps. This invention solves the problem of pseudo-label reliability assessment by combining these two types of uncertainty.
[0112] For example Figure 3 As shown, Figure 3 This is a schematic diagram of the training process of an image classification model provided in this application. First, self-supervised learning based on contrastive learning is performed using unlabeled data. Then, supervised learning is performed using labeled data. Finally, pseudo-labeled data is determined based on semi-supervised learning using a Bayesian neural network. The pseudo-labeled data is added to the labeled data, and the next round of supervised learning with labeled data is performed to obtain the pre-trained image classification model.
[0113] In this embodiment, the image classification device first acquires the target image to be classified, and then inputs the target image into a pre-trained image classification model. The pre-trained image classification model is generated through self-supervised learning, supervised learning, and semi-supervised learning in sequence. Self-supervised learning is trained on an unlabeled dataset, supervised learning is trained on a labeled dataset, and semi-supervised learning is trained on both a pseudo-labeled dataset and a labeled dataset. The pseudo-labeled dataset is generated based on the unlabeled dataset. Finally, the image category corresponding to the target image is output. Because this application first uses a massive unlabeled dataset in a specific scenario to perform self-supervised learning on the neural network model, allowing the model to "access" global data as much as possible to obtain more comprehensive feature extraction capabilities, and then uses a small amount of labeled data in a specific and similar scenario to perform supervised learning on the pre-trained model to further optimize the model, and finally generates a pseudo-labeled dataset from the unlabeled dataset and combines it with the labeled dataset to further train the model, the utilization rate of unlabeled data is improved, resulting in higher classification accuracy.
[0114] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the embodiments of the apparatus of the present invention, please refer to the embodiments of the method of the present invention.
[0115] Please see Figure 4 This diagram illustrates the structure of an image classification device provided in an exemplary embodiment of the present invention. The image classification device can be implemented as all or part of a terminal through software, hardware, or a combination of both. The device 1 includes an image acquisition module 10, an image input module 20, and a category output module 30.
[0116] Image acquisition module 10 is used to acquire the target image to be classified;
[0117] The image input module 20 is used to input the target image into a pre-trained image classification model. The pre-trained image classification model is generated by training through self-supervised learning, supervised learning, and semi-supervised learning in sequence. Self-supervised learning is trained on an unlabeled dataset, supervised learning is trained on a labeled dataset, and semi-supervised learning is trained on a pseudo-labeled dataset and a labeled dataset. The pseudo-labeled dataset is generated based on the unlabeled dataset.
[0118] The category output module 30 is used to output the image category corresponding to the target image.
[0119] It should be noted that the image classification device provided in the above embodiments is only illustrated by the division of the above functional modules when performing the image classification method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image classification device and the image classification method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.
[0120] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0121] In this embodiment, the image classification device first acquires the target image to be classified, and then inputs the target image into a pre-trained image classification model. The pre-trained image classification model is generated through self-supervised learning, supervised learning, and semi-supervised learning in sequence. Self-supervised learning is trained on an unlabeled dataset, supervised learning is trained on a labeled dataset, and semi-supervised learning is trained on both a pseudo-labeled dataset and a labeled dataset. The pseudo-labeled dataset is generated based on the unlabeled dataset. Finally, the image category corresponding to the target image is output. Because this application first uses a massive unlabeled dataset in a specific scenario to perform self-supervised learning on the neural network model, allowing the model to "access" global data as much as possible to obtain more comprehensive feature extraction capabilities, and then uses a small amount of labeled data in a specific and similar scenario to perform supervised learning on the pre-trained model to further optimize the model, and finally generates a pseudo-labeled dataset from the unlabeled dataset and combines it with the labeled dataset to further train the model, the utilization rate of unlabeled data is improved, resulting in higher classification accuracy.
[0122] The present invention also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implement the image classification methods provided in the above-described method embodiments.
[0123] The present invention also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the image classification methods of the various method embodiments described above.
[0124] Please see Figure 5 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. Figure 5 As shown, terminal 1000 may include: at least one processor 1001, at least one network interface 1004, user interface 1003, memory 1005, and at least one communication bus 1002.
[0125] The communication bus 1002 is used to realize the connection and communication between these components.
[0126] The user interface 1003 may include a display screen and a camera. Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.
[0127] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).
[0128] The processor 1001 may include one or more processing cores. The processor 1001 connects to various parts within the electronic device 1000 using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1005, and by calling data stored in the memory 1005. Optionally, the processor 1001 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 1001 may integrate one or more of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip, without being integrated into the processor 1001.
[0129] The memory 1005 may include random access memory (RAM) or read-only memory. Optionally, the memory 1005 may include a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. Figure 5 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an image classification application.
[0130] exist Figure 5 In the terminal 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and to obtain the user's input data; while the processor 1001 can be used to call the image classification application stored in the memory 1005 and specifically perform the following operations:
[0131] Obtain the target image to be classified;
[0132] The target image is input into a pre-trained image classification model. The pre-trained image classification model is generated by training through self-supervised learning, supervised learning, and semi-supervised learning in sequence. Self-supervised learning is trained on an unlabeled dataset, supervised learning is trained on a labeled dataset, and semi-supervised learning is trained on both a pseudo-labeled dataset and a labeled dataset. The pseudo-labeled dataset is generated based on the unlabeled dataset.
[0133] Output the image category corresponding to the target image.
[0134] In one embodiment, when generating a pre-trained image classification model, the processor 1001 performs the following operations:
[0135] Obtain the dataset; the dataset includes both unlabeled and labeled datasets;
[0136] Create a classification model, and build an encoder and a momentum encoder based on the model parameters of the classification model;
[0137] The trained encoder is obtained by performing self-supervised learning based on the unlabeled dataset, the encoder, and the momentum encoder.
[0138] The parameters of the trained encoder are loaded into the classification model, and the fully connected layers of the classification model are initialized to obtain the first classification model.
[0139] The labeled dataset is input into the first classification model for supervised learning, resulting in the trained first classification model.
[0140] The first classification model after preprocessing training is obtained;
[0141] The labeled dataset is input into the preprocessed first classification model for semi-supervised learning to obtain a pre-trained image classification model.
[0142] In one embodiment, when processor 1001 performs self-supervised learning based on the unlabeled dataset, the encoder, and the momentum encoder to obtain the trained encoder, it specifically performs the following operations:
[0143] Initialize a queue of a preset size;
[0144] Divide the unlabeled dataset into multiple sub-datasets;
[0145] To identify a target subset of data from multiple subsets;
[0146] Image transformations are performed on the target subset to obtain first transformed data and second transformed data;
[0147] The first transformation data and the second transformation data are respectively input into the encoder and the momentum encoder, and the first embedded characterization result and the second embedded characterization result are output.
[0148] The first embedded representation result and the second embedded representation result are respectively subjected to dimensional expansion to obtain the first expansion result and the second expansion result;
[0149] Calculate the feature similarity of positive samples based on the first and second expansion results;
[0150] The features in the queue are permuted to obtain the permutation matrix, and the similarity of negative sample features is calculated based on the permutation matrix and the first embedded representation result.
[0151] The self-supervised learning loss value is calculated based on the similarity of positive sample features and the similarity of negative sample features. When the self-supervised learning loss value reaches the preset value, the trained encoder is obtained.
[0152] In one embodiment, when the processor 1001 obtains the trained encoder when the self-supervised learning loss value reaches a preset value, it specifically performs the following operations:
[0153] When the self-supervised learning loss value does not reach the preset value, the encoder is backpropagated based on the self-supervised learning loss value to update the encoder parameters;
[0154] Continue executing the step of identifying a target subset from multiple subsets until the self-supervised learning loss value reaches the preset value.
[0155] In one embodiment, when the processor 1001 executes the pre-processed first classification model and obtains the pre-processed first classification model, it specifically performs the following operations:
[0156] Determine the backbone network and the first fully connected layer of the first classification model after training;
[0157] Construct a second fully connected layer with the same structure as the first fully connected layer;
[0158] The second fully connected layer is connected to the last layer of the backbone network to obtain the second classification model;
[0159] The parameters of the backbone network and the first fully connected layer in the second classification model are fixed, and all Dropout layers in the second classification model are enabled to obtain the preprocessed first classification model.
[0160] In one embodiment, when processor 1001 performs semi-supervised learning by inputting a labeled dataset into a preprocessed first classification model to obtain a pre-trained image classification model, it specifically performs the following operations:
[0161] The labeled data is input into the preprocessed first classification model for multiple parallel calculations, and multiple first target prediction values are output.
[0162] Calculate the first mean and the first standard deviation for each first target prediction value, and calculate the semi-supervised learning loss value based on the first mean and the first standard deviation;
[0163] When the semi-supervised learning loss value reaches the preset value, a third classification model is obtained;
[0164] Enable the Dropout layer in the second fully connected layer of the third classification model;
[0165] Disabling the Dropout layers in all layers except the second fully connected layer in the third classification model yields the preprocessed third classification model.
[0166] The unlabeled data is input into the preprocessed third classification model for multiple parallel calculations, and multiple second target probability values and random uncertainty parameters are output.
[0167] The second mean and second standard deviation are calculated based on each second target probability value;
[0168] The pseudo-label dataset is obtained based on the random uncertainty parameter, the second mean, and the second standard deviation.
[0169] When the pseudo-labels meet multiple preset conditions, the pseudo-label dataset is added to the labeled dataset to obtain the target dataset;
[0170] The target dataset is input into the first classification model for supervised learning to obtain a pre-trained image classification model.
[0171] In one embodiment, when the processor 1001 obtains a third classification model when the semi-supervised learning loss value reaches a preset value, it specifically performs the following operations:
[0172] If the semi-supervised learning loss value does not reach the preset value, continue to execute the step of inputting labeled data into the preprocessed first classification model for multiple parallel calculations.
[0173] In this embodiment, the image classification device first acquires the target image to be classified, and then inputs the target image into a pre-trained image classification model. The pre-trained image classification model is generated through self-supervised learning, supervised learning, and semi-supervised learning in sequence. Self-supervised learning is trained on an unlabeled dataset, supervised learning is trained on a labeled dataset, and semi-supervised learning is trained on both a pseudo-labeled dataset and a labeled dataset. The pseudo-labeled dataset is generated based on the unlabeled dataset. Finally, the image category corresponding to the target image is output. Because this application first uses a massive unlabeled dataset in a specific scenario to perform self-supervised learning on the neural network model, allowing the model to "access" global data as much as possible to obtain more comprehensive feature extraction capabilities, and then uses a small amount of labeled data in a specific and similar scenario to perform supervised learning on the pre-trained model to further optimize the model, and finally generates a pseudo-labeled dataset from the unlabeled dataset and combines it with the labeled dataset to further train the model, the utilization rate of unlabeled data is improved, resulting in higher classification accuracy.
[0174] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The image classification program can be stored in a computer-readable storage medium, and when executed, the program can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.
[0175] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.
Claims
1. An image classification method, characterized in that, The method includes: Obtain the target image to be classified; The target image is input into a pre-trained image classification model; the pre-trained image classification model is generated according to the following steps: Obtain a dataset, which includes an unlabeled dataset and a labeled dataset; create a classification model and construct an encoder and a momentum encoder based on the model parameters of the classification model; perform self-supervised learning based on the unlabeled dataset, the encoder, and the momentum encoder to obtain a trained encoder; load the parameters of the trained encoder onto the classification model and initialize the fully connected layers of the classification model to obtain a first classification model; input the labeled dataset into the first classification model for supervised learning to obtain a trained first classification model; preprocess the trained first classification model to obtain a preprocessed first classification model; input the labeled dataset into the preprocessed first classification model for semi-supervised learning to obtain a pre-trained image classification model. The preprocessed first classification model obtained after training includes: The backbone network and the first fully connected layer of the trained first classification model are determined; a second fully connected layer with the same structure as the first fully connected layer is constructed; the second fully connected layer is connected to the last layer of the backbone network to obtain the second classification model; the parameters of the backbone network and the first fully connected layer in the second classification model are fixed, and all Dropout layers in the second classification model are enabled to obtain the preprocessed first classification model. The step of inputting the labeled dataset into the preprocessed first classification model for semi-supervised learning to obtain a pre-trained image classification model includes: The labeled data is input into a preprocessed first classification model for multiple parallel computations, outputting multiple first target prediction values. A first mean and a first standard deviation are calculated for each first target prediction value, and a semi-supervised learning loss value is calculated based on the first mean and the first standard deviation. When the semi-supervised learning loss value reaches a preset value, a third classification model is obtained. The Dropout layer in the second fully connected layer of the third classification model is enabled. The Dropout layers in all layers of the third classification model except the second fully connected layer are disabled, resulting in a preprocessed third classification model. The unlabeled data is input into the preprocessed third classification model for multiple parallel computations, outputting multiple second target probability values and random uncertainty parameters. A second mean and a second standard deviation are calculated for each second target probability value. A pseudo-label dataset is obtained based on the random uncertainty parameter, the second mean, and the second standard deviation. When the pseudo-labels meet multiple preset conditions, the pseudo-label dataset is added to the labeled dataset, resulting in a target dataset. The target dataset is input into the first classification model for supervised learning, resulting in a pre-trained image classification model. Output the image category corresponding to the target image.
2. The method according to claim 1, characterized in that, The step of performing self-supervised learning based on the unlabeled dataset, the encoder, and the momentum encoder to obtain the trained encoder includes: Initialize a queue of a preset size; The unlabeled dataset is divided into multiple sub-datasets; A target subset of data is determined from the plurality of subsets. The target subset of data is subjected to image transformation to obtain first transformed data and second transformed data; The first transformation data and the second transformation data are respectively input into the encoder and the momentum encoder, and the first embedded characterization result and the second embedded characterization result are output. The first embedded representation result and the second embedded representation result are respectively subjected to dimensional expansion to obtain the first expansion result and the second expansion result; Calculate the feature similarity of positive samples based on the first expansion result and the second expansion result; The features in the queue are permuted to obtain a permutation matrix, and the negative sample feature similarity is calculated based on the permutation matrix and the first embedded representation result. The self-supervised learning loss value is calculated based on the similarity of the positive sample features and the similarity of the negative sample features. When the self-supervised learning loss value reaches a preset value, the trained encoder is obtained.
3. The method according to claim 2, characterized in that, The step of obtaining the trained encoder when the self-supervised learning loss value reaches a preset value includes: When the self-supervised learning loss value does not reach the preset value, the encoder is backpropagated according to the self-supervised learning loss value to update the encoder parameters; Continue executing the step of determining a target subset of the datasets from the plurality of subsets until the self-supervised learning loss value reaches a preset value.
4. The method according to claim 1, characterized in that, When the semi-supervised learning loss value reaches a preset value, a third classification model is obtained, including: If the semi-supervised learning loss value does not reach the preset value, the step of inputting the labeled data into the preprocessed first classification model for multiple parallel calculations continues.
5. An image classification apparatus implemented using the method according to any one of claims 1-4, characterized in that, The device includes: The image acquisition module is used to acquire the target image to be classified. An image input module is used to input the target image into a pre-trained image classification model; The category output module is used to output the image category corresponding to the target image.
6. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the method steps as claimed in any one of claims 1-4.
7. A terminal, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the method steps as claimed in any one of claims 1-4.