An unsupervised transfer learning image classification method for environmental changes

Abstract

The invention discloses an unsupervised transfer learning image classification method facing environmental changes. The method comprises the following steps: obtaining a to-be-classified image, and if no model facing the application environment exists, transferring learning a new model from the learning environment; collecting training data and initializing the model; calculating the classification loss of labeled samples and the distribution shift loss of the learning environment and the application environment data on a pseudo label generator, and generating some pseudo labels; designing multiple joint classifiers to output two-dimensional joint probabilities and simultaneously predict the main task and self-supervised task labels of the image, and calculating the classification loss of the joint classifier; integrating the marginal probability of the self-supervised task label in the output of the joint classifier to obtain different concept labels; calculating the consistency loss between different concept labels of the application environment samples; accumulating all losses and updating parameters by gradient back propagation; iterative training to a maximum number of iterations; and predicting the mean of the concept labels of the to-be-classified image by all the joint classifiers.

Description

Technical Field

[0001] The invention relates to an unsupervised transfer learning image classification method oriented to environmental changes, and belongs to the technical field of image classification. Background Art

[0002] Image classification is one of the most basic artificial intelligence application scenarios. Especially with the development of deep learning, deep learning image classification algorithms have been widely used in autonomous driving, short video personalized recommendation, e-commerce search, etc. Deep learning image classification systems require a large amount of labeled data for supervised learning, but in many practical application environments, the time cost and economic cost of manually labeling training data are very high. Therefore, in many scenarios, image classification models can only be trained from similar learning environments. For example, in autonomous driving tasks, it is very time-consuming and laborious to use cars to collect supervised data sets in real application scenarios. At this time, if the autonomous driving model can be trained in a computer-simulated environment, it can effectively save time, save energy and reduce emissions, and it is also of great significance to ensure the safety of testers. However, there is a difference between the computer-simulated environment and the real application environment. There is a data distribution offset between the data between them, which leads to the fact that the effect achieved by using the model trained in the simulated learning environment directly in the real application environment is often very unsatisfactory. Therefore, how to transfer learning from a supervised learning environment to an unsupervised application environment is a very worthy issue to study. Summary of the invention

[0003] Purpose of the invention: The learning environment and application environment of the deep image classification system are often different. The data distribution shift caused by environmental changes will make the model learned in the learning environment perform poorly in the application environment. To address this problem, the present invention proposes an unsupervised transfer learning image classification method for environmental changes. Based on the joint learning of the target task and different self-supervised auxiliary tasks, the conceptual labels of the unsupervised samples in the application environment are obtained from multiple perspectives, and the true labels of the samples in the application environment are learned by exploring the consistency between different conceptual labels. The transfer learning image classification system aims to solve the problem of image classification model performance degradation caused by data distribution shift caused by environmental changes from a supervised learning environment to an unsupervised application environment. This method can avoid the need to re-collect labeled supervised samples in a new application environment, which is of great significance in terms of time cost, economic cost, energy conservation and emission reduction.

[0004] Technical solution: An unsupervised transfer learning image classification method for environment changes. Through the following steps, a model that is effective in the application environment is learned from a learning environment with offset data distribution:

[0005] Step (1) obtaining a new image sample to be classified in an application environment;

[0006] Step (2) determines whether a completed image classification model for the application environment already exists. If not, proceed to step (3) to perform transfer learning from the learning environment to the application environment to construct an image classification model; if so, proceed to step (13);

[0007] In step (3), if the model update iteration number reaches the maximum iteration number, proceed to step (13); if not, proceed to step (4);

[0008] Step (4) obtaining a large amount of labeled image data from the learning environment as supervised samples, and obtaining a large amount of unlabeled image data from the application environment as unsupervised samples, to construct a training set;

[0009] Step (5) initializing image classification model parameters, hyperparameters in the model training process, and several self-supervised tasks based on data transformation;

[0010] Step (6) Use a residual neural network connected to a fully connected bottleneck network to extract features of the image data;

[0011] Step (7) Use the pseudo-label generator to predict the unlabeled samples and select some pseudo-labels with high confidence;

[0012] Step (8) calculates the classification loss of labeled samples (including pseudo labels) on the pseudo label generator, and calculates the data distribution offset loss of the learning environment and the application environment in the feature space;

[0013] Step (9) combines the original labels of the labeled samples with different self-supervised task labels of the current samples to generate different two-dimensional joint labels. Accordingly, a multi-layer fully connected neural network is used to construct multiple joint classifiers to output two-dimensional joint probabilities to predict the joint labels, and the joint labels are used to calculate the average classification loss of all joint classifiers; the labeled samples include pseudo-label samples;

[0014] Step (10) integrates the marginal probability of the self-supervised task label distribution in the two-dimensional joint probability of the output of each joint classifier to obtain different concept labels for the target task by each joint classifier;

[0015] Step (11) on all application environment samples, calculate the consistency loss between the concept labels given by each joint classifier to align the concept labels;

[0016] Step (12) accumulates all losses and updates the model parameters using gradient back propagation; the number of iterations increases by one; and the process goes to step (3);

[0017] Step (13) transforms the data using a corresponding data transformation set according to each data transformation-based self-supervised task type;

[0018] Step (14) inputs the data transformed by different data transformation sets into the joint classifier corresponding to the corresponding self-supervised task to obtain a two-dimensional joint probability prediction of the target task label and the corresponding self-supervised task label;

[0019] Step (15) integrates the marginal distribution corresponding to the self-supervision in the two-dimensional joint probability distribution output by each joint classifier to obtain different concept labels for the target task by each joint classifier;

[0020] Step (16) predicts the label of the image to be classified by taking the mean of all concept labels as the consistent part of the concept labels.

[0021] Beneficial effects: Compared with the prior art, the present invention can avoid the need to re-collect labeled supervised learning samples in the application environment to retrain the model, thereby effectively saving time and labor costs. In many application scenarios, it can provide effective assistance to energy conservation, emission reduction and safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of how to classify pictures of an application environment in an example of the present invention;

[0023] Figure 2 A flowchart of constructing an unsupervised transfer learning model for image classification according to an embodiment of the present invention;

[0024] Figure 3 The present invention provides a flowchart of concept label generation in the prediction process of an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0026] The flow chart of the present invention for classifying pictures in the application environment is as follows Figure 1 First, we obtain the image to be classified in the application environment; if there is no model for the application environment, we first perform transfer learning from the learning environment to the application environment to build a new model, that is, Figure 2The sub-processes are as follows: obtain a large number of labeled supervised samples from the learning environment, obtain a large number of unlabeled unsupervised samples from the application environment, and construct a training set; initialize model parameters, hyperparameters in the model training process, and several self-supervised auxiliary tasks to help model training; use a residual convolutional neural network connected to a fully connected bottleneck network to extract the features of image data; use a pseudo-label generator to predict unlabeled samples and screen out some pseudo-labels with high confidence; calculate the classification loss of labeled samples (including pseudo-labels) on the pseudo-label generator, and calculate the data distribution offset loss of the learning environment and the application environment in the feature space; compare all labeled samples (including pseudo-labels) with the sample The labels of different self-supervised tasks generate multiple two-dimensional joint labels, and correspondingly design multiple joint classifiers to output two-dimensional joint probabilities to predict these joint labels, and use the joint labels to calculate the average classification loss of all joint classifiers; in the two-dimensional joint probability output by each joint classifier, the marginal probability of the self-supervised task label distribution is integrated to obtain different concept labels for the target task for each joint classifier; align the concept labels given by each joint classifier on all application environment samples, and calculate the consistency loss between them; accumulate all losses and update the model parameters by gradient propagation; iterate the training process until the maximum number of iterations; when the model training is completed, enter Figure 3 Sub-process: Use the mean of the concept labels of all joint classifiers for the current image to be classified as the consistent concept label to predict the true label of the current sample. The specific implementation steps are as follows:

[0027] Step (0) is the starting action;

[0028] Step (1) obtaining a new image sample to be classified in an application environment;

[0029] Step (2) determines whether a completed image classification model for the application environment already exists. If not, proceed to step (3) to transfer learning from the learning environment to the application environment to build an image classification model; if so, proceed to step (13);

[0030] In step (3), if the model update iteration number reaches the maximum iteration number, proceed to step (13); if not, proceed to step (4);

[0031] Step (4) obtaining a large amount of labeled image data from the learning environment as supervised samples, and obtaining a large amount of unlabeled image data from the application environment as unsupervised samples, to construct a training set;

[0032] Step (5) Initialize model parameters, hyperparameters in the model training process, and several self-supervised tasks based on data transformation;

[0033] Step (6) Use a residual neural network connected to a fully connected bottleneck network to extract features of the image data;

[0034] Step (7) Use the pseudo-label generator to predict the unlabeled samples and select some pseudo-labels with high confidence;

[0035] Step (8) calculates the classification loss of labeled samples (including pseudo labels) on the pseudo label generator, and calculates the data distribution offset loss of the learning environment and the application environment in the feature space;

[0036] Step (9) combines the original labels of the labeled samples with different self-supervised task labels of the current samples to generate different two-dimensional joint labels. Accordingly, a multi-layer fully connected neural network is used to construct multiple joint classifiers to output two-dimensional joint probabilities to predict the joint labels, and the joint labels are used to calculate the average classification loss of all joint classifiers; the labeled samples include pseudo-label samples;

[0037] Step (10) integrates the marginal probability of the self-supervised task label distribution in the two-dimensional joint probability output of each joint classifier to obtain different concept labels for the target task by each joint classifier;

[0038] Step (11) on all application environment samples, calculate the consistency loss between the concept labels given by each joint classifier to align the concept labels;

[0039] Step (12) accumulates all losses and updates the model parameters using gradient propagation; the number of iterations increases by one; and the process is switched to step (3);

[0040] Step (13) transforms the data using a corresponding data transformation set according to each data transformation-based self-supervised task type;

[0041] Step (14) inputs the data transformed by different data transformation sets into the joint classifier corresponding to the corresponding self-supervised task to obtain a two-dimensional joint probability prediction of the target task label and the corresponding self-supervised task label;

[0042] Step (15) integrates the marginal distribution corresponding to the self-supervision in the two-dimensional joint probability distribution output by each joint classifier to obtain different concept labels for the target task by each joint classifier;

[0043] Step (16) predicts the label of the image to be classified by taking the mean of all concept labels as the consistent part of the concept labels.

[0044] In order to better explain the specific implementation of the present invention, it is assumed that the input space of the training data is The latent feature space shared across environments is The target task (main task) label space, that is, the category space of the classification task is The label space of the i-th self-supervised auxiliary task is In addition, there is a feature extractor shared across environments And each self-supervised task has a joint classifier, such as the i-th joint classifier is written as Will Writing C i , which means that the feature extractor is connected to the i-th joint classifier. In addition, each self-supervised task corresponds to a data transformation set. For example, the data transformation set corresponding to the i-th joint classifier is

[0045] The process of building an image classification model by transferring learning from the learning environment to the application environment Figure 2 In the learning environment, we first collect n s Source domain dataset with labeled samples represents the i-th image in the source domain, The label of the i-th image in the source domain is n, which is collected from the application environment. t target domain dataset of unlabeled samples represents the i-th image in the target domain. Next, determine p self-supervised tasks based on data transformation and the data transformation set corresponding to each self-supervised task Each data transformation set (such as the i-th one) has Data transformation function And the corresponding self-supervised label space

[0046] There is a pseudo-label generator in the application-oriented model In addition, Writing C PL , which means the feature extractor is connected to the pseudo-label generator. First, use the pseudo-label generator to generate some pseudo-labels for samples in the application environment: in represents the i-th image in the target domain, represents the pseudo label of the i-th image in the target domain. Then the pseudo label selection function is:

[0047]

[0048] To select pseudo labels in application environment samples Samples with high confidence: Where τ represents the confidence threshold.

[0049] Calculate the classification loss on labeled samples as follows:

[0050]

[0051] in represents the cross entropy classification loss, represents an image dataset with n labeled samples, x i represents the i-th picture in the dataset D, y i Represents the label of the i-th picture in the dataset D; C PL It means that the feature extractor is connected to the pseudo label generator; the distribution shift loss of the learning environment and the application environment is obtained:

[0052]

[0053] Among them, k.(·,·) is the Gaussian kernel function; and They represent the i-th source domain image and the j-th target domain image in the latent feature space after being processed by the feature extractor F. The hidden features obtained in and They represent the label predictions of the pseudo-label generator for the target task for the i-th source domain image and the j-th target domain image, respectively.

[0054] Then use the target task's labeled space Make a Cartesian product with the label space of all self-supervised tasks, such as the label space of the i-th self-supervised task is Cartesian product result

[0055] Where m means that the target task has m categories. It means that the i-th self-supervised task has categories; Indicates that the current sample is in the jth category of the target task, and the label is y j , and belongs to the kth category in the i-th self-supervised task, and the label is Compute the average loss for a 2D classifier in represents the data transformation set corresponding to the i-th self-supervised auxiliary task, D s represents the source domain dataset, represents a dataset consisting of target domain samples with pseudo labels. Writing C i , which means that the feature extractor is connected to the i-th joint classifier, then the classification loss calculation function of the i-th joint classifier on the data set D is as follows:

[0056]

[0057] Next, we integrate the marginal probability distribution of the self-supervised task in the joint classifier output to obtain the concept label:

[0058]

[0059] in, represents the two-dimensional probability output by the i-th joint classifier, represents the marginal probability distribution of the i-th self-supervised task label.

[0060] The transformation function in each transformation set is used uniformly based on data transformation, that is, the marginal probability distribution of the i-th self-supervised task label is a uniform distribution:

[0061]

[0062] Integrate before the activation function, then the concept label obtained after integrating the output of the i-th joint classifier is:

[0063]

[0064] in, Indicates that the i-th data transformation set is applied to x The kth data transformation method in [:,k] means taking the kth column of the two-dimensional output.

[0065] Then, the consistency loss of the concept labels given by the p joint classifiers is calculated on all application environment data

[0066]

[0067] Among them, P i (y|x k ) and P j (y|x k ) represent the concept labels of the kth target domain sample by the i-th joint classifier and the j-th joint classifier respectively.

[0068] In summary, all losses

[0069] L=L PL +L trans +L joint +L CC

[0070] Use gradient back propagation to iteratively update model parameters until the maximum number of iterations.

[0071] When predicting the application environment image to be classified, the mean of all concept labels is used as the consistent part of the concept label to predict the label of the image to be classified, that is, where x t Indicates the application environment image currently to be classified.

Claims

1. An unsupervised transfer learning image classification method for environmental changes, characterized in that: The steps include: Step (1) obtaining a new image sample to be classified in an application environment; Step (2) determines whether a completed image classification model for the application environment already exists. If not, proceed to step (3) to perform transfer learning from the learning environment to the application environment to construct an image classification model. If it exists, go to step (13); In step (3), if the model update iteration number reaches the maximum iteration number, proceed to step (13); if not, proceed to step (4); Step (4) obtaining labeled image data from the learning environment as supervised samples, obtaining unlabeled image data from the application environment as unsupervised samples, and constructing a training set; Step (5) initializing image classification model parameters, hyperparameters in the model training process, and several self-supervised tasks based on data transformation; Step (6) Use a residual neural network connected to a fully connected bottleneck network to extract features of the image data; Step (7) predict the unlabeled samples using the pseudo-label generator and select pseudo-labels with high confidence; Step (8) calculating the classification loss of the labeled samples on the pseudo-label generator, and calculating the data distribution offset loss of the learning environment and the application environment in the feature space; the labeled samples include pseudo-label samples; Step (9) combines the original labels of the labeled samples with different self-supervised task labels of the current samples to generate different two-dimensional joint labels; accordingly, a multi-layer fully connected neural network is used to construct multiple joint classifiers to output two-dimensional joint probabilities to predict the joint labels, and the joint labels are used to calculate the average classification loss of all joint classifiers; the labeled samples include pseudo-label samples; Step (10) integrates the marginal probability of the self-supervised task label distribution in the two-dimensional joint probability of the output of each joint classifier to obtain different concept labels for the target task by each joint classifier; Step (11) on all application environment samples, calculate the consistency loss between the concept labels given by each joint classifier to align the concept labels; Step (12) accumulates all losses and updates the model parameters using gradient back propagation; the number of iterations increases by one; and the process goes to step (3); Step (13) transforms the data using a corresponding data transformation set according to each data transformation-based self-supervised task type; Step (14) inputs the data transformed by different data transformation sets into the joint classifier corresponding to the corresponding self-supervised task to obtain a two-dimensional joint probability prediction of the target task label and the corresponding self-supervised task label; Step (15) integrates the marginal distribution corresponding to the self-supervision in the two-dimensional joint probability distribution output by each joint classifier to obtain different concept labels for the target task by each joint classifier; Step (16) predicts the label of the image to be classified by taking the mean of all concept labels as the consistent part of the concept labels; The input space of training data is The latent feature space shared across environments is The target task label space, that is, the category space of the classification task is The label space of the i-th self-supervised auxiliary task that helps train the image classification model is Feature extractors shared across environments And each self-supervised task has a joint classifier, and the i-th joint classifier is written as Will Writing C i , It means that the feature extractor F is connected to the i-th joint classifier G i ; In addition, each self-supervised task corresponds to a data transformation set, and the data transformation set corresponding to the i-th joint classifier is In step (8), the classification loss on the labeled samples is calculated as follows: in represents the cross entropy classification loss, represents an image dataset with n labeled samples, x i represents the i-th picture in the dataset D, y i Represents the label of the i-th picture in the dataset D; C PL It means that the feature extractor is connected to the pseudo label generator; the distribution shift loss of the learning environment and the application environment is obtained: Among them, k.(·,·) is the Gaussian kernel function; and They represent the i-th source domain image and the j-th target domain image in the latent feature space after being processed by the feature extractor F. The hidden features obtained in and They represent the predictions of the target task labels of the i-th source domain image and the j-th target domain image by the pseudo-label generator; In step (9), the label space of the target task is used Taking the Cartesian product of the label spaces of all self-supervised tasks, the label space of the i-th self-supervised task is Cartesian product result Where m means that the target task has m categories. It means that the i-th self-supervised task has categories; Indicates that the current sample is in the jth category of the target task, and the label is y j , and belongs to the kth category in the i-th self-supervised task, and the label is Compute the average loss for a 2D classifier in represents the data transformation set corresponding to the i-th self-supervised auxiliary task, D s represents the source domain dataset, represents a dataset consisting of target domain samples with pseudo labels; Writing C i , which means that the feature extractor is connected to the i-th joint classifier, then the classification loss calculation function of the i-th joint classifier on the data set D is as follows: Where D represents a dataset containing n labeled samples, which is D s or Represents the data transformation set of the i-th self-supervised task The kth data transformation method in It represents the label in the i-th self-supervised task corresponding to this transformation; In step (10), the marginal probability distribution of the self-supervised task is integrated in the output of the joint classifier to obtain the concept label: in, represents the two-dimensional probability output by the i-th joint classifier, represents the marginal probability distribution of the i-th self-supervised task label; The transformation function in each transformation set is used uniformly based on data transformation, that is, the marginal probability distribution of the i-th self-supervised task label is a uniform distribution: Integrate before the activation function, then the concept label obtained after integrating the output of the i-th joint classifier is: in, Indicates that the i-th data transformation set is applied to x The kth data transformation method in [:,k] means taking the kth column of the two-dimensional output; In the step (11), the consistency loss of the concept labels given by the p joint classifiers is calculated on all application environment data. Among them, P i (y|x k ) and P j (y|x k ) represent the concept labels of the k-th target domain sample by the i-th joint classifier and the j-th joint classifier respectively; In the steps (12)-(13), all losses are accumulated L=L PL +L trans +L joint +L CC Use gradient back propagation to iteratively update model parameters until the maximum number of iterations; When predicting the application environment image to be classified, the mean of all concept labels is used as the consistent part of the concept label to predict the label of the image to be classified, that is, where x t Indicates the application environment image currently to be classified.

2. The unsupervised transfer learning image classification method for environmental changes according to claim 1, characterized in that: In step (4), first collect n s Source domain dataset with labeled samples represents the i-th image in the source domain, The label of the i-th image in the source domain is n, which is collected from the application environment. t target domain dataset of unlabeled samples represents the i-th image in the target domain.

3. The unsupervised transfer learning image classification method for environmental changes according to claim 1, characterized in that: In step (5), p self-supervisory tasks based on data transformation and a data transformation set corresponding to each self-supervisory task are determined. The i-th data transformation set has Data transformation function And the corresponding self-supervised label space 4. The unsupervised transfer learning image classification method for environmental changes according to claim 1, characterized in that: In step (7), there is a pseudo-label generator in the model Will Writing C PL , which means that the feature extractor is connected to the pseudo-label generator; first use the pseudo-label generator to generate pseudo-labels for samples in the application environment: in represents the i-th image in the target domain, represents the pseudo label of the i-th image in the target domain; then the pseudo label selection function is used: To select pseudo labels in application environment samples Samples with high confidence: Where τ represents the confidence threshold.

Images