Domain generalization image classification method based on mixture of multiple latent domains

Abstract

The present invention is based on a domain generalization image classification method for a mixture of multiple potential domains, the steps of which include: 1) obtaining a domain generalization data set, performing data enhancement and data set expansion; 2) constructing an AlexNet network, loading pre-training weights, and performing adversarial training on the AlexNet network; 3) extracting shallow features of the AlexNet network, obtaining convolutional feature statistics, and calculating domain discriminant features; 4) clustering the convolutional feature statistics into several clusters, and assigning pseudo-domain labels to the data according to the clusters; and updating the pseudo-domain labels at the same time; 5) inputting the pseudo-domain labels into a domain discriminator to complete the training of the domain discriminator, and simultaneously completing the training of the classification network of the overall image, and inputting the data that did not participate in the training into the classification network as a test set to obtain a classification result. The method of the present invention improves the generalization performance of the network and has better classification and prediction performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of transfer learning and relates to a domain generalization image classification method based on a mixture of multiple potential domains. Background Art

[0002] Currently, deep learning models are widely used in various fields of computer vision, including image classification. With the rise of deep neural networks and the powerful computing power of GPUs, image classification methods based on deep learning continue to flourish. However, the training and updating of high-performance deep network models rely on a large amount of labeled data. Although massive amounts of image data are readily available, this data is often only in rudimentary form, and labeling data requires significant manpower and material resources, which has become a major factor restricting the development of computer vision. Furthermore, in the field of machine learning, the network's training data and test data are drawn from the same distribution, satisfying the independent and identically distributed assumption. However, in actual engineering applications, data distributions often differ across different scenarios, resulting in poor model generalization in new scenarios.

[0003] Domain generalization relaxes the fundamental assumption of independent and identically distributed (IID) in traditional machine learning. Its goal is to transfer knowledge from similar, well-labeled domains to an unknown target domain. It primarily addresses the problem of using limited source domain data to train a network model with good generalization performance when there is a mismatch between training and test data and target domain data is difficult to obtain. Consequently, domain generalization methods have garnered increasing attention in recent years, resulting in several noteworthy research results.

[0004] Existing image classification algorithms utilize deep learning networks as their backbone, extracting features from input images for classification. Many current supervised image classification algorithms can achieve excellent accuracy on most public datasets, but these algorithms rely on large-scale labeled data for network training. In real life, large-scale labeled data is not always available in other image domains. Because of these issues, supervised image classification algorithms have certain limitations when the application scenario changes. The problem of domain generalization research is specifically to learn a highly generalizable model from several datasets with different data distributions in order to achieve good results on unknown test sets. This involves training a network model using richly labeled source domain data and testing it on an unknown target domain that has a certain correlation with the source domain data, achieving high target recognition accuracy. Summary of the Invention

[0005] The purpose of the present invention is to provide a domain generalization image classification method based on a mixture of multiple potential domains, which solves the problem that when the existing technology has source domain data with rich labels and the target domain data is difficult to obtain, it has a great impact on the performance of the deep model in image classification, resulting in low accuracy of the trained network model in predicting the category of unknown target domain data.

[0006] The technical solution adopted by the present invention is a domain generalization image classification method based on a mixture of multiple potential domains, which is specifically implemented according to the following steps:

[0007] Step 1: Obtain domain generalization dataset, including source domain dataset Ds and target domain dataset D T , perform data enhancement and dataset expansion;

[0008] Step 2: Build the AlexNet network, load the pre-trained weights, and input the source domain dataset Ds data into the AlexNet network to perform adversarial training on the AlexNet network.

[0009] Step 3: Extract shallow features of the AlexNet network, obtain convolutional feature statistics, and calculate domain discriminant features;

[0010] Step 4: Use the standard K-means clustering algorithm to cluster the convolution feature statistics into several clusters, and assign pseudo-domain labels to the data according to the clusters; at the same time, the pseudo-domain labels are updated as the network training process progresses;

[0011] Step 5: Input the pseudo domain label output by the clustering algorithm into the domain discriminator to complete the training of the domain discriminator and the training of the classification network of the entire image. T The data is input into the classification network as a test set, and the classification result is finally obtained.

[0012] The beneficial effects of the present invention are: 1) target domain data is not used during the training process, and only source domain data is used to train the network model, thereby improving the generalization performance of the network while ensuring the approximation of the global domain distribution. 2) taking into account more realistic scenarios, multiple source domain data are not necessarily completely separated, but are more likely to be mixed in the same source domain. After training with entropy loss, domain adversarial loss and classification loss as the overall loss function, a standard clustering algorithm is used to perform cluster analysis on the source domain data mixed with multiple potential domains, and the cluster labels after cluster analysis are used as pseudo-domain labels of the samples, and are further updated and optimized through the objective function to complete the training of the domain discriminator. 3) The unknown target domain data, that is, the target domain data that has not participated in the training, is input into the network model as a test set to verify the generalization performance of the network model, so that it can show better classification and prediction performance in the target domain test. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a network structure diagram of the method of the present invention;

[0014] Figure 2 It is a structural block diagram of the deep feature extraction network used in the method of the present invention. DETAILED DESCRIPTION

[0015] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] The present invention is based on a domain generalization image classification method based on a mixture of multiple potential domains, and is specifically implemented according to the following steps:

[0017] Step 1: Obtain domain generalization dataset, including source domain dataset Ds and target domain dataset D T , perform data enhancement and data set expansion, the specific process is,

[0018] The source domain dataset Ds follows a certain distribution P s (x), the class label is L s , m is the number of samples, that is

[0019] Target domain dataset D T Subordinate distribution P T (x), the class label is L T ,Right now

[0020] The source domain dataset Ds and the target domain dataset D T The data are input into the random data augmentation network in batches (the random data augmentation network is an existing conventional technology). The random data augmentation network performs a random operation on the source domain dataset Ds and the target domain dataset D T After rotating, cropping and adding Gaussian white noise transformation, the samples are restored to the original input size to form new samples, which are re-added to the source domain dataset Ds to achieve the purpose of dataset expansion.

[0021] Step 2: Build the AlexNet network, load the pre-trained weights, and input the source domain dataset Ds (composed of multiple potential domain samples) into the AlexNet network to perform adversarial training on the AlexNet network.

[0022] The structure of the AlexNet network is composed of an input layer, five convolutional layers, two fully connected layers and an output layer. Figure 2Each convolutional layer is followed by a nonlinear activation function, and each nonlinear activation function is followed by a maximum pooling layer (a total of five nonlinear activation functions and five maximum pooling layers); the last maximum pooling layer is followed by a fully connected layer, followed by another fully connected layer (a total of two fully connected layers), and then a Softmax function is added after the last fully connected layer.

[0023] The training process of the AlexNet network is:

[0024] 2.1) Reference Figure 2 , build the AlexNet network architecture, load the pre-trained model (pre-trained model is an existing conventional technology), and then extract a batch of data X from the source domain dataset Ds and input it into the AlexNet network;

[0025] 2.2) The AlexNet network performs convolution and pooling operations on the input data X (input image), calculates the loss function and gradient, backpropagates the classification error of the AlexNet network, iteratively updates the parameters of the AlexNet network, and completes the training of the AlexNet network.

[0026] The operation of the AlexNet network includes the calculation of three loss functions, namely the classification loss L cls (F f ,F c ), the adversarial loss L of the domain discriminator in adversarial learning adv (F f ,F d ), entropy loss L ent (F f ,F c ), the calculation formulas are as follows:

[0027] a) Classification loss L cls (F f ,F c ) is calculated as:

[0028]

[0029] Among them, F f is the feature extractor, F c is the classifier, N s Indicates the number of input samples, C represents the label set of the data sample, y i represents the label of the i-th sample;

[0030] b) Adversarial loss L adv (F f ,F d ) is calculated as:

[0031]

[0032] Among them, F d is the domain discriminator, represents a pseudo-domain, K represents a known number of domains, k represents the kth domain, represents a pseudo-domain label;

[0033] c) Entropy loss L ent (F f ,F c ) is calculated as:

[0034]

[0035] Where H(·) represents the cross entropy loss function;

[0036] At this point, we get two final optimization functions, which are expressed as follows:

[0037]

[0038]

[0039] Step 3: Extract the shallow features of the AlexNet network, obtain convolutional feature statistics, and calculate domain discriminant features. The specific process is as follows:

[0040] 3.1) Samples of the source domain dataset Ds The AlexNet network is used as the feature extractor F(·), and the shallow features extracted are called convolutional feature statistics;

[0041] 3.2) The extracted convolutional feature statistics are used to calculate the style loss. The mean μ(x) and standard deviation σ(x) are calculated independently across the spatial dimensions for each channel c. The calculation formulas are:

[0042]

[0043]

[0044] Where H and W represent the height and width of the feature vector respectively, c represents the channel of the feature vector, and x is the input data sample;

[0045] 3.3) After calculating the mean μ(x) and standard deviation σ(x), in order to combine the multi-scale style features obtained from different convolutional layers, the stack of the mean μ(x) and standard deviation σ(x) is defined as the domain discriminant feature ddf(x). The calculation formula of the domain discriminant feature ddf(x) is:

[0046] ddf(x)={μ(φ1(x)),σ(φ1(x)),…,μ(φM (x)),σ(φ M (x))} (8)

[0047] Where μ represents the calculation of the mean in equation (5), σ represents the calculation of the standard deviation in equation (6), and φ(·) represents the mapping function MMD.

[0048] Step 4: Use the standard K-means clustering algorithm to cluster the convolution feature statistics into several clusters, and assign pseudo-domain labels to the data based on the clusters. At the same time, the pseudo-domain labels are updated as the network training progresses. The specific process is as follows:

[0049] 4.1) After obtaining the domain discriminant feature ddf(x), cluster analysis is performed using the K-means algorithm.

[0050] First, the cluster center of the source domain data samples is calculated. Then, the sum of the squares of the distances between the sample points in each cluster set and the cluster center is minimized. The cluster center is updated repeatedly. The K-means algorithm is considered to have converged until the cluster center no longer changes or a certain number of iterations is reached.

[0051] The objective function of the K-means algorithm is:

[0052]

[0053] Where x is each data sample in the given source domain dataset Ds S i Represents the cluster set N={N1,N2,…,N k}, a subset of i is a set S i The center point of the cluster is also the cluster center;

[0054] 4.2) Based on the cluster centers obtained by the K-means algorithm, the data of multiple potential mixed source domain datasets Ds are divided into clusters, and use the clusters to allocate a i As a pseudo-domain label Assign the clustering result a i Convert to pseudo-domain label The process is iterated repeatedly to update. The expression of the objective function required for the update process is:

[0055]

[0056] Among them, N s is the sth cluster set, Indicates the pseudo domain label before an epoch, a i Indicates the Nth sThe cluster centers of the cluster sets, π represents the Kuhn-Munkres algorithm.

[0057] Step 5: Input the pseudo domain label output by the clustering algorithm into the domain discriminator to complete the training of the domain discriminator and the training of the classification network of the entire image. T The data is input into the classification network as a test set, and the classification result is finally obtained.

[0058] The specific process is:

[0059] 5.1) Input the pseudo domain label output in step 4 into the domain discriminator and combine it with the domain-invariant features extracted by the feature extractor to complete the training of the domain discriminator. The specific loss function is defined as shown in formula (2);

[0060] 5.2) Complete the training of the image classification network described in the method of the present invention, and set the target domain D that has not participated in the training T The data is input into the classification network as a test set, and the classification result is finally obtained.

[0061] Experimental verification:

[0062] To verify the effectiveness of the method of the present invention, we conducted a performance test on a currently available PACS dataset using the method of the present invention and two existing methods, CIDDG and MLDG. The specific results are shown in Table 1. As can be seen from Table 1, the accuracy of the method of the present invention is improved on the PACS dataset, indicating that the method of the present invention has higher accuracy and better reliability.

[0063] Table 1. Experimental results of domain generalization method on PACS dataset

[0064]

Claims

1. A domain generalization image classification method based on multiple potential domain mixtures, characterized by: Follow these steps to implement it: Step 1: Obtain domain generalization dataset, including source domain dataset and target domain dataset , perform data enhancement and data set expansion, the specific process is: The source domain dataset Obey a certain distribution , the category label is , is the number of samples, that is ; The target domain dataset Obey the distribution , the category label is ,Right now ; The source domain dataset , target domain dataset The data is input into the random data augmentation network in batches, and the random data augmentation network is used to extract the source domain dataset. and target domain dataset After rotating, cropping and adding Gaussian white noise transformation, the samples are restored to the original input size to form new samples and added back to the source domain dataset. In the implementation, data set expansion is realized; Step 2: Build the AlexNet network, load the pre-trained weights, and convert the source domain dataset into The data is input into the AlexNet network and adversarial training is performed on the AlexNet network; Step 3: Extract shallow features of the AlexNet network, obtain convolutional feature statistics, and calculate domain discriminant features; Step 4: Use the standard K-means clustering algorithm to cluster the convolution feature statistics into several clusters, and assign pseudo-domain labels to the data according to the clusters; at the same time, the pseudo-domain labels are updated as the network training process progresses; Step 5: Input the pseudo domain labels output by the clustering algorithm into the domain discriminator to complete the training of the domain discriminator and the training of the classification network of the entire image. The data is input into the classification network as a test set, and the classification result is finally obtained.

2. The domain generalization image classification method based on multiple potential domain mixtures according to claim 1, characterized in that In step 2, the specific process is: The structure of the AlexNet network is composed of an input layer, five convolutional layers, two fully connected layers and an output layer; each convolutional layer is followed by a nonlinear activation function, and each nonlinear activation function is followed by a maximum pooling layer; the last maximum pooling layer is followed by a fully connected layer, followed by another fully connected layer, and then a Softmax function is added after the last fully connected layer. The training process of the AlexNet network is: 2.1) Build the AlexNet network architecture, load the pre-trained model, and then Extract a batch of data , input to the AlexNet network; 2.2) AlexNet network for input data Perform convolution and pooling operations, calculate loss functions and gradients at the same time, backpropagate the classification error of the AlexNet network, iteratively update the parameters of the AlexNet network, and complete the training of the AlexNet network. The operation of the AlexNet network includes the calculation of three loss functions, namely classification loss Adversarial Loss for Domain Discriminators in Adversarial Learning , entropy loss , the calculation formulas are as follows: a) Classification loss The calculation formula is: (1) in, is the feature extractor, is the classifier, represents the number of input samples, represents the label set of the data sample, Indicates the The labels of the samples; b) Adversarial Loss The calculation formula is: (2) in, is the domain discriminator, Represents a pseudo-domain, represents a domain of known quantity, Indicates the domains, represents a pseudo-domain label; c) Entropy loss The calculation formula is: (3) in represents the cross entropy loss function; At this point, we get two final optimization functions, which are expressed as follows: (4) (5)。 3. The domain generalization image classification method based on multiple potential domain mixtures according to claim 1, characterized in that In step 3, the specific process is: 3.1) Source domain dataset Sample , and use the AlexNet network as the feature extractor ,The extracted shallow features are called convolutional feature statistics; 3.2) The extracted convolution feature statistics are used to calculate the style loss for each channel Compute means independently across spatial dimensions and standard deviation , the calculation formulas are: (6) (7) in, denote the height and width of the eigenvector respectively, represents the channel of the feature vector, is the input data sample; 3.3) Calculate the mean and standard deviation Later, in order to combine the multi-scale style features obtained from different convolutional layers, the mean and standard deviation The stack of is defined as the domain discriminant feature , domain discriminant features The calculation formula is: (8) in The calculation of the mean in expression (5) is: The standard deviation calculation in expression (6) is: Represents the mapping function MMD.

4. The domain generalization image classification method based on multiple potential domain mixtures according to claim 1, characterized in that In step 4, the specific process is: 4.1) Obtain domain discriminant features Then, cluster analysis was performed using the K-means algorithm. First, the cluster center of the source domain data samples is calculated. Then, the sum of the squares of the distances between the sample points in each cluster set and the cluster center is minimized. The cluster center is updated repeatedly. The K-means algorithm is considered to have converged until the cluster center no longer changes or a certain number of iterations is reached. The objective function of the K-means algorithm is: (9) in, For a given source domain dataset Each data sample in , Represents a cluster set A subset of is a collection The center point of the cluster is also the cluster center; 4.2) According to the cluster centers obtained by the K-means algorithm, multiple potential mixed source domain data sets are The data is divided into clusters and use cluster allocation As a pseudo-domain label , the cluster assignment results of the cluster Convert to pseudo-domain label The process is iterated repeatedly to update. The expression of the objective function required for the update process is: (10) in, For the A set of clusters, Indicates the pseudo domain label before an epoch, Indicates the The cluster centers of the cluster sets, Represents the Kuhn-Munkres algorithm.

5. The domain generalization image classification method based on multiple potential domain mixtures according to claim 1, characterized in that In step 5, the specific process is: 5.1) Input the pseudo domain label output in step 4 into the domain discriminator and combine it with the domain-invariant features extracted by the feature extractor to complete the training of the domain discriminator. The specific loss function is defined as shown in formula (2); 5.2) Complete the training of the image classification network described in the method of the present invention, and set the target domain that did not participate in the training The data is input into the classification network as a test set, and the classification result is finally obtained.

Images