A training method for an image classification model based on transfer learning

Abstract

A training method for an image classification model, wherein the image classification model extracts sample features corresponding to samples through a feature extractor and classifies samples according to the sample features through a classifier, the method comprising: S1, obtaining an initial training set, which includes samples from multiple source domains and labels indicating the source domain to which each sample belongs and the category to which it belongs, wherein the samples are images, the label space of each source domain is the same, and one label corresponds to one subdomain; S2, training the feature extractor to extract sample features using the initial training set, and updating the parameters of the feature extractor based on the loss value calculated based on a first loss function, wherein the first loss function is configured to penalize the differences between sample features corresponding to samples in the same subdomain of different source domains based on the domain attributes and labels; S3, performing augmentation processing on the samples in the initial training set to add augmented samples outside the multiple source domains, and using the samples and their augmented samples to perform adversarial training on the feature extractor and two classifiers.

Description

Technical Field

[0001] This invention relates to the field of computer vision, specifically to the field of image classification technology in machine learning, and more specifically, to a training method for an image classification model based on transfer learning. Background Technology

[0002] The most basic assumption in traditional machine learning, especially image classification models, is that the training data (source domain) and the test data (target domain) are independent and identically distributed. Under this assumption, the performance of the model on the test data is optimized by minimizing the error of the training data. However, the target domain data collected in a new environment usually has a different distribution than the source domain data. Specifically, when image classification models are faced with data that is unrelated to the training distribution, for example, when real product images under different viewing angles and lighting conditions are used as tests, the model trained on clean product images will show poor performance if traditional machine learning methods [1] are used to minimize the source domain experience loss.

[0003] Transfer learning is an important method to mitigate the disparity in target domain distribution. It trains a model using existing source domain data with a large number of labels to correctly identify target domains with unknown or few labels. In practice, the labeled datasets used for training are collected in different environments, and these datasets from different source domains often follow different distributions. This means that multi-source domain transfer learning in a single domain does not maximize the information advantage of multiple source domains. Furthermore, multi-source domain transfer learning tends to produce an overfitted model, making the model's performance unpredictable in unknown target domains.

[0004] Domain generalization usually focuses on the source domain data used for training, while domain adaptation usually focuses on both the source domain data used for training and the target domain data used for testing. The emergence of domain generalization is to minimize the generalization risk caused by dataset drift. Its typical technical feature is to learn domain-invariant representations. It collects data with different labels (one label corresponds to one subdomain) from different source domains to obtain more contextual information of objects, thereby learning the invariant association features between different data and labels, i.e., essential features. Existing domain adaptation methods, such as an adversarial source domain alignment method using a domain discriminator [2] and a maximum classifier difference source domain alignment method [3], have achieved good performance in the multi-source domain generalization problem. The main approach of current domain generalization is to align the feature distribution on the whole source domain (hereinafter referred to as: general domain alignment or source domain alignment) to obtain domain-invariant features. Aligning feature distributions on a per-source-domain basis (forced general domain alignment) may cause the model to learn over-compressed domain-invariant features with fuzzy class boundaries. Not only is domain invariance lost, but representations from different classes are also confused, resulting in performance issues in both the source and target domains.

[0005] In domain generalization adversarial methods, such as a deep domain adversarial image generation method for domain generalization [4], forced alignment of the general domain may cause the model to learn overcompressed domain-invariant features with fuzzy class classification boundaries. Although in domain adaptation, subdomain alignment can usually achieve better adaptive performance than the traditional global alignment method, in domain generalization, due to the lack of information in the target domain, it has been proven that general domain-invariant features will increase the span of the latent representation, especially for distant target domains; and if the sample features corresponding to the samples of the same subdomain of different source domains are directly aligned (i.e., the sample features of the same class are aligned, hereinafter referred to as subdomain alignment), the performance of the model in the unknown target domain (especially the target domain with a large data distribution difference from the source domain) is worse than the performance of the model obtained by the source domain alignment scheme, which makes it difficult to generalize subdomain alignment to the invisible target domain in domain generalization.

[0006] In a feature-styled domain-aware contrastive learning method for domain generalization[5], the idea of ​​contrastive learning is used to align the source domain, but the alignment of the subdomains of the source domain is not considered.

[0007] In summary, existing domain generalization methods cannot effectively align sample features from subdomains within different source domains when solving image classification problems involving multiple source domains, such as the PACS dataset. Furthermore, the unknown target domain poses a risk, leading to poor classification results.

[0008] In summary, among the existing domain generalization methods, aligning the entire source domain leads to the model learning over-compressed domain-invariant features, reducing the model's performance in both the source and target domains. On the other hand, directly aligning the sample features corresponding to samples in the same subdomain of different source domains is difficult to generalize to the invisible target domain.

[0009] References:

[0010] [1]Vapnik V N.An overview of statistical learning theory[J].IEEE transactions on neural networks,1999,10(5):988-999.

[0011] [2] Ganin Y, Ustinova E, Ajakan H, et al. Domain-adversarial training of neural networks [J]. The journal of machine learning research, 2016, 17(1): 2096-2030.

[0012] [3]Saito K,Watanabe K,Ushiku Y,et al.Maximum classifier discrepancy for unsupervised domain adaptation[C] / / Proceedings of the IEEE conference oncomputer vision and pattern recognition.2018:3723-3732.

[0013] [4]Zhou K, Yang Y, Hospedales T, et al.Deep domain-adversarial image generation for domain generalization[C] / / Proceedings of the AAAI Conference on Artificial Intelligence.2020,34(07):13025-13032.

[0014] [5]Jeon S, Hong K, Lee P, et al.Feature stylization and domain-awarecontrastive learning for domain generalization[C] / / Proceedings of the 29thACM International Conference on Multimedia.2021:22-31. Summary of the Invention

[0015] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a training method for an image classification model.

[0016] According to a first aspect of the present invention, a training method for an image classification model is provided. The image classification model includes a feature extractor and two classifiers. The feature extractor extracts sample features corresponding to samples, and the classifiers classify samples based on the sample features. The method includes: S1, obtaining an initial training set, which includes samples from multiple source domains and labels indicating the domain to which each sample belongs and the category to which it belongs, wherein the samples are images, the label spaces of each source domain are the same, and one label corresponds to one subdomain; S2, training the feature extractor to extract sample features using the initial training set, and updating the parameters of the feature extractor based on the loss value calculated by a first loss function, wherein the first loss function is configured to penalize the differences between sample features corresponding to samples in the same subdomain of different source domains based on the domain attributes and labels; S3, performing augmentation processing on the samples in the initial training set to add augmented samples outside the multiple source domains, and using the samples and their augmented samples to perform adversarial training on the feature extractor and the two classifiers.

[0017] In some embodiments of the present invention, step S3 includes: during adversarial training, updating the parameters of the two classifiers based on a loss calculated using a second loss function, wherein the second loss function is configured to penalize the classification bias of each classifier and reward the sum of the differences in confidence probabilities of the two classifiers for augmented samples in each class; and during adversarial training, updating the parameters of the feature extractor based on a loss calculated using a third loss function, wherein the third loss function is configured to penalize the classification bias of each classifier and penalize the sum of the differences in confidence probabilities of the two classifiers for augmented samples in each class.

[0018] In some embodiments of the present invention, the loss calculated by the second loss function is configured to be positively correlated with the deviation between the confidence probabilities of the two classifiers for the sample and its augmented sample in each class and the corresponding label, and negatively correlated with the sum of the differences in the confidence probabilities of the two classifiers for the augmented sample in each class.

[0019] Preferably, the second loss function is as follows:

[0020]

[0021] in, This represents the mean deviation between the confidence probabilities of the two classifiers for each sample and its augmented sample in each class and their corresponding labels. β represents the mean of the sum of the differences in confidence probabilities of the two classifiers for augmented samples in each class. Preset weights.

[0022] In some embodiments of the present invention, the loss calculated by the third loss function is configured to be positively correlated with the first loss function, positively correlated with the deviation between the confidence probability of each classifier for the sample and its augmented sample in each class and the corresponding label, and positively correlated with the sum of the differences in the confidence probabilities of the two classifiers for the augmented sample in each class.

[0023] Preferably, the third loss function is as follows:

[0024]

[0025] in, Denotes the first loss function. This represents the mean deviation between the confidence probabilities of the two classifiers for each sample and its augmented sample in each class and their corresponding labels. Let α represent the mean of the sum of the differences in confidence probabilities for augmented samples in each class between the two classifiers. The preset weights, μ, are denoted as Preset weights.

[0026] In some embodiments of the present invention Calculate as follows:

[0027]

[0028] Where 2N represents the number of samples used during training or augmentation, and L ce (·) represents the cross-entropy loss between the confidence probability output by the classifier and the corresponding label, G i (·) represents the confidence probability of the i-th classifier classifying the sample based on the sample features, where i=1 represents the first classifier and i=2 represents the second classifier. F(·) represents the sample features extracted by the feature extractor. m Let m represent the sample. y represents the augmented sample corresponding to sample m. m This represents the label corresponding to sample m.

[0029] In some embodiments of the present invention Calculate as follows:

[0030]

[0031] Where ||·||1 represents the L1 distance, M represents the number of augmented samples used during training, and C represents the total number of classes. Represents classifier G 1 Predicted Enhanced Samples The confidence probability of belonging to category c Represents classifier G2 Predicted Enhanced Samples The confidence probability of belonging to category c.

[0032] In some embodiments of the present invention, in step S2, N samples are taken from the initial training set as anchor samples for each training iteration, and samples with the same label but different domain attributes as each anchor sample are taken as positive samples of that anchor sample, forming a training batch with a batch size of 2N, and the loss value is calculated according to the following first loss function:

[0033]

[0034] Where i represents the number of an anchor sample in the training batch, and i+N represents the number of the positive sample of the anchor sample with number i in the training batch. The sub-loss l(i, i+N) or l(i+N, i) is calculated as follows:

[0035]

[0036] Where, when a = i, b = i + N, and when a = i + N, b = i; exp(·) represents an exponential function with the natural number e as the base; and sim(·) represents a function for calculating cosine similarity. Let a, b, and l represent the sample features corresponding to samples a, b, and l, respectively. Sample l is the sample with number l in the training batch. τ represents the temperature hyperparameter, k represents the domain attribute corresponding to the sample, y represents the label corresponding to the sample, and w(a, l) represents the... The weighting values ​​are set.

[0037] In some embodiments of the present invention, w(a, l) is determined as follows:

[0038]

[0039] Among them, M neg (a, l), M neg+ (a, l) represents the value determined based on the domain attributes and labels of samples a and l, where samples a and l have the same domain attributes but different labels. neg+ (a, l) = 1, otherwise M neg+ (a, l) = 0; when the domain attributes and labels of samples a and l are different, M neg (a, l) = 1, otherwise M neg (a, l) = 0; This represents the number of samples in the current training batch that have different domain attributes but the same label as sample a. This indicates the number of samples in the current training batch that have the same domain attributes as sample a but different labels.

[0040] In some embodiments of the present invention, in step S3, the image classification model is subjected to multiple rounds of adversarial training, and in each round, each sample is randomly enhanced using at least one of the multiple image enhancement methods, including: left-right flipping, color distortion, Gaussian blur, and daylighting, or a combination thereof.

[0041] In some embodiments of the present invention, the multi-source domain includes an art image domain, a cartoon domain, a photographic domain, and a hand-drawn domain, or a combination thereof, and the label corresponding to the sample indicates the category of the object contained in the image.

[0042] According to a second aspect of the present invention, an image classification method is provided, the method comprising: acquiring an image sample to be classified; and predicting the classification of the image sample to be classified using an image classification model, wherein the image classification model comprises a feature extractor obtained by the training method according to the first aspect of the present invention and one of two classifiers.

[0043] Compared with the prior art, the advantages of the present invention are as follows:

[0044] 1. Based on contrastive learning, dynamic domain weighted contrastive loss is used to align subdomains of multiple source domains, avoiding over-compression and having better adaptive capabilities.

[0045] 2. By constructing an augmentation domain during model training to simulate the potential target domain, and aligning the subdomains of the augmentation domain with the subdomains of the multi-source domain, the model expansion of subdomain alignment is achieved, reducing the risk of unknown target domains. Attached Figure Description

[0046] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0047] Figure 1 This is a schematic diagram of the structure of an image classification model without a head-mounted projection head according to an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of a training method for an image classification model according to an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of aligning the same subdomain of different source domains without a projection head according to an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of the structure of an image classification model including a projection head according to an embodiment of the present invention;

[0051] Figure 5 This is a schematic diagram of aligning the same subdomain of different source domains, including a projection head, according to an embodiment of the present invention.

[0052] Figure 6 This is a schematic diagram illustrating the training of an image classification model according to an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0054] As mentioned in the background section, existing domain generalization methods that align the entire source domain can lead to the model learning over-compressed domain-invariant features, reducing model performance in both the source and target domains. Directly aligning sample features from the same subdomain of different source domains is difficult to generalize to unseen target domains. Therefore, the inventors utilize the domain attributes and labels in the initial training set to align sample features from the same subdomain of different source domains. Furthermore, by augmenting the samples in the initial training set to add augmented samples from outside the source domains, an augmented training set is obtained. This augmented training set is then used to conduct adversarial training on the feature extractor and two classifiers. The aim is to allow the feature extractor and classifiers to continuously encounter samples from outside the source domains (i.e., augmented samples) during the adversarial process, thereby better learning more fundamental sample features. This improves the model's classification performance for samples in the unknown target domain while maintaining its performance in the known source domain.

[0055] To better understand this invention, the following detailed description, in conjunction with the accompanying drawings and embodiments, covers four aspects: model structure, training samples, model training, and application scenarios.

[0056] I. Model Structure

[0057] According to one embodiment of the present invention, see Figure 1This invention provides an image classification model, which includes a feature extractor and two classifiers: Classifier 1 and Classifier 2. According to one embodiment of the invention, the feature extractor is used to extract features from samples to generate sample features, and the classifiers are used to classify samples based on these features. The feature extractor can be an existing neural network backbone, such as a ResNet model (e.g., ResNet18, ResNet34, or ResNet50), an AlexNet model, a VGG model (e.g., VGG16, VGG19), a Transformer network backbone, or a custom feature extractor defined by the implementer. For example, in a ResNet model, the backbone network consisting of conv1, conv2_x, conv3_x, conv4_x, conv5_x, and an average pooling layer can be used as the feature extractor. Similarly, in a VGG model, the backbone network consisting of the penultimate fully connected layer (FC-4096) and all layers preceding it can be used as the feature extractor. According to one embodiment of the present invention, classifier 1 and classifier 2 may employ the same structure. Each classifier includes a linear layer (such as one or more fully connected layers) for linearly transforming sample features to obtain classification features, and a Softmax layer for calculating the confidence probability of each category based on the classification features. The classifier ultimately outputs the category with the highest confidence probability as the predicted classification result.

[0058] II. Training Samples

[0059] According to one embodiment of the present invention, the dataset of the present invention can adopt an existing image classification dataset, which includes samples from multiple source domains and labels indicating the domain attributes and category of each sample. The samples are images, and the label space of each source domain is the same, with one label corresponding to one subdomain. The data distribution corresponding to samples from different source domains is different. Since data from multiple source domains is required, the cost of collecting sample data is high. For testing purposes, publicly available datasets can be used. For example, the PACS image dataset, which is a commonly used dataset for multi-domain adaptation in transfer learning, includes image data from multiple source domains, including Artpainting, Cartoon, Photo, Sketch, or combinations thereof. The samples from each source domain are images, and the labels corresponding to the samples indicate the category of the objects contained in the image. Labels include: Dog, Elephant, Giraffe, Guitar, Horse, House, and Person. It should be understood that the image classification model described in this invention can be applied to various classification scenarios. The above classification scenarios are merely illustrative, and those skilled in the art can adjust and set them as needed. For example, they can add labels such as "cow" and "bird," or set a custom label space, such as labels including "horse," "cow," "dog," "cat," "mouse," "bird," "car," "bicycle," "electric car," "house," and "person." This invention does not impose any limitations in this regard. For ease of explanation and testing of the effectiveness of this invention, the following descriptions will use a PACS image dataset as an example.

[0060] Before training, the dataset can be further divided into training and validation sets, or training, validation, and test sets. Typically, this is done in a specific ratio, such as an 8:2 ratio, or an 8:1:1 or 7:2:1 ratio. If using a PACS image dataset, the initial training set is obtained by dividing the PACS image dataset according to this ratio. Additionally, if the size of the samples in the PACS image dataset is inconsistent with the input size of the selected feature extractor, the sample size can be adjusted first. For example, assuming the input size is 224×224, to facilitate the use of samples in subsequent training, when obtaining the initial training set, all image samples are first uniformly resized to 227×227, and then 224×224 cropped pixels are obtained from the center of each image.

[0061] III. Model Training

[0062] According to one embodiment of the present invention, see Figure 2 The present invention provides a method for... Figure 1A training method for an image classification model is provided, comprising performing one or more steps: S1, S2, and S3. To better understand this invention, each step is described in detail below with reference to specific embodiments.

[0063] In step S1, an initial training set is obtained, which includes samples from multiple source domains and domain attributes indicating the source domain to which each sample belongs and labels indicating the category to which it belongs. The samples are images, and the label space of each source domain is the same, with one label corresponding to one subdomain.

[0064] In step S2, a feature extractor is trained using the initial training set to extract sample features, and the parameters of the feature extractor are updated based on the loss value calculated using a predetermined first loss function. This first loss function is configured to penalize the differences between sample features corresponding to samples in the same subdomain from different source domains based on domain attributes and labels. Thus, sample features corresponding to samples in the same subdomain from different source domains are aligned based on domain attributes.

[0065] To better align subdomains from different source domains, we can selectively utilize the domain attributes and labels in the initial training set. If a sample is selected as the anchor sample, comparing its domain attributes and labels with those of the corresponding sample can classify samples into different types. This allows for better utilization of different sample types during subdomain alignment, thus improving the alignment effect. For example, any sample from the initial training set can be used as the anchor sample. Samples with different domain attributes but the same labels as anchor samples For positive samples (k1≠k2, Where k is the domain attribute and y is the label), and the sample has the same domain attribute as the anchor sample but a different label. Type I negative sample Samples that are different from anchor samples in both domain attributes and labels Type II negative samples

[0066] To better understand the first loss function, the relevant parameters involved are explained below. In step S2, each training iteration takes N samples from the initial training set as anchor samples and takes samples with the same label but different domain attributes as positive samples of each anchor sample, forming a training batch with a batch size of 2N. For example, this invention divides the samples in the initial training set into several groups, each containing N samples, and uses these N samples as anchor samples. For each sample, select a positive sample pair (y) with it. i =y i+N , k≠k i+N A sample of size ) constitutes a sample of size ) The initial training set sample batch. In each initial training set sample batch, the positive sample corresponding to each sample i (where i∈[1,N]) is sample i+N (where i+N∈[1+N,2N]). This invention enhances the sample weights of the first type of negative samples, wherein the weighted value (also called weight) corresponding to the sample pair formed by the two samples is calculated through the domain attributes and label categories of the samples, and the set of weighted values ​​constitutes the domain weighted value matrix (also called the domain weighted weight matrix). In order to obtain the domain weighted value matrix, for different sample pairs, some values ​​determined according to the domain attributes and labels of the two samples in the sample pair can be set in the following way to calculate the weighted values ​​corresponding to different sample pairs: when sample i (where i∈[1,N]) and sample j (where j∈[1,2N]) form a positive sample pair, M pos (i, j) = 1, otherwise M pos When (i, j) = 0, and two samples i and j form a negative sample pair of the first type, M neg+ (i, j) = 1, otherwise M neg+ When (i, j) = 0, and two samples i and j form a negative sample pair of the second type, M neg (i, j) = 1, otherwise M neg (i, j) = 0, further,

[0067] According to an embodiment of the present invention, in step S2, the loss value is calculated according to the following first loss function:

[0068]

[0069] Where i represents the index of an anchor sample in the training batch, i+N represents the index of the positive sample of anchor sample i in the training batch, and l(i, i+N) and l(i+N, i) represent sub-losses. During training, the parameters of the feature extractor are updated based on the loss value calculated by the first loss function. The purpose is to minimize the loss value calculated by the first loss function, and the corresponding formula is as follows:

[0070]

[0071] According to one embodiment of the present invention, the sub-loss l(i, i+N) or l(i+N, i) is calculated as follows:

[0072]

[0073] Where, when a = i, b = i + N, and when a = i + N, b = i; exp(·) represents an exponential function with the natural number e as the base; and sim(·) represents a function for calculating cosine similarity. Let a, b, and l represent the sample features corresponding to samples a, b, and l, respectively. Sample l is the sample with number l in the training batch. τ represents the temperature hyperparameter, k represents the domain attribute corresponding to the sample, y represents the label corresponding to the sample, and w(a, l) represents the... The set weighting value. The temperature hyperparameter is usually set to a value less than 1 to adjust the distribution of the output sample features, increase the similarity difference between similar and dissimilar sample pairs to make the distribution smoother, thereby amplifying the class similarity and improving the model's discriminative ability. Of course, the temperature hyperparameter is not necessary; it can be set to 1, which is equivalent to no adjustment; or the temperature hyperparameter can be directly canceled. Subsequent embodiments involving temperature hyperparameters are similar and will not be elaborated further. The technical solution of this embodiment can at least achieve the following beneficial technical effects: updating the parameters of the feature extractor by using the loss value calculated by the first loss function can shorten the distance between sample features corresponding to samples in the same subdomain of different source domains in the latent space (i.e., the feature space corresponding to sample features), and widen the distance between sample features corresponding to samples in different subdomains. This aligns the sample features corresponding to samples in the same subdomain of different source domains based on domain attributes and labels, enabling the feature extractor to better learn the more essential features in the initial training set of multi-source domains that characterize whether samples are classified the same or different.

[0074] According to one embodiment of the present invention, the weighting value can be a value preset by the implementer, for example, setting the weighting value when sample l is a positive sample of sample a, a first-class negative sample, and a second-class negative sample respectively. Since the training process typically uses mini-batch gradient descent, the initial training set is divided into multiple batches, and the image classification model is trained and its parameters are updated in batches. To better ensure the subdomain alignment effect, w(a, l) can be dynamically determined according to the actual situation of different types of samples in a batch. According to one embodiment of the present invention, w(a, l) is determined in the following manner:

[0075]

[0076] Among them, M neg (a, l), M neg+ (a, l) represents the value determined based on the domain attributes and labels of samples a and l, where samples a and l have the same domain attributes but different labels. neg+ (a, l) = 1, otherwise M neg+ (a, l) = 0; when the domain attributes and labels of samples a and l are different, M neg (a, l) = 1, otherwise M neg (a, l) = 0; This represents the number of samples in the current training batch that have different domain attributes but the same label as sample a. This represents the number of samples in the current training batch that have the same domain attributes as sample a but different labels. The technical solution of this embodiment can achieve at least the following beneficial technical effects: by calculating w(a, l), the weight of samples in a batch that have the same domain attributes as each sample a but different labels can be increased, thus extracting sample features with better domain removal characteristics. It should be noted that a and l in w(a, l) correspond to the domain attributes mentioned above. i and j in M; neg (a, l) corresponds to M mentioned above. neg (i, j), M ne (a, l) corresponds to M mentioned above. neg+ (i, j).

[0077] If the implementer has a specific backbone network they want to use, they can directly construct an image classification model and use the sample features output by the backbone network directly in the first loss function to calculate the loss value, for example, see [link to relevant documentation]. Figure 3 This invention generates sample features corresponding to samples x in the initial training set through a feature extractor. z The parameters of the feature extractor are updated based on the loss value calculated using a predetermined first loss function, so as to align the sample features corresponding to samples in the same subdomain of different source domains based on domain attributes. Figure 3 Among the parameters involved, This represents an anchor sample in the initial training set. This indicates the sample features generated by the feature extractor for the anchor sample. This represents the negative sample of the anchor sample. This indicates the sample features generated by the feature extractor from the negative sample. This indicates the positive sample (y) of the anchor sample. j1 =y j2 (k1≠k2), This represents the sample features generated by the feature extractor for the positive sample, where j represents the sample number, k represents the domain attribute of the sample, and y represents the label of the sample.

[0078] Since different existing backbone networks can be used, if the implementer wants to select the optimal backbone network from multiple backbone networks to construct the model, the dimensions of the sample features output by different backbone networks may be inconsistent, making comparison inconvenient. Therefore, a projection head can be designed (the structure of the image classification model described in this invention after adding the projection head is as follows). Figure 4 As shown, a projection head is used to transform the sample features extracted by the backbone network to a predetermined feature dimension, resulting in the transformed sample features. This facilitates comparison between different backbone networks. According to an embodiment of the present invention, see... Figure 5After adding the projection head, the cosine similarity of the sample features calculated in the first loss function uses the sample features transformed by the projection head. The projection head can be implemented using linear layers (such as a single-layer fully connected layer or a multilayer perceptron with a single hidden layer). In step S2, when training the feature extractor, this invention first generates corresponding sample features z′ from the sample x in the initial training set through the backbone network, and then performs dimensionality reduction through the projection head to obtain the transformed sample features z (in this case, the transformed sample features z is the final output of feature extraction). The parameters of the backbone network and the projection head are then updated using the loss value calculated by the first loss function based on the transformed sample features. Figure 5 Among the parameters involved, z′ represents the corresponding sample features generated through the backbone network, z represents the sample features z transformed by the projection head after dimensionality reduction. The descriptions of the remaining parameters are the same as those in the previous section. Figure 3 The same applies, so I will not repeat it here.

[0079] The parameters of the feature extractor and the projection head are updated by minimizing the first loss function through the above steps to align the subdomains of multi-source domain samples in the current batch (i.e., align the sample features corresponding to samples in the same subdomain of different source domains based on domain attributes). This makes the distribution of sample features with the same label in the latent space as consistent as possible, reduces the diameter of the convex hull of the subdomain, and increases the distance between the centers of the convex hulls of different subdomains. It should be noted that the convex hull of the multi-source domain refers to the intersection of all convex sets of sample features corresponding to all initial training set samples in the latent space from a geometric perspective, and the convex hull of the subdomain refers to the intersection of all convex sets of sample features corresponding to samples with the same label. For each training batch, adversarial training of the feature extractor and classifier is performed under the premise of aligning the sample features corresponding to samples in the same subdomain of different source domains based on domain attributes.

[0080] In step S3, the samples in the initial training set are augmented to add augmented samples outside the multiple source domains, and the feature extractor and two classifiers are trained adversarially using the samples and their augmented samples.

[0081] According to one embodiment of the present invention, the present invention enhances samples in the initial training set to add enhanced samples outside the multiple source domains, and uses the samples and their enhanced samples to perform adversarial training on the feature extractor and classifier. To ensure that the novel style of the dataset satisfies diversity and rationality, and that the enhanced domain formed by the enhanced samples can approximately represent the target domain, the enhanced domain is required to be as diverse as possible while maintaining semantic invariance. Simultaneously, to ensure style realism and not deviate from the distribution of the true source styles in the initial training set, the present invention randomly employs at least one of the multiple image enhancement methods for each sample during the adversarial training process in step S3. These multiple image enhancement methods include: left-right flipping, color distortion, Gaussian blur, and daylighting, or combinations thereof. For example, assuming a sample is x, the present invention can define a transformation M based on the SimCLR framework to achieve image enhancement, obtaining the enhanced sample:

[0082] According to one embodiment of the present invention, the adversarial training can be performed in one or more epochs. In each epoch, the initial training set can be divided into multiple batches, and image augmentation is performed on the samples in each batch to obtain augmented samples. For example, the size of each batch is 2N (which is the size of one batch in step S2). Moreover, preferably, each sample is augmented randomly using at least one of the multiple image augmentation methods each time. Thus, in different epochs, the augmented samples of the same sample may use different image augmentation methods, so that the image classification model sees augmented samples with a different style from the source domain samples in adversarial training, thereby further improving the domain generalization and enhancing the performance of the image classification model in the potential target domain.

[0083] After obtaining the augmented samples for each batch through the above steps, the first loss value calculated in step S2 is directly obtained from the initial training set samples of the current batch. Then, the samples of the current batch and their augmented samples are input into the feature extractor for feature extraction to generate sample features. These sample features are then input into two classifiers to output the predicted classification results. The parameters of the two classifiers are updated based on the loss calculated using a predetermined second loss function, and the parameters of the feature extractor are updated based on the loss calculated using a predetermined third loss function. The second and third loss functions are explained below.

[0084] According to one embodiment of the present invention, the loss calculated by the second loss function is configured to be positively correlated with the deviation between the confidence probabilities of the two classifiers for samples and their augmented samples in each category and their corresponding labels, and negatively correlated with the sum of the differences in the confidence probabilities of the two classifiers for augmented samples in each category. This is used to penalize the classification bias of each classifier and reward the sum of the differences in the confidence probabilities of the two classifiers for augmented samples in each category. The technical solution of this embodiment can achieve at least the following beneficial technical effects: The loss calculated by the second loss function of the present invention is configured to be positively correlated with the deviation between the confidence probabilities of the two classifiers for samples and their augmented samples in each category and their corresponding labels, thus allowing the two classifiers to learn correct classification knowledge. Simultaneously, the loss calculated by the second loss function of the present invention is configured to be negatively correlated with the sum of the differences in the confidence probabilities of the two classifiers for augmented samples in each category, which is equivalent to rewarding the two classifiers for the differences in their confidence probabilities in each category, provided that they classify correctly. This allows both classifiers to learn more generalized classification knowledge, which is superior to image classification models containing a single classifier and helps improve the performance of image classification models.

[0085] According to one embodiment of the present invention, the second loss function is as follows:

[0086]

[0087] in, This represents the mean deviation between the confidence probabilities of the sample and its augmented samples in each class and their corresponding labels (in this invention, it is also referred to as the expected cross-entropy loss of the sample and its augmented samples on the two classifiers). β represents the mean of the sum of the confidence probabilities of the two classifiers for augmented samples in each class (also referred to as the L1 classification difference distance in this invention), where β is denoted as... Preset weights. Preferably, in this invention, β is preset to 0.5.

[0088] It should be understood that the purpose of updating the classifier's parameters is to minimize the second loss function, as shown in the following formula:

[0089]

[0090] According to one embodiment of the present invention, the loss calculated by the third loss function is configured to be positively correlated with the first loss function, positively correlated with the bias between the confidence probability of each classifier for a sample and its augmented sample in each class and the corresponding label, and positively correlated with the sum of the differences in the confidence probabilities of the two classifiers for the augmented sample in each class. This loss is used to penalize the classification bias of each classifier and to penalize the sum of the differences in the confidence probabilities of the two classifiers for the augmented sample in each class. It should be noted that the third loss function is only used to update the parameters of the feature extractor and not the parameters of the classifier. By minimizing the third loss function, the feature extractor is forced to generate features with stronger generalization ability (i.e., to minimize the difference in classification results of the augmented sample between the two classifiers), thereby completing one adversarial training of the feature extractor against the two classifiers.

[0091] According to one embodiment of the present invention, the third loss function is as follows:

[0092]

[0093] in, Denotes the first loss function. This represents the mean deviation between the confidence probabilities of the two classifiers for each sample and its augmented sample in each class and their corresponding labels. Let α represent the mean of the sum of the differences in confidence probabilities for augmented samples in each class between the two classifiers. The preset weights, μ, are denoted as The preset weights are given, where μ ∈ [0, 1]. Preferably, in this invention, α is preset to 1 and μ is preset to 0.4.

[0094] It should be understood that in step S3, the purpose of updating the parameters of the feature extractor is to minimize the third loss function, the corresponding formula of which is as follows:

[0095]

[0096] According to one embodiment of the present invention, Calculate as follows:

[0097]

[0098] Where 2N represents the number of samples used during training or augmentation, and L ce (·) represents the cross-entropy loss between the confidence probability output by the classifier and the corresponding label, G i (·) represents the confidence probability of the i-th classifier classifying the sample based on the sample features, where i=1 represents the first classifier and i=2 represents the second classifier. F(·) represents the sample features extracted by the feature extractor.m Let m represent the sample. y represents the augmented sample corresponding to sample m. m This represents the label corresponding to sample m.

[0099] In the above calculation During the process, each time a sample and its corresponding augmented sample are input into two classifiers G respectively. 1 and G 2 The expected cross-entropy loss is calculated, and the calculation is performed on each sample in the multi-source domain sample and its corresponding augmented sample to obtain the expected cross-entropy loss of the entire sample batch.

[0100] When training with each batch of dataset, by minimizing To ensure that the initial training set samples are correctly and identically classified on both classifiers (i.e., the expected loss of source domain empirical risk is minimized), adversarial training is performed on the feature extractor and classifier to enhance the samples.

[0101] According to one embodiment of the present invention, Calculate as follows:

[0102]

[0103] Where ||·||1 represents the L1 distance, M represents the number of augmented samples used during training, and C represents the total number of classes. Represents classifier G 1 Predicted Enhanced Samples The confidence probability of belonging to category c Represents classifier G 2 Predicted Enhanced Samples The confidence probability of belonging to class c. It should be noted that, for mini-batch training, training here refers to training in a single batch.

[0104] In the above calculation During the process, each augmented sample is input into two classifiers G. 1 and G 2 The L1 classification difference loss (i.e., the absolute difference between the probability outputs of the augmented sample on the two classifiers) is calculated, and this calculation is repeated for each augmented sample to obtain the overall L1 classification difference loss (also known as the L1 classification difference distance) for the entire batch of augmented samples. Since the two classifiers have similar structures, it is difficult to produce different classification results on the augmented samples. Therefore, it is necessary to increase the sum of the confidence probability differences of the two classifiers for each category of the augmented sample so that the augmented sample can produce different classification results on the two classifiers, facilitating each classifier to learn the different features of the augmented sample.

[0105] It should be noted that the parameters mentioned in this invention refer to the parameters updated by the backpropagation algorithm for the feature extractor F and / or classifier G. 1 G 2 The trainable parameters (also referred to as weight parameters in some literature) in neural networks are currently updated using various methods, such as SGD, Momentum, and Adam. This invention preferably uses the Adam optimizer to optimize the neural network parameters.

[0106] The following is based on Figure 6 To more clearly illustrate the training process of this invention, class A and class B represent different categories (corresponding to different subdomains, distinguished by different gray depths, with class B having a deeper gray depth than class A), where D... S Representing the source domain, denoted by D S1 D S2 D S3 To distinguish different source domains, using D Aug This represents the domain corresponding to the augmented sample (abbreviated as augmentation domain). Figure 6 ab illustrates subdomain alignment by describing the changes in the distribution corresponding to the sample features of a subdomain during training. During training, step S1 of this invention is followed first. Figure 6 (Not shown) Obtain the initial training set, at which point the distribution of sample features in this subdomain corresponds to Figure 6 As shown in the initial state (a), it can be seen that the overall data distribution diameter (hereinafter referred to as convex hull diameter) of the sample features in this subdomain from multiple source domains is relatively dispersed, the convex hull diameter is large, and the data distribution corresponding to the sample features of the same subdomain from different source domains is also far apart. Then, the parameters of the feature extractor are updated according to step S2. The first loss function penalizes the differences between the sample features corresponding to the samples in the same subdomain from different source domains based on domain attributes and labels, thereby narrowing the distance between the data distributions of the sample features corresponding to the samples in the same subdomain from different source domains, aligning the sample features corresponding to the samples in the same subdomain from different source domains, reducing the convex hull diameter of this subdomain, and obtaining the corresponding... Figure 6 The state of b; then, following step S3, adversarial training is performed using the samples and their augmented samples, so that the model can see augmented samples that do not belong to any source domain, while simultaneously narrowing the distance between the data distribution of the sample features of the augmented samples and the sample features corresponding to samples from multiple source domains, thus obtaining the corresponding... Figure 6 The state of c. Additionally, to demonstrate the distribution changes of sample features in different subdomains corresponding to the augmented domain formed by the augmented samples after adversarial training, the following is given: Figure 6d shows the changes in the subdomains of class A and class B formed by the augmented sample-composed augmented domain after step S3. It can be seen that after adversarial training, the distance between the data distribution of sample features of the subdomains of the augmented domain and the corresponding subdomains of the source domain can be narrowed. At the same time, the distance between the data distribution of sample features of different subdomains corresponding to the augmented domain is also increased.

[0107] It should be understood that although the foregoing embodiments of the present invention provide a specific implementation detail of the first loss function, the second loss function, and the third loss function, they are not the only implementation means. Those skilled in the art can use alternative means that can achieve the same technical effect according to the principles of the present invention. For example, for the first loss function, the traditional triplet loss function can also be used, which groups the samples into triplets including anchor samples, positive samples, and negative samples, in the form of: {anchor sample, positive sample, negative sample}. The positive sample is a sample that belongs to the same subdomain as the anchor sample but to a different source domain, and the negative sample is a sample that belongs to the same or different source domain as the anchor sample but to a different subdomain. In step S2, the parameters of the feature extractor are updated with the loss value calculated by the triplet loss function. For the second loss function, it is used to penalize the classification bias of each classifier and reward the sum of the differences in the confidence probabilities of the two classifiers for the augmented samples in each category. Therefore, under the premise of ensuring this, there are many feasible alternative implementation methods, such as changing the second loss function to: Of course, some sub-loss coefficients in the second loss function can also be modified, for example: In Change to Similarly, the third loss function is used to penalize the classification bias of each classifier and the sum of the differences in confidence probabilities of the two classifiers for the augmented samples in each class. Therefore, while ensuring this premise, there are several feasible alternative implementations, such as changing the third loss function to: Of course, some sub-loss coefficients in the third loss function can also be modified, for example: In Change to Alternatively, for the first or third loss function, the weights w(a,l) can also be... Change to

[0108] IV. Application Scenarios

[0109] According to an embodiment of the present invention, the present invention also provides a classification method, the method comprising: obtaining a prediction model composed of a feature extractor and one of two classifiers in an image classification model trained according to the training method of the image classification model; obtaining a sample to be predicted; using the prediction model to predict the classification to which the sample to be predicted belongs; and obtaining the corresponding classification result.

[0110] To verify the effectiveness of the invention, the inventors conducted experiments. Since collecting samples from multiple source domains and an unknown target domain is costly, in the experiments, the inventors selected one domain from the PACS dataset as the unknown target domain and used other domains as source domains. For example, a hand-drawn domain could be used as the unknown target domain, while artistic image, cartoon, and photograph domains could be set as multiple source domains. Samples from the hand-drawn domain were removed from the original PACS dataset to obtain a modified PACS dataset. An initial training set and a test set were then obtained from the modified PACS dataset. The initial training set was used to train the image classification model, and the test set was used to test the accuracy of the trained image classification model. Before classifying and predicting the data to be predicted for the actual unknown target domain using the image classification model, the classification results of two classifiers could be tested using the test set. If the classification results of the two classifiers were the same, either classifier could be selected to classify and predict the unknown target domain. If the classification results of the two classifiers were different, the classifier with the more accurate classification result would be selected to classify and predict the unknown target domain. Finally, the five existing methods mentioned in the background technology were compared with the solution of the present invention. The classification accuracy results are shown in Table 1. It can be seen that the solution provided by the present invention has higher classification accuracy and average classification accuracy when classifying each source domain as an unknown target domain than the existing methods. It can achieve a more accurate classification effect than the existing methods.

[0111] Table 1

[0112]

[0113]

[0114] Note:

[0115] Method 1: A method for minimizing the source domain empirical loss [1]

[0116] Method 2: An adversarial source domain alignment method using a domain discriminator [2]

[0117] Method 3: A method for aligning the differential source domains of a maximum classifier [3]

[0118] Method 4: A deep domain adversarial image generation method oriented towards domain generalization [4]

[0119] Method 5: A feature stylization and domain-aware contrastive learning method for domain generalization [5]

[0120] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0121] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0122] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0123] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A training method for an image classification model, characterized in that, The image classification model includes a feature extractor and two classifiers. The feature extractor extracts sample features corresponding to the samples, and the classifiers classify the samples based on these features. The method includes: S1. Obtain the initial training set, which includes samples from multiple source domains and labels indicating the source domain to which each sample belongs and the category to which it belongs. The samples are images, and the label space of each source domain is the same. One label corresponds to one subdomain. S2. Train a feature extractor using the initial training set to extract sample features, and update the parameters of the feature extractor based on the loss value calculated by the first loss function. The first loss function is configured to penalize the differences between sample features corresponding to samples in the same subdomain of different source domains based on domain attributes and labels; wherein, each training iteration takes samples from the initial training set. Each sample is used as an anchor sample, and samples with the same label but different domain attributes as each anchor sample are taken as positive samples for that anchor sample, forming a batch size of [number missing]. The training batches are calculated using the following first loss function: in, This represents the ID of an anchor sample in a training batch. This indicates that the batch numbered in the training batch is... The anchor sample's positive sample number, sub-loss or Calculate as follows: in, , , Represented by natural numbers An exponential function with base 0. This represents the function for calculating cosine similarity. , , Representing samples respectively , ,sample Corresponding sample features, sample For the training batch numbered The sample, This indicates temperature hyperparameters. This represents the domain attribute corresponding to the sample. This indicates the label corresponding to the sample. Represented as The set weighting value; S3. The samples in the initial training set are augmented to add augmented samples outside the multiple source domains. The samples and their augmented samples are used to perform adversarial training on the feature extractor and the two classifiers.

2. The method according to claim 1, characterized in that, Step S3 includes: During adversarial training, the parameters of the two classifiers are updated based on a loss calculated using a second loss function, which is configured to penalize the classification bias of each classifier and reward the sum of the differences in confidence probabilities of the two classifiers for augmented samples in each class; and During adversarial training, the parameters of the feature extractor are updated based on the loss calculated using a third loss function, which is configured to penalize the bias of each classifier's classification and the sum of the differences in confidence probabilities of the two classifiers for augmented samples in each class.

3. The method according to claim 2, characterized in that, The loss calculated by the second loss function is configured to be positively correlated with the deviation between the confidence probabilities of the two classifiers for the sample and its augmented sample in each class and the corresponding labels, and negatively correlated with the sum of the differences in the confidence probabilities of the two classifiers for the augmented sample in each class.

4. The method according to claim 3, characterized in that, The second loss function is as follows: in, This represents the mean deviation between the confidence probabilities of the two classifiers for each sample and its augmented sample in each class and their corresponding labels. This represents the mean of the sum of the differences in the confidence probabilities of the two classifiers for augmented samples in each class. Represented as Preset weights.

5. The method according to claim 2, characterized in that, The loss calculated by the third loss function is configured to be positively correlated with the first loss function, positively correlated with the deviation between the confidence probability of each classifier for the sample and its augmented sample in each class and the corresponding label, and positively correlated with the sum of the differences in the confidence probabilities of the two classifiers for the augmented sample in each class.

6. The method according to claim 5, characterized in that, The third loss function is as follows: in, Denotes the first loss function. , This represents the mean deviation between the confidence probabilities of the two classifiers for each sample and its augmented sample in each class and their corresponding labels. This represents the mean of the sum of the differences in the confidence probabilities of the two classifiers for augmented samples in each class. Represented as Preset weights Represented as Preset weights.

7. The method according to claim 4 or 6, characterized in that, Calculate as follows: in, This indicates the number of samples used during training or the number of augmented samples. The cross-entropy loss represents the relationship between the confidence probability output by the classifier and the corresponding label. Indicates the first The confidence probability of a classifier classifying a sample based on its features. Time represents the first classifier. The time represents the second classifier. This represents the sample features extracted by the feature extractor. Indicates sample , Indicates sample The corresponding augmented samples, Indicates sample The corresponding tags.

8. The method according to claim 4 or 6, characterized in that, Calculate as follows: in, Let L represent the L1 distance, and M represent the number of augmented samples used during training. Indicates the total number of categories. Represents classifier Predicted Enhanced Samples Category The confidence probability, Represents classifier Predicted Enhanced Samples Category The confidence probability.

9. The method according to claim 8, characterized in that, Determined in the following manner: in, , Indicates based on sample and samples The domain attributes and the values ​​determined by the tags, where the sample and samples When the domain attributes are the same but the tags are different =1, otherwise ;sample and samples When the domain attributes and tags are all different =1, otherwise =0; Indicates the samples in the current training batch. The number of samples with different domain attributes but the same label. Indicates the samples in the current training batch. The number of samples with the same domain attributes but different labels.

10. The method according to any one of claims 1-6, characterized in that, In step S3, the image classification model is subjected to multiple rounds of adversarial training, and in each round, each sample is randomly enhanced using at least one of a variety of image enhancement methods, including: left and right flipping, color distortion, Gaussian blur and daylighting, or a combination thereof.

11. The method according to any one of claims 1-6, characterized in that, Multiple source domains include artistic image domains, cartoon domains, photographic domains, and hand-drawn domains, or combinations thereof, and the labels corresponding to the samples indicate the categories of objects contained in the image.

12. An image classification method, characterized in that, The method includes: Obtain image samples to be classified; An image classification model is used to predict the category to which an image sample to be classified belongs, wherein the image classification model consists of a feature extractor obtained by the training method according to any one of claims 1-11 and one of two classifiers.

13. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1-11 and 12.

14. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the electronic device to perform the steps of the method as described in any one of claims 1-11 and 12.

Images