Cross-domain long-tail image classification method based on self-supervised learning and self-training mechanism

Abstract

The application discloses a cross-domain long-tail image classification method based on self-supervised learning and self-training mechanism, comprising the following steps: firstly, acquiring a dataset of cross-domain long-tail images, and dividing the dataset into a training set and a test set; then, training a double-branch network model by using the cross-domain long-tail image data in the training set; for the unsupervised adaptive branch, using a self-supervised learning and global distribution alignment method for representation learning; for the supervised adaptive branch, self-training and mutual information maximization are performed on the classifier; using a dynamic weighted ensemble training strategy to automatically adjust the learning weights of the two branches, so as to realize the simultaneous optimization of the learning objectives of the two branches; finally, classifying the cross-domain long-tail image data in the test set by using the trained model. The application solves the problem that the existing classification method is prone to cross-domain long-tail image classification errors due to insufficient extraction of image discriminative features, model decision boundary deviation, error accumulation in the self-training process and the like.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a cross-domain long-tail image classification method based on self-supervised learning and self-training mechanisms. Background Technology

[0002] In transfer learning, the data with abundant labeled information is typically referred to as the source domain, while the unlabeled data to be processed is called the target domain. Domain adaptation, as a representative transfer learning technique, eliminates the distributional differences between data by mapping the labeled source domain and the unlabeled target domain to a shared feature space, enabling the classifier trained in the source domain to be well-suited for the target task. Traditional domain adaptation methods assume that the label distributions between the source and target domains are the same. However, in practical applications, not only are there differences in probability distributions between the source and target domains, but label distribution shifts are also common. This is because the cost of acquiring labels often differs across domains. For example, the types and numbers of prevalent diseases vary in different regions, leading to discrepancies in the number of patient categories in different regions. Traditional domain adaptation methods, by only considering the assumption of identical label distributions between the source and target domains, fail in practical cross-domain long-tail image classification tasks, and may even exhibit severe negative transfer phenomena.

[0003] Since the learning objectives of deep domain adaptation methods can be divided into two main aspects—classification loss and domain difference loss—the former utilizes supervised information from the source domain to train the classifier, for example, by minimizing cross-entropy loss. The latter achieves distribution alignment by reducing the difference between the source and target domains. For example, the most commonly used method for measuring distribution difference is Maximum Mean Difference (MMD), which uses kernel tricks to map the original features to the regenerating kernel Hilbert space and then calculates the average embedding distance between different domains as a measure of domain difference. Therefore, imbalanced label distributions also have two effects on cross-domain long-tail image classification:

[0004] For classifier model training, since only the source domain has supervised information, the quality of the labels directly affects its classification performance. Unlike balanced classes, imbalanced label distributions cause the classifier's decision boundary to be biased towards the majority class. While rebalancing strategies (such as resampling and reweighting) provide solutions for imbalanced learning, these strategies can disrupt the inherent feature representations by altering the data distribution, thus decomposing the learning process into representation learning and classifier learning. Furthermore, considering the superiority of self-supervised learning in unsupervised representation learning, the effectiveness of this type of approach in imbalanced learning has been proven. Therefore, while ensuring the quality of feature representations, constructing a shared feature space between the source and target domains becomes crucial for cross-domain imbalanced learning.

[0005] For distribution alignment, directly reducing the differences between domains is inappropriate when label distribution is offset. This is because label inconsistency affects distribution alignment, causing traditional feature alignment methods to fail. This is especially true for subdomain adaptation methods that rely on label information to achieve distribution alignment within the same category. These methods attempt to address the domain distribution offset problem by using noisy target domain pseudo-labels for self-training or minimizing conditional entropy. However, class imbalance directly leads to reduced classifier performance, particularly in the initial training phase. Early pseudo-label classification errors accumulate iteratively, disastrously impacting the entire domain adaptation process and ultimately leading to incorrect classification. Summary of the Invention

[0006] The purpose of this invention is to provide a cross-domain long-tail image classification method based on self-supervised learning and self-training mechanisms, so as to achieve accurate classification of cross-domain long-tail images. This solves the problem that existing classification methods are prone to errors in cross-domain long-tail image classification due to insufficient extraction of image discriminative features, model decision boundary offset, and error accumulation during self-training.

[0007] To address the aforementioned technical problems, the present invention employs a cross-domain long-tail image classification method based on self-supervised learning and self-training mechanisms, comprising the following steps:

[0008] Step S1: Obtain the dataset of cross-domain long-tail images, which includes source domain and target domain long-tail image data, and divide the dataset into training set and test set;

[0009] Step S2: Train the dual-branch network model using cross-domain long-tail image data from the training set, including the following steps;

[0010] Step S21: For the unsupervised adaptive branch of the dual-branch network model, representation learning is performed using self-supervised learning and global distribution alignment methods;

[0011] Step S22: For the supervised adaptive branch of the dual-branch network model, the classifier is self-trained and mutual information is maximized to further improve the classifier's classification ability on the target task.

[0012] Step S23: Automatically adjust the learning weights of the two branches using a dynamic weighted ensemble training strategy, thereby simultaneously optimizing the learning objectives of the two branches;

[0013] Step S3: Use the trained model to classify the cross-domain long-tail image data in the test set.

[0014] Further, step S21 includes the following steps:

[0015] Step S211: Preprocess the input training set data;

[0016] Step S212: Minimize the distribution distance between the long-tailed image data of the source and target domains using a global distribution alignment method;

[0017] Step S213: Use self-supervised representation to learn the intrinsic features of long-tailed image data;

[0018] Step S214: Adjust the ratio between global distribution alignment and self-supervised representation learning to obtain the optimization objective of the unsupervised adaptive branch and find the optimal shared feature representation.

[0019] Furthermore, the data preprocessing in step S211 specifically includes: adjusting the image size of the cross-domain long-tailed image, normalization processing, and data augmentation;

[0020] Data augmentation refers to enhancing all long-tail image data in both the source and target domains using both strong and weak enhancement methods. The strong enhancement includes random color jitter, random grayscale, random Gaussian blur, and random horizontal flipping. The weak enhancement refers to random horizontal flipping.

[0021] Furthermore, the global distribution alignment method in step S212 refers to using MMD to measure the distribution difference between the long-tailed image data of the source and target domains after weak enhancement in image preprocessing, and using a Gaussian kernel as the kernel function in MMD, and by adjusting the distribution distance... The network parameters of the feature extractor are minimized to enable it to extract shared feature representations from the source and target domains; where, for features from the probability distribution P s and P t The MMD distribution distance for the samples is expressed as follows:

[0022]

[0023] In the formula, the mapping function By using a feature kernel function k instead, that is... Thus, the source domain D with a sample size of n s The i-th sample and the target domain D with a sample size of m t The j-th sample This kernel function k maps to the reproducing kernel Hilbert space H, where x s x t These represent samples in the source and target domains, respectively.

[0024] Furthermore, in step S213, the self-supervised representation extracts the intrinsic features of cross-domain long-tail image data through a dictionary lookup task: when a query and a key value are different augmentations of the same image, the query and the key value are matched; wherein the query adopts a strong augmentation strategy, and the key value adopts a weak augmentation strategy.

[0025] The self-supervised loss function in the self-supervised representation process is:

[0026]

[0027] Where N is the number of negative examples, τ is the temperature parameter, Q is the query, and K... + K is the key value. n This represents the nth negative example.

[0028] Furthermore, the optimization objective of the unsupervised adaptive branch in step S214 is expressed as:

[0029]

[0030] Among them, L con For self-supervised loss function, Let the probability distribution be P s and P t The distance of the MMD distribution, where α is a trade-off parameter.

[0031] Further, step S22 specifically includes:

[0032] Step S221: Preprocess the source domain data with real labels and the long-tailed image data of the target domain with high confidence and pseudo labels;

[0033] Step S222: Train the classifier using a self-training method;

[0034] Step S223: Maximize and optimize pseudo-labels through mutual information;

[0035] Step S224: Adjust the ratio between self-training loss and mutual information to obtain the optimization objective of the supervised adaptive branch;

[0036] Step S222 specifically involves:

[0037] For a given target domain long-tailed image sample Assume that the highest probability value among the output probabilities for each category is Secondly, a high probability value If the difference between the highest probability value and the second highest probability value is greater than a threshold, the target domain sample is considered a high-confidence sample, and the category with the highest probability value is added as a pseudo-label to the classifier training. If the condition is not met, the sample is considered a low-confidence long-tail image sample, that is, it is considered as a null value and is not included in the classifier training.

[0038] After selecting high-confidence target domain long-tail image samples, resampling is used to construct the classifier f. cl The training set is balanced and includes source domain long-tail image samples with real labels and target domain long-tail image samples with pseudo-labels and high confidence; for a number of instances n c The class label c, the sampling probability P of resampling c for:

[0039]

[0040] Where, η c This represents the reciprocal of the number of samples in the c-th class relative to the total number of training samples, calculated as follows: C represents the total number of class labels for the classification task, n l It is the number of samples of the l-th class in the training set; according to the sampling probability P c During training, an equal number of samples from each class are sampled to achieve rebalancing;

[0041] Ultimately, by minimizing the classification loss L cl To train the classifier:

[0042] L cl =L ce (x train ,y train )

[0043] Among them, L ce The cross-entropy loss is applied to the balanced training set, x train y train These represent source domain image samples with real labels and target domain image samples with high confidence and pseudo-labels, respectively.

[0044] Furthermore, in step S223, maximizing mutual information refers to increasing the diversity of the target domain output results by calculating the mutual information loss of the target domain, thereby improving the quality of the pseudo-labels; wherein, the mutual information loss of the target domain is expressed as:

[0045]

[0046] in, It is a long-tailed image sample of the target domain. The predicted probability obtained after using Softmax It is the average predicted probability of all long-tailed image samples in the c-th class, and <·,·> is the inner product operation.

[0047] Furthermore, the optimization objective of the supervised adaptive branch in step S224 is expressed as:

[0048] L loab =L cl -βL mi

[0049] Among them, L cl For classification loss, L mi β represents the mutual information loss in the target domain, and β is the trade-off factor.

[0050] Furthermore, the dynamic weighted ensemble training strategy in step S23 refers to automatically controlling the weights of representation learning and classifier learning, allowing the model to first perform representation learning and then gradually focus on classifier learning. The dynamic weighted ensemble factor μ of the dynamic weighted ensemble training strategy is calculated as follows:

[0051]

[0052] Where iter is the current iteration number, and T is the total number of iterations. μ gradually approaches 0 as the number of iterations increases. The learning strategy is dynamically adjusted by using μ as the weight of the loss function, which is the inverse function of the current iteration number iter.

[0053] The loss function of the dual-branch network model is:

[0054] L dbdan =μL lfab +(1-μ)L loab

[0055] Among them, L lfab For the optimization objective of unsupervised adaptive branching, L loab The optimization objective is for supervised adaptive branching.

[0056] The beneficial effects of this invention are:

[0057] (1) The unsupervised adaptive branching representation learning method of the present invention solves the problem of insufficient feature extraction of minority classes by the feature extractor due to label distribution differences in existing methods. It obtains the intrinsic feature representation of the data by constructing a simple dictionary lookup task using self-supervised learning; and uses MMD to measure the global distribution distance and minimizes the distance to achieve feature alignment between cross-domain long-tail image data.

[0058] (2) The supervised adaptive branch classifier learning method of the present invention solves the problem of classifier decision boundary shift caused by class imbalance in existing methods. By adopting a self-training approach, a balanced training set is constructed by resampling high-confidence target domain data and source domain data for classification learning; and mutual information maximization is used to ensure the quality of target domain pseudo-labels, thereby improving the accuracy of cross-domain long-tail image classification.

[0059] (3) The dynamic weighted ensemble training strategy of the present invention solves the problems of error accumulation caused by the introduction of pseudo-labels in the target domain and the inability of neural networks to be trained end-to-end in the existing self-training process. By automatically adjusting the weights of representation learning and classifier learning, more attention is paid to representation learning of cross-domain long-tail data in the early stage of training, and more attention is paid to classification learning in the later stage of training, which alleviates the accumulation of self-training errors caused by insufficient representation ability. At the same time, it realizes simultaneous optimization of different representation learning and classifier learning, and is trainable end-to-end. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a schematic diagram of a cross-domain long-tail image classification method based on a self-supervised learning and self-training mechanism according to an embodiment of the present invention.

[0062] Figure 2 These are t-SNE feature visualizations of different methods in embodiments of the present invention on the DomainNet dataset, wherein (a) is a t-SNE feature visualization of ResNet50 on the DomainNet dataset, (b) is a t-SNE feature visualization of DAN on the DomainNet dataset, (c) is a t-SNE feature visualization of JAN on the DomainNet dataset, and (d) is a t-SNE feature visualization of the method of the present invention on the DomainNet dataset.

[0063] Figure 3 This is a confusion matrix diagram of the cross-domain long-tail image classification method based on self-supervised learning and self-training mechanism in this invention on the DomainNet dataset. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] This invention discloses a cross-domain long-tail image classification method based on self-supervised learning and self-training mechanisms, such as... Figure 1 As shown, a dual-branch network structure model is constructed, which decomposes the training process of the deep network model into representation learning and classification learning. First, the unsupervised adaptive branch performs representation learning using self-supervised learning and global distribution alignment methods. Second, the supervised adaptive branch performs self-training and mutual information maximization on the learned features to further improve the classification ability on the target task. Finally, a dynamic weighted ensemble training strategy is designed to automatically adjust the learning weights of different representation learning and classifier learning, thereby achieving simultaneous optimization of the two representation learning and classifier learning objectives.

[0066] Specifically as follows:

[0067] S1. Unsupervised adaptive branching representation learning method

[0068] a. Image Preprocessing: In the embodiments of this invention, the preprocessing of the input long-tail image data of the source and target domains in the unsupervised representation learning method can be divided into three steps: 1) Adjusting the image size of the cross-domain long-tail images: The image size of all input cross-domain long-tail image data is uniformly adjusted and randomly cropped to 224×224. 2) Normalization: The input cross-domain long-tail images are normalized. Based on the mean of the three channels being {0.485, 0.456, 0.406} and the standard deviation being {0.229, 0.224, 0.225}, the mean is subtracted from each pixel in each channel, and then divided by the standard deviation to obtain the normalized result. 3) Data Augmentation: Since the self-supervised learning method needs to utilize the augmentation invariance of data to establish a dictionary lookup task, a "strong-weak" data augmentation method is adopted for all input long-tail image data of the source and target domains in this method. The "strong enhancement" method employs random color jitter, random grayscale, random Gaussian blur, and random horizontal flipping to improve the robustness of the learned feature representations; the "weak enhancement" method only uses random horizontal flipping to retain more original image feature information. Since the DomainNet dataset used in this embodiment has been pre-divided into training and test sets, this embodiment uses the original dataset partitioning and trains the network parameters of the feature extractor using the training sets of the source and target domains with a batch size of 32.

[0069] b. Global Distribution Alignment Method: In this step, this embodiment uses MMD only to measure the distribution difference between the long-tailed image data of the source and target domains after "weak enhancement" in image preprocessing. This is because "strong enhancement" will increase the MMD value to some extent, making the measurement of the distribution difference between the source and target domains inaccurate. As a non-parametric measurement method, MMD does not require any prior assumptions about the distribution and does not involve any supervision information. Although the global distribution difference measurement will still be biased in the case of imbalance, compared with the calculation of intra-class distribution difference, MMD is less affected by label offset and can better measure the distribution difference in the initial case. Specifically, MMD measures the distribution difference between the source and target domains by embedding the mean in the kernel space into RKHS. Formally, for data from a probability distribution of P... s and P t The MMD distance method for the samples is shown in equation (1):

[0070]

[0071] Wherein, mapping function This can be replaced by using a feature kernel function k, i.e. Where x s x t Let D represent the samples in the source domain and the target domain, respectively, so that the source domain D with a sample size of n is... s The i-th sample and the target domain D with a sample size of m t The j-th sample This kernel function k is mapped to the reproducing kernel Hilbert space H. In this embodiment, a Gaussian kernel is used as the kernel function in MMD, and the distribution distance is mapped... Minimizing the network parameters trained on the feature extractor enables the feature extractor to extract shared feature representations of the source and target domains of cross-domain long-tailed images.

[0072] c. Self-supervised representation learning method: To obtain robust feature representations while minimizing the negative impact of label distribution shift, this embodiment utilizes self-supervised learning to establish an unsupervised feature extraction method. It follows a simple dictionary lookup task to extract the intrinsic features of cross-domain long-tail image data. For example, if a query and a key value are different augmentations of the same image, then the query and key value are matched. Specifically, for a given sample x∈{D} s ∪D t The goal is to learn its intrinsic representation Q as a query, which can be derived from a set of encoded features {K1, K2, K3, ..., K}. n} (i.e., negative examples, where K) n (representing the key value K of the sample after augmentation in the nth negative example) + (i.e., positive examples) are distinguished. Therefore, by using the dot product to measure similarity, a self-supervised loss function, also known as InfoNCE, is proposed, as shown in equation (2):

[0073]

[0074] Where N is the number of negative examples and τ is a temperature parameter used to control the model's ability to distinguish negative instances.

[0075] This embodiment employs a queue (size 65536) to expand the capacity of negative examples, thereby effectively enhancing the diversity of contrasting samples. Furthermore, by using two parallel encoders, the query encoder updates gradients through backpropagation, while the key-value encoder's parameters are updated using a momentum update strategy to prevent convergence fluctuations in the initial stage. This invention uses the query encoder as the feature extractor (backbone network) of a deep domain adaptive model. Different augmentation strategies are applied to the original samples for both query and key, with the query employing a "strong augmentation" strategy to improve the generalization ability of the learned representation, while the key-value pair employs a "weak augmentation" strategy to preserve the original feature information of the instance.

[0076] d. Unsupervised representation learning optimization objective: A tradeoff parameter α, which is 0.1 in this embodiment, is added between MMD and self-supervised loss to adjust the ratio between global distribution alignment and self-supervised learning, i.e., importance. Therefore, the optimization objective of the unsupervised adaptive branch can be expressed as Equation (3), and the optimal shared feature representation is obtained by minimizing this objective:

[0077]

[0078] S2. Supervised Adaptive Branching Classifier Learning Method

[0079] a. Image Preprocessing: "Warming up" the network parameters using unsupervised representation learning methods enhances the feature extraction capability of the neural network to some extent and indirectly increases the confidence of the target domain pseudo-labels. During classifier learning, the feature extractor obtained from the unsupervised representation learning method is used to extract features from the input data. The input consists of source domain long-tail image data with true labels and target domain long-tail image data with high confidence and pseudo-labels. The image resizing and normalization methods used are the same as in the unsupervised representation learning method. The difference lies in that this classifier learning method only uses a "weak enhancement" approach to uniformly process the input source domain long-tail image data and target domain long-tail image data for subsequent classification training.

[0080] b. Self-training-based classifier learning method: The self-training method improves the model's learning ability for the target domain and increases the classifier's prediction confidence by adding high-confidence long-tail image samples of the target domain with pseudo-labels to the model training. Therefore, the selection of high-confidence samples is particularly important. Considering that the classifier's predictions are easily affected by imbalance under class distribution shift conditions, the selection of long-tail image samples of the target domain is crucial. Output probability In this context, the confidence level of a sample is defined by the difference between the highest and second-highest probability values. For a given target domain sample... Assume that the highest probability value among the output probabilities for each category is Secondly, a high probability value If condition (4) is satisfied, then Considered to have a false label High-confidence samples.

[0081]

[0082] Where δ is the threshold and c is the specific class label. If the above condition is met, i.e., the difference between the highest probability value and the second highest probability value is greater than the threshold, then the sample is considered a high-confidence sample, and the class with the highest probability value is added to the classifier training as its pseudo-label; if the condition is not met, then the sample is considered a low-confidence sample, i.e., it is considered null and does not participate in the classifier training. In the experiments of this invention, the threshold is set to 0.9 to ensure the reliability of the assigned pseudo-labels.

[0083] After selecting high-confidence long-tailed target domain image samples, resampling is performed on source domain image samples with true labels and high-confidence target domain image samples with pseudo-labels to construct the classifier f. cl Balanced training set {x} train ,y train Specifically, the sampling probability of each class is adjusted; that is, the class with fewer instances has a higher sampling probability. Let n be the number of samples in the c-th class. c Then the sampling probability P of the c-th class is c It can be calculated using equation (5):

[0084]

[0085] Where, η c This represents the reciprocal of the number of samples in the c-th class relative to the total number of training samples, and it is calculated as follows: C represents the total number of class labels for this classification task, and n l This is the number of samples in the l-th class in the training set. Based on the sampling probability P...c During training, rebalancing can be achieved by sampling an equal number of samples from each class. Ultimately, this is achieved by minimizing the classification loss L. cl To train the classifier:

[0086] L cl =L ce (x train ,y train (6)

[0087] Among them, L ce The cross-entropy loss is calculated on a balanced training set, x. train y train These represent source domain image samples with real labels and target domain image samples with high confidence and pseudo-labels, respectively.

[0088] c. Pseudo-label optimization method based on maximizing mutual information: Generally, when selecting long-tailed image samples in a high-confidence target domain, the higher the predicted confidence score, the greater the probability of correctly classifying the samples in the target domain. However, excessively high confidence can easily lead to trivial solutions, meaning the model will overfit to information from a single class, lacking learning of information from other similar classes, resulting in severe overfitting. Therefore, this embodiment calculates the mutual information loss L in the target domain... mi And maximize it to increase the diversity of the target domain output results, thereby optimizing the quality of pseudo-labels, as shown in Equation (7).

[0089]

[0090] in, It is a target domain sample The predicted probability obtained after using Softmax It is the average predicted probability of all samples in the c-th class, and <·,·> is the inner product operation.

[0091] d. Supervised classifier learning optimization objective: By providing β as a tradeoff factor between classification loss and mutual information loss (0.1 in this invention), the ratio between self-training loss and mutual information is adjusted, i.e., importance. The optimization objective of the supervised classifier learning method can be expressed as Equation (8):

[0092] L loab =L cl -βL mi (8)

[0093] S3. Dynamically Weighted Ensemble Training Strategy

[0094] Based on the different roles that representation learning and classifier learning play in the network's learning phase, this model needs to obtain robust feature representations before training the classifier to minimize the negative impact of label distribution bias on the model. Therefore, this embodiment employs a dynamically weighted ensemble training strategy to automatically control the weights of representation learning and classifier learning, allowing the model to first perform representation learning and then gradually focus on classifier learning. Specifically, the dynamic weighted ensemble factor μ is calculated as follows:

[0095]

[0096] Where iter is the current iteration number, and T is the total number of iterations. μ will gradually approach 0 as the number of iterations increases.

[0097] The learning strategy is dynamically adjusted by using μ as the weight of the loss function, with the current iteration number iter as the variable. Therefore, the loss function of this model can be expressed as Equation (10). In this embodiment, mini-batch SGD with a momentum of 0.9 is used to optimize the entire model parameters to obtain the optimal model parameters, so as to realize the shared feature extraction and accurate classification of cross-domain long-tail images.

[0098] L dbdan =μL lfab +(1-μ)L loab (10)

[0099] This embodiment selects some cutting-edge deep domain adaptation methods to verify the effectiveness of the present invention. Considering that the overall average accuracy is not suitable for evaluating classification performance in the case of class imbalance, this embodiment uses the average accuracy on each class to verify the model performance.

[0100] This embodiment presents the experimental results of existing related methods on the DomainNet and OfficeHome cross-domain long-tail image classification tasks, which have label distribution shifts, as shown in Tables 1 and 2. Overall, the method of this invention achieves good classification results on most tasks, with a slight lead on DomainNet. Out of all 12 tasks, this invention achieves performance superiority on 9 tasks. In OfficeHome, this invention wins all transfer learning tasks and outperforms state-of-the-art algorithms in related fields by more than 2%. These excellent experimental results demonstrate the superior performance of the method of this invention for label distribution shift problems and show that this invention can still learn more transferable information under this special data distribution condition.

[0101] Table 1. Class-average accuracy of the DomainNet dataset with label distribution offset.

[0102]

[0103]

[0104] Table 2. Class-average accuracy of the label distribution offset dataset OfficeHome.

[0105] source 70.74 44.24 67.33 38.68 53.51 51.85 54.39 BSP 72.80 23.82 66.19 20.05 32.59 30.36 40.97 PADA 60.77 32.28 57.09 26.76 40.71 38.34 42.66 BBSE 61.10 33.27 62.66 31.15 39.70 38.08 44.33 MCD 66.03 33.17 62.95 29.99 44.47 39.01 45.94 DAN 69.35 40.84 66.93 34.66 53.55 52.09 52.90 F-DANN 68.56 40.57 67.32 37.33 55.84 53.67 53.88 JAN 67.20 43.60 68.87 39.21 57.98 48.57 54.24 DANN 71.62 46.51 68.40 38.07 58.83 58.05 56.91 MDD 71.21 44.78 69.31 42.56 52.10 52.70 55.44 COAL 73.65 42.58 73.26 40.61 59.22 57.33 58.40 InstaPBM 75.56 42.93 70.30 39.32 61.87 63.40 58.90 MDD+IA 76.08 50.04 74.21 45.38 61.15 63.15 61.67 SENTRY <![CDATA[ 76.12 ]]> <![CDATA[ 56.80 ]]> <![CDATA[ 73.60 ]]> <![CDATA[ 54.75 ]]> <![CDATA[ 65.94 ]]> <![CDATA[ 64.29 ]]> <![CDATA[ 65.25 ]]> This invention - w / o.μ 75.56 52.31 73.45 49.52 63.68 63.84 63.06 This invention 79.34 57.57 75.86 54.95 67.59 67.71 67.17

[0106] To demonstrate the contribution of different modules in this invention, this embodiment decomposes the model and conducts ablation experiments on six cross-domain imbalanced learning tasks on OfficeHome. This embodiment uses the cross-entropy loss after class resampling as a baseline and compares it with experimental results from different combinations (such as domain difference loss, contrastive loss, and mutual information loss), as shown in Table 3. It can be observed that, in the imbalanced case, class resampling is beneficial for classifier training, improving average performance by nearly 4%. In contrast, domain difference loss and mutual information loss contribute significantly to the overall adaptation process because their combination yields experimental results similar to the final outcome.

[0107] Table 3 Ablation experiments of DBDAN on the OfficeHome dataset.

[0108]

[0109] Secondly, this embodiment utilizes t-SNE to visualize the feature representations learned by ResNet50, DAN, JAN, and the present invention on the DomainNet task CR, such as... Figure 2 As shown in (a)-(d), the figures demonstrate that this invention can output features that are more distinctive than other features. Figure 2 (d) exhibits a clearer clustering structure, characterized by smaller intra-cluster distances and larger inter-cluster distances. Figure 2 A comparison of (a) and (b) shows that, in the case of LDS, simply aligning the global distribution does not yield good results, potentially leading to worse domain fitting than training using only the source domain. In contrast, JAN achieves better adaptation performance than DAN, but it is clear that... Figure 2 In (c), a large number of instances are scattered among the class clusters.

[0110] Finally, the confusion matrix can effectively reflect the model's performance under class imbalance conditions. From Figure 3 As can be seen, the confusion matrix of this invention not only achieves consistent prediction results across each class, but also exhibits high confidence in the diagonal of the confusion matrix. These experimental results demonstrate the effectiveness of this invention in cross-domain long-tail image classification tasks.

[0111] The cross-domain long-tail image classification method based on self-supervised learning and self-training mechanisms provided in this invention has been well validated on two datasets with offset label distributions. In the DomainNet and OfficeHome datasets used, not only are there significant differences in data distribution between different domains, but their label distributions also differ significantly. Traditional domain adaptation methods only consider the case where the label distributions are the same between the source and target domains, leading to the failure of traditional methods in practical cross-domain long-tail image classification tasks, and even severe negative transfer phenomena. Therefore, the technical solution proposed in this invention not only improves the model's shared feature extraction capability in cross-domain long-tail image classification tasks, but also achieves accurate classification under such complex data distribution conditions.

[0112] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0113] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A cross-domain long-tail image classification method based on self-supervised learning and self-training mechanisms, characterized in that, Includes the following steps: Step S1: Obtain the dataset of cross-domain long-tail images, which includes source domain and target domain long-tail image data, and divide the dataset into training set and test set; Step S2: Train the dual-branch network model using cross-domain long-tail image data from the training set, including the following steps: Step S21: For the unsupervised adaptive branch of the dual-branch network model, representation learning is performed using self-supervised learning and global distribution alignment methods; Step S22: For the supervised adaptive branch of the dual-branch network model, the classifier is self-trained and mutual information is maximized to further improve the classifier's classification ability on the target task. Step S23: Automatically adjust the learning weights of the two branches using a dynamic weighted ensemble training strategy, thereby simultaneously optimizing the learning objectives of the two branches; Step S3: Use the trained model to classify the cross-domain long-tail image data in the test set; Step S22 specifically involves: Step S221: Preprocess the source domain data with real labels and the long-tailed image data of the target domain with high confidence and pseudo labels; Step S222: Train the classifier using a self-training method; Step S223: Maximize and optimize pseudo-labels through mutual information; Step S224: Adjust the ratio between self-training loss and mutual information to obtain the optimization objective of the supervised adaptive branch; Step S222 specifically involves: For a given target domain long-tailed image sample Assuming that the highest probability value among the output probabilities for each category is The second highest probability value is If the difference between the highest and second-highest probability values ​​is greater than a threshold, the target domain sample is considered a high-confidence sample, and the category with the highest probability value is added as a pseudo-label to the classifier training. If the condition is not met, the sample is considered a low-confidence long-tail image sample, i.e., it is considered a null value. It does not participate in classifier training; After selecting high-confidence target domain long-tail image samples, resampling is used to construct the classifier. The training set is balanced and includes source domain long-tail image samples with real labels and target domain long-tail image samples with pseudo-labels and high confidence; for a number of instances... Class label The sampling probability of resampling for: in, Indicates the first The number of samples in each class is the reciprocal of the total number of training samples, calculated as follows: , Represented as the total number of class labels for the classification task. It is the first in the training set The number of samples in each class; based on the sampling probability. During training, an equal number of samples from each class are sampled to achieve rebalancing; Ultimately, by minimizing the classification loss To train the classifier: in, It involves balancing the cross-entropy loss on the training set. , These represent source domain image samples with real labels and target domain image samples with high confidence and pseudo-labels, respectively.

2. The cross-domain long-tail image classification method based on self-supervised learning and self-training mechanism according to claim 1, characterized in that, Step S21 includes the following steps: Step S211: Preprocess the input training set data; Step S212: Minimize the distribution distance between the long-tailed image data of the source and target domains using a global distribution alignment method; Step S213: Use self-supervised representation to learn the intrinsic features of long-tailed image data; Step S214: Adjust the ratio between global distribution alignment and self-supervised representation learning to obtain the optimization objective of the unsupervised adaptive branch and find the optimal shared feature representation.

3. The cross-domain long-tail image classification method based on self-supervised learning and self-training mechanism according to claim 2, characterized in that, The data preprocessing in step S211 specifically includes: adjusting the image size of cross-domain long-tailed images, normalization processing, and data augmentation; Data augmentation refers to enhancing all long-tail image data in both the source and target domains using both strong and weak enhancement methods. The strong enhancement includes random color jitter, random grayscale, random Gaussian blur, and random horizontal flipping. The weak enhancement refers to random horizontal flipping.

4. The cross-domain long-tail image classification method based on self-supervised learning and self-training mechanism according to claim 2, characterized in that, The global distribution alignment method in step S212 refers to using MMD to measure the distribution difference between the long-tailed image data of the source and target domains after weak enhancement in image preprocessing, and using a Gaussian kernel as the kernel function in MMD, and by adjusting the distribution distance... The network parameters of the feature extractor are minimized to enable it to extract shared feature representations from the source and target domains; where, for features from the probability distribution are... and The MMD distribution distance for the samples is expressed as follows: In the formula, the mapping function By using the feature kernel function To replace, that is Thus, the sample size is source domain The first in Sample and the number of samples is target domain The first in Sample Through this kernel function Mapped to the regenerating kernel Hilbert space Among them , These represent samples in the source and target domains, respectively.

5. A cross-domain long-tail image classification method based on self-supervised learning and self-training mechanism according to claim 2, characterized in that, In step S213, the self-supervised representation extracts the intrinsic features of cross-domain long-tail image data through a dictionary lookup task: when a query and a key value are different augmentations of the same image, the query and the key value are matched. The query uses a strong enhancement strategy, while the key-value pair uses a weak enhancement strategy. The self-supervised loss function in the self-supervised representation process is: in, N It is the number of negative examples. It's a temperature parameter. For query, As key-value pair Indicates the first One negative example.

6. The cross-domain long-tail image classification method based on self-supervised learning and self-training mechanism according to claim 2, characterized in that, The optimization objective of the unsupervised adaptive branch in step S214 is expressed as: in, For self-supervised loss function, The probability distribution is and MMD distribution distance, To weigh the parameters.

7. The cross-domain long-tail image classification method based on self-supervised learning and self-training mechanism according to claim 6, characterized in that, In step S223, maximizing mutual information refers to increasing the diversity of the target domain output results by calculating the mutual information loss of the target domain, thereby improving the quality of pseudo-labels; whereby the mutual information loss of the target domain is expressed as: in, It is a long-tailed image sample of the target domain. The predicted probability obtained after using Softmax All long-tailed image samples are in the first... The average predicted probability across all classes It is an inner product operation.

8. A cross-domain long-tail image classification method based on self-supervised learning and self-training mechanism according to claim 6, characterized in that, The optimization objective of the supervised adaptive branch in step S224 is expressed as: in, For classifying losses, For the loss of mutual information in the target domain, This is a weighting factor.

9. A cross-domain long-tail image classification method based on self-supervised learning and self-training mechanism according to claim 1, characterized in that, The dynamic weighted ensemble training strategy in step S23 refers to automatically controlling the weights of representation learning and classifier learning, allowing the model to first perform representation learning and then gradually focus on classifier learning. The dynamic weighted ensemble factor in the dynamic weighted ensemble training strategy... The calculation method is as follows: in, It is the current iteration number, and T It represents the total number of iterations. It will gradually approach 0 as the number of iterations increases; by using the current iteration number... For the inverse function of the variable, use The learning strategy is dynamically adjusted using the weights of the loss function. The loss function of the dual-branch network model is: in, The optimization objective for unsupervised adaptive branching is... The optimization objective is for supervised adaptive branching.

Images