Knowledge distillation passive domain unsupervised field adaptive learning method

Abstract

The invention provides a knowledge distillation passive domain unsupervised domain self-adaptive method, which realizes classification of a target domain without using source domain data by designing a teacher-student model and a competition mechanism. Firstly, a teacher network composed of a feature extractor, a classifier and a rotation classifier is constructed, source domain data is used for training, classifier parameters are fixed, a source model is finely adjusted through a self-supervised rotation sampling method, and a pseudo label of target domain data is generated through a clustering method. And then designing a student network composed of a feature extractor and a classifier, generating a pseudo tag only by using target domain data, and optimizing the discrimination and diversity of the tag by maximizing mutual information loss. And finally, screening the pseudo labels of the teacher-student network by using a competition module, and extracting the pseudo labels with higher confidence to update the student network, thereby completing knowledge migration. According to the method, source domain data is not needed, data storage and transmission cost is reduced, privacy and data security are improved, and unsupervised classification learning on unlabeled target domain data is supported.

Description

Technical Field

[0001] The present invention belongs to the technical field of domain adaptive pattern classification, and relates to a passive domain unsupervised domain adaptive learning method of knowledge distillation. Background Art

[0002] Machine learning is an important research direction in the era of artificial intelligence. Its research results have been widely applied to various fields and have played an increasingly important role in people's daily lives. Current machine learning technology usually relies on a closed environment, and its learning process involves three basic assumptions: [1] : The first is the independent and identically distributed assumption: the training and test data have the same feature space and data distribution; the second is the closed category label space: the training and test data category space is the same; the third is the big data assumption: there are enough available labeled training samples to ensure that a better classification model is learned. However, the machine learning process in an open environment does not meet the above three basic assumptions. Machine learning in which the learning environment does not meet the above three assumptions is called machine learning in an open environment. When these three assumptions are not met, it is necessary to study machine learning problems in an open environment. Adapting machine learning methods to open environments has become one of the key issues that need to be solved in the research of the new generation of artificial intelligence technology.

[0003] Domain Adaptation (DA [2][3] DA can learn a model with good performance on the source domain data, adapt to the target domain data with different distribution or even different category label space, and solve the learning task of the target domain data with no labeled samples or a small number of labeled samples. DA can adapt to the application scenarios of open environments and is one of the main paradigms of machine learning in open environments. [4] DA is the main way to solve the current limitations of machine learning in closed environments (including out-of-distribution scene learning, closed label category space, no labels and few labels, etc.). DA helps large pre-trained models handle various applications and promote the development of machine learning and even artificial intelligence.

[0004] However, existing domain adaptation methods generally rely on the accessibility of source domain data. However, in real application scenarios, the following problems often exist. (1) In the era of big data, data security and privacy protection are particularly important. For example, private data sets such as medical data and facial data are not suitable for frequent disclosure and use in model training; (2) The training of deep neural networks requires a large amount of data support. Over-reliance on source domain data will also lead to high costs for data storage and transmission; (3) At the same time, a large amount of source domain data will also increase the computational burden of computers. Therefore, in order to solve the above problems and realize the "decoupling" of source domain data in unsupervised domain adaptation methods, researchers proposed passive domain adaptation. [5](Source-Free DA, SFDA). Existing SFDA [5] Based on closed set domain adaptation, the pre-trained source model is migrated to the unlabeled target domain without accessing any source data to complete the learning task of the target domain. Existing SFDA methods are divided into two categories: white-box SFDA and black-box SFDA. [6] , the difference between the two is whether the parameters of the pre-trained model are available.

[0005] In passive unsupervised domain adaptation, a source domain model that has been trained with source domain data is usually provided. In the subsequent target domain data training process, the source domain data is no longer involved, and the downstream tasks are completed by only providing the source domain model without providing the source domain data. This effectively solves the above-mentioned problems such as data security, data storage and computing burden.

[0006] [1]Z.Yao,C.Liu,CYSuen.Towards robust pattern recognition:areview.Proceedings of the IEEE,2020,108(6):894-922.

[0007] [2]SJPan,Q.Yang.A survey on transfer learning.IEEE Transactions OnKnowledge Data Engineering.2010,22(10):1345-1359.

[0008] [3] L. Zhang,

[0009] [4] Yuan Xiaotong, Zhang Xuyao, Liu Xi, Cheng Zhen, Liu Chenglin. Research progress on the theory of machine learning for open environments. Pattern Recognition and Artificial Intelligence, 2023, 36(12): 1059-1071.

[0010] [5]J.Liang,D.Hu,J.Feng.Do we really need to access the source data? Source hypothesis transfer for unsupervised domain adaptation. Proceedings of the 37th International Conference on Machine Learning, 2020.

[0011] [6] Y. Fang, P. Yap, W. Lin, H. Zhu, M. Liu. Source-free unsupervised domain adaptation: a survey. arXiv preprint arXiv: 2301.00265, 2022. Summary of the invention

[0012] The purpose of the present invention is to provide a source domain unsupervised domain adaptive learning method for knowledge distillation, which solves the problems of existing domain adaptive methods relying on the accessibility of source domain data, lack of privacy protection and data security, and high cost of data storage and transmission, and realizes privacy protection and data security protection of source domain data, and performs unsupervised classification learning on unlabeled data in the target domain.

[0013] The technical solution adopted by the present invention is

[0014] The passive domain unsupervised domain adaptive learning method of knowledge distillation does not use source domain data, and designs a student model and a competition mechanism to realize knowledge distillation and classification of the target domain. It is characterized by first designing a teacher network composed of a feature extractor, a classifier, and a rotation classifier, inputting the source domain data into the teacher network for training to obtain a source model, then fixing the classifier parameters of the source model, designing a rotation classifier, inputting the target domain data, and fine-tuning the source model using a self-supervised method of rotation sampling, and obtaining the pseudo-labels of the target domain data of the teacher network by a clustering method. Then, a student network composed of a feature extractor and a classifier is designed, and only the target domain data is input into the student network to obtain the pseudo-labels of the student network, and the student network is optimized by maximizing the mutual information loss, so that the pseudo-labels of the student network are more distinguishable and diverse. Finally, a competition module is constructed, and the pseudo-labels of the teacher-student network are respectively input into the competition module to obtain pseudo-labels with higher confidence, and the student network is fed back and updated to complete the knowledge transfer.

[0015] The beneficial effects of the present invention are

[0016] 1. The new method uses the teacher-student model and competition mechanism to alleviate the source domain offset problem, fully mine the target domain information, and realize cross-domain knowledge distillation and transfer.

[0017] 2. The new method does not need to access the source domain data when training the target domain, thus achieving data security and privacy protection of the source domain data.

[0018] 3. In the teacher network, a rotation classifier is designed, and a self-supervised method of rotation sampling is constructed to fine-tune the source model. The pseudo-label of the target domain data of the teacher network is obtained through clustering, which can extract rich target domain feature information.

[0019] 4. The method can design the student network to maximize the mutual information loss and design the competition module so that the teacher and student networks can complete the knowledge distillation and complete the high-precision learning task in the target domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is the overall framework diagram of the method of the present invention;

[0021] Figure 1 Medium, F Tea and F Stu denote the feature extractors of the teacher network and the student network, respectively, and C Tea and C Stu Represent the classifiers of the teacher network and the student network, C Rot represents the classifier of sample rotation, B represents the batch size, represents the data distribution of source domain samples in the teacher network source model training phase, represents the pseudo labels of the target domain samples in the teacher network domain adaptation phase, is the data distribution corresponding to the sample, represents the pseudo labels of the target domain samples of the student network in the domain adaptation phase, is the data distribution corresponding to the sample, Represents the target domain pseudo label obtained through the competition mechanism. DETAILED DESCRIPTION

[0022] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] The entire training process of the method is divided into two steps, in which the teacher network and the student network can select networks such as ResNet as the basic backbone network.

[0024] The first step is to train the source domain model. The labeled source domain data is input into the teacher network and the cross entropy loss is used. To optimize network parameters, Defined as:

[0025]

[0026] Where B is Batch_size,

[0027] Then the second step is domain adaptation. In this process, the trained source model is combined with a separate rotation classifier as the teacher network, and the classifier parameters of the source model are fixed, the teacher network feature extractor parameters are updated as usual, and the teacher-student network only inputs the target domain data.

[0028] For the teacher network, we design the network to maximize the mutual information loss To constrain the target domain data, so that the results are distinguishable and diverse. The mutual information loss is divided into two parts. The first part is the entropy minimization loss. The entropy minimization loss reduces the unknown nature of the student network output and makes the pseudo-labels more distinguishable. Defined as:

[0029]

[0030] The other part is the diversity maximization loss It alleviates the problem of degenerate solution caused by entropy minimization, making the network output diversified and the classification effect better. Defined as:

[0031]

[0032] in express The result after averaging.

[0033] Combining equations (2) and (3) gives the mutual information loss Its expression is:

[0034]

[0035] In order to make the teacher network classification more accurate, this method introduces self-supervision methods such as k-means clustering and sample rotation.

[0036] In the k-means clustering method, we first need to calculate the class prototype of each class as the centroid. The centroid expression is:

[0037]

[0038] The pseudo label is obtained by judging the cosine distance from the centroid, and its expression is:

[0039]

[0040] The class centroid is updated twice in each epoch, and the updated centroid expression is:

[0041]

[0042] The pseudo labels obtained by clustering are used for cross entropy loss, and the expression is:

[0043]

[0044] in

[0045] In the sample rotation method, the target domain data will be rotated 90 degrees, 180 degrees and 270 degrees respectively, and the rotated samples and the original samples will be combined into a four-image picture with the label The small data set is then input into the teacher network, through the feature extractor and a separate rotation classifier, and the cross entropy loss is calculated:

[0046]

[0047] in

[0048] For the student network, we only input the target domain data and calculate the cross entropy loss The expression is:

[0049]

[0050] in

[0051] Similar to the teacher network, the student network is also designed to maximize the mutual information loss, expressed as:

[0052]

[0053] However, since the labels of both the teacher network and the student network are unreliable in the early stage of training, the teacher network at least has the constraint of the correct label, while the student network will converge prematurely under the influence of unreliable pseudo labels. Therefore, it is necessary to ensure the priority of the teacher network label selection in the early stage and introduce the threshold T p The expression is:

[0054]

[0055] Here, γ is set to 10, and p is the ratio of the current number of iterations to the total number of iterations, so the value of p varies from 0 to 1. p , 1] to ensure the priority of the selected label, so the final competition module output is:

[0056]

[0057] The threshold Tp Defined by formula (15).

[0058] Under the influence of the competition mechanism, in the initial stage of training, the student network will be guided and constrained by the teacher network and learn the knowledge of the teacher network. As the training progresses, the student network gradually surpasses the teacher network and completes knowledge distillation and knowledge transfer.

Claims

1. The source-domain unsupervised domain adaptive learning method of knowledge distillation does not use source domain data, but designs a generation model and competition mechanism to achieve knowledge distillation and target domain classification. It is characterized by First, a teacher network consisting of a feature extractor, a classifier and a rotation classifier is designed, and the source domain data is input into the teacher network for training to obtain a source model. Subsequently, the classifier parameters of the source model are fixed, and a rotation classifier is designed. The target domain data is input and the source model is fine-tuned using a self-supervised method of rotation sampling, and the pseudo-labels of the target domain data of the teacher network are obtained by clustering. Then, a student network consisting of a feature extractor and a classifier is designed, and only the target domain data is input into the student network to obtain the pseudo-labels of the student network. The student network is optimized by maximizing the mutual information loss, so that the pseudo-labels of the student network are more distinguishable and diverse. Finally, a competition module is constructed, and the pseudo-labels of the above teacher-student network are respectively input into the competition module to obtain pseudo-labels with higher confidence, and the student network is fed back and updated to complete the knowledge transfer, which at least includes the following steps: (1) Step 1: Train the source domain model, input the labeled source domain data into the teacher network, and design the cross entropy loss Optimize network parameters, cross entropy loss Defined as Where B is Batch_size, (2) Step 2: Domain Adaptation: The trained source model is combined with a separate rotation classifier as the teacher network. The classifier parameters of the source model are fixed, and the teacher network feature extractor parameters are updated. The teacher and student networks only input the target domain data. In step 2, the teacher network construction process includes at least the following steps: First, we design a loss that maximizes the mutual information In order to constrain the target domain data and make the results discriminative and diverse, the mutual information loss is divided into two parts: the first part is the entropy minimization loss The entropy minimization loss reduces the unknown of the student network output, making the pseudo-labels more distinguishable, where the entropy minimization loss Defined as Another part of mutual information loss is diversity maximization loss It alleviates the problem of degenerate solution caused by entropy minimization, makes the network output diversified, the classification effect is better, and the diversity maximization loss Defined as in express The result after averaging; Minimize the entropy loss and diversity maximization loss Combined to get mutual information loss 2. The passive domain unsupervised domain adaptive learning method of knowledge distillation is characterized by The teacher network is designed to realize the knowledge distillation process. In order to improve the classification ability of the teacher network, the method constructs the k-means clustering method and the sample rotation self-supervision method to optimize the teacher network. The specific steps are as follows: (1) The design steps of the k-means clustering method include: First, we need to calculate the class prototype of each class as the centroid. The centroid expression is: Then get the pseudo-label, pseudo-label It is obtained by determining the cosine distance from the centroid, defined as: The class centroid is updated twice in each epoch, and the updated centroid expression is: Pseudo labels obtained by clustering and forming cross entropy loss for in (2) The implementation steps of the sample rotation self-supervision method include: First, the target domain data is rotated by 90 degrees, 180 degrees, and 270 degrees respectively, and the rotated samples and the original samples are combined into a new data set of four samples, whose labels are Then input it into the teacher network, pass through the feature extractor and a separate rotation classifier, and calculate the cross entropy loss after rotation as in Then, for the student network, only the target domain data is input and the cross entropy loss is calculated for in 3. The passive domain unsupervised domain adaptive learning method of knowledge distillation is characterized by Make the student network compete with the teacher network and realize knowledge distillation and transfer. The construction process of the student network is similar to that of the teacher network. The steps of constructing the student network include at least the following: (1) Designing a student network to maximize mutual information loss The mutual information loss Include entropy minimization loss and diversity maximization loss Two parts, defined as: where the entropy minimization loss is defined as Diversity maximization loss Defined as (2) Introducing threshold T p Prevent the student network from converging too early; since the labels of the teacher network and the student network are unreliable in the early stage of training, the student network will converge too early under the influence of the unreliable pseudo-labels of the teacher network. Therefore, the threshold Tp is introduced in the early training stage to ensure the priority of the teacher network label selection. The threshold Tp is defined as: Where γ is set to 10, p is the ratio of the current number of iterations to the total number of iterations, so the value of p varies from 0 to 1; the teacher network ensures the priority of selecting labels within the range of [Tp,1], so the final competitive module output is The threshold (3) Design a competition module. Under the influence of the competition mechanism, in the initial stage of training, the student network will be guided and constrained by the teacher network and learn the knowledge of the teacher network. As the training progresses, the student network gradually surpasses the teacher network, completes knowledge distillation and knowledge transfer, and achieves efficient learning tasks in the target domain.

Images