Test time domain adaptive classification method based on feature matching and three-branch attention

Abstract

Effective recognition of a known class of a source domain and an unknown class of a target domain is achieved through a passive universal field adaptive method of generative prototype comparison clustering and three-branch attention, and the method is characterized in that firstly, source domain data are not used, and a generative network is designed to generate a source domain feature prototype so that field distribution alignment can be conducted on the feature level; then, designing a target private class identification processing method of contrast clustering to screen a target private class, and establishing corresponding pseudo label distribution strategies and processing methods for different special samples; secondly, designing a feature matching algorithm based on a feature distance to preferentially associate the generated features and the target features; and finally, constructing a three-branch attention module to extract inter-domain invariant features, and completing cross-domain feature extraction and target domain classification. The invention aims to provide a passive universal field adaptive method for generating prototype contrast clustering and three-branch attention, and solves the problems of field distribution offset, category label offset, data privacy security, data storage and the like of an existing field adaptive method.

Description

Technical Field

[0001] The present invention belongs to the technical field of domain - adaptive pattern classification, and relates to a test - time domain - adaptive classification method based on feature matching and three - branch attention. Background Art

[0002] Machine learning is an important research direction in the era of artificial intelligence. Its research results are widely applied to various fields and play an increasingly important role in people's daily lives. Current machine - learning technologies should face an open environment. The machine - learning process in an open environment does not satisfy the following three basic assumptions [1]: First, the test and training data do not satisfy the independent and identically - distributed assumption, and there is a distribution - shift problem in machine learning; second, the class - label space of the test and training data does not satisfy the closed - class - label - space assumption, and there is a class - label - space - shift problem in machine learning; third, it does not satisfy the assumption of having enough large - scale labeled data. There are no class labels or very few available class labels in the test domain, and the cost of manual annotation of labels is expensive or even difficult to achieve.

[0003] Domain Adaptation (DA [2][3]) can learn a well - performing model on source - domain data and adapt it to target - domain data with different distributions or even different class - label spaces, and solve the learning tasks of target - domain data with unlabeled samples or a small number of labeled samples. DA can adapt to the application scenarios of an open environment and is one of the main paradigms of machine learning in an open environment. [1] .

[0004] Test - time Domain Adaptation [4] is a branch of DA. It uses a pre - trained model to adapt to unlabeled test data during testing. Traditional unsupervised domain adaptation usually performs offline model adaptation. The model is fixed after training and no adjustment is made during testing, while test - time domain adaptation only uses test samples and makes certain adjustments to the model during inference and prediction so that the model can adapt to the unknown data probability distribution at test time. Because the test - time domain - adaptation method does not require any data from the source domain for adaptation, in some studies, test - time domain adaptation is also called source - free domain adaptation. [5] , but it is a different setting from general source - free domain adaptation.

[0005] Although test - time domain adaptation can improve the performance of the model, there are also some disadvantages and limitations. First, the computational and data requirements of test - time domain adaptation are relatively large. [6]. During testing, domain adaptation methods require additional test data and additional calculations and adjustments to the model during testing, which increases computational costs and time consumption. Secondly, existing domain adaptation methods during testing adjust and change the trained model, which causes the model to gradually lose the knowledge learned from pre-training, and the scalability of the model is weak. Moreover, domain adaptation during testing [7] cannot obtain the target domain distribution. It cannot calculate the statistics of the target distribution, cannot align the distributions of the source domain and the target domain like active domain adaptation, and cannot assign pseudo-labels to target samples by clustering. Therefore, it is impossible to align the data distributions of the target domain and the source domain, and only the target samples can be used to fine-tune the model continuously. To solve these problems, the patent proposes a domain adaptation classification method during testing based on feature matching and three-branch attention. First, the source class centers are stored during the pre-training process of the source domain. Then, the source classifier and the stored source class centers are used to generate feature prototypes, and a conditional generation network is constructed and trained to fit the inaccessible source domain distribution. Finally, a distribution alignment strategy between the fitted pseudo-source domain and the target domain is established, and the trained generation network is used to generate feature prototypes to replace the source domain data to complete domain adaptation with the target data. During domain adaptation, the test-time target data in each batch is first feature-extracted by a feature extractor, then paired with the feature prototypes based on distance matching in the feature space, and then the feature pairs are input into the three-branch attention module to achieve domain alignment.

[0006] [1] Yuan Xiaotong, Zhang Xuyao, Liu Xi, Cheng Zhen, Liu Chenglin. Research Progress of Machine Learning Theory for Open Environments. Pattern Recognition and Artificial Intelligence, 2023, 36(12): 1059-1071.

[0007] [2] S. J. Pan, Q. Yang. A survey on transfer learning. IEEE Transactions On Knowledge Data Engineering. 2010, 22(10): 1345-1359.

[0008] [3] L. Zhang, X. Gao. Transfer adaptation learning: a decade survey. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35(1): 23-44.

[0009] [4] Y. Sun, X. Wang, Z. Liu, J. Miller, A. Efros, and M. Hardt. Test-time training with self-supervision for generalization under distribution shifts. International Conference on Machine Learning, 2020, pp. 9229-9248.

[0010] [5] J. N. Kundu, N. Venkat, R. M V, R. V. Babu. Universal source-free domain adaptation. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020, pp. 4544-4553.

[0011] [6] Y. Zhang, X. wang, K. Jin, Z. Zhang, L. Wang, R. Jin, T. Tan. AdaNPC: Exploring non-parametric classifier for test-time adaptation. International Conference on Machine Learning, 2023.

[0012] [7] D. Mario, R. A. Marsden, B. Yang. Robust mean teacher for continual and gradual test-time adaptation. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023. Summary of the Invention

[0013] The object of the present invention is to provide a test-time domain adaptation classification method based on feature matching and three-branch attention, which solves problems such as domain distribution shift, class label shift, source domain data privacy and security, and high costs of data storage and transmission in existing domain adaptation methods, enhances the extraction of cross-domain features of source domain and target domain data, improves the recognition accuracy of known classes in the source domain, increases the model's ability to discover unknown classes, realizes the indirect utilization of source domain data in the pre-training stage, promotes the privacy protection and data security protection of source domain data, and performs unsupervised classification learning on unlabeled data in the target domain.

[0014] The technical solution adopted by the present invention is

[0015] A test-time domain adaptation classification method based on feature matching and three-branch attention, which is characterized in that first, source class centers are stored during the pre-training process in the source domain, and then feature prototypes are generated by using the source classifier and the stored source class centers, and a conditional generation network is constructed and trained to fit the inaccessible source domain distribution; then, each batch of online target data is first subjected to feature extraction by a feature extractor, a matching method based on feature distance is designed to complete paired feature matching; finally, a three-branch attention mechanism is designed to extract target domain features to achieve domain alignment, and the trained generation network is used to generate feature prototypes to replace the source domain data to complete domain adaptation classification with the target data.

[0016] The beneficial effects of the present invention are

[0017] 1. Construct a memory to store the source domain class centers to assist in the generation of feature prototypes in the next stage. The method can exchange a very small amount of data transmission volume for an improvement in the quality of the generated prototypes. When testing, only the target domain data is used without using the source domain data, which can effectively reduce the transmission of the source domain data and protect the security and privacy of the source domain data.

[0018] 2. Design a conditional generation network to generate source domain feature prototypes and simulate the source distribution, pair with the features of online target samples, and set the pseudo-labels of the online target samples to the labels of the generated prototypes paired with them, so as to achieve efficient domain alignment at the feature level.

[0019] 3. Construct a three-branch attention module to extract inter-domain invariant features, design two self-attention modules to extract generated features and target features respectively, and design a cross-attention module to extract fused features. The cross-attention module will assign lower weights to dissimilar block pairs in the feature pair, filter out pseudo-label noise, can achieve the alignment of the target domain and the pseudo-source domain distribution with high precision, filter out the paired feature pairs with noisy pseudo-labels, and improve the classification accuracy. Description of the Drawings

[0020] Figure 1 is the overall framework diagram of the method of the present invention;

[0021] Figure 2 It is the structural diagram of the three-branch attention module. Specific implementation manners

[0022] The present invention will be described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0023] The general process of the method is divided into three stages: the pre-training stage, the generating feature prototype stage, and the domain adaptation stage.

[0024] (1) First is the pre-training stage. The method constructs a memory to store the source domain class centers. The general process is as follows: First, store the source class centers during the source domain pre-training process. Then, use the source classifier and the stored source class centers to generate feature prototypes, which are used to assist the generation of feature prototypes in the next stage. By using a very small amount of data transmission volume, the quality of the generated prototypes is improved. Construct and train a conditional generation network to fit the inaccessible source domain distribution. Then, use the trained generation network to generate feature prototypes to replace the source domain data and complete domain adaptation with the target data.

[0025] (2) Then is the generating feature prototype stage. Design a conditional generation network to generate source domain feature prototypes instead of generating images, so as to perform domain alignment at the feature level. The specific process is as follows: Given the class label and random noise, train the generation network by using the standard cross-entropy loss through the source model classifier to generate the feature prototypes of the inaccessible source domain and simulate the source distribution. After being paired with the features of the online target samples, they are used as the input of the attention module. Then, introduce a repository to record the class centers of each type of source class during the training of the source model, so that the generated prototype features are more accurate. And construct a consistency loss to assist the training during the process of generating prototype features, and add a contrast loss between the prototype features to assist the training to make the feature prototypes more compact.

[0026] (3)Finally, it is the domain adaptation stage. This stage includes two parts: feature matching and a three-branch attention module. To appropriately associate the generated feature prototypes and the target features, a feature matching algorithm based on feature distance is designed to calculate the cosine similarity between the features of the online target samples in this batch and all the generated prototype feature vectors, and find the most similar feature from the generated prototypes for each target sample feature to pair as the input of the domain adaptation part. Since the online target samples cannot guide the pseudo-label assignment of the target domain by calculating the class centroids of all target samples, the pseudo-labels of the online target samples are set to the labels of the paired generated prototypes. Then, to achieve domain alignment, a three-branch attention module is designed, in which two self-attention modules are designed to extract the generated features and the target features respectively, and a cross-attention module is designed to extract the fused features. The cross-attention module will assign lower weights to the dissimilar block pairs in the feature pairs, thus alleviating the problem of pseudo-label noise. In addition, this method introduces a distillation loss, using the output of the cross-attention module branch to guide the training of the target branch, and only the target branch is retained during testing.

Claims

1. A test-time domain-adaptive classification method based on feature matching and three-branch attention, characterized by First, without using source domain data, a conditional generative network is designed to generate source domain feature prototypes to align domain distribution at the feature level; the specific process is as follows: first, the source class centers are stored during the source domain pre-training process, and then the source classifier and the stored source class centers are used to generate feature prototypes to assist in the generation of feature prototypes in the next stage, using a very small amount of data transmission in exchange for improved quality of generated prototypes; then, given class labels and random noise, the source model classifier is used to train the generative network using standard cross entropy loss to generate feature prototypes of the inaccessible source domain and simulate the source distribution, and the trained generative network is used to generate feature prototypes to replace source domain data and target data to complete domain adaptation, wherein a repository is introduced to record the various centers of the source class in the source model and training to make the generated prototype features more accurate, and in the process of generating prototype features, consistency loss and contrast loss are constructed to assist training to make the feature prototype more compact.

2. A test-time domain-adaptive classification method based on feature matching and three-branch attention, characterized by Secondly, a feature matching algorithm based on feature distance is designed to preferentially associate the generated features with the target features. This stage includes three parts: assigning pseudo labels, prototype target matching, and domain alignment. The specific steps are as follows: First, a feature matching algorithm based on feature distance is designed to match the feature prototype with the target features processed by class recognition; Then, the cosine similarity between the target feature and the generated prototype feature vector in the same category is calculated, and the most similar feature pairing is found for the target feature of each common class as the input of the domain adaptation part; since the online target samples cannot guide the pseudo-label assignment of the target domain by calculating the class center of all target samples, the pseudo-label of the online target sample is set to the label of the generated prototype paired with it; finally, for samples of the target unknown class, only the target branch is trained.

3. A test-time domain-adaptive classification method based on feature matching and three-branch attention, characterized by Finally, a three-branch attention module is constructed to achieve domain adaptation; the attention weights of the source domain, target domain, and cross-domain are calculated respectively. Enhanced feature alignment effect extracts invariant features between domains, filters pseudo-label noise, realizes the alignment of target domain and pseudo-source domain distribution, and completes cross-domain feature extraction and target domain classification; the specific process is as follows: construct a three-branch attention module combined with a classifier to realize target domain feature extraction and cross-domain classification. The three-branch attention module includes two self-attention modules and one cross-attention module. The two self-attention modules extract generated features and target features respectively, while the cross-attention module is used to extract fusion features and assign lower weights to feature pairs with inconsistent pseudo-labels to alleviate the problem of pseudo-label noise. At the same time, it is proposed to introduce distillation loss, and use the output of the cross-attention module branch to guide the training of the target branch. Only the target branch is retained during testing.

Images