Passive field adaptive image recognition method for difficult sample division processing

Abstract

The invention discloses a passive field adaptive image recognition method for difficult sample division processing. The method is characterized by comprising the following steps: firstly, not accessing source domain data and training a source domain model; and then constructing a target domain model, training an optimal feature generator of the target domain, designing mutual information maximization loss and a weighted clustering algorithm to learn feature representation of the target domain, and implicitly aligning cross-domain features. And finally, the difficult samples are divided and processed, the target domain samples are divided into simple samples, low-confidence samples and high-confidence samples, and an effective weight distribution mechanism is designed to be combined with a strategy for promoting feature learning so as to realize excellent domain migration ability. The invention aims to provide a high-precision passive field adaptive image recognition method, which solves the problems of data security and privacy protection of existing field adaptation, solves the problem of dependence of deep learning on a large amount of annotation data, reduces sample annotation cost and hardware implementation cost in image recognition, realizes cross-field knowledge migration, and improves image recognition efficiency. And high-precision image recognition of the target domain is completed.

Description

Technical Field

[0001] The invention belongs to the technical field of image recognition and domain adaptive learning, and relates to a passive domain adaptive image recognition method for difficult sample division processing. Background Art

[0002] Deep learning has become a research hotspot in machine learning [1][2] , deep learning, especially deep convolutional neural networks, can automatically learn to extract rich features from image big data, and its research results have been widely applied to various fields. Although deep learning methods have achieved great results, they have the following problems: 1) Deep learning requires a large amount of labeled training sample data, and many domain experts are very expensive to label, and some labeled samples are scarce; 2) Deep learning requires a large amount of memory and computing resources, otherwise the training process will be extremely time-consuming; 3) Training deep neural networks often becomes very complicated due to overfitting and convergence problems, and it is often necessary to repeatedly adjust the learning parameters in the framework or network. The high performance of deep learning is closely related to a large number of labeled samples. The high requirements for the hardware configuration lead to high hardware costs. At the same time, the large amount of labeled data requires a high cost of labeling. Therefore, how to use the existing labeled domain data knowledge to label the unlabeled domain data for classification is of great significance in various application fields.

[0003] Domain Adaptation is one of the main methods to solve the above domain knowledge migration. It can overcome the differences in data distribution and label space between the two domains, migrate the labeled source domain knowledge to the unlabeled target domain, and complete the learning task of the target domain. [3] ) can learn a model with good performance on the source domain data, adapt to the target domain data with different distribution or even different category label space, and solve the learning task of the target domain data with no labeled samples or a small number of labeled samples. At present, domain adaptation can be roughly divided into two categories. One method is based on statistical moment alignment, which uses the distance function to minimize the distribution difference between the source domain and the target domain, including the maximum mean deviation (MMD) [4] , Center moment deviation (CMD) [5] and second-order statistical alignment. The other is an adversarial approach that shortens the distance between the two domains through adversarial training and obtains effective cross-domain features. Existing adversarial domain adaptation methods are mainly divided into three categories: [3] : Gradient reversal-based methods, minimax optimization-based methods, and generative adversarial network-based methods. The first two methods minimize the domain distribution difference through feature-level domain confusion supervised by a domain discriminator, and the latter method achieves pixel-level domain transfer by synthesizing indistinguishable target domain images.

[0004] Existing domain adaptation methods use source domain data, but the security and privacy of source domain data are difficult to guarantee. At the same time, source domain data needs to be transmitted and saved, which increases the workload and overhead. In order to protect the security and privacy of source domain data, passive domain adaptation is proposed. Passive domain adaptation does not use source domain data, but can achieve cross-domain knowledge transfer and complete related learning tasks on the unlabeled target domain. It can reduce the transmission and storage of source domain data and protect the security and privacy of source domain data. Existing passive domain adaptation is divided into two types: white box and black box. [6] , the difference lies in whether the source domain data is available when pre-training the model. Among them, white-box SFDA has two major branches: data generation-based methods [8] and model fine-tuning based methods [7] Among them, the data generation-based methods mostly focus on mining the source domain distribution information contained in the source classifier, and transform the problem into a traditional unsupervised domain adaptation problem by fitting the source domain distribution. However, these methods ignore the indirect use of source domain data in the pre-training stage. At the same time, difficult samples are an important factor causing the model transfer effect, which will have a serious negative transfer effect on the knowledge transfer process. The above methods do not pay attention to and deal with difficult samples.

[0005] In order to solve the above problems, a passive domain adaptive image recognition method is proposed: a passive domain adaptive image recognition method with difficult sample division and processing. Difficult samples are divided and processed, which can not only solve the problem of insufficient image annotation samples in image recognition applications, but also realize cross-domain knowledge transfer and high-precision image classification.

[0006] [1] K.He,

[0007] [2] C. Szegedy, S. Ioffe, V. Vanhoucke, A. Alemi. Inception-v4, inception-resNet and the impact of residual connections on learning, in: AAAI Conference on Artificial Intelligence, 2016.

[0008] [3]L.Zhang,X.Gao.Transfer adaptation learning:a decade survey.IEEETransactions on Neural Networks and Learning Systems,2024,35(1):23-44.

[0009] [4]Tzeng,E.,Hoffman,J.,Zhang,et al.Deep domain confusion:Maximizingfor domain invariance[J].Computing Research Repository,2014.

[0010] [5]Zellinger W,Grubinger T,Lughofer,E,et al.Central momentdiscrepancy(CMD)for domain-invariant representation learning[C].InternationalConference on Learning Representations,2017.

[0011] [6]Y.Fang,P.Yap,W.Lin et al.Source-free unsupervised domainadaptation:a survey.

[0012] Neural Networks.2024:106230.

[0013] [7]J.Lee,G.Lee.Feature alignment by uncertainty and self-training forsource-free unsupervised domain adaptation.Neural Networks,2023:682-692.

[0014] [8]VKKurmi,VKSubramanian,VPNboodiri.Domain impression:asource data free domain adaptation method.IEEE / CVF Winter Conference onApplications of Computer Vision,2021:615-625. Summary of the invention

[0015] The purpose of the present invention is to provide an image recognition method for knowledge transfer: a passive domain adaptive image recognition method for difficult sample partitioning processing, which solves the data security and privacy protection problems of existing domain adaptation, solves the dependence of deep learning on a large amount of labeled data, reduces the sample annotation cost and hardware implementation cost in image recognition, realizes cross-domain knowledge transfer, and completes target domain passive domain and high-precision image recognition.

[0016] The technical solution adopted by the present invention is

[0017] A passive domain adaptive image recognition method for difficult sample partitioning is characterized in that the source domain model is first trained without accessing the source domain data, and the classifier module of the source model is frozen; then the target domain model is constructed, the optimal feature generator of the target domain is trained, so that the target domain data distribution matches the source domain data distribution, and the mutual information maximization loss and weighted clustering algorithm are designed to learn the feature representation of the target domain and implicitly align cross-domain features. Finally, the difficult samples are partitioned and processed, the target domain samples are divided into simple samples, low-confidence and high-confidence difficult samples, and an effective weight allocation mechanism is designed in combination with a strategy to promote feature learning to achieve excellent domain transfer capabilities.

[0018] The beneficial effects of the present invention are

[0019] 1. The passive domain adaptive image recognition method for difficult sample partitioning processes trains the source domain model without accessing the source domain data, effectively protecting the security and privacy of the source domain data and reducing the data transmission and storage costs of the source domain data.

[0020] 2. A passive domain adaptive image recognition method for difficult sample partitioning, which designs mutual information maximization loss and weighted clustering algorithm, can learn the optimal feature representation of the target domain. In the case of unlabeled samples for image recognition, it can achieve unsupervised and high-precision cross-domain image recognition tasks in the target domain.

[0021] 3. The passive domain adaptive image recognition method with difficult sample division and processing divides the difficult samples that affect knowledge transfer, designs an effective weight allocation mechanism combined with a strategy to promote feature learning, can achieve a high degree of domain knowledge transfer capability and improve the accuracy of cross-domain image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is the overall structural framework of the method of the present invention;

[0023] like Figure 1 As shown, and is the labeled source domain and the unlabeled target domain, where n s and n t represents the number of samples in the source domain and the target domain, y s is the label set of the source domain, x s and x t Represents the source domain and target domain data, L CE is the supervision loss of source domain samples, L IM is the mutual information loss of the target domain samples, and L SCE is the loss for optimizing the feature learning of difficult samples in the target domain. In SFDA, the label spaces of the source and target domains are consistent, while the source domain data cannot be directly accessed. DETAILED DESCRIPTION

[0024] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] The whole process of the method is divided into three stages: training the source domain model, building the target domain model, and dividing and processing difficult samples.

[0026] The first step is to train the source domain model. The general process of this step is as follows.

[0027] First, a source domain model is trained using only all source domain data and combined with a supervision loss L CE Perform network optimization. Input source domain data x s To the network feature extractor G s Then the generated data features are input into the classifier F s , and finally use the label smoothed cross entropy function to back propagate the loss, L CE The definitions are as follows.

[0028]

[0029] Among them, SM(·) represents the Softmax function, γ is the smoothing factor and K is the number of categories. The label smoothing strategy can promote samples to be tightly and evenly distributed in clusters.

[0030] Then the second step is to build the target domain model. The target domain model is initialized using the pre-trained source domain model, and the target domain classification module is frozen as a way to implicitly obtain source domain data information. The process of training the target domain network is to expect to learn the optimal feature generator G t , so that the target domain data distribution matches the source domain data distribution. The general process of this step is as follows.

[0031] (1) Design mutual information maximization loss

[0032] Without access to the source data, it is difficult to achieve feature-level alignment. Therefore, we design the mutual information maximization loss L IM , making individual outputs distinguishable and globally diverse, and maximizing the mutual information loss L IM The definitions are as follows.

[0033]

[0034] in, yes Find the expected result. IM The trivial solution of having the same one-hot representation for all unlabeled data can be avoided to promote the diversity of target features.

[0035] (2) Constructing a weighted K-means clustering algorithm

[0036] Maximizing mutual information helps the target domain model to align the target data with the unseen source data well, but may match the wrong source hypothesis to some extent. In order to alleviate this effect, a pseudo-labeling strategy is adopted to better supervise the target data. However, the pseudo-labels generated by the source hypothesis are usually noisy, so a new self-supervised pseudo-labeling strategy is proposed: the weighted k-means clustering algorithm, the general process of which is as follows.

[0037] First, the centroid of each category in the target domain is calculated as shown in the following formula. These centroids can robustly describe the distribution of different categories in the target domain.

[0038]

[0039] Then, the pseudo label of the target sample is obtained through the nearest centroid classifier as shown in the following formula.

[0040]

[0041] Where Dis(·) represents the measurement criterion, and the cosine similarity is used here.

[0042] After multiple iterations, the class centroid tends to be stable and the generated pseudo-labels are more reliable. In practice, usually two class centroid updates are enough to obtain sufficiently accurate pseudo-labels, which lays the foundation for promoting feature learning of difficult samples.

[0043] The last third step is to divide and process difficult samples, dividing the target domain samples into simple samples, low-confidence difficult samples and high-confidence difficult samples, and processing them. The general steps of this step are as follows.

[0044] (1) First, the target domain samples are divided into simple samples, low-confidence difficult samples, and high-confidence difficult samples. The sample classification entropy is used as the criterion to determine whether the sample is difficult to classify, and the sample x t The classification entropy is defined as

[0045]

[0046] Among them, p c (x t ) represents the sample x t The probability of being classified into category c, where C represents the total number of categories. By calculating the classification entropy of a sample, the difficulty of classifying the sample can be effectively identified.

[0047] (2) Then, the mean and standard deviation of the sample classification entropy are continuously updated according to the number of iterations. Each batch of samples is sorted according to the classification entropy, and the mean of the sample classification entropy is used as the dynamic threshold for division. Samples greater than the threshold are classified as difficult samples. However, difficult samples with low confidence levels mostly show obvious discrimination ability after a certain number of iterations in the source sample division. Therefore, the standard deviation of the sample classification entropy is used as the boundary to divide the target sample into high-confidence and low-confidence difficult samples. Then the sample x t The division criterion is shown below.

[0048]

[0049] Among them, H(x t ) represents the sample x t The classification entropy value of , the dynamic threshold μ is the mean of the classification entropy, and the standard deviation σ is the tolerance boundary.

[0050] (3) Secondly, we will focus on how to promote the feature learning of difficult samples and assign different weights to the divided target domain samples, as shown in the following formula.

[0051] w L +w H +w E =1, (7)

[0052] Among them, w L and w HRepresent the weights of low-confidence and high-confidence difficult samples, w E represents the weight of simple samples.

[0053] In formula (7), we define weight strategies for different types of samples and propose more allocation details. First, we assign larger weights to difficult samples. Second, we should assign larger weights to difficult samples with low confidence, and relatively smaller weights to difficult samples with high confidence. In practice, whether it is a fixed or adaptive weight allocation strategy, w H All are smaller than w L Finally, w L 、w H and w E The combination gives the weight matrix w, which is then used as a coefficient with the target sample x t The predicted distribution of Combined with the pseudo labels obtained by the weighted K-means clustering strategy, the optimization objective is defined as

[0054]

[0055] Among them, w refers to the weight matrix generated by the sample classification entropy, and m refers to the number of target samples in a batch.

[0056] Through the indirect guidance of the source domain model and the promotion of feature learning of difficult samples, the target domain model not only learns rich sample features, but also has a certain ability to identify the features of difficult samples.

[0057] In summary, the overall optimization goal is as follows.

[0058] L object =αL IM +βL SCE , (9)

[0059] Among them, α and β are hyperparameters used to control the mutual information loss and promote the loss of difficult sample learning.

Claims

1. A passive domain adaptive image recognition method for difficult sample segmentation processing, characterized in that First, without accessing the source domain data, train the source domain model; the general process of this step is as follows: (1) First, train a source domain model, using only all source domain data and combining it with the supervision loss L CE Perform network optimization; (2) Then input the source domain data x s To the network feature extractor G s In the above example, the generated data features are input into the classifier F s ; (3) Finally, the label smoothed cross entropy function is used to back propagate the loss, L CE The definition is as follows: Where SM(·) represents the Softmax function, γ is the smoothing factor, K is the number of categories, and the label smoothing strategy can promote samples to be tightly and evenly distributed in clusters.

2. A passive domain adaptive image recognition method for difficult sample segmentation processing, characterized in that Secondly, the target domain model is constructed; the target domain model is initialized using the pre-trained source domain model, and the target domain classification module is frozen as a way to implicitly obtain source domain data information; The process of training the target domain network is to expect to learn the optimal feature generator G t , so that the target domain data distribution matches the source domain data distribution; the general process of this step is as follows: (1) Design mutual information maximization loss Without access to the source data, it is difficult to achieve feature-level alignment. Therefore, we design the mutual information maximization loss L IM , making individual outputs distinguishable and globally diverse, and maximizing the mutual information loss L IM The definition is as follows: in yes Find the expected result, L IM It can avoid the trivial solution of having the same one-hot for all unlabeled data to promote the diversity of target features; (2) Constructing a weighted K-means clustering algorithm Maximizing mutual information helps the target domain model to align the target data with the unseen source data well, but it may match the wrong source hypothesis to a certain extent. To alleviate this effect, a pseudo-label strategy is adopted to better supervise the target data. However, the pseudo-labels generated by the source hypothesis are usually noisy, so a new self-supervised pseudo-labeling strategy is proposed: the weighted k-means clustering algorithm. The general process is as follows: First, the centroid of each category in the target domain is calculated, as shown in the following formula. These centroids can robustly describe the distribution of different categories in the target domain; Then, the pseudo label of the target sample is obtained through the nearest centroid classifier, as shown in the following formula: Where Dis(·) represents the measurement criterion, and cosine similarity is used here; After many iterations, the class centroid tends to be stable and the generated pseudo-labels are more reliable; in practice, usually two class centroid updates are enough to obtain sufficiently accurate pseudo-labels, which lays the foundation for promoting feature learning of difficult samples.

3. A passive domain adaptive image recognition method for difficult sample segmentation processing, characterized in that Finally, the difficult samples are divided and processed. The target domain samples are divided into simple samples, difficult samples with low confidence, and difficult samples with high confidence, and processed. The general steps of this step are as follows: (1) First, the target domain samples are divided into simple samples, low-confidence difficult samples, and high-confidence difficult samples. The sample classification entropy is used as the criterion to determine whether the sample is difficult to classify, and the sample x t The classification entropy is defined as where p c (x t ) represents the sample x t The probability of being classified into category c, where C represents the total number of categories. By calculating the classification entropy of samples, the difficulty of sample classification can be effectively identified. (2) Then, the mean and standard deviation of the sample classification entropy are continuously updated according to the number of iterations. Each batch of samples is sorted according to the classification entropy, and the mean of the sample classification entropy is used as the dynamic threshold for division. Samples greater than the threshold are classified as difficult samples. However, difficult samples with low confidence levels generally show obvious discrimination ability after a certain number of iterations in the source sample division. Therefore, the standard deviation of the sample classification entropy is used as the boundary to divide the target samples into high-confidence and low-confidence difficult samples. Then, the sample x t The division criteria are as follows: Where H(x t ) is the sample x t The classification entropy value, the dynamic threshold μ is the mean of the classification entropy, and the standard deviation σ is the tolerance boundary; (3) Secondly, we will focus on how to promote the feature learning of difficult samples and assign different weights to the divided target domain samples, as shown in the following formula: In L +in H +in E =1, (7) where w L and w H represents the weights of low-confidence and high-confidence difficult samples, w E is the weight of the simple sample; The method will define weight strategies for different types of samples and propose more allocation details; first, assign larger weights to difficult samples; second, assign larger weights to difficult samples with low confidence, and assign relatively smaller weights to difficult samples with high confidence; In practice, whether it is a fixed or adaptive weight allocation strategy, w H All are smaller than w L ; L 、w H and w E The combination gives the weight matrix w, which is then used as a coefficient with the target sample x t The predicted distribution of Combined with the pseudo labels obtained by the weighted K-means clustering strategy, the optimization objective is defined as Where w refers to the weight matrix generated by the sample classification entropy, and m refers to the number of target samples in a batch; Through the indirect guidance of the source domain model and the promotion of feature learning of difficult samples, the target domain model not only learns rich sample features, but also has a certain ability to identify the features of difficult samples; In summary, the overall optimization goal is as follows: L object =αL IM +βL SCE , (9) Among them, α and β are hyperparameters used to control mutual information loss and promote learning loss of difficult samples.

Images