A cross-domain image classification method based on class prototype-assisted pseudo-label generation

Abstract

The present invention discloses a cross-domain image classification method based on class prototype-assisted pseudo-label generation, which utilizes class prototypes to realize pseudo-label generation of a target domain in the field of domain adaptation, optimizes the strategy of pseudo-label generation, takes the class average features of each category of source domain data as the source domain global class prototype, and takes the class average features of the target domain as the target domain global class prototype, and uses the relationship between all global class prototypes and target domain samples to assign pseudo-labels to target domain samples, so that the model makes full use of the connection between samples at different levels, thereby improving the accuracy of pseudo-labels and ultimately achieving enhanced generalization performance of the model.

Description

Technical Field

[0001] The present invention belongs to the field of image technology, and in particular relates to a cross-domain image classification method. Background Art

[0002] With the development of deep learning, the performance of image classification has achieved leapfrog development and has been successfully applied to network security, intelligent monitoring and other fields. However, the lighting and style of many actual application scenarios are changeable, and due to factors such as privacy, it is difficult to collect training data that is completely consistent with the actual application. Therefore, deep learning models that rely heavily on training data will cause serious performance degradation in actual application scenarios. In order to solve this problem, scholars have proposed cross-domain image classification research, aiming to utilize the correlation between multiple domain data, reduce the model's dependence on the training domain, and improve the model's generalization ability. In this field, specific research includes unsupervised domain adaptation, and the main research directions are divided into data enhancement, feature representation and learning strategies. In domain adaptive image classification based on learning strategies, semi-supervised technology occupies an important position. Its principle is to pseudo-label the target domain data based on the source domain data, thereby achieving the effect of supervised training. Although the current method has achieved relatively good results, there are still errors in the accuracy of pseudo-labeling, and how to solve the negative impact of errors is still an important issue. Summary of the invention

[0003] In order to overcome the shortcomings of the prior art, the present invention provides a cross-domain image classification method based on class prototype-assisted pseudo-label generation, which utilizes class prototypes to realize pseudo-label generation of the target domain in the field of domain adaptation, optimizes the strategy of pseudo-label generation, and uses the class average features of each category of source domain data as the global class prototype of the source domain, and the class average features of the target domain as the global class prototype of the target domain. The relationship between all global class prototypes and target domain samples is used to assign pseudo-labels to the target domain samples, so that the model makes full use of the connection between samples at different levels, thereby improving the accuracy of pseudo-labels and ultimately achieving enhanced generalization performance of the model.

[0004] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0005] Step 1: Initialize the source domain global class prototype and the target domain global class prototype by category, denoted as S_po and T_po respectively. The dimensions of the two are the same, both [c, d], where c is the number of categories and d is the feature dimension;

[0006] Step 2: Use the feature extraction network to extract the features of the source domain data. According to the sample labels, take the average of the features of the same category samples to update the source domain global class prototype. The update strategy is the exponential moving average strategy S po′ =γ*S po+(1-γ)*Stmp, γ represents the degree of forgetting of the global class prototype of the old source domain, and its value is between 0 and 1. The larger the value, the less forgotten; S po represents the global class prototype of the old source domain, S po′ Represents the updated source domain global class prototype, Stmp represents the temporary source domain global class prototype obtained in the current batch;

[0007] Step 3: Calculate the cosine distance between the target domain sample and the global class prototype of the source domain, and assign the class label with the largest similarity and greater than the threshold a to the target domain sample;

[0008] Step 4: Calculate the cosine distance between target domain samples, and assign the category label with the largest similarity greater than the threshold b to the unlabeled sample;

[0009] Step 5: Calculate the cosine distance between the target domain sample and the target domain global class prototype, and assign the class label with the largest similarity to the unlabeled sample;

[0010] Step 6: According to the pseudo-labels of the target domain samples, take the average of the features of the samples of the same category to update the global class prototype of the target domain. The update strategy is the exponential moving average strategy T po′ =γ*T po +(1-γ)*Ttmp, γ represents the degree of forgetting of the global class prototype of the old target domain, and its value is between 0 and 1. The larger its value is, the less forgotten it is; T po Represents the old target domain global class prototype, T po′ represents the updated target domain global class prototype, and Ttmp represents the temporary target domain global class prototype obtained in the current batch;

[0011] Step 7: Repeat steps 2 to 6 to complete the multi-level pseudo-labeling process.

[0012] Preferably, in the multi-level pseudo-label assignment process, the contrastive learning loss function used is as follows:

[0013] loss InfoNCE =loss sinfo +loss tinfo ,

[0014]

[0015] Among them, loss sinfo represents the contrastive learning loss between the source domain sample and the source domain global class prototype, loss tinfo represents the contrastive learning loss between the target domain sample and the source domain global class prototype, x s represents the source domain sample, k +represents the global class prototype of the same type in the source domain, 2M represents the number of global class prototypes in the source domain and the target domain, represents all source domain samples; τ represents a temperature parameter;

[0016]

[0017] Among them, x t represents the target domain samples whose pseudo-label confidence is greater than 0.5, k + represents the global class prototype of the target domain of the same type, 2M represents the number of global class prototypes of the source domain and the target domain, represents all target domain samples; τ represents a temperature parameter.

[0018] Preferably, the multi-level pseudo-label assignment process introduces two feature alignment loss functions. The DAN function realizes feature alignment between the source domain and the target domain, and is used for direct alignment of source domain samples and target domain samples after feature extraction. The OT function realizes alignment of target domain feature distribution and classifier weights, and is used for target domain sample features and the last layer of classifier weights.

[0019] The beneficial effects of the present invention are as follows:

[0020] 1. The present invention can take into account the update of the source domain class prototype and the target domain class prototype, and at the same time utilize the relationship between the class prototypes of the samples to shorten the distance between the samples and the same class prototypes in the feature space, and to increase the distance between the samples and the samples of different classes, thus overcoming the domain differences of the class prototypes to a certain extent.

[0021] 2. The present invention makes full use of class prototypes to label target domain samples, and establishes different priorities for assigning pseudo labels and thresholds for assigning pseudo labels. This can fully explore the relationship between source domain class prototypes and target domain samples, between target domain samples, and between target domain class prototypes and target domain samples, thus improving the accuracy of pseudo labels.

[0022] 3. The present invention combines two commonly used alignment loss functions to improve the quality of feature extraction.

[0023] 4. The method of the present invention can improve the generalization performance of some domain adaptation models on domain adaptation datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Schematic diagram of the pseudo-labeling process of the present invention. DETAILED DESCRIPTION

[0025] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0026] like Figure 1 As shown, the present invention sets up a three-level anti-counterfeiting label mechanism based on the idea of ​​nearest neighbor distance, and the specific steps are as follows:

[0027] (1) Initialize the source domain global class prototype and the target domain global class prototype by category, denoted as S_po and T_po respectively. The dimensions of the two are the same, both [c, d], where c is the number of categories and d is the feature dimension;

[0028] (2) Use the feature extraction network to extract the features of the source domain data. According to the sample labels, take the average value of the features of samples of the same category to update the source domain global class prototype. The update strategy is the exponential moving average strategy S po =γ*S po +(1-γ)*Stmp;

[0029] (3) Calculate the cosine distance between the target domain sample and the global class prototype of the source domain, and assign the class label with the largest similarity and greater than the threshold a to the target domain sample;

[0030] (4) Calculate the cosine distance between samples in the target domain, and assign the category label with the largest similarity greater than the threshold b to the unlabeled sample;

[0031] (5) Calculate the cosine distance between the target domain sample and the target domain global class prototype, and assign the class label with the greatest similarity to the unlabeled sample;

[0032] (6) According to the pseudo-labels of the target domain samples, the average of the features of the samples of the same category is taken to update the global class prototype of the target domain. The updating strategy is the exponential moving average strategy, which is the same as S po ;

[0033] The above process is a specific iterative pseudo-label assignment process in the training process, which is repeated continuously. Through this multi-level pseudo-label assignment process, the relationship between different sample levels can be fully utilized to improve the quality of pseudo-labels.

[0034] In this process, an improved contrastive learning loss function is designed to improve the quality of class prototypes, which aims to narrow the distance between samples and source domain class prototypes or target domain class prototypes.

[0035] loss InfoNCE =loss sinfo +loss tinfo ,

[0036]

[0037] loss tinfo and loss sinfo The formula structure is the same, but the source domain sample x s Replace with target domain samples x whose pseudo label confidence is greater than 0.5 t . 2M represents the number of global class prototypes in the source domain and the target domain, ki represents the global class prototype, and τ represents a temperature parameter.

[0038] Figure 1 Medium grey represents unlabeled, blue represents class 1, and green represents class 2. Specific embodiment:

[0040] In order to improve the accuracy of pseudo-label assignment, a multi-level regularization constraint is designed based on the cross entropy loss, which is the most basic in the training process. This constraint includes the feature alignment DAN function and the optimal transmission loss OT function.

[0041] (1) DAN function

[0042] Directly use the feature alignment function DAN which is widely used in domain generalization,

[0043]

[0044] Among them, λ is greater than 0 and is a penalty factor. l1 and l2 represent the layers of the network. In the specific experiment, l1 is taken as 6 and l2 is taken as 8. It is the MK-MMD value of the source domain data and the target domain data represented in the lth layer of the network. This constraint can make the representation of the source domain data and the target domain data in the l1th and l2th hidden layers in the network similar, thereby achieving domain feature alignment.

[0045] (2)OT function

[0046] A concise version of the bidirectional optimal transmission loss function is used as follows:

[0047] loss OT =L t->w +L w->t ,

[0048] Where L t->w represents the expectation from the target data feature distribution to the classifier weight distribution, L w->t represents the expectation from the classifier weight distribution to the target data feature distribution,

[0049] (3) Total loss function

[0050] The total loss consists of the common classification loss, the contrastive learning loss in the pseudo-label assignment stage, and two alignment loss functions as follows:

[0051] Loss = loss CE +loss OT +loss DAN +α*loss InfoNCE ,

[0052] Among them, α has two meanings. One is the impact of InfoNCE loss in the total loss, which changes with the training process; the other is the threshold at which the InfoNCE loss of the target domain data affects the overall InfoNCE loss. α gradually changes from 0 to 1 as the amount of data increases during training. By introducing this parameter, the negative impact of the class prototype on the model in the early stage of training can be effectively reduced, and when the class prototype gradually stabilizes in the later stage, its impact on the model is also correspondingly strengthened.

Claims

1. A cross-domain image classification method based on class prototype-assisted pseudo-label generation, characterized in that: The steps include: Step 1: Initialize the source domain global class prototype and the target domain global class prototype by category, denoted as S_po and T_po respectively. The dimensions of the two are the same, both [c, d], where c is the number of categories and d is the feature dimension; Step 2: Use the feature extraction network to extract the features of the source domain data. According to the sample labels, take the average of the features of the same category samples to update the source domain global class prototype. The update strategy is the exponential moving average strategy S po′ =γ*S po +(1-γ)*Stmp, γ represents the degree of forgetting of the global class prototype of the old source domain, and its value is between 0 and 1. The larger the value, the less forgotten; S po represents the global class prototype of the old source domain, S po′ Represents the updated source domain global class prototype, Stmp represents the temporary source domain global class prototype obtained in the current batch; Step 3: Calculate the cosine distance between the target domain sample and the global class prototype of the source domain, and assign the class label with the largest similarity and greater than the threshold a to the target domain sample; Step 4: Calculate the cosine distance between target domain samples, and assign the category label with the largest similarity greater than the threshold b to the unlabeled sample; Step 5: Calculate the cosine distance between the target domain sample and the target domain global class prototype, and assign the class label with the largest similarity to the unlabeled sample; Step 6: According to the pseudo-labels of the target domain samples, take the average of the features of the samples of the same category to update the global class prototype of the target domain. The update strategy is the exponential moving average strategy T po′ =γ*T po +(1-γ)*Ttmp, γ represents the degree of forgetting of the global class prototype of the old target domain, and its value is between 0 and 1. The larger its value is, the less forgotten it is; T po Represents the old target domain global class prototype, T po′ represents the updated target domain global class prototype, and Ttmp represents the temporary target domain global class prototype obtained in the current batch; Step 7: Repeat steps 2 to 6 to complete the multi-level pseudo-labeling process.

2. According to claim 1, a cross-domain image classification method based on class prototype-assisted pseudo-label generation is characterized in that: In the multi-level pseudo-label assignment process, the contrastive learning loss function used is as follows: loss InfoNCE =loss sin fo +loss t in fo , Among them, loss sin fo represents the contrastive learning loss between the source domain sample and the source domain global class prototype, loss t inf o represents the contrastive learning loss between the target domain sample and the source domain global class prototype, x s represents the source domain sample, k + represents the global class prototype of the same type in the source domain, 2M represents the number of global class prototypes in the source domain and the target domain, represents all source domain samples; τ represents a temperature parameter; Among them, x t represents the target domain samples whose pseudo-label confidence is greater than 0.5, k + represents the global class prototype of the target domain of the same type, 2M represents the number of global class prototypes of the source domain and the target domain, represents all target domain samples; τ represents a temperature parameter.

3. The cross-domain image classification method based on class prototype-assisted pseudo-label generation according to claim 1, characterized in that: The multi-level pseudo-label assignment process introduces two feature alignment loss functions. The DAN function realizes feature alignment between the source domain and the target domain, and is used for direct alignment of source domain samples and target domain samples after feature extraction. The OT function realizes alignment of target domain feature distribution and classifier weights, and is used for target domain sample features and the last layer of classifier weights.

Images