Black box domain adaptation method based on double-teacher distillation and sub-network correction

Abstract

The invention discloses a black box domain adaptation method based on double-teacher distillation and sub-network correction, and the method comprises the steps: carrying out the preprocessing of obtained target domain image data, and inputting the data into a black box source model and a CLIP model for prediction; fusing the prediction results of the two to generate an initial pseudo label; in the first stage, an initial pseudo label is used as a supervision signal to train a target model; constructing a sub-network which shares a parameter subset with the target model, and correcting the target model based on the output of the sub-network; in the second stage, based on the target model trained in the first stage, image features of target domain image data are extracted; generating a correction pseudo label according to the similarity between the image features and the corresponding category prototype, and training a target model by taking the correction pseudo label as a supervision signal; and iterating the first stage and the second stage until the target model converges, and applying a downstream image classification task to the trained target model. According to the method, the semantic information is fully fused to effectively suppress the noise interference of the pseudo tag.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of computer vision and transfer learning, and more specifically to a black-box domain adaptation method based on dual-teacher distillation and sub-network correction. Background Technology

[0002] The emergence of domain adaptation tasks is primarily aimed at addressing the problem of "models performing well in the source domain (training data domain), but experiencing a significant performance drop in the target domain (real-world application domain) due to differences in data distribution." Its background can be summarized from three aspects: data, technology, and application. At the data level, significant distribution differences often exist between source and target domain data in reality. These differences may stem from variations in collection conditions such as lighting, angle, and weather, or from differences in the source of labeled data in the laboratory versus unlabeled data in real-world scenarios. Furthermore, the labeling costs in fields such as medical imaging and industrial inspection are extremely high, making it difficult to label sufficient training data for all target domain scenarios. This is the fundamental motivation behind the emergence of domain adaptation tasks. At the technology level, traditional computer vision models have limited generalization capabilities and are prone to overfitting the details or even noise of the source domain data during training, failing to effectively extract cross-domain commonalities. Using features and having fixed parameters after training makes it difficult to quickly adjust to new target domains, and retraining requires a lot of computing power and time, making it difficult to cope with dynamically changing scenarios. At the application level, as computer vision is applied to fields such as autonomous driving, facial recognition, and remote sensing image analysis, the demand for cross-scenario deployment has surged. For example, autonomous driving models need to adapt to different urban road environments from test sites, and facial recognition models need to adapt to facial features of different races and age groups. At the same time, the requirements of edge devices for lightweight model adaptation further drive the development of domain adaptation technology to bridge the performance gap between the source domain and the target domain and meet the application needs of real-world scenarios.

[0003] To address the distribution shift problem encountered when transferring knowledge from a labeled source domain to an unlabeled target domain, unsupervised domain adaptation has been proposed. However, traditional unsupervised domain adaptation methods assume simultaneous access to both source and target domain data, an assumption that is often difficult to uphold in practice. To mitigate this limitation, passive domain adaptation relies solely on a source model pre-trained on labeled source domain data and adapts it to the unlabeled target domain. However, even when the source model is available, it is still possible to reconstruct the original source samples using generative techniques, posing a continued risk of privacy breaches. Black-box domain adaptation further overcomes these limitations, assuming that neither source domain data nor the source model can be accessed; only the prediction results output by the black-box source model can be queried. This scenario is more relevant to real-world situations, such as in cross-institutional collaborations and commercial model adaptation, where data sharing or model exposure is often prohibited. Existing mainstream black-box domain adaptation methods are mainly based on entropy-based self-training methods, feature separation, and regularization strategies. However, existing black-box domain adaptation technologies suffer from three core problems that constrain their adaptation performance: 1. Severe noise from false labels: Due to the difference in distribution between the source and target domains, the prediction results of the black-box source model for the target sample are noisy; relying solely on black-box prediction to generate false labels will cause the target model to overfit to the wrong supervision signal.

[0004] 2. Insufficient utilization of semantic priors: Although the visual-language model CLIP possesses strong cross-domain generalization capabilities through large-scale image-text contrastive learning and can provide high-level semantic information to compensate for the semantic deficiencies in black-box domain adaptation that rely solely on black-box prediction, some methods, when introducing visual-language models such as CLIP, merely switch between black-box prediction and CLIP prediction using fixed rules (such as confidence selection), failing to fully integrate the complementarity of the two and neglecting to optimize the semantic adaptability of CLIP for the target domain.

[0005] 3. Model overfitting and poor generalization: The lack of an effective regularization mechanism makes the target model prone to overfitting to noisy pseudo-labels; at the same time, the model does not make full use of the structural information of the target data itself, resulting in limited generalization ability of the model in the target domain.

[0006] Therefore, how to effectively suppress noise interference from pseudo-labels by fusing general visual semantic knowledge under the technical constraints of not being able to access source model parameters and source data, so as to achieve stable and accurate cross-domain adaptation in specific image classification systems, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] In view of the above problems, the present invention is proposed to provide a black-box domain adaptation method based on dual-teacher distillation and subnetwork correction to overcome or at least partially solve the above problems.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: This invention provides a black-box domain adaptation method based on dual-teacher distillation and sub-network correction, comprising: After preprocessing the acquired target domain image data, it is input into the black box source model and CLIP model respectively to obtain the prediction results of the source model and the prediction results of the CLIP model. The prediction results of the source model and the prediction results of the CLIP model are fused to generate initial pseudo-labels; The first stage of training involves using the initial pseudo-labels as supervision signals to train the target model through knowledge distillation; simultaneously, a sub-network sharing a subset of parameters with the target model is constructed, and the target model is regularized based on the output of the sub-network. Second-stage training: Based on the target model trained in the first stage, image features of the target domain image data are extracted; correction pseudo-labels are generated according to the similarity between the image features and the corresponding category prototypes, and the correction pseudo-labels are used as supervision signals to self-train the target model; Iterate through the first and second stages of training until the target model converges, and then apply the trained target model to the downstream image classification task.

[0009] Furthermore, the preprocessing includes data augmentation and normalization.

[0010] Furthermore, the black-box source model includes a feature extractor and a classifier; The feature extractor is used to extract numerical feature vectors from an image in the form of a digital matrix. The classifier is used to learn the mapping relationship between the numerical feature vector and the preset semantic category label.

[0011] Further, the step of fusing the prediction results of the source model and the prediction results of the CLIP model to generate initial pseudo-labels specifically includes: Calculate the prediction entropy of the black-box source model and the CLIP model for the target domain image data, respectively. Based on the predicted entropy and the size of the target domain image dataset, the fusion weights of the source model prediction results and the CLIP model prediction results are dynamically allocated; The prediction results of the source model and the CLIP model are fused according to the fusion weights, and the fusion result is used as the initial pseudo-label.

[0012] Furthermore, in the first phase of training, the loss function for knowledge distillation includes KL divergence loss, Mixup consistency loss, and information maximization loss.

[0013] Furthermore, the regularization correction of the target model based on the output of the sub-network specifically includes: The JS divergence between the subnetwork and the target model output is calculated as the divergence loss; Calculate the cosine similarity between the gradients of the parameters corresponding to the subnetwork and the target model, and construct a weighted gradient difference loss by minimizing this similarity; The target model is regularized based on the divergence loss and the weighted gradient difference loss.

[0014] Furthermore, it also includes: after each training phase in the first stage, using the prediction results of the target model, iteratively optimizing the initial pseudo-labels through exponential moving average, and based on the optimized initial pseudo-labels, fine-tuning the prompt words of the CLIP model through consistency maximization loss.

[0015] Furthermore, the category prototype is obtained through the following steps: The class prototype of a class is obtained by averaging all image features of the same class predicted by the target model after the first stage of training.

[0016] Furthermore, the step of generating corrected pseudo-labels based on the similarity between the image features and the corresponding category prototype specifically includes: The nearest category prototype is determined based on the cosine distance between the image features and the corresponding category prototype, and the category corresponding to the nearest category prototype is used as the correction pseudo-label.

[0017] Furthermore, the downstream image classification tasks include cross-institutional classification and recognition of medical images, cross-production line defect classification in industrial visual quality inspection, and classification adaptation of private datasets using commercial API services.

[0018] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a black-box domain adaptation method based on dual-teacher distillation and sub-network correction, which has the following beneficial effects: This invention addresses the problems of noise in pseudo-labels and insufficient utilization of semantic information. By introducing the CLIP model as an auxiliary teacher, its powerful semantic understanding capabilities are leveraged to improve the reliability of pseudo-labels. Simultaneously, a sub-network regularization strategy is employed to reduce overfitting of the target model to noisy pseudo-labels. This largely overcomes the noise problem of pseudo-labels.

[0019] Based on the number of target domain samples and prediction uncertainty, this invention dynamically integrates the task-specific knowledge of the black-box source model and the general semantic knowledge of CLIP to generate more reliable pseudo-labels, making full use of semantic information. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the black-box domain adaptation method based on dual-teacher distillation and sub-network correction provided in an embodiment of the present invention.

[0022] Figure 2 This is a schematic diagram of the first-stage training process provided in an embodiment of the present invention.

[0023] Figure 3This is a schematic diagram of the second-stage training process provided in an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] This invention discloses a black-box domain adaptation method based on dual-teacher distillation and sub-network correction, such as... Figure 1 As shown, it includes the following steps: S1. After preprocessing the acquired target domain image data, input it into the black box source model and CLIP model respectively to obtain the prediction results of the source model and the prediction results of the CLIP model. S2. Fuse the prediction results of the source model and the prediction results of the CLIP model to generate initial pseudo-labels; S3. First stage training: Using the initial pseudo-labels as supervision signals, the target model is trained through knowledge distillation; at the same time, a sub-network that shares a subset of parameters with the target model is constructed, and the target model is regularized and corrected based on the output of the sub-network. S4. Second-stage training: Based on the target model trained in the first stage, extract image features from the target domain image data; generate correction pseudo-labels based on the similarity between the image features and the corresponding category prototypes, and use the correction pseudo-labels as supervision signals to self-train the target model; S5. Iterate through the first and second stages of training until the target model converges, and then apply the trained target model to the downstream image classification task.

[0026] The labels S1 to S5 are used for ease of explanation and do not specify the execution order of the steps. The following sections will explain each step in detail.

[0027] In step S1 above, the acquired target domain image data is preprocessed and then input into the black box source model and CLIP model respectively to obtain the prediction results of the source model and the prediction results of the CLIP model. In this embodiment of the invention, the target domain image datasets used are Office-31, Office-Home, and VisDA datasets. Office-31 contains 3 domains, 6 transfer tasks, and 31 categories; Office-Home contains 4 domains, 12 transfer tasks, and 65 categories; VisDA contains 2 domains, 1 transfer task, and 12 categories. The image data is then preprocessed, including data augmentation and normalization operations. The processed images are converted into tensor format for subsequent use.

[0028] The aforementioned black-box source model is a source domain pre-trained model, comprising a feature extractor and a classifier. The feature extractor transforms the image, which exists in the form of a numerical matrix, into a set of more refined and meaningful numerical feature vectors through a series of algorithms for subsequent classification. The classifier "flattens" the extracted high-level features and learns the mapping relationship between the numerical feature vectors and the predefined semantic category labels.

[0029] In step S2 above, the prediction results of the source model and the prediction results of the CLIP model are fused to generate initial pseudo-labels. Specifically, the prediction entropy of the black-box source model and the CLIP model for the target domain image data is calculated respectively. According to the prediction entropy and the size of the target domain image dataset, the fusion weights of the prediction results of the source model and the CLIP model are dynamically allocated. The prediction results of the source model and the CLIP model are fused according to the fusion weights, and the fusion result is used as the initial pseudo-label.

[0030] In step S3 above, such as Figure 2 As shown, the first stage of training involves using initial pseudo-labels as supervision signals and training the target model through knowledge distillation to ensure global diversity in target domain predictions and suppress overfitting of the model to noisy pseudo-labels. The loss function of knowledge distillation includes KL divergence loss, Mixup consistency loss, and information maximization loss, expressed as:

[0031] in, The loss function represents knowledge distillation; E represents the mathematical expectation, which can be understood as taking the average. x t Represents the target domain image sample, belonging to X t ; X t Represents the target domain image dataset; Represents the KL divergence loss function; This represents pseudo-label predictions in a hybrid softmax format; This represents the target model's prediction of the target domain image data; Simultaneously, a sub-network sharing a subset of parameters with the target model is constructed. For example, the sub-network is initialized using 84% of all parameters of the target model, and the target model is regularized based on the output of the sub-network to prevent overfitting. Specifically, this includes: calculating the JS divergence between the sub-network and the target model outputs as the bifurcation loss; calculating the cosine similarity of the gradients of the corresponding parameters of the sub-network and the target model, and constructing a weighted gradient difference loss by minimizing this similarity; and performing regularization on the target model based on the bifurcation loss and the weighted gradient difference loss. The bifurcation loss function... Represented as:

[0032] in, This represents the Jensen–Shannon (JS) divergence loss function; This represents the subnetwork's prediction of the target domain image data; This represents the target model's prediction of the target domain image data; Represents the parameters of the subnetwork; Represents the parameters of the target model; After each training phase, the initial pseudo-labels are iteratively optimized using the prediction results of the target model through exponential moving average. Based on the optimized initial pseudo-labels, the prompt words of the CLIP model are fine-tuned through consistency maximization loss.

[0033] In step S4 above, as Figure 3 As shown, the second stage of training involves: extracting image features from the target domain image data based on the target model trained in the first stage; obtaining the predicted labels corresponding to the image features output by the target model; grouping and statistically analyzing image features by category, and taking the average value of image features belonging to the same category as the category prototype; determining the nearest category prototype based on the cosine distance between the image features and the corresponding category prototype, and using the category corresponding to the nearest category prototype as the correction pseudo-label, and using the correction pseudo-label as the supervision signal, and performing self-training of the target model through the cross-entropy loss function; expressed as:

[0034] in, This represents the second-stage training result of the target model; Represents the cross-entropy loss function; This indicates the corrected pseudo-label; This represents the predicted label output by the target model for the target domain image data.

[0035] In step S5 above, the first and second phases of training are iterated until the target model converges. After the target model is trained, experiments are conducted on various transfer tasks across all datasets, recording the classification accuracy of the target domain samples. Ablation experiments are also performed to verify the effectiveness of different modules.

[0036] The trained target model is applied to downstream image classification tasks, including: 1) cross-institutional classification and recognition of medical images, for example, Hospital A's model is used as a black box, and Hospital B has no permission to obtain source data and model parameters, and only adapts to local data by querying prediction results; 2) cross-production line defect classification in industrial visual quality inspection, for example, calling a third-party AI classification API (which only returns prediction results) to adapt to the enterprise's internal private dataset, without relying on the API provider's source data or model parameters; 3) classification adaptation of private datasets using commercial API services, for example, the target model structure can be designed independently, without compatibility with the black box source model, and can be lightweightly deployed to edge devices (such as mobile phones and cameras) to complete classification adaptation for specific scenarios (such as cross-production line adaptation of factory quality inspection images).

[0037] In summary, the black-box domain adaptation method based on dual-teacher distillation and sub-network correction provided by this invention does not depend on the source data and source model structure, and the target model does not need to be the same as the source model. It can effectively adapt to tasks in different domains on resource-constrained devices. For example, it can be flexibly applied in cross-institutional collaboration or business model adaptation scenarios, without being limited by data and models, and has strong adaptability. In addition, this invention achieves stable model training through strategies such as adaptive prediction fusion, sub-network correction, and category prototype self-training. For example, in experiments on the Office-31 and Office-Home datasets, the model accuracy improves rapidly in the early stage of training, then converges stably, and is insensitive to hyperparameters, exhibiting strong robustness. Furthermore, through experiments on multiple benchmark datasets (Office-31, Office-Home, VisDA-17), this method outperforms other black-box domain adaptation methods on all tasks, and has a leading average accuracy.

[0038] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0039] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A black-box domain adaptation method based on double-teacher distillation and subnetwork correction, characterized in that, The method comprises the following steps: After preprocessing the obtained target domain image data, input the preprocessed target domain image data into a black box source model and a CLIP model respectively to obtain a source model prediction result and a CLIP model prediction result; Fusing the source model prediction result and the CLIP model prediction result to generate an initial pseudo label; First stage training: taking the initial pseudo label as a supervision signal, training the target model through knowledge distillation; Meanwhile, a subnetwork sharing a subset of parameters with the target model is constructed, and the target model is regularized and corrected based on the output of the subnetwork; Second stage training: based on the target model after the first stage training, extracting image features of the target domain image data; generating a corrected pseudo label according to the similarity between the image features and the corresponding category prototype, and taking the corrected pseudo label as a supervision signal to train the target model; Iterating the first stage training and the second stage training until the target model converges, and applying the trained target model to a downstream image classification task.

2. The black-box domain adaptation method based on double-teacher distillation and subnetwork correction of claim 1, wherein, The preprocessing comprises data enhancement processing and normalization processing.

3. The black-box domain adaptation method based on double-teacher distillation and subnetwork correction of claim 1, wherein, The black box source model comprises a feature extractor and a classifier. The feature extractor is configured to extract a numerical feature vector from an image in the form of a digital matrix. The classifier is configured to learn a mapping relationship between the numerical feature vector and a preset semantic category label.

4. The black-box domain adaptation method based on double-teacher distillation and subnetwork correction of claim 1, wherein, The fusing the source model prediction result and the CLIP model prediction result to generate an initial pseudo label comprises the following steps: Specifically comprising: Respectively calculating the prediction entropy of the target domain image data of the black box source model and the CLIP model; According to the prediction entropy and the size of the target domain image data set, the fusion weights of the source model prediction result and the CLIP model prediction result are dynamically allocated; According to the fusion weights, the source model prediction result and the CLIP model prediction result are fused, and the fusion result is taken as the initial pseudo label.

5. The black-box domain adaptation method based on double-teacher distillation and subnetwork correction of claim 1, wherein, In the first stage training, the loss function of knowledge distillation comprises KL divergence loss, Mixup consistency loss and information maximization loss.

6. The black-box domain adaptation method based on double-teacher distillation and subnetwork correction of claim 1, wherein, The regularizing and correcting the target model based on the output of the subnetwork comprises the following steps: Calculating the JS divergence of the output of the subnetwork and the target model as a divergence loss; Calculating the cosine similarity of the corresponding parameter gradients of the subnetwork and the target model, and constructing a weighted gradient difference loss by minimizing the similarity; According to the divergence loss and the weighted gradient difference loss, the target model is regularized and corrected.

7. The black-box domain adaptation method based on double-teacher distillation and subnetwork correction of claim 1, wherein, Further comprising: After the first stage training is completed each time, the initial pseudo label is iteratively optimized by using the prediction result of the target model through exponential moving average, and the prompt word of the CLIP model is fine-tuned through consistency maximization loss based on the optimized initial pseudo label.

8. The black-box domain adaptation method based on double-teacher distillation and subnetwork correction of claim 1, wherein, The category prototype is obtained by the following steps: The category prototype of the category is obtained by averaging all image features of the same category predicted by the target model after the first stage training.

9. The black-box domain adaptation method based on double-teacher distillation and subnetwork correction of claim 1, wherein, The generating a corrected pseudo label according to the similarity between the image features and the corresponding category prototype comprises the following steps: A nearest class prototype is determined according to a cosine distance between the image feature and a corresponding class prototype, and a class corresponding to the nearest class prototype is taken as the correction pseudo label.

10. The black-box domain adaptation method based on double-teacher distillation and subnetwork correction of claim 1, wherein, The downstream image classification task includes cross-institution classification and identification of medical images, cross-production line defect classification in industrial visual quality inspection, and classification adaptation of a private data set by using a commercial API service.

Images