Passive domain adaptive federal learning method based on self-supervised knowledge distillation

Abstract

The invention discloses a passive domain adaptive federal learning method based on self-supervised knowledge distillation, which improves the generalization ability of a model in a target domain by optimizing pseudo label generation, self-supervised learning and knowledge distillation strategies. The method comprises the following steps: a client firstly uses source domain data to train a local model, and a server aggregates to generate a global source domain model; then, on the basis of the global source domain model, the client side generates an initial pseudo label for target domain data, the quality of the pseudo label is optimized through self-supervised learning, a knowledge distillation strategy is introduced, the teacher model uses the pseudo label to guide the student model to learn, and the student model updates parameters and then sends the parameters to the server; and the server aggregates and generates globally updated target model parameters and broadcasts the globally updated target model parameters back to the client, and the client continuously trains until the model performance reaches the standard or converges. The method does not need to depend on source domain data, only uses the unmarked data of the target domain to generate the pseudo tag, combines the federated learning framework to aggregate the model parameters, significantly enhances the adaptability and accuracy of the model to the target domain, effectively protects the data privacy, and reduces the storage cost.

Description

Technical Field

[0001] The present invention relates to federated learning, passive domain adaptation and self-supervised learning technologies, and specifically to a passive domain adaptation federated learning method combined with self-supervised knowledge distillation. Background Art

[0002] With the rapid development of artificial intelligence, deep learning has made significant progress in fields such as image recognition, natural language processing, and speech recognition. However, when faced with new target domain data, deep learning models experience significant performance degradation due to domain shift. Domain adaptation technology has emerged to address this issue, aiming to transfer knowledge learned from the source domain to the target domain. Traditional domain adaptation methods require access to data from both the source and target domains, minimizing the distribution difference between the two domains to achieve knowledge transfer. However, these methods suffer from two major issues: First, directly accessing source data containing sensitive information may violate data privacy policies; second, source datasets are often very large, and storing and loading this data is costly, limiting their application on resource-constrained devices.

[0003] Passive domain adaptation technology aims to avoid using source data during the target domain adaptation process, relying only on a pre-trained source model and unlabeled data from the target domain. Existing passive domain adaptation methods mainly focus on aligning the feature distributions of the source and target domains, ignoring the model's ability to focus on object regions in the target domain, resulting in insufficient generalization of the model in the target domain. Secondly, existing self-supervised methods are not mature enough when dealing with complex, multimodal target domain data, making it difficult to effectively utilize unlabeled data from the target domain to improve the model's generalization ability. At the same time, although existing knowledge distillation methods guide the learning of student models by leveraging the knowledge of the teacher model, in actual applications, the data in the target domain is often distributed across different devices. Existing methods cannot effectively process distributed data, cannot protect data privacy, and cannot reduce storage and computing costs. Summary of the Invention

[0004] To address the issues of distributed data storage and processing and data privacy, this paper discloses a passive domain-adaptive federated learning method based on self-supervised knowledge distillation. This method significantly improves the generalization ability and adaptability of the model in the target domain by optimizing the pseudo-label generation process, introducing a self-supervised learning mechanism, and a knowledge distillation strategy. At the same time, it uses a federated learning method to protect data privacy and reduce storage costs. This invention adopts the following technical solution, which includes the following steps:

[0005] Step 1: Each client uses its own source domain data to train a local source domain model. These local models are trained independently on their respective clients to ensure data privacy, while using the cross-entropy loss function to optimize the model classification ability;

[0006] Step 2: Send the parameters of the local source domain models trained by all clients to the server. The server aggregates these parameters to generate a global source domain model and repeats the federated learning process of steps 1 and 2, saving the optimal global model at each iteration.

[0007] Step 3: Use the optimal global source domain model as the initial target model. This model does not use the source data in the target domain adaptation process and only relies on the unlabeled data of the target domain.

[0008] Step 4: On each client, use the initial target model to generate initial pseudo-labels for the local target domain data. Through a self-supervised learning mechanism, convert the model output into a probability distribution, calculate entropy and weights, find the predicted labels and calculate the initial accuracy, normalize the features, use cosine similarity as the distance metric, update the cluster centers and predicted labels through two iterations, calculate the distance between the features and the cluster centers, and redistribute the labels to obtain the predicted labels, thus generating the teacher model. Entropy loss is also introduced during the optimization process to encourage the model to generate high-confidence predictions, reduce uncertainty, and improve the stability of the pseudo-labels.

[0009] Step 5: On each client, a knowledge distillation strategy is introduced to guide the learning of the student model through the pseudo-labels generated by the teacher model, enabling the student model to better focus on the key features in the target domain and improve its generalization ability;

[0010] Step 6: Each client calculates the total loss based on the optimized pseudo-labels and updates the parameters of the student model, sending the updated parameters to the server. The server aggregates the parameter updates from each client, generates the globally updated target model parameters, and broadcasts them back to each client, completing one round of federated learning iteration.

[0011] Step 7: Repeat steps 4 to 6, and continuously monitor the performance of the model on the target domain dataset during each iteration until 10 rounds of federated learning are completed.

[0012] Furthermore, step 1 specifically includes:

[0013] Assume that the number of clients participating in the training in the federated learning process is N, then N={n1,n2,n3,…,n N}, the source domain data of the i-th client is in, is the input data, is the corresponding label. Each client trains a local source domain model The cross entropy loss function formula (1) is used to express:

[0014]

[0015] By optimizing Each client updates its local model parameters The pre-trained model used for local model training consists of the last layer of ResNet50 plus a Transformer. After layers of convolution and downsampling, the final layer of the ResNet50 network outputs feature maps with semantic information. These feature maps contain information such as local texture and shape of the image, but their ability to capture long-range dependencies is limited. Therefore, the output of the last layer of ResNet50 serves as the input to the Transformer. The Transformer module effectively captures long-range dependencies between features through its self-attention mechanism, modeling global information. In the Transformer, input features first pass through a multi-head self-attention layer to calculate attention weights between features at different positions, thereby enhancing the representation of important features. The features are then further transformed and nonlinearly mapped through a feedforward neural network layer.

[0016] Furthermore, step 2 specifically includes:

[0017] After the local model is trained, the server receives the model parameters from all clients. Where M is the number of clients. The server calculates the global model parameters θ s , its calculation formula is expressed by formula (2):

[0018]

[0019] Assume that after T rounds of iterations, the optimal global model parameters saved are

[0020] Furthermore, step 3 specifically includes:

[0021] The optimal global source domain model parameters As the initial target model parameters The model does not use source data during the target domain adaptation process and only relies on unlabeled data of the target domain.

[0022] Furthermore, step 4 specifically includes:

[0023] Assume that the target domain data of the i-th client is expressed as formula (3):

[0024]

[0025] in, is the input data of the target domain, is the number of target domain data samples of the i-th client. The pseudo labels are generated using the initial target model, and the pseudo label calculation formula is expressed as formula (4):

[0026]

[0027] in, is the generated pseudo label, is the initial target model Input The predicted output.

[0028] Furthermore, step 5 specifically includes:

[0029] On each client, a knowledge distillation strategy is introduced. The pseudo labels generated by the teacher model are used to guide the learning of the student model. The knowledge distillation loss function is expressed using formula (5):

[0030]

[0031] Among them, L kd is the knowledge distillation loss, It is a student model Input The predicted output is obtained by optimizing L kd Update the student model parameters. In addition, the information entropy loss is introduced to measure the uncertainty of the target domain samples, encouraging the model to output more confident predictions. The information entropy loss function is expressed using formula (6):

[0032]

[0033] in, is the number of data samples of the i-th client in the target domain, is the jth unlabeled sample in the target domain, The student model (i.e. the target model currently being trained) is The predicted output is a probability distribution that represents the probability that the sample belongs to each category. To calculate the information entropy of the student model, we take the logarithm of the predicted probability. The information entropy is calculated for the predicted probability of each class, which measures the uncertainty of the predicted distribution.

[0034] Furthermore, step 6 specifically includes:

[0035] Each client calculates the total loss based on local data and optimized pseudo labels. The total loss function formula is expressed using formula (7):

[0036] L total =L kd +λ1L ent +λ2Lce (7)

[0037] Among them, L kd is the knowledge distillation loss, ensuring that the student model learns the knowledge of the teacher model and updates the parameters of the student model based on this loss function, L ent is the entropy loss, which improves the confidence of the model in the target domain and avoids excessive uncertainty in prediction. λ1 is a hyperparameter that controls the weight of information entropy loss, balancing target domain adaptability and knowledge distillation. λ2 controls the weight of cross entropy loss. Based on this, the student model parameters are updated and the updated parameters are sent to the server. The formula for updating the student model parameters is expressed using formula (8):

[0038]

[0039] in, represents the student model parameters of the i-th client, η is the learning rate, which controls the step size of parameter update, and L is the total loss function used to optimize the target domain model. It is the gradient of the loss function with respect to the student model parameters, guiding the adjustment of the parameters to reduce the loss.

[0040] The server receives the updated student model parameters from all clients and calculates the globally updated target model parameters, which are expressed using formula (9):

[0041]

[0042] in, is the target model parameter, N represents the number of clients, Represents the student model parameters of the i-th client, and the server broadcasts the globally updated target model parameters back to each client.

[0043] Furthermore, step 7 specifically includes:

[0044] On each client, the process of pseudo-label generation, self-supervised knowledge distillation, and model parameter update is repeated. Each client updates the parameters of the student model based on local data and optimized pseudo-labels and sends the updated parameters to the server. The server aggregates the parameter updates from each client, generates the globally updated target model parameters, and broadcasts them back to each client, completing one round of federated learning iteration. This process is repeated to improve model performance.

[0045] Through the above steps, the present invention discloses a passive domain adaptation federated learning method based on self-supervised knowledge distillation. This method enhances the model's generalization ability to target domain data by introducing a self-supervised learning mechanism without relying on source domain data. On each client, pseudo labels are first generated using unlabeled data from the target domain. Pseudo labels are generated by having a teacher model predict the target domain data and using the predictions as training labels for a student model. This method, similar to the pseudo-labeling technique used in self-supervised learning, can effectively utilize unlabeled data to guide model training. After pseudo-label generation is completed on each client, the model is further trained using self-supervised knowledge distillation. Knowledge distillation is a model compression technique that transfers knowledge from the teacher model to the student model, enabling the student model to learn richer feature representations. This not only improves the student model's performance but also enables effective learning of target domain data without relying on source domain data. A federated learning framework is used to aggregate model parameter updates from each client, protecting data privacy and further improving model performance on the target domain by aggregating model parameter updates from each client. The present invention generates pseudo labels from unlabeled data in the target domain and uses them to guide the training of the student model, thereby achieving effective domain adaptation without the need for source domain data, significantly improving the adaptability and accuracy of the model in the target domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 Schematic diagram of the process of the passive domain adaptation federated learning method based on self-supervised knowledge distillation of the present invention;

[0048] Figure 2 This is a module structure diagram of the passive domain adaptation federated learning method based on self-supervised knowledge distillation in the present invention. DETAILED DESCRIPTION

[0049] To make the technical solutions and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention:

[0050] The present invention provides a passive domain adaptation federated learning method based on self-supervised knowledge distillation. A new method is proposed. By independently generating pseudo labels and performing self-supervised knowledge distillation on the target domain, the model parameter updates of each client are aggregated using the federated learning framework. This method significantly improves the adaptability and accuracy of the model on the target domain while protecting data privacy.

[0051] like Figure 1 The flowchart of the method of the present invention is shown. Experiments were conducted using the public office-home dataset. In the embodiment, three of the four domains of Art, Clipart, Product, and RealWorld were used as source domains, and the other domain was used as the target domain for experiments. The entire dataset contains 15,500 images, and each of the four domains contains 65 categories of images. 10% of the data from each client is extracted as server data. The ratio of the training set to the test set of the source domain is 9:1. The images in the target domain are unlabeled. The classification performance of the model is improved through each round of adaptation. The training includes the following steps:

[0052] Step 1: Select three domains from the four domains of the office-home dataset as source domains, and another domain as the target domain. The three domains in the source domain are used as clients. 10% of the data from each domain is used as server data. The remaining 90% of the data is divided into training and test sets in a ratio of 9:1. Each client trains its own local model locally and tests it on its own test set.

[0053] Step 2: The server aggregates the local source domain models trained by all clients to generate a global source domain model. This global model is tested on the server's dataset to obtain the corresponding accuracy to judge the model's performance, and finally the best performing global model is saved.

[0054] Step 3: Distribute the best global source domain model as the initial target model for target domain training to all clients, and the clients use this model to start adaptive training.

[0055] Step 4: Each client inputs the input data into the target model to extract features and obtain classification output, converts the model output into a probability distribution, calculates entropy and weights, finds the predicted label and calculates the initial accuracy, normalizes the features, uses cosine similarity as the distance metric, and determines the initial cluster center by transposing the softmax output and dot product with the feature, then dividing by the output row sum (to prevent division by zero). Subsequent iterations update the distance between the calculated features and the cluster center and redistribute the labels to obtain the predicted labels to generate the teacher model.

[0056] Step 5: The global model distributed by the server to the client is used as the student model. After loading the student model weights, its parameters are frozen. The teacher model transfers the learned soft label knowledge to the student model. An exponential moving average strategy is used to dynamically update the teacher model parameters based on the student model parameters. The performance of the two is regularly evaluated to obtain the difference delta, which is used to adjust the update strategy and provide feedback on the training status to ensure that the teacher model continues to effectively guide the student model learning.

[0057] Step 6: Combining the local data and the optimized pseudo-labels, the total loss used to guide model optimization is calculated. This total loss comprehensively considers knowledge distillation loss, entropy loss, and cross-entropy loss, with each loss weighted to 1 to balance their impact on model training. The client then uses the optimizer to update the student model parameters based on the calculated total loss. After the update is complete, the client sends the updated student model parameters to the server, which aggregates them to obtain the globally updated target model parameters and broadcasts these parameters back to each client.

[0058] Step 7: Repeat steps 3-6 to continuously monitor the model's performance on the target domain dataset and complete ten rounds of federated training.

[0059] In this experiment, the federated learning training process was set up as follows: a total of 10 federated rounds were conducted, with each client performing 20 epochs of training locally. During source domain training, the batch size was set to 32. This parameter helps balance computational efficiency and model convergence during training. The learning rate was set to 0.005, and the weight decay coefficient was set to 0.0001. These values ensure fast training while effectively avoiding exploding or vanishing gradients. During target domain training, the batch size was set to 32, the learning rate was set to 0.0005, and the stochastic gradient descent algorithm was used for optimization. To further enhance optimization, the momentum parameter was set to 0.9 to accelerate convergence and reduce oscillations during parameter updates. The momentum decay was also set to 0.001. As training progressed, the momentum was gradually adjusted to stabilize the model in the later stages of training, thereby effectively optimizing the model and improving its performance.

[0060] The present invention uses accuracy as the model performance evaluation indicator, and the accuracy calculation formula is expressed using formula (10):

[0061]

[0062] Among them, TP represents true positive examples, that is, the number of samples that are actually positive and correctly predicted as positive by the model; TN represents true negative examples, that is, the number of samples that are actually negative and correctly predicted as negative by the model; FP represents false positive examples, that is, the number of samples that are actually negative but incorrectly predicted as positive by the model; FN represents false negative examples, that is, the number of samples that are actually positive but incorrectly predicted as negative by the model.

[0063] According to the above steps, a passive domain adaptation federated learning architecture for self-supervised knowledge distillation is built, such as Figure 2 As shown in Table 1, the data clearly shows that the evaluation indicators of the proposed method are significantly better than those of the centralized training method for passive domain adaptation tasks. The clients participating in federated learning correspond to the three domains in the office-home dataset, and the target domain is another domain in the office-home dataset.

[0064] Table 1 Performance comparison with the compared methods on the office-home dataset

[0065]

[0066] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes based on the technical solutions and inventive concepts of the present invention within the technical scope disclosed in the present invention, and these changes should be covered by the scope of protection of the present invention.

Claims

1. A passive domain adaptation federated learning method based on self-supervised knowledge distillation, characterized by include: S1: Each client independently trains a local source domain model using its own source domain data to ensure data privacy; S2: Each client uploads the local source domain model parameters to the server, which aggregates and generates a global source domain model. Steps S1 and S2 are repeated, and the optimal global model is retained in each iteration. S3: The optimal global source domain model is used as the initial target model, which only relies on the unlabeled data in the target domain when adapting to the target domain; S4: Each client generates initial pseudo labels for local target domain data using the initial target model and uses a self-supervised learning mechanism to optimize the quality of the pseudo labels. S5: Each client introduces a knowledge distillation strategy, using pseudo labels generated by the teacher model to guide the student model's learning and improve the student model's generalization ability; S6: Each client updates the student model parameters using the optimized pseudo-labels and uploads them to the server. The server aggregates the parameter updates of each client, generates the globally updated target model parameters, and then broadcasts them to each client, completing one round of federated learning iteration. S7: Repeat steps S4 to S6 until the model performance in the target domain reaches the expected level or converges.

2. The passive domain adaptation federated learning method based on self-supervised knowledge distillation according to claim 1, characterized in that: Each client generates initial pseudo labels for local target domain data with the help of the initial target model: when generating pseudo labels, the target domain data of the i-th client is represented as Use the initial target model to generate the initial pseudo label according to formula (1): in, is the generated pseudo label, is the initial target model Input The predicted output.

3. The passive domain adaptation federated learning method based on self-supervised knowledge distillation according to claim 1, characterized in that: Each client uses a self-supervised learning mechanism to optimize the quality of pseudo labels: S31: When introducing a self-supervised learning mechanism to optimize the quality of pseudo labels, the initial pseudo labels are corrected and screened in terms of label accuracy and consistency with the target domain data distribution. S32: By calculating the difference between the feature distribution of the target domain data and the feature distribution corresponding to the pseudo label, the quality of the pseudo label is optimized using the calculation formula (2): Among them, L ent is the information entropy loss term, which is used to measure the uncertainty of model prediction to enhance the confidence of pseudo labels. is the number of target domain data samples of the i-th client, For the model to sample The predicted category probability distribution is usually output by Softmax. is the logarithm of the probability of the category.

4. The passive domain adaptation federated learning method based on self-supervised knowledge distillation according to claim 1, characterized in that: Each client introduces a knowledge distillation strategy, using pseudo labels generated by the teacher model to guide the student model's learning. During the knowledge distillation process, the teacher model is used to guide the student model's learning: S41: Each client uses the teacher model to generate more stable pseudo labels to guide the student model to learn. The calculation formula is as follows: Among them, L kd is the knowledge distillation loss, KL represents the Kullback-Leibler divergence, and are the output probability distributions of the teacher model and the student model respectively; S42: The teacher model adopts the exponential moving average (EMA) strategy to improve the stability of pseudo labels.

5. The passive domain adaptation federated learning method based on self-supervised knowledge distillation according to claim 1, characterized in that: Each client uses the optimized pseudo-labels to update the student model parameters and uploads them to the server. The server aggregates the parameter updates of each client to generate the globally updated target model parameters. During the federated aggregation training process: S61: Each client calculates the total loss based on the target domain data to guide model optimization and improve model performance. The total loss is expressed by formula (4): L total =L kd +λ1L ent +λ2L ce (4) Among them, L ce is the cross entropy loss, used to supervise the learning of pseudo labels, L ent is the information entropy loss, which is used to measure the uncertainty of the prediction and optimize the quality of the pseudo label, L kd is the knowledge distillation loss, which is used to guide the student model to learn the characteristics of the teacher model. λ1 and λ2 are the weights of information entropy loss and cross entropy loss respectively; S62: Use the optimizer to update the student model parameters and send the updated parameters to the server. The updated student model parameters are expressed using formula (5): in, represents the student model parameters of the i-th client, η is the learning rate, which controls the step size of parameter update, and L is the total loss function used to optimize the target domain model. It is the gradient of the loss function with respect to the student model parameters, guiding it to adjust the parameters to reduce the loss.

Images