Non-IID federal learning method based on dynamic knowledge distillation and dynamic regularization

Abstract

The invention discloses a Non-IID federated learning method based on dynamic knowledge distillation and dynamic regularization. The method comprises the steps that after a local model of a client is trained, distribution Ps of local data tags is output; uploading local model parameters to a server; the server gathers a group of client model parameters obtained by random sampling, and outputs global model parameters for the next round of use; the client downloads global model parameters, loads the global model parameters to a local model and outputs global data label distribution Pt; performing dynamic knowledge distillation at the client, integrating Ps, Pt and a dynamic function d (r), and calculating a local loss function of the client; and updating the local model of the client through back propagation. According to the method, the convergence speed and accuracy of the model can be remarkably improved under the condition of non-independent and identically-distributed Non-IID client data.

Description

Technical Field

[0001] The present invention relates to the technical field of federated learning, and specifically to a Non-IID federated learning method based on dynamic knowledge distillation and dynamic regularization. Background Art

[0002] Federated learning (FL) is a well-known distributed learning paradigm first introduced by Google in 2016 with the algorithm FedAvg. The core of FL is to collaboratively train a shared model using local data from geographically distributed clients. A key advantage of FL is that it can be trained on multiple clients simultaneously without the local data leaving the device, thus effectively protecting data privacy. Under this framework, clients exchange intermediate variables and parameters through a centralized server, maintaining collaborative training while respecting data privacy. As a result, federated learning technology has shown broad application potential in many cutting-edge fields, covering key areas such as healthcare, autonomous driving, blockchain, and Industrial Internet of Things.

[0003] Despite the efficiency and privacy advantages of federated learning (FL), it still faces challenges in practical deployment. These challenges include the limitation of communication resources, the variation of client data structure, and the heterogeneity of system architecture. Particularly challenging is the presence of non-independent and isotopic (non-iid) features in client data, which can cause the local objective function to be inconsistent with the global optimization direction, thus affecting the convergence of the model. In addition, some researchers have also highlighted client drift as a major challenge of FL. In scenarios with heterogeneous datasets, due to the difference in local optimal points, the locally trained model may deviate from the global objective, which may lead to convergence problems or model divergence.

[0004] Various strategies have been proposed to mitigate client drift. For example, regularization terms are added to the optimization loss function during local training to stabilize parameter updates. In addition, adjustments to the objective function are aimed at making the local model and the global model more consistent. Knowledge distillation techniques have also been explored to address the client data drift problem. For example, Yao et al. applied knowledge distillation techniques to maintain consistency between local and global models. Here, the global model acts as a teacher to guide the training of the local model, ensuring consistency in the parameter space and optimization direction. Other studies use data-free knowledge distillation methods to alleviate the data heterogeneity problem, but this method incurs computational and communication overheads due to the use of generators.

[0005] Despite the progress, existing methods often ignore the dynamic nature of client forgetting during training. In the context of non-homogeneous data, during the cross-client local model training process, a large amount of knowledge previously learned from other clients’ training data will be forgotten, a phenomenon widely known as “client catastrophic forgetting”. Figure 3 As shown in Figure 2, the root cause can be traced back to the huge difference between the unique distribution of the local dataset and the overall distribution of the global dataset. Therefore, how to alleviate client drift while solving the catastrophic forgetting problem is one of the important issues currently faced in the research of federated learning. Summary of the invention

[0006] In view of the problems of client drift and catastrophic forgetting in the prior art, the present invention provides a Non-IID federated learning method based on dynamic knowledge distillation and dynamic regularization. Dynamic knowledge distillation technology is used to perform adaptive gradient correction, and dynamic regularization technology is combined to constrain and adjust local and global objectives, thereby solving different degrees of client forgetting problems during training and alleviating client drift, thereby achieving the effect of improving the convergence speed and accuracy of the model.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] A Non-IID federated learning method based on dynamic knowledge distillation and dynamic regularization includes the following steps:

[0009] S10: After the local model training of client i in the rth round, the distribution P of local data labels is output s ;

[0010] S20, the local model parameter w i r Upload to the server;

[0011] S30, the server summarizes a set of client model parameters w1 obtained by random sampling r ,w2 r ,…,w M r , M is the total number of clients, and outputs the global model parameter w0 r+1 , for the next round;

[0012] S40, client i downloads global model parameter w0 r+1 ;

[0013] S50, the global model parameter w0 r+1 Load it into the client's local model and output the distribution P of the global data label t ;

[0014] S60, perform dynamic knowledge distillation on client i and integrate P s , P t And the dynamic function d(r), calculate the local loss function of client i in round r Expressed as

[0015]

[0016] In the formula, represents the prediction loss of client i, is the model parameter, w r-1 represents the global model parameters of the previous round, b i represents a small batch dataset, represents the gradient of the model parameters of client i in the last round of training, α represents the penalty coefficient of the proximal term, and KL represents the KL divergence;

[0017] S70, back propagation updates the local model of client i, where the update rule of client i is expressed as

[0018]

[0019] In the formula, η represents the learning rate, represents the corrected gradient, Represents the gradient of the model parameters in the previous round of training.

[0020] Specifically, the dynamic function d(r) is used to adaptively adjust the gradient correction in the training round, expressed as

[0021]

[0022] Where R is the number of communication rounds.

[0023] Specifically, the corrected gradient By solving the corrected objective function L for client i i 1 Obtained, expressed as

[0024]

[0025] Where b i =(x i ,y i )∈D i Represents a mini-batch dataset.

[0026] Specifically, the corrected objective function of the client i is expressed as

[0027]

[0028] In the formula, represents the prediction loss of client i, and the dynamic function d(r) is a function of the number of communication rounds r.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention integrates dynamic knowledge distillation and dynamic regularization technology to solve the inconsistency problem between the client and the server. By dynamically updating the local objective function of each client, the realization of an accurate global model is accelerated. The gradient is improved by dynamic knowledge distillation, which effectively solves the client drift problem. The global and local models are dynamically adjusted by regularization to improve the convergence speed. Compared with the existing federated learning method, the present invention shows faster convergence speed and higher generalization accuracy in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the overall process framework of an embodiment of the present invention.

[0032] Figure 2 This is the changing trend of the dynamic function in the communication round in the embodiment of the present invention.

[0033] Figure 3 The figure shows the comparison of data loss between the present invention and the traditional method in case of catastrophic forgetfulness of the client. DETAILED DESCRIPTION

[0034] The present invention is further described below in conjunction with the accompanying drawings and examples. The embodiments of the present invention include but are not limited to the following examples.

[0035] Example

[0036] like Figures 1 to 3 As shown, the Non-IID federated learning method based on dynamic knowledge distillation and dynamic regularization includes the following steps:

[0037] S10: After the local model training of client i in the rth round, the distribution P of local data labels is output s ;

[0038] S20, the local model parameter w i r Upload to the server;

[0039] S30, the server summarizes a set of client model parameters w1 obtained by random sampling r ,w2 r ,…,w M r , M is the total number of clients, and outputs the global model parameter w0 r+1 , for the next round;

[0040] S40, client i downloads global model parameter w0 r+1 ;

[0041] S50, the global model parameter w0 r+1 Load it into the client's local model and output the distribution P of the global data label t ;

[0042] S60, perform dynamic knowledge distillation on client i and integrate P s , P t And the dynamic function d(r), calculate the local loss function of client i in round r Expressed as

[0043]

[0044] In the formula, represents the prediction loss of client i, is the model parameter, w r-1 represents the global model parameters of the previous round, b i represents a small batch dataset, represents the gradient of the model parameters of client i in the last round of training, α represents the penalty coefficient of the proximal term, and KL represents the KL divergence;

[0045] S70, back propagation updates the local model of client i, where the update rule of client i is expressed as

[0046]

[0047] In the formula, η represents the learning rate, represents the corrected gradient, Represents the gradient of the model parameters in the previous round of training.

[0048] Based on knowledge distillation, the present invention adds a dynamic distillation term to the objective function to reduce client drift caused by label differences in the data. When the data does not meet the independent identical distribution condition, that is, there is non-independent and identically distributed Non-IID data, the present invention regards the global model of the previous round as the teacher model and the current local model as the student model, uses KL divergence to constrain the label distribution of the two models, and adjusts the gradient in each iteration, thereby effectively reducing client drift.

[0049] Therefore, the objective function of gradient correction is

[0050]

[0051] Where P t represents the label predicted by the global model, P srepresents the label predicted by the local model, and the dynamic function d(r) is a function of the number of communication rounds r. By scaling the KL divergence gradient correction, not only the client drift problem is solved, but also the model generalization energy is enhanced, which promotes faster convergence. Solution The gradient of Expressed as

[0052]

[0053] Where b i =(x i ,y i )∈D i Represents a small batch dataset, which is from the data distribution D i In practice, D i The distribution and size of may vary from client to client, which indicates that the client data has the non-independent and identically distributed (non-iid) property.

[0054] Then, by implementing dynamic constraints, we limit the local drift accumulation and model parameters, ensuring that when the local model solves the optimal solution, the local drift accumulation tends to zero. This stabilizes the global model and resolves the difference between the local model and the global objective. The objective function is further defined as

[0055]

[0056] In the formula, represents the prediction loss of client i, represents the gradient of the model parameters in the previous round of training, and α represents the penalty coefficient of the proximal term.

[0057] Therefore, the loss function of client i is expressed as

[0058]

[0059] The update rule for client i is expressed as

[0060]

[0061] In addition, considering the entire model training cycle, the initial model parameters may have greater heterogeneity and benefit from exploring different gradient directions. However, as training progresses, it is necessary to increase the gradient correction near the optimal point to prevent deviation. Specifically, the dynamic function d(r) is used to adaptively adjust the gradient correction during the training round, expressed as

[0062]

[0063] Where R is the number of communication rounds. Figure 2As shown, the gradient correction can be dynamically adjusted according to different training stages, thereby effectively reducing client drift and ensuring the consistency of global model parameters.

[0064] Figure 3 The data loss comparison between the FedKDDR method of the present invention and the traditional FedAvg method in the case of catastrophic forgetting of the client in the CIFAR10 dataset is shown in the figure. r-1 The data D calculated from the local data of client i in round r-1 i loss, loss(r,i) represents the average loss of the same data calculated using local training parameters in round r. The loss(r,i) of the traditional FedAvg method is significantly higher than the loss(r-1,i), which indicates that after the local training stage, other clients have forgotten the knowledge of the previous round of training data; in contrast, the gap between loss(r-1,i) and loss(r,i) of the FedKDDR method of the present invention is significantly narrowed. It can be seen that the present invention adopts dynamic knowledge distillation to adjust the label distribution between the global model of round r-1 and the local model of round r, thereby alleviating the forgetting problem in the local training stage.

[0065] The effect of the present invention is verified by experiments below. The federated learning method of the present invention is named FedKDDR.

[0066] The experiments use the MNIST, CIFAR10, and CIFAR100 datasets, and use convolutional neural networks to complete classification tasks; data preprocessing follows the procedures outlined in FedDyn; and the training and test sets are separated using standard methods.

[0067] In order to simulate the data imbalance between clients in the Non-IID scenario, Dirichlet distribution and pathological distribution are used. Specifically, parameters μ = 0.3 and μ = 0.6 are defined for two Dirichlet distributions (labeled as D1 and D2) on the CIFAR10 and CIFAR100 datasets, parameters c = 3 and c = 6 are defined for two pathological distributions (labeled as P1 and P2) on the CIFAR10 dataset, and parameters c = 10 and c = 20 are defined for two pathological distributions (labeled as P3 and P4) on the CIFAR100 dataset. The network architecture used to train CIFAR10 and CIFAR100 includes two convolutional layers with 5x5 kernels, one maximum pooling layer, and three fully connected layers.

[0068] A series of traditional and state-of-the-art algorithms, including FedAvg, FedProx, Scaffold, FedDyn, FedVRA and FedDisco, are selected as the baseline to evaluate the efficacy of the FedKDDR of the present invention.

[0069] Hyperparameter setting: A standard federated learning framework is adopted, and multiple clients train models using their own local datasets and upload updates to the central server at the end of each communication cycle. The central server aggregates these updates to update the global model; the experimental parameters are configured as follows: learning rate is 0.1, learning rate decay is 0.998, weight decay coefficient is 0.001, batch size is 50, and each round of local iterations is 5 times. For the CIFAR10 dataset, a total of 1000 rounds of training are performed; for the CIFAR100 dataset, a total of 600 rounds of training are performed; for the MNIST dataset, a total of 300 rounds of training are performed. In order to evaluate the robustness of FedKDDR under different circumstances, two experimental configurations are designed: one is 100 clients, of which 10% and 15% participated in the training, and the other is 500 clients, of which 2% participated in the training.

[0070] Table 1 shows the comparison of the accuracy of FedKDDR and the baseline algorithm on the MNIST, CIFAR10 and CIFAR100 datasets under the first configuration, expressed as mean ± variance. Table 2 shows the comparison of the accuracy of FedKDDR and the baseline algorithm on the CIFAR10 / CIFAR100 datasets under the second configuration, i.e., 2% client participation and 500 clients with data heterogeneity, expressed as mean ± variance. Table 3 shows the comparison of the corresponding number of communication rounds for different algorithms to achieve the same target accuracy. D1 and D2 on the left side of the table correspond to the Dirichlet distribution, and P1 and P2 on the right side of the table correspond to the pathological distribution. R# indicates the number of communication rounds required for each method to achieve the target accuracy, S↑ indicates the corresponding convergence rate relative to FedAvg, and the symbol > indicates that the target accuracy cannot be achieved within the communication limit.

[0071]

[0072] Table 1 Comparison of the accuracy of FedKDDR and baseline algorithms in multiple rounds of testing on MNIST, CIFAR10 and CIFAR100 datasets

[0073]

[0074] Table 2 Comparison of the accuracy of FedKDDR and baseline algorithms in multiple rounds of testing on CIFAR10 / CIFAR100 datasets

[0075]

[0076] Table 3 Comparison of the number of communication rounds for different algorithms to achieve the same target accuracy

[0077] As can be seen from Table 1, on the CIFAR10 dataset with a partial client participation rate of 10%, the FedKDDR of the present invention achieved an accuracy of 83.69% under the D1 setting, which is 3.49% higher than FedAvg; similarly, under the P2 setting, the accuracy of FedKDDR is 3.48% higher than that of FedAvg. It is worth noting that on the CIFAR100 dataset, the accuracy of FedKDDR is also significantly better than that of FedAvg, with an increase of 5%-10%. These significant improvements highlight the effectiveness of FedKDDR of the present invention in processing complex data distributions. It can be seen that the dynamic distillation learning mechanism of FedKDDR ensures stability and consistent performance under different data environments, and is particularly suitable for highly heterogeneous dataset scenarios.

[0078] As can be seen from Table 2, when there are 500 clients and the client participation rate is 2%, the FedKDDR of the present invention can also achieve the highest accuracy compared with other baseline algorithms; for example, on the CIFAR10 dataset, the accuracy of FedKDDR is 3% to 6% higher than that of FedAvg, and on the CIFAR100 dataset, the accuracy of FedKDDR is 1% to 2% higher than that of FedAvg. These findings confirm the robustness of FedKDDR and its superiority in various experimental scenarios. It can be seen that by balancing global and local model updates, FedKDDR effectively solves the problem of client forgetting, accelerates convergence, and improves model accuracy.

[0079] Table 3 compares the convergence speed of FedKDDR with other baseline algorithms in detail. The experimental results show that FedKDDR can solve the problem of client data drift and significantly speed up the model convergence. Specifically, compared with the baseline algorithm, FedKDDR only needs fewer communication rounds to achieve the preset target accuracy. Among them, on the CIFAR10 dataset, with 100 clients and 10% partial client participation under the D1 setting, FedKDDR only used 174 rounds of communication to achieve 80% accuracy, while FedAvg required more than 1,000 rounds to achieve the same accuracy under the same conditions. Therefore, compared with FedAvg, FedKDDR converges more than 5.74 times faster. This significant improvement stems from FedKDDR's clever handling of local data drift and effective optimization of the target, which directly translates into a reduction in communication costs.

[0080] In summary, the FedKDDR of the present invention not only performs well in terms of model performance, but also has significant advantages in terms of convergence speed, especially in dealing with the challenges brought by data heterogeneity and imbalance. These findings emphasize the effectiveness and practicality of FedKDDR in dealing with complex data distributions.

[0081] The above embodiments are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any changes made by adopting the design principles of the present invention and performing non-creative work on this basis should fall within the protection scope of the present invention.

Claims

1. A Non-IID federated learning method based on dynamic knowledge distillation and dynamic regularization, characterized in that: The following steps are involved: S10: After the local model training of client i in the rth round, the distribution P of local data labels is output s ; S20, the local model parameter w i r Upload to the server; S30, the server summarizes a set of client model parameters w1 obtained by random sampling r ,w2 r ,…,w M r , M is the total number of clients, and outputs the global model parameter w0 r+1 , for the next round; S40, client i downloads global model parameter w0 r+1 ; S50, the global model parameter w0 r+1 Load it into the client's local model and output the distribution P of the global data label t ; S60, perform dynamic knowledge distillation on client i and integrate P s , P t And the dynamic function d(r), calculate the local loss function of client i in round r Expressed as In the formula, represents the prediction loss of client i, is the model parameter, w r-1 represents the global model parameters of the previous round, b i represents a small batch dataset, represents the gradient of the model parameters of client i in the last round of training, α represents the penalty coefficient of the proximal term, and KL represents the KL divergence; S70, back propagation updates the local model of client i, where the update rule of client i is expressed as In the formula, η represents the learning rate, represents the corrected gradient, Represents the gradient of the model parameters in the previous round of training.

2. The Non-IID federated learning method based on dynamic knowledge distillation and dynamic regularization according to claim 1, characterized in that: The dynamic function d(r) is used to adaptively adjust the gradient correction in the training round, expressed as Where R is the number of communication rounds.

3. The Non-IID federated learning method based on dynamic knowledge distillation and dynamic regularization according to claim 2, characterized in that: The corrected gradient By solving the corrected objective function L for client i i 1 Obtained, expressed as Where b i =(x i ,y i )∈D i Represents a mini-batch dataset.

4. The Non-IID federated learning method based on dynamic knowledge distillation and dynamic regularization according to claim 3, characterized in that: The corrected objective function of client i is expressed as In the formula, represents the prediction loss of client i, and the dynamic function d(r) is a function of the number of communication rounds r.

Images