Federal learning method combining sharpness perception training and knowledge distillation

Abstract

The invention provides a federated learning method combining acutance perception training and knowledge distillation, and belongs to the technical field of privacy protection. The method comprises the steps that a central server initializes global model parameters and distributes the initial global model parameters to participating clients; after each participating client receives the global model parameters, acutance perception training is executed based on the local data set, and a local updated global model is obtained and serves as a teacher model; through a knowledge distillation method, on the basis of a local training set and in combination with a soft label of a teacher model, obtaining student model parameter variation; performing privacy protection operation on the parameter variation of the student model, and uploading the parameter variation to a central server; the central server aggregates the parameter variation uploaded by each participating client after privacy protection, and updates a global model; the updated global model is used as an initial global model of the next round of communication. According to the method, privacy and efficiency are balanced, sensitive information leakage can be prevented, and the practicability of a federal learning global model is maintained.

Description

Technical Field

[0001] The present invention relates to the field of privacy protection technology, and in particular to a federated learning method combining sharpness perception training and knowledge distillation. Background Art

[0002] With the rapid development of big data and artificial intelligence technologies, machine learning models have been widely used in various industries. However, traditional centralized learning methods face serious privacy leakage risks due to the need to upload data to a central server. To address this issue, Federated Learning (FL) has emerged as a distributed learning paradigm. It allows multiple clients to train models locally and only upload model parameters to a server aggregation, thereby avoiding direct sharing of raw data. However, federated learning still has privacy risks. Attackers may be able to restore sensitive information from model parameters through reverse reasoning. To this end, Differential Privacy (DP) was introduced into the federated learning framework. By adding random noise or clipping gradients in model updates, it ensures the indistinguishability of individual participants' data, thereby providing strict privacy protection.

[0003] To enhance privacy protection, existing research has proposed client-side differentially private federated learning (DPFL) methods. These techniques typically employ the following mechanisms: 1) adding Gaussian or Laplace noise to gradients or model parameters to perturb data features; 2) limiting the norm of updates through gradient clipping to prevent privacy leaks; and 3) encrypting uploaded model updates using secure aggregation techniques, such as homomorphic encryption or multi-party computation. Representative methods, such as CN116629388B, improve model generalization through gradient perturbation and parameter averaging strategies, attempting to balance privacy protection with model performance. Furthermore, some studies have attempted to dynamically adjust the noise level or clipping threshold to mitigate the conflict between privacy protection and model utility.

[0004] However, existing DPFL methods still have significant limitations: First, noise addition and gradient clipping introduce additional biases, making it difficult for the model to converge, especially in complex tasks (such as non-independent and identically distributed Non-IID data), with poor performance and a significant drop in accuracy and efficiency. Second, noise interference weakens the model's ability to extract data features, reduces generalization and robustness, and makes it ineffective in new data or dynamic environments. Finally, in scenarios with high privacy budgets (low noise constraints) or complex data sets, existing methods find it difficult to strike a balance between privacy protection strength and model performance. For example, secure aggregation improves privacy but increases computational and communication overhead. Therefore, there is an urgent need for a federated learning method that strikes a balance between privacy protection strength and model efficiency. Summary of the Invention

[0005] In light of this, the present invention provides a federated learning method that combines sharpness-aware training with knowledge distillation. This method integrates differentially private sharpness-aware training and knowledge distillation techniques. While effectively protecting user privacy, it optimizes the model training process through differentially private sharpness-aware training, enhancing its ability to find flat minima, thereby improving the model's generalization and robustness to noise. Furthermore, knowledge distillation is used to guide the training of the student model through the teacher model, effectively improving the student model's performance even in the presence of noise.

[0006] To this end, the present invention provides the following technical solutions:

[0007] A federated learning method that combines sharpness-aware training with knowledge distillation, including:

[0008] The central server initializes the global model parameters and distributes the initial global model parameters to participating clients;

[0009] After receiving the global model parameters, each participating client performs sharpness perception training based on the local dataset to obtain a locally updated global model as the teacher model;

[0010] The knowledge distillation method is used to obtain the parameter changes of the student model based on the local training set and the soft labels of the teacher model;

[0011] Perform privacy protection operations on the changes in student model parameters and upload them to the central server;

[0012] The central server aggregates the privacy-protected parameter changes uploaded by each participating client and updates the global model; the updated global model serves as the initial global model for the next round of communication.

[0013] Furthermore, performing sharpness perception training based on the local data set includes:

[0014] In each iteration, the direction of the maximum loss value in the neighborhood of the model parameters is calculated, and the model parameters are updated in the reverse direction to find the flat minimum.

[0015] Furthermore, the knowledge distillation method is used to obtain the parameter changes of the student model based on the local training set and the soft labels of the teacher model, including:

[0016] Use the teacher model to generate soft labels for training samples;

[0017] The soft labels and true labels are used to jointly supervise the training of the student model to obtain the changes in the student model parameters.

[0018] Furthermore, the privacy protection operation includes:

[0019] Tailor the changes in student model parameters;

[0020] Noise is added to the clipped parameter variation.

[0021] Furthermore, the soft labels and the true labels are used to jointly supervise the student model training, including:

[0022] The main loss is represented by the cross entropy loss between the student model prediction value and the true label;

[0023] Introducing a temperature parameter to construct a distillation loss based on the soft labels output by the teacher model;

[0024] The distillation loss and cross entropy loss are aggregated to generate the total loss function of the student model.

[0025] Furthermore, a temperature parameter is introduced to construct a distillation loss based on the soft labels output by the teacher model, including:

[0026]

[0027] Among them, s represents the output of the student model, t represents the output of the teacher model, and T is the temperature parameter used to soften the output of the softmax function.

[0028] Furthermore, the trimming of the variation of the student model parameters includes:

[0029]

[0030] Where C represents the preset clipping threshold, Δt i represents the change in the student model parameters, and ‖·‖2 represents the l2 norm.

[0031] Furthermore, the adding noise to the clipped parameter variation comprises:

[0032]

[0033] Where σ is the noise factor, σ 2 C 2 / m is the variance of the noise, C is the clipping threshold, I d is the identity matrix, and m is the number of participating clients in this round.

[0034] Furthermore, the noise coefficient satisfies:

[0035]

[0036] Among them, ε represents the privacy budget, which is used to quantify the strength of privacy protection, and δ represents the probability of privacy protection failure.

[0037] Furthermore, the central server aggregates the privacy-protected parameter changes uploaded by each participating client and updates the global model, including:

[0038]

[0039] Where 1 / m is a scaling factor, w t represents the unupdated global model weight, w t+1 Represents the updated global model weight.

[0040] Advantages and positive effects of the present invention:

[0041] This paper uses sharpness-aware training to precisely locate flat minima on the loss function surface. Flat minima typically correspond to a more stable parameter space and are insensitive to parameter perturbations, resulting in a model that exhibits stronger generalization capabilities on test data. Furthermore, because flat minima can buffer parameter shifts caused by noise, the model remains robust even under the noise introduced by differential privacy, significantly reducing the negative impact of privacy-preserving mechanisms on performance.

[0042] This invention uses knowledge distillation, where the teacher model transfers refined knowledge (such as soft labels or feature representations) to the student model without requiring the student to directly access the raw data. This allows the student model to bypass the complex feature extraction stage and converge quickly. This distillation mechanism not only shortens training time (the teacher model's knowledge compresses the learning path), but also, thanks to the teacher model's noise filtering capabilities, enables the student model to efficiently absorb key features even in noisy environments, thereby improving overall performance.

[0043] The client-level differentially private federated learning provided by this invention adds noise to gradients or parameters during the local training phase and re-implements privacy protection during global aggregation, making it impossible for attackers to restore the original data from model updates through reverse reasoning. Furthermore, because the student model in knowledge distillation only learns the "knowledge summary" (not the original data) output by the teacher, data privacy is protected even if the student model is attacked. This combined strategy achieves a balance between privacy and performance—it prevents the leakage of sensitive information at the micro level while maintaining the practicality of the global federated learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0045] Figure 1 This is a flow chart of the federated learning method combining sharpness perception training and knowledge distillation in Example 1 of the present invention;

[0046] Figure 2 Schematic diagram of the structure of the federated learning method combining sharpness perception training and knowledge distillation in Example 1 of the present invention;

[0047] Figure 3 Schematic diagram of comparative experimental results of the method of the present invention on the MNIST dataset. DETAILED DESCRIPTION

[0048] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0049] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0050] The present invention provides a federated learning method that combines sharpness-aware training with knowledge distillation. The inventive concept is as follows: introducing knowledge distillation technology to guide student model training through a teacher model to reduce performance loss under noise interference; improving the sharpness-aware training method to calculate the disturbance term based on historical gradients and optimize generalization in combination with periodic parameter averaging; and adopting a composite privacy protection strategy to superimpose clipping and noise addition in the model update after knowledge distillation to achieve stricter client-level differential privacy.

[0051] Example 1

[0052] Combine Figure 1 As shown, the method of the present invention is described with the client as the execution subject:

[0053] S1: The client receives the initialized training model and corresponding hyperparameter configuration from the central server as the starting point of the training process.

[0054] U represents the set of all clients, where |U|=M means there are M clients in the set U, and each client i∈U has its own unique local dataset D i .

[0055] The entire training set consists of the local datasets of all agents, expressed as: D = ∑ i∈U D i .

[0056] In each round, the central server randomly selects a certain number of clients to participate in the training. At the beginning of each round t (where t belongs to the set {0,1,...,T-1}), the server randomly selects a subset of clients S t ∈U, and the current global model w t Distribute to S t The client in .

[0057] S2: Participating clients perform model training tasks on their respective local datasets and use local data to iteratively optimize model weights.

[0058] Each participating client i∈S t After receiving the global model, it performs k (where k belongs to the set {1, ..., K}) iterations on its local dataset based on the selected optimizer to update its local model.

[0059] S3: Integrate knowledge distillation technology into the system architecture to extract and transfer the knowledge of the teacher model from the student model to improve the generalization ability and performance of the model.

[0060] S31: Performs a forward pass using the teacher model for each batch of data in the loader, without computing gradients to avoid changing the teacher model's parameters. This generates outputs called soft labels, which represent the probability distribution for each class. If a maximum length is specified, the system verifies that the concatenated soft label tensor meets this length. Finally, a padded soft label tensor is returned for subsequent training of the learner model.

[0061] S32: Let x denote the input data, l denote the true label, and f denote the student model. The input data x is forward-propagated through the student model f to generate a prediction corresponding to the target task. Running statistics (mean and variance) of the batch normalization layer in the model are enabled to track the statistics of the dataset during training. The main loss is calculated using the cross-entropy loss between the model predictions and the true labels (converted to long integers for compatibility):

[0062]

[0063] Where N is the number of samples in the batch, Q is the number of categories, and l ij is the jth element of the i-th sample (the true label after one-hot encoding), pred ij is the probability that the i-th sample belongs to the j-th class predicted by the student model.

[0064] S33: Jointly optimize the outputs of the teacher and student models by introducing a temperature parameter T. The student model's original prediction f(x) and the teacher model's soft labels S are temperature-scaled separately, and then the softmax function is applied to both simultaneously to achieve probability normalization. Finally, the KL divergence (Kullback-Leibler Divergence) is used to measure the information difference between the probability distribution of the temperature-scaled student model output f(x) and the soft labels S generated by the teacher model.

[0065] Let s represent the output of the student model (student output), t represent the output of the teacher model (teacher output), y represent the true label, and T is the temperature parameter used to soften the output of the softmax function. The distillation loss part can be expressed as:

[0066]

[0067] The KL divergence is defined as:

[0068]

[0069] S34: Aggregate the distillation loss and cross entropy loss to generate the total loss function:

[0070] L=α·L kD +(1.0-α)·L ce

[0071] α is a key hyperparameter, ranging from [0, 1], that balances the distillation loss and the student model's own loss. This composite loss is backpropagated to compute the gradients with respect to the model parameters, enabling optimization via gradient descent. By combining the true labels with the teacher model's soft labels, the student model can effectively absorb richer information, improving its performance and generalization. Knowledge distillation guides student model training through the teacher model's soft labels, reducing direct data dependence that is sensitive to noise. Sharpness-aware training further enhances noise robustness by perturbing the gradient to optimize the model's flat minimum. This combination enables the student model to both absorb the teacher's knowledge and converge stably under perturbations.

[0072] S4: Integrate sharpness-aware training methods to fine-tune model updates, improve the model's sensitivity to data features, and further optimize the model's training results.

[0073] S41: The local loss function on each client is defined as follows:

[0074]

[0075] in, Is an expectation operator, which means that i The random sample η is used to find the expectation, which describes the i Average performance on H i represents the local loss function of client i; w t,k (i) represents the model parameters of the kth iteration of the tth round; δ i is the disturbance term calculated based on the historical gradient, ξ i is the local data batch. The disturbance term δ i The design goal of is to find a flat minimum in the parameter space and evaluate the sharpness of the model by maximizing the loss value after perturbation. ρ is a predefined constant used to control the radius of the perturbation, while ‖·‖2 is the l2 norm, and U represents the set of clients.

[0076] S42: Under the existing parameter configuration, the backpropagation algorithm is applied to calculate the gradient of the loss function with respect to each model parameter. This gradient clearly identifies the direction in which the loss function value decreases most sharply at the current parameter position. Next, small adjustments are made to the parameters along the determined gradient direction to explore the sharpness characteristics in the parameter space, that is, to evaluate the sensitivity of the loss function to small changes in the parameters.

[0077] S43: For each client i∈U, at each local iteration k∈{1,2...,K} in each communication round t∈{0,1,...,T-1}, the kth internal iteration of client i is performed as follows:

[0078]

[0079] Among them, g t-1,k Represents the gradient of the global model in the previous round.

[0080] S5: Participating clients send differentially private model updates to the central server, which aggregates the received updates to update the global model.

[0081] S51: The participating client submits its model update, which is the difference between the local model after the Kth iteration and the global model received in the previous round. The difference is expressed as:

[0082] Δt i =w t,k (i)-w t,0 (i)

[0083] By uploading only the differences instead of the entire model, communication overhead can be significantly reduced. Typically, the differences are much smaller than the data size of the full model, making the transmission more efficient and reducing the consumption of network resources.

[0084] S52: The system clips the model update uploaded by each participating client to limit its size. This is achieved by applying a clipping function that adjusts the size of the update according to a clipping threshold C. The clipping method is as follows:

[0085]

[0086] Specifically, if the l2 norm (i.e., the Euclidean norm, which represents the "length" of a vector) of an update exceeds C, then the update will be scaled so that its l2 norm is exactly equal to C. Doing so prevents the update of any single client from having too large an impact on the global model, thereby enhancing the overall robustness of the system.

[0087] S53: In this step, the system adds Gaussian noise to the clipped updates.

[0088] Gaussian noise (also known as normally distributed noise) is a widely used privacy protection method. Its advantage is that it can conceal subtle differences in the original data, thereby preventing sensitive information about the client data from being inferred through model updates. Adding Gaussian noise to local updates ensures client-level differential privacy (DP):

[0089]

[0090] Where σ is the noise factor, σ 2 C 2 / m is the variance of the noise, C is the clipping threshold, I d is the identity matrix, and m is the number of clients participating in the update. By adjusting σ 2 The value of can control the intensity of noise, thus finding a balance between privacy protection and model performance. 2 In the case of , the cumulative privacy budget ε can be calculated based on the sampled Gaussian mechanism. To ensure (ε,δ)-differential privacy, the noise coefficient σ needs to be adjusted according to the sensitivity C / m. Specifically, σ should satisfy:

[0091]

[0092] Among them, ε represents the privacy budget, which is used to quantify the strength of privacy protection, and δ represents the probability of privacy protection failure.

[0093] S54: The server collects the model updates of each client after clipping and noise processing. After the collection is complete, the global model parameters are updated by calculating the weighted average of these updates. The obtained weighted average is combined with the current global model weight w t In this way, the global model is updated:

[0094]

[0095] Where m=|s t | is the number of participating clients in the current round, which is used to ensure that the sum of updates matches the update size of a single client on average.

[0096] For all clients i∈S participating in this update t Updated after noise addition Perform the summation:

[0097]

[0098] This summation is added to the global model weights w t Thus, we get the new global model weight w t+1 This process is repeated until the global model reaches a satisfactory performance level or other stopping conditions are met.

[0099] Example 2

[0100] Combine Figure 2 The method of the present invention is further described:

[0101] 1. Model Distribution: First, server initialization begins, marking the beginning of the federated learning process. During this stage, the server prepares an initial machine learning model and related training parameters, including but not limited to the learning rate and batch size. This initial model serves as the foundation for training on all clients. After initialization, the server distributes this initial model and training parameters to participating clients, ensuring that all clients begin training from the same starting point.

[0102] 2. Knowledge Distillation: A well-trained, high-performing teacher model is used to generate soft labels. These soft labels contain richer information than hard labels (i.e., explicit category labels) and reflect the teacher model's predicted probabilities for different categories. The student model is trained by combining the true labels with the soft labels provided by the teacher model. In this way, the student model not only learns the knowledge derived from the true labels, but also absorbs the "knowledge" of the teacher model, potentially achieving better performance and generalization.

[0103] 3. Sharpness-aware training: This improves model robustness by exploring the sharpness of the loss function in parameter space. Sharpness-aware training assesses the model's sensitivity to small parameter changes by adding small perturbations to the model parameters and observing the changes in the loss function. This allows the model to learn smoother and more robust parameter configurations, reducing the risk of overfitting and improving generalization to new data.

[0104] 4. Differential Privacy: After local training is complete, each client calculates the difference between its trained model and the initial model, which is called a model update. This update reflects the knowledge learned by the client on the local dataset. To protect the privacy of client data, the system performs pruning and noise processing on the model updates. Pruning limits the impact of individual updates on the global model, while noise is a differential privacy protection technique that adds random noise to the updates to mask details of the original data.

[0105] 5. Model Aggregation: The server collects all client-side cropped and noisy model updates and aggregates them to update the global model. This global model serves as the starting point for the next round of federated learning and is subsequently distributed to clients for training. In this way, the federated learning system can continuously learn and improve model performance while protecting user privacy.

[0106] Specifically:

[0107] 1. Initialization: Initialize the global model w 0 , the total number of clients is M, the proportion of clients selected in each communication is q, the total number of communication rounds in the entire training process is T, the threshold for data clipping is set to C, and the step size of local learning (i.e., learning rate) is l γ ,In each round of communication, each selected client will perform K local iterative updates.

[0108] 2. Perform T rounds of iterations: randomly select a group of participating clients with a number of m = qM without replacement.

[0109] 3. Each participating client trains for K rounds: the global parameters are copied to each participating client as the initial values of their local parameters;

[0110] 1) For each participating client i, the local parameter w local t (i) Equal to the global parameter w t .

[0111] 2) Calculate the distillation loss as an important basis for guiding local training;

[0112]

[0113] 3) The participating client randomly extracts a small batch of data samples η from the local dataset and calculates the current parameter w t,k (i) The gradient g of the data under t,k (i) Sharpness-aware training can be enabled to process gradients and enhance the generalization performance of the model.

[0114] 4) Client participation basis Update local parameters to complete the current iteration.

[0115] 4. After each participating client completes K rounds of training, the client calculates the change Δt of its local parameters after K iterations. i , the change is specifically expressed as the parameter w of the last iteration t,k (i) With the initial parameter w t,0 The difference between (i).

[0116] To ensure the security of data privacy, participating clients trim the changes in local training parameters to limit their range of activity and introduce noise into them.

[0117] The cropping operation is as follows:

[0118]

[0119] The operation of introducing noise is as follows:

[0120]

[0121] This process is the core of differential privacy, aiming to prevent the precise derivation of any specific information about any individual training data point from model updates. This ensures that the privacy of individual training data points is effectively protected even during model updates, thereby enhancing overall data privacy and security.

[0122] After completing these privacy protection measures, the participating clients will update the parameters after processing The updates are sent to the central server, which aggregates them to update the global model parameters.

[0123] 5. The central server collects parameter updates from each participating client Calculate their average value and add this average value to the current global parameter w t Thus, we get the new global parameter w t+1 The specific calculation method is as follows:

[0124]

[0125] After one round of global model update, return to step 2 to continue the next round of iteration until the preset total number of communication rounds T is reached.

[0126] Example 3

[0127] The beneficial effects of the method of the present invention are further illustrated by comparative experiments;

[0128] The control groups include: DP-FedAvg (Differentially Private Federated Averaging), Fed-SMP (Federated Learning with Sparsification and Model Perturbation), and DP-FedAvg-blurs (Differentially Private Federated Averaging with Blurring Mechanisms).

[0129] The experimental results are as follows Figure 3 As shown in the figure: Based on the MNIST dataset (Modified National Institute of Standards and Technology database), the concentration parameter (Concentration Parameter) is set to 0.3 when the Dirichlet distribution generates non-independent and identically distributed (Non-IID) data partitions.

[0130] The accuracy of the proposed model reaches 78.04%, which is 4.94% higher than DP-FedAvg, and 2.29% and 1.79% higher than Fed-SMP and DP-FedAvg-blurs, respectively.

[0131] This invention contributes a novel framework to the field of federated learning, cleverly integrating sharpness-aware training with the advanced concepts and technical advantages of knowledge distillation. This innovative combination not only provides a unique solution to the long-standing conflict between privacy protection and performance optimization in federated learning, but also significantly promotes technological progress in this field. By integrating these advanced technologies, the framework of this invention not only protects user data privacy but also significantly improves the generalization and robustness of the model. This means that in practical application scenarios, the federated learning model of this invention can demonstrate higher reliability, stability, and effectiveness, thereby better meeting the needs of various complex tasks.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A federated learning method combining sharpness perception training and knowledge distillation, characterized in that: include: The central server initializes the global model parameters and distributes the initial global model parameters to participating clients; After receiving the global model parameters, each participating client performs sharpness perception training based on the local dataset to obtain a locally updated global model as the teacher model; The knowledge distillation method is used to obtain the parameter changes of the student model based on the local training set and the soft labels of the teacher model; Perform privacy protection operations on the changes in student model parameters and upload them to the central server; The central server aggregates the privacy-protected parameter changes uploaded by each participating client and updates the global model; the updated global model serves as the initial global model for the next round of communication.

2. The federated learning method combining sharpness perception training and knowledge distillation according to claim 1, characterized in that: The performing sharpness perception training based on the local data set includes: In each iteration, the direction of the maximum loss value in the neighborhood of the model parameters is calculated, and the model parameters are updated in the reverse direction to find the flat minimum.

3. The federated learning method combining sharpness perception training and knowledge distillation according to claim 1, characterized in that: The knowledge distillation method is based on the local training set and the soft labels of the teacher model to obtain the changes in the student model parameters, including: Use the teacher model to generate soft labels for training samples; The soft labels and true labels are used to jointly supervise the training of the student model to obtain the changes in the student model parameters.

4. The federated learning method combining sharpness perception training and knowledge distillation according to claim 1, characterized in that: The privacy protection operation includes: Trimming the changes in student model parameters; Noise is added to the clipped parameter variation.

5. The federated learning method combining sharpness perception training and knowledge distillation according to claim 2, characterized in that: The soft labels and the true labels are used to jointly supervise the student model training, including: The main loss is represented by the cross entropy loss between the student model prediction value and the true label; Introducing a temperature parameter to construct a distillation loss based on the soft labels output by the teacher model; The distillation loss and cross entropy loss are aggregated to generate the total loss function of the student model.

6. The federated learning method combining sharpness perception training and knowledge distillation according to claim 5, characterized in that: Introducing the temperature parameter, constructing the distillation loss based on the soft labels output by the teacher model, including: Among them, s represents the output of the student model, t represents the output of the teacher model, and T is the temperature parameter used to soften the output of the softmax function.

7. The federated learning method combining sharpness perception training and knowledge distillation according to claim 4, characterized in that: The step of trimming the variation of the student model parameters includes: Where C represents the preset clipping threshold, Δt i represents the change in the student model parameters, and ‖·‖2 represents the l2 norm.

8. The federated learning method combining sharpness perception training and knowledge distillation according to claim 7, characterized in that: The adding noise to the clipped parameter variation comprises: Where σ is the noise factor, σ 2 C 2 / m is the variance of the noise, C is the clipping threshold, I d is the identity matrix, and m is the number of participating clients in this round.

9. The federated learning method combining sharpness perception training and knowledge distillation according to claim 8, characterized in that: The noise figure satisfies: Among them, ε represents the privacy budget, which is used to quantify the strength of privacy protection, and δ represents the probability of privacy protection failure.

10. The federated learning method combining sharpness perception training and knowledge distillation according to claim 1, characterized in that: The central server aggregates the privacy-protected parameter changes uploaded by each participating client and updates the global model, including: Where 1 / m is a scaling factor, w t represents the unupdated global model weight, w t+1 Represents the updated global model weight.

Images