Federal learning dual optimization method and system based on cluster knowledge distillation and adaptive local differential privacy

Abstract

The invention belongs to the technical field of federated learning dual optimization, and discloses a federated learning dual optimization method and system based on cluster knowledge distillation and adaptive local differential privacy, and the method comprises the steps: carrying out the data statistical feature extraction of a client; clustering the clients; local training of the client is carried out; applying an adaptive local differential privacy mechanism (ALDP); carrying out model aggregation FedAvg in the cluster; distilling structural knowledge; updating the global model; and evaluating model performance and privacy protection capability. The invention provides a federated learning optimization method which combines client clustering, structural knowledge distillation and a self-adaptive local differential privacy mechanism aiming at the problems of client data heterogeneity, unstable model distillation and difficulty in considering privacy protection and performance in the existing federated learning method. According to the method, client data statistical feature clustering and structural relationship knowledge distillation are combined, and a disturbance mechanism with a dynamic adjustment capability is introduced, so that efficient protection of privacy information is realized while the model performance is improved.

Description

Technical Field

[0001] The present invention belongs to but is not limited to the field of federated learning dual optimization technology, and in particular relates to a federated learning dual optimization method and system based on cluster knowledge distillation and adaptive local differential privacy. Background Art

[0002] With the rapid development of artificial intelligence and big data technologies, more and more industries are leveraging machine learning models for intelligent analysis and decision-making on distributed data. In data-sensitive sectors such as finance, healthcare, and communications, traditional centralized machine learning approaches are struggling to adapt to current data privacy regulations, as data has strong privacy attributes and compliance constraints. To address this, Google first proposed federated learning (FL), a new distributed learning paradigm, in 2016. FL allows data to remain locally, communicating only model parameters or gradients, with a centralized server performing global model aggregation. This enables multi-party collaborative modeling while protecting privacy. Federated learning is widely applicable in scenarios such as cross-bank fraud detection in the financial industry, multi-center disease prediction models in the healthcare industry, and cross-carrier customer service quality evaluation in the communications industry. These scenarios share common characteristics: decentralized data, the involvement of multiple parties, and extremely high privacy requirements. However, despite its theoretical adaptability and privacy-friendly nature, federated learning faces significant technical challenges in practical application, primarily in the following two areas:

[0003] (1) Data heterogeneity problem (Non-IID): In federated learning scenarios, the data sources, distribution, and quality of different clients vary significantly, often showing non-independent and identically distributed (Non-IID) behavior. For example, the behavioral habits of different users, the diagnostic styles of medical institutions, and the consumption patterns of bank customers may all be inconsistent, resulting in large differences in the gradient directions of local models, thereby affecting the convergence speed and accuracy of the global model. Algorithms such as FedNova, FedProx, SCAFFOLD, and FedSiKD have alleviated the problem of data heterogeneity to a certain extent, but they still have limitations such as unstable performance and easy to fall into local optimality when facing strongly heterogeneous data.

[0004] (2) Privacy protection issues: Although federated learning was designed to protect user privacy, studies have shown that simply uploading model parameters or gradients still carries the risk of attackers recovering the original data through methods such as "gradient inversion" and "member inference." For example, DLG or GAN inversion techniques can restore the private data of participants to a certain extent. To this end, researchers have proposed security enhancement mechanisms such as differential privacy, homomorphic encryption, and secure multi-party computation to improve the anti-attack capabilities of federated learning. However, these mechanisms often have trade-offs such as high communication overhead and loss of model performance.

[0005] 1. Personalized Federated Learning

[0006] In practical applications, data between clients often suffer from non-independent and identically distributed (Non-IID) phenomena, which seriously affects the generalization ability and convergence efficiency of the model. To improve the adaptability of the model to local data, researchers have proposed a variety of personalized federated learning strategies:

[0007] (1) FedPer: aggregates the shared feature extraction layer, and each client retains a local classifier.

[0008] (2) FedBN: Shares parameters except the Batch Normalization layer and retains the BN parameters locally.

[0009] (3) FedALA: Distinguishes between global sharing and local personalized parameters, and adjusts them flexibly.

[0010] (4) Meta-FL: Combines meta-learning strategies to dynamically optimize client models to adapt to differences.

[0011] Although the above methods have alleviated the Non-IID problem to a certain extent, it is still difficult to balance personalization and generalization in highly heterogeneous data scenarios.

[0012] 2. Client Clustering

[0013] To further address the problem of data heterogeneity, some studies have introduced client clustering strategies to group clients with similar feature distributions into the same group, thereby improving model consistency and training stability within the group. Common methods include:

[0014] (1) Similarity matrix clustering: Clustering is performed after constructing a similarity matrix based on model output or gradient

[0015] (2) Gaussian mixture model (GMM) + EM algorithm: automatically divide client groups.

[0016] Clustering strategies can improve local model training effects, but clustering accuracy has a significant impact on the final model performance. Incorrect clustering can easily lead to model drift or performance degradation.

[0017] 3. Federated Knowledge Distillation Technology

[0018] Knowledge distillation transfers knowledge from the teacher model to the student model and has been widely used to alleviate model differences in federated learning. Typical methods include:

[0019] (1) Adversarial knowledge distillation: using generative adversarial networks to enhance the distillation effect.

[0020] (2) Dual KD: Clients act as teachers and students to each other, transferring knowledge in both directions.

[0021] (3) FedSiKD: Based on client clustering, teacher models within the same group are selected for distillation.

[0022] These methods have shown good results for dealing with non-IID data, but they also incur high computational and communication costs. In addition, if the teacher model is not selected properly, the distillation results are prone to fluctuations, affecting stability.

[0023] 4. Privacy protection mechanism

[0024] To prevent privacy leaks, federated learning is often combined with the following privacy-enhancing technologies:

[0025] (1) Secure Multi-party Computation (SMC): It completes collaborative computing without leaking the data of all parties. It has high security but high overhead.

[0026] (2) Homomorphic encryption (HE): It supports direct operations on encrypted data and has strong security, but it requires extremely high computing and communication resources.

[0027] (3) Differential Privacy (DP): It prevents re-identification attacks by adding noise. The traditional DP strategy has fixed noise, which often leads to a decrease in model accuracy, especially in Non-IID data.

[0028] In recent years, dynamic differential privacy (Adaptive DP) has gradually attracted attention. It balances privacy and performance by adaptively adjusting noise intensity. However, this method still faces challenges such as complex implementation and insufficient stability.

[0029] Defects and shortcomings of existing technology:

[0030] Current personalized federated learning methods still have significant limitations when dealing with highly heterogeneous data distribution across clients. This makes it difficult to effectively improve the adaptability of the global model, leading to performance degradation on some client models and insufficient personalized generalization capabilities. Furthermore, client-side clustering methods rely heavily on the accuracy of the initial clustering algorithm. Improper clustering can easily lead to poor model convergence or even negative transfer, impacting overall system performance. While federated knowledge distillation can help improve model consistency, the additional communication and training steps incur significant computational overhead. Furthermore, the lack of a unified standard for teacher model selection makes it difficult to ensure stable distillation results in heterogeneous environments. Regarding privacy protection, while SMC and HE offer high security, their high computational and communication costs make them difficult to apply to resource-constrained devices. Traditional differential privacy methods lack dynamicity, making it difficult to achieve a good trade-off between privacy protection and model performance. While dynamic differential privacy mechanisms hold promise, their applicability and stability in complex real-world scenarios require further verification. Summary of the Invention

[0031] In response to the problems existing in the existing technology, the present invention provides a dual optimization method and system for federated learning based on cluster knowledge distillation and adaptive local differential privacy.

[0032] The present invention is implemented as follows: a dual optimization method for federated learning based on cluster knowledge distillation and adaptive local differential privacy, the method comprising:

[0033] S1: client data statistical feature extraction;

[0034] S2: client clustering;

[0035] S3: client-side local training;

[0036] S4: Apply the adaptive local differential privacy mechanism ALDP;

[0037] S5: Intra-cluster model aggregation FedAvg;

[0038] S6: Structural knowledge distillation;

[0039] S7: Global model update: the server aggregates the student models and teacher models in all clusters using FedAvg to obtain a new global model.

[0040] S8: Evaluate model performance and privacy protection capabilities.

[0041] Furthermore, the S1 specifically includes:

[0042] During the initialization phase, the client extracts statistical features from the local dataset, including: mean μ i , standard deviation σ i , skewness γ i , and then uploaded to the server to form the client feature triplet:

[0043] client_statistics={(μ1,σ1,γ1),…,(μ n ,σ n ,γ n )};

[0044] The S2 specifically includes:

[0045] The server uses the K-means clustering algorithm to cluster the client data statistics to form K clusters, minimizing the intra-cluster squared error:

[0046]

[0047] Among them, C kis the kth cluster, x is the client feature, c k is the cluster center;

[0048] Note: The initialization phase includes steps 1 and 2. After the initialization phase is completed, the server completes the clustering operation on the clients and assigns a corresponding cluster label to each client.

[0049] In the subsequent training process, starting from step 3, the client only needs to perform local training, model perturbation and upload according to the established process, without repeating the clustering operation; after receiving the model uploaded by the client, the server directly assigns it to the cluster determined in the initialization phase and continues with the following operations; in other words, the clustering operation is only performed once in the initialization phase, and the clustering results are used in subsequent rounds of training to reduce computational overhead and maintain clustering stability.

[0050] Furthermore, the S3 specifically includes:

[0051] The client downloads the global model sent by the server Use stochastic gradient descent (SGD) for training locally to update local model parameters

[0052]

[0053] where η t is the learning rate, is the loss function, D i It is the client's local dataset;

[0054] The S4 specifically includes:

[0055] For each layer weight w l , perform the following processing:

[0056] (1) Calculate the center value c l and the disturbance radius r l :

[0057]

[0058] in, Respectively represent the maximum and minimum values ​​of the l-th layer parameters

[0059] (2) Calculate the offset δ l :

[0060] δ l =r l -w l

[0061] Sampling a Bernoulli random variable b, the probability Pr that the Bernoulli variable b takes the value 1 is calculated by the following equation:

[0062]

[0063] For each weight w of each layer of the model l Add disturbance and get the disturbance value M(w l )

[0064]

[0065] Finally upload the perturbed model parameters

[0066] Furthermore, the S5 specifically includes: performing weighted averaging on the client models in the same cluster to form a cluster aggregation model

[0067]

[0068] The S6 specifically includes:

[0069] Calculate the cosine similarity between the cluster model and the global model:

[0070]

[0071] Select the clustering model with the largest cosine similarity as the teacher model Others as student models

[0072]

[0073] (3) Define the distillation loss function as L:

[0074] L=L cls +L inter +L intra

[0075] in:

[0076] Classification loss L cls :

[0077]

[0078] Inter-class structure loss L inter :

[0079]

[0080] Intra-class consistency loss L intra :

[0081]

[0082] Among them, the Pearson correlation coefficient ρ is defined as:

[0083]

[0084] Furthermore, the S8 specifically includes:

[0085] The final model is evaluated using the following metrics:

[0086] (1) Precision, F1 score, and recall rate;

[0087] (2) Performance under privacy budgets of ε = 3, 4, 5, 10, and 20;

[0088] (3) Protection effect against anti-gradient attacks, such as IDLG.

[0089] Furthermore, the ALDP mechanism has the following mathematical properties:

[0090] (1) The proof formula for satisfying ε-LDP is as follows:

[0091]

[0092] (2) The perturbation expectation is an unbiased estimate:

[0093]

[0094] (3) The disturbance variance is:

[0095]

[0096] Another object of the present invention is to provide a federated learning dual optimization system based on cluster knowledge distillation and adaptive local differential privacy based on the federated learning dual optimization method based on cluster knowledge distillation and adaptive local differential privacy, the system specifically comprising:

[0097] Feature extraction module: extracts client data statistics and sends them to the server;

[0098] Clustering module: The server performs K-means clustering to cluster data statistics into k clusters;

[0099] Local training module: performs local model updates;

[0100] ALDP perturbation module: client implements perturbation mechanism;

[0101] Intra-cluster aggregation module: FedAvg aggregation model;

[0102] Structural knowledge distillation module: Structural knowledge distillation trains the student model;

[0103] Global update module: forms a new round of global model;

[0104] Evaluation module: Evaluates model accuracy, robustness, and privacy.

[0105] Another object of the present invention is to provide a computer device, characterized in that the computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the federated learning dual optimization method based on cluster knowledge distillation and adaptive local differential privacy.

[0106] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the dual optimization method of federated learning based on cluster knowledge distillation and adaptive local differential privacy.

[0107] Another object of the present invention is to provide an information data processing terminal, which is used to implement the federated learning dual optimization system based on cluster knowledge distillation and adaptive local differential privacy.

[0108] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0109] First, this invention addresses the challenges of existing federated learning methods, such as client data heterogeneity, unstable model distillation, and the difficulty in balancing privacy protection and performance. This invention proposes a federated learning optimization method that combines client clustering, structural knowledge distillation, and an adaptive local differential privacy mechanism. By combining client data statistical feature clustering with structural relationship knowledge distillation and introducing a perturbation mechanism with dynamic adjustment capabilities, this method improves model training efficiency while achieving efficient protection of privacy information. This invention has the following significant advantages:

[0110] (1) Client clustering can be completed efficiently without relying on high-dimensional model parameters or original data. By using simple statistics such as the mean, standard deviation, and skewness of the client's local data for clustering, rather than directly relying on model output or original features, the communication and computing overhead brought by the traditional clustering process is avoided, and the clustering efficiency and robustness are improved. It is particularly suitable for distributed devices in resource-constrained environments.

[0111] (2) Using structural knowledge distillation mechanism, without relying heavily on softlabel distillation

[0112] The structural knowledge distillation employed in this paper effectively enhances semantic alignment between models by introducing structural loss terms such as classification loss, inter-class structural loss, and intra-class consistency loss. This makes the knowledge transfer between the teacher and student models more stable and generalizable. This mechanism improves the consistency of models in heterogeneous scenarios and avoids model degradation or failure.

[0113] (3) Adopting an adaptive local differential privacy mechanism to effectively balance privacy protection and model performance. Compared with the traditional differential privacy mechanism that uniformly adds fixed noise and causes model performance loss, the perturbation mechanism designed in this paper dynamically adjusts the perturbation amplitude of each parameter according to the parameter distribution of each layer, making the perturbation more targeted. When the privacy budget ε = 3, it can still maintain good model accuracy and effectively resist anti-gradient attacks, achieving the coordinated optimization of privacy security and learning ability.

[0114] (4) The system structure is modular, highly adaptable and flexible in deployment

[0115] The method of the present invention has a clear architecture, which limits client clustering to the initialization stage. In the subsequent training process, only the established cluster labels are relied upon for distillation and aggregation, which significantly reduces the repeated calculation cost during system operation.

[0116] (5) The model has high accuracy and strong generalization ability

[0117] Experimental results on typical datasets (such as FMNIST, CIFAR-10, and CIFAR100) show that this invention can significantly improve model accuracy while ensuring privacy, with an accuracy increase of up to 15.8% compared to traditional methods. It also has good generalization capabilities and is suitable for typical federated learning application scenarios such as personalized modeling, collaborative recommendation, and multi-party financial risk control.

[0118] Second, the present invention can be widely used in privacy-sensitive industries such as finance, healthcare, and communications. By improving the comprehensive performance of federated learning in privacy protection, personalized modeling, and model performance, it helps build a secure and intelligent data collaboration platform. It has good commercial implementation and industrialization prospects, and has significant economic value and promotion potential.

[0119] This invention integrates client clustering, structural knowledge distillation and adaptive local differential privacy mechanism into a unified framework for the first time, and proposes a new solution that can achieve efficient clustering without high-dimensional model data and only based on statistical features, solving the problems of low efficiency and poor adaptability of traditional methods.

[0120] Federated learning has long struggled to balance heterogeneous data processing and privacy protection. This paper, through clustering, structured distillation, and an adaptive differential privacy perturbation mechanism, improves model performance and effectively mitigates privacy risks in a non-IID environment, resolving a long-standing problem in this field.

[0121] This invention breaks the conventional wisdom that effective clustering in federated learning requires raw data or high-dimensional gradients and that differential privacy inevitably sacrifices model accuracy. It demonstrates that efficient and secure collaborative modeling can be achieved through low-dimensional statistical feature clustering and adaptive perturbation strategies, providing a new technical path and mindset for the development of federated learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0122] Figure 1 This is a flow chart of a dual optimization method for federated learning based on cluster knowledge distillation and adaptive local differential privacy provided by an embodiment of the present invention.

[0123] Figure 2 This is a framework diagram of the dual optimization method of federated learning based on cluster knowledge distillation and adaptive local differential privacy in an embodiment of the present invention.

[0124] Figure 3 4 is a performance comparison chart of the models provided by the embodiments of the present invention.

[0125] Figure 4 The model performance comparison and privacy protection capabilities of the embodiments of the present invention under different privacy budgets are demonstrated.

[0126] Figure 5 This is a comparison of the test accuracy of five algorithms on the FMNIST, CIFAR10 and CIFAR100 datasets provided by an embodiment of the present invention; (a): test accuracy comparison of the FMNIST dataset under α=0.05; (b): test accuracy comparison of the FMNIST dataset under α=0.1; (c): test accuracy comparison of the FMNIST dataset under α=0.5; (d): test accuracy comparison of the CIFAR10 dataset under α=0.05; (e): test accuracy comparison of the CIFAR10 dataset under α=0.1; (f): test accuracy comparison of the CIFAR10 dataset under α=0.5; (g): test accuracy comparison of the CIFAR100 dataset under α=0.05; (h): test accuracy comparison of the CIFAR100 dataset under α=0.1; (i): test accuracy comparison of the CIFAR100 dataset under α=0.5. DETAILED DESCRIPTION

[0127] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0128] An embodiment of the present invention provides a dual optimization method for federated learning based on client clustering, structural knowledge distillation, and an adaptive local differential privacy mechanism, which is suitable for multi-party collaborative modeling scenarios with privacy protection requirements and data heterogeneity.

[0129] The method comprises the following basic steps:

[0130] Step S1: Client data statistical feature extraction

[0131] During the initialization phase, the client extracts statistical features from the local dataset, including: mean μ i , standard deviation σ i , skewness γ i , forming client feature triples and then uploading them to the server for subsequent clustering.

[0132] client_statistics={(μ1,σ1,γ1),…,(μ n ,σ n ,γ n )}

[0133] Step S2: Client clustering

[0134] The server uses the K-means clustering algorithm to cluster client data statistics to form K clusters. The clustering results are used to establish structured client clusters, and the data distribution within each cluster is similar:

[0135]

[0136] Among them, C k is the kth cluster, x is the client feature, c k is the center of the cluster.

[0137] Note: This clustering operation is performed only once during the initialization phase and will not change in subsequent training rounds.

[0138] Step S3: Client local training

[0139] The client downloads the global model sent by the server Use stochastic gradient descent (SGD) to train locally and update local model parameters

[0140]

[0141] where η t is the learning rate, is the loss function, D i The client's local dataset.

[0142] Step S4: Adaptive Local Differential Privacy Perturbation (ALDP)

[0143] Each client performs adaptive perturbation on each layer parameter:

[0144] δ l =r l -w l

[0145]

[0146] Step S5: Model aggregation within the cluster

[0147] The server performs a weighted average (FedAvg) of the client models within the cluster:

[0148]

[0149] Step S6: Structural relationship knowledge distillation

[0150] Calculate the cosine similarity between the cluster model and the global model:

[0151]

[0152] Select the clustering model with the largest cosine similarity as the teacher model Others as student models

[0153]

[0154] The distillation loss function is:

[0155] L=L cls +L inter +L intra

[0156] Classification loss L cls , inter-class structure loss L inter , intra-class consistency loss L intra It is designed based on the Pearson correlation coefficient ρ correlation coefficient.

[0157] Step S7: Global model update

[0158] Aggregate the student and teacher models in all clusters again using FedAvg to obtain a new round of global models

[0159] Step S8: Evaluate model performance and privacy protection capabilities

[0160] Use accuracy, F1, and AUC metrics to evaluate model performance;

[0161] The privacy protection capability of the model is tested using IDLG attack simulation.

[0162] (2) Description of equipment or system structure

[0163] The present invention also provides a federated learning system device, whose system structure includes the following core functional modules:

[0164] Table 1: Core functional modules of the system

[0165]

[0166]

[0167] (3) Information processing terminal description

[0168] The information processing terminal includes:

[0169] Processor: used to execute program logic and implement client model training, perturbation, and upload;

[0170] Storage: stores model parameters, client data features, training logs and other information;

[0171] Communication interface: responsible for interactive communication with the server, such as model upload and global parameter download.

[0172] Figure 3 We compared the test accuracy of the four algorithms on the FMNIST, CIFAR10, and CIFAR100 datasets. FedCKD-ALDP consistently achieved comparable or even superior accuracy to existing methods on the FMNIST, CIFAR10, and CIFAR100 datasets with varying levels of data heterogeneity (α = 0.05, 0.1, and 0.5). Notably, in the highly heterogeneous CIFAR10 dataset (α = 0.1), FedCKD-ALDP achieved an accuracy of 64.4%, significantly outperforming the baselines (FedSiKD: 58.1%, FedMD: 56.9%, and FedGKD: 58.4%). These findings highlight the robustness and effectiveness of our approach in challenging non-IID settings.

[0173] Figure 4The left side shows the performance of ALDP, AFLDP, LAD and DP under different privacy budgets (ε=3, 4, 5, 10, 20) on the CIFAR10 dataset. ALDP achieves an accuracy of 0.626 when ε=3, which is only 0.013 lower than the noise-free case, and is significantly better than AFLDP (0.624 when ε=10) and LAD (0.628 when ε=20). This shows that ALDP still maintains excellent performance under low privacy budgets. The performance difference stems from the different control strategies of the methods for noise distribution. DP and LAD use a fixed noise mechanism, which lacks flexibility and can easily lead to performance degradation; although AFLDP adjusts the learning rate to control ε, the noise is still uniformly distributed and has limited flexibility. In contrast, ALDP adaptively distributes noise and applies different degrees of perturbation to the weight differences of each layer, significantly reducing performance loss and achieving a better privacy-utility trade-off. In addition, from Figure 4 As can be seen on the right, ALDP demonstrates good robustness against IDLG attacks. As ε decreases, the attack becomes significantly less effective, especially when ε = 3, where the IDLG attack essentially fails. This demonstrates that ALDP can effectively defend against reconstruction attacks while preserving privacy.

[0174] An embodiment of the present invention provides a computer device, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor performs the steps of the federated learning dual optimization method based on cluster knowledge distillation and adaptive local differential privacy.

[0175] An embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor performs the steps of the dual optimization method of federated learning based on cluster knowledge distillation and adaptive local differential privacy.

[0176] An embodiment of the present invention provides an information data processing terminal, which is used to implement the federated learning dual optimization system based on cluster knowledge distillation and adaptive local differential privacy.

[0177] Figure 5The test accuracy of the five algorithms was compared on the FMNIST, CIFAR10, and CIFAR100 datasets. The results show that FedCKD-ALDP significantly outperforms FedAvg under non-independent and identically distributed (Non-IID) conditions, effectively alleviating the Non-IID problem. Although FedProx outperforms FedAvg to a certain extent, its performance is still limited in highly Non-IID environments. SCAFFOLD performs better in adapting to Non-IID data, but still fails to reach the performance level of FedCKD-ALDP at low α values ​​(i.e., high Non-IID levels). In addition, Table 2 compares the performance of FedCKD-ALDP with and without knowledge distillation (NKD) on the FMNIST and CIFAR10 datasets. It can be seen that the introduction of knowledge distillation significantly improves model performance, especially when the Non-IID level is high (α = 0.05). By introducing knowledge distillation, the average accuracy of FMNIST is improved by up to 9.33%, while that of CIFAR10 is improved by up to 15.96%.

[0178] Table 2 Performance comparison of FedCKD-ALDP and NKD models on FMNIST and CIFAR10 datasets at different levels of non-IID

[0179]

[0180] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A dual optimization method for federated learning based on cluster knowledge distillation and adaptive local differential privacy, characterized by: The following steps are involved: Step 1: The client extracts statistical features from the local data set, including calculating the mean, standard deviation, and variance of each data dimension, and uploads the statistical features to the server; Step 2: The server clusters the clients using a clustering algorithm based on the statistical features uploaded by the clients to form multiple clusters and assign a cluster identifier to each client; Step 3: The client downloads the global model sent by the server, and uses the stochastic gradient descent method to train and update the model parameters locally; In step 4, the client performs adaptive local differential privacy perturbation processing on the model parameters obtained through local training layer by layer. Specifically, the client calculates the maximum and minimum values ​​of the weights of each layer to obtain the center value and perturbation radius based on them, samples random variables according to the preset privacy budget, calculates the perturbation value based on the center value, perturbation radius and random variables, and adds noise to the original weights to obtain the perturbed model parameters. Step 5: The server receives the perturbed model parameters uploaded by all clients in the same cluster, and obtains the cluster aggregation model by weighted average based on the amount of data from each client. Step 6: The server calculates the cosine similarity between each cluster aggregation model and the current global model parameter vector, uses the cluster aggregation model with the largest similarity as the teacher model, and the remaining cluster aggregation models as student models. A distillation loss function is defined based on classification loss, inter-class structure loss, and intra-class consistency loss, and the student model is trained using the distillation loss function. Step 7: The server performs a weighted average of the teacher models and student models of all clusters according to the amount of data in each cluster to obtain a new global model. Step 8: Perform performance evaluation on the new global model under different privacy budget conditions, including accuracy, recall, and F1 score tests, and verify the privacy protection capability in the face of reverse gradient attack scenarios.

2. The dual optimization method for federated learning based on cluster knowledge distillation and adaptive local differential privacy according to claim 1 is characterized in that: The clustering algorithm in step 2 is the K-means algorithm, and the clustering goal is to minimize the square error between all client statistical features and the corresponding cluster centers. The server completes clustering in the initialization phase and does not repeat the clustering operation in subsequent training.

3. The dual optimization method for federated learning based on cluster knowledge distillation and adaptive local differential privacy according to claim 1 is characterized in that: The perturbation process of the weight vector of each layer in step 4 includes the following process: first, the maximum and minimum values ​​of the weight of the layer are calculated to obtain the center value and perturbation radius accordingly; second, Bernoulli random variables are sampled according to the privacy budget; finally, a single weight perturbation value is calculated based on the center value, perturbation radius and Bernoulli random variables.

4. The dual optimization method for federated learning based on cluster knowledge distillation and adaptive local differential privacy according to claim 1 is characterized in that: The classification loss in step six is ​​the cross entropy loss, the inter-class structure loss is one minus the Pearson correlation coefficient between the student model output and the teacher model output, and the intra-class consistency loss is the average similarity of the Pearson correlation coefficients between the corresponding channel outputs of the student model and the teacher model within the cluster.

5. The dual optimization method for federated learning based on cluster knowledge distillation and adaptive local differential privacy according to claim 1 is characterized in that: When the privacy budget in step eight is set to three, four, five, ten, and twenty, the accuracy, recall, and F1 score of the new round of global model are evaluated respectively, and the robustness of the model under the anti-gradient attack is tested.

6. A federated learning dual optimization system based on cluster knowledge distillation and adaptive local differential privacy, characterized by: include: Feature extraction module, used by the client to extract local data statistical features and send them to the server; Clustering module, used by the server to perform clustering operations based on client statistical features and generate cluster labels; The local training module is used by the client to download the global model sent by the server and perform stochastic gradient descent training on local data; The perturbation module is used by the client to perform layer-by-layer perturbation processing on the model parameters obtained through local training using the adaptive local differential privacy method; The cluster aggregation module is used by the server to obtain a cluster aggregation model by weighted averaging the perturbed model parameters uploaded within the same cluster according to the data volume; The knowledge distillation module is used by the server to select a teacher model based on the cosine similarity between the cluster aggregation model and the global model and define the distillation loss to train the student model; The global update module is used by the server to generate a new round of global models by weighted averaging the teacher models and student models in all clusters according to the amount of data in each cluster; The evaluation module is used to evaluate the performance and privacy protection capabilities of the new round of global models under multiple privacy budget conditions.

7. The federated learning dual optimization system based on cluster knowledge distillation and adaptive local differential privacy according to claim 6 is characterized in that: The clustering module uses the K-means algorithm to cluster the clients during the system initialization phase, and reuses the initial clustering results in subsequent training processes.

8. The federated learning dual optimization system based on cluster knowledge distillation and adaptive local differential privacy according to claim 6 is characterized in that: The perturbation module is implemented as follows: the maximum and minimum values ​​of the weight vector of each layer are calculated to obtain the center value and perturbation radius accordingly, Bernoulli random variables are sampled according to the privacy budget, and the perturbation values ​​of each weight are calculated based on the center value, perturbation radius and Bernoulli random variables and replace the original weights.

9. A computer-readable storage medium storing an executable program, on which computer-executable instructions for executing the method according to claim 1 are stored, and when the instructions are executed by a processor, the processor completes client data statistical feature extraction, server-side clustering, local model training, adaptive local differential privacy perturbation, cluster model aggregation, structured knowledge distillation, and global model update and evaluation.

10. The computer-readable storage medium storing an executable program according to claim 9, wherein: The computer-executable instructions further include instructions for implementing adaptive local differential privacy perturbation, which are executed according to the following process: calculating the maximum and minimum values ​​of the weights of each network layer to obtain the center value and perturbation radius accordingly, sampling random variables according to the preset privacy budget, calculating the perturbation value based on the center value, perturbation radius and random variables, and replacing the original weight.

Images