Federal learning method and system based on bounded unbiased differential privacy protection

Abstract

This invention discloses a federated learning method and system based on bounded unbiased discrete privacy protection, primarily addressing the difficulty in balancing privacy protection and model performance in existing technologies. The implementation includes: constructing a teacher-student distillation model architecture via a server; training both the teacher and student models locally, using soft labels containing class relationships generated by the teacher model to guide the student model in jointly optimizing cross-entropy and KL divergence loss; constructing a perturbation function including a baseline function and a dome function, using the baseline function to construct a uniform perturbation basis, and using the dome function to adjust the expected unbiased perturbation to achieve bounded unbiased discrete privacy protection; and performing weighted average aggregation on the perturbated student models. This invention, through the collaborative design of a bounded unbiased discrete privacy mechanism and a federated distillation architecture, can significantly improve model performance and training efficiency in non-IID data scenarios while ensuring strict privacy protection, and can be applied in fields such as healthcare, finance, smart cities, and the Industrial Internet of Things.

Description

Technical Field

[0001] This invention belongs to the field of data privacy protection technology, and in particular relates to a federated learning method that can be used in sensitive data scenarios such as medical image analysis and financial risk control, as well as in high privacy risk scenarios such as finance and healthcare, and in smart cities and industrial IoT. Background Technology

[0002] Traditional differential privacy protection typically employs Laplace or Gaussian mechanisms to add random noise to the model gradient. These methods, due to their use of unbounded probability distributions, are prone to extreme perturbations under small privacy budgets, leading to significant deviations in gradient update directions and affecting model convergence. Furthermore, in non-independent and identically distributed (Non-IID) data scenarios, the gradient update directions of different clients are severely inconsistent due to data distribution differences, causing serious client drift problems. Existing methods struggle to effectively coordinate these gradient differences while ensuring privacy. This limitation makes it difficult for traditional solutions to achieve a good balance between privacy protection and model performance, hindering the practical application of federated learning in sensitive data scenarios. To find solutions to these problems, researchers have proposed several methods, including:

[0003] Patent document CN202510058542.2 discloses "A decentralized learning method based on self-gradient / cross-gradient collaborative computation and similarity weighted correction," which achieves privacy protection in Non-IID data scenarios through Gaussian noise injection and momentum update mechanisms. While this scheme can effectively improve the learning efficiency of heterogeneous data under a decentralized architecture, it has two limitations: First, because it uses a Gaussian difference privacy mechanism with unbounded noise characteristics, it may cause significant deviations in gradient directions during deep neural network training, especially when the privacy budget is small, which can severely damage model convergence. Second, because its processing of Non-IID data mainly relies on single-round gradient similarity calculation and lacks modeling of the consistency of gradient distribution across rounds, it is difficult to fundamentally solve the client drift problem caused by differences in data distribution.

[0004] Patent application CN202411412885.6 discloses a "Data Privacy Protection Method Based on Multi-Party Secure Computation," which proposes a privacy protection scheme combining Paillier semi-homomorphic encryption and federated learning. This scheme achieves privacy-preserving training of support vector machine models through homomorphic kernel matrix computation and a secure aggregation protocol. While this scheme can complete kernel function computation and model optimization in encrypted form, effectively protecting data privacy, it also has two shortcomings: First, because the Laplace differential privacy mechanism it employs lacks boundedness, it risks generating extreme noise values ​​and causing data distortion when the privacy budget is low. Second, because the secure aggregation protocol it uses is based on a globally unified kernel space mapping, it does not consider the impact of differences in local data distribution among participating parties on kernel matrix similarity, which may lead to support vector selection bias and decision boundary offset problems in Non-IID scenarios.

[0005] The challenges in addressing these issues and shortcomings are as follows: The mathematical modeling complexity of bounded differential privacy mechanisms is high, requiring simultaneous satisfaction of triple conditions: ε-differential privacy constraints, unbiasedness constraints, and strict boundary constraints. The difficulty of searching for feasible solutions in the parameter space increases exponentially. The dual-model collaborative training within the federated distillation framework involves dynamic coupling of gradient propagation paths, which can easily lead to gradient conflicts between the teacher and student models under Non-IID data distributions, resulting in decreased knowledge transfer efficiency. Furthermore, achieving a balance between privacy protection and model performance is difficult.

[0006] Traditional methods suffer from a sharp increase in noise variance when the privacy budget ε < 1, which seriously affects model convergence. Summary of the Invention

[0007] To address the shortcomings of the existing technologies, this invention proposes a federated learning method based on bounded and unbiased differential privacy protection. One objective is to achieve boundedness and unbiasedness of noise output while ensuring that differential privacy requirements are met. Another objective is to protect data privacy while effectively improving the model's generalization ability on Non-IID data and significantly reducing client drift.

[0008] The techniques to achieve the above objectives are as follows: A bounded and unbiased perturbation mechanism is established, which constructs a perturbation function by linearly combining the baseline function and the dome function. Through triple constraints of boundary control, unbiased protection, and privacy proof, the boundedness and unbiasedness of the noise output are ensured while meeting differential privacy requirements. Furthermore, a federated distillation mechanism based on a localized teacher-student model architecture is implemented. Each client deploys a teacher model and a student model locally. The teacher model generates soft labels containing inter-class relationships, and the student model learns this semantic information using KL divergence loss to protect data privacy, improve the model's generalization ability on Non-IID data, and reduce client drift.

[0009] Based on the above approach, this invention is a bounded, unbiased, privacy-preserving federated learning method, characterized by comprising:

[0010] Obtain the dataset, divide it into data categories, and assign 1 to 2 data shards to each client to make it a Non-IID case;

[0011] The server constructs a global teacher model and a global student model. The teacher model adopts a fully localized design with a deep network structure, while the student model adopts a lightweight architecture. Both models and their initialization parameters are broadcast to all participating clients.

[0012] Each client initializes local teacher and student model copies according to the parameters sent, and deploys the teacher-student distillation model architecture locally. The teacher model is trained on local private data and generates soft labels containing inter-category similarity information through a temperature scaling mechanism to guide the training of the student model. The student model fits the real labels through cross-entropy loss on the one hand, and matches the distribution of soft labels output by the teacher model through KL divergence loss on the other hand, so as to learn the deep semantic features extracted by the teacher model.

[0013] The client uses a bounded, unbiased privacy mechanism to perturb the gradient of the uploaded student model before uploading it to the server.

[0014] The server only aggregates the student models and calculates a weighted average of the perturbed student model gradients uploaded by each client. After updating the global student model, a new training cycle is started until the model converges or reaches the preset number of cycles to complete the federated learning process.

[0015] Furthermore, the client employs a bounded, unbiased, privacy-partitioning mechanism to perturb the gradient of the uploaded student model, the implementation of which includes:

[0016] Construct a perturbation function that includes a baseline function and a vault function;

[0017] The gradient tensor of the student model, which is to be uploaded to the server for aggregation, is flattened into a one-dimensional vector gradient tensor.

[0018] For each gradient, the parameter combination (p,q,y,L1,L2) that meets the constraints is found through the differential evolution algorithm and then passed into the perturbation function;

[0019] For each gradient value in the flattened gradient vector, it is converted into a bounded and unbiased random value by the perturbation function with its corresponding optimized parameters, which is the perturbed gradient.

[0020] The perturbated gradients are reassembled into the original tensor structure.

[0021] Furthermore, the construction of the perturbation function, which includes a baseline function and a vault function, includes:

[0022] Define the baseline function G(x);

[0023] Define a quadratic function as the apex function H(x);

[0024] Adding the baseline function to the dome function H(x) yields the perturbation function P(x).

[0025] Furthermore, the step of converting each gradient value in the flattened gradient vector into a bounded and unbiased random value using its corresponding parameter-optimized perturbation function includes:

[0026] The differential evolution algorithm is used to optimize the parameter combination of the perturbation function to satisfy all the set constraints, and then the optimized parameters are input into the perturbation function.

[0027] 1000 points are uniformly sampled in the interval [-L1,L2], and the probability density of each point is calculated; then the discrete cumulative distribution function (CDF) is generated by cumulative summation and normalization.

[0028] Generate uniform random numbers u ~ U(0,1), and use binary search to determine Δ = F. -1 (u) directly outputs Δ as the gradient value after perturbation.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] First, it achieves a better balance between privacy and utility:

[0031] Because this invention sets up a bounded, unbiased privacy mechanism, the noise amplitude can be controlled within a preset range, allowing the model to maintain high baseline accuracy under strong privacy protection with a small privacy budget. Therefore, under the same privacy budget, it significantly improves data availability compared to traditional Laplace or Gaussian mechanisms. At the same time, it strictly satisfies the mathematical definition of ε-difference privacy, ensuring the rigor and reliability of this invention in terms of privacy protection.

[0032] Secondly, it effectively solves the client drift problem caused by Non-IID data:

[0033] This invention employs a fully localized teacher-student distillation architecture, enabling lightweight student models to learn deep semantic features and inter-class relationships extracted by teacher models. In extreme Non-IID scenarios, it can significantly improve model accuracy, enhance training efficiency and generalization performance in heterogeneous data environments, and significantly reduce client drift.

[0034] Third, it significantly improves the system's usability and deployment flexibility in resource-constrained environments:

[0035] Because this invention adopts a lightweight student model architecture, it can achieve efficient adaptation to various resource-constrained terminal devices such as mobile smart terminals, edge computing nodes, and IoT sensing devices without sacrificing model performance. This feature makes the method not only suitable for high privacy risk scenarios such as finance and healthcare, but also for federated learning application scenarios that require large-scale device collaboration, such as smart cities and industrial IoT.

[0036] In summary, this invention significantly improves the model performance of federated learning in Non-IID data environments while ensuring data privacy and security. Attached Figure Description

[0037] Figure 1 This is the overall flowchart of the implementation of the privacy-preserving federated learning method based on bounded unbiased division in this invention;

[0038] Figure 2 This is a schematic diagram illustrating the principle of each client deploying and training a teacher-student distillation model locally in the method of this invention;

[0039] Figure 3 This is a flowchart illustrating the implementation of local federated distillation in this invention.

[0040] Figure 4 This is a flowchart illustrating the privacy protection implementation process in this invention.

[0041] Figure 5 This is a schematic diagram of the perturbation function constructed in this invention;

[0042] Figure 6 This is a performance comparison chart of the federated distillation + bounded unbiased perturbation and the ordinary federated distillation + Laplace perturbation of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments.

[0044] Example 1: A Privacy-Preserving Federated Learning Method Based on Bounded Unbiased Division

[0045] Reference Figure 1 The specific implementation steps of this example include the following:

[0046] Step 1: The server builds a global teacher model and a global student model and initializes the model parameters.

[0047] 1.1) Obtain the dataset, divide it into data categories, and assign 1 to 2 data shards to each client to make it a Non-IID case;

[0048] 1.2) The server constructs a global teacher model and a global student model. The teacher model adopts a fully localized design with a deep network structure to enhance feature extraction capabilities; the student model adopts a lightweight architecture to adapt to the limitations of terminal devices.

[0049] 1.2.1) Select the model architecture based on the data type:

[0050] For the simple MNIST dataset and the FashionMNIST dataset, the teacher model uses a 3-layer convolutional structure, and the student model uses a 2-layer convolutional structure.

[0051] For the more complex CIFAR10 dataset, the teacher model uses an enhanced convolutional structure with 128-256 channels, while the student model uses a lightweight structure with 64-128 channels.

[0052] 1.2.2) Initialize model parameters:

[0053] For the teacher model, its convolutional layer parameters are initialized using Kaiming_normal_ and follow the rules. Distribution, where n l Where is the number of input channels, W is the weight parameter matrix of the convolutional layer, and N is a normal distribution.

[0054] For the student model, its parameters are initialized using PyTorch's default settings;

[0055] 1.2.3) Broadcast these two models and their initialization parameters to all participating clients;

[0056] 1.3) Set key hyperparameters such as privacy budget ε and distillation temperature τ, and each client receives the initialization parameters and prepares for local training.

[0057] The fully localized design of the teacher model has three aspects: First, the local teacher model is only initially distributed by the server once and does not participate in any parameter aggregation on the server side; second, the teacher model is only trained on the client side based on local data and extracts deep features to generate soft labels containing inter-category relationships, which are used to guide the training of the local student model; third, the teacher model parameters and intermediate calculation results are always kept on the client side and are not transmitted or shared in any form.

[0058] Step 2: Train the dual models locally.

[0059] Each client initializes local copies of the teacher and student models based on the distributed parameters, and deploys a teacher-student distillation model architecture locally. The teacher model employs a more complex network structure with stronger feature extraction capabilities; its parameter count is 2-3 times that of the student model, and it is fully localized to guide the student model. The student model, on the other hand, adopts a lightweight design to ensure efficient operation on resource-constrained devices. During local co-training, the teacher model learns the deep feature distribution of the data and generates soft labels containing rich inter-class relationships. These soft labels not only contain predicted class probabilities but also reflect similarity information between different classes. The student model is trained by simultaneously performing knowledge distillation through joint optimization of two loss functions. Specifically, cross-entropy loss based on real labels ensures basic classification performance, while KL divergence-driven distillation loss forces the student model to learn the soft label distribution output by the teacher model, thereby capturing implicit semantic relationships between data. This dual-supervision mechanism significantly improves the model's generalization ability in heterogeneous data environments, especially in class imbalance scenarios, where the inter-class similarity information conveyed by the soft labels effectively alleviates cognitive biases in local models.

[0060] Reference Figure 2 and Figure 3 This step specifically includes:

[0061] 2.1) The teacher model is trained on local private data and generates soft labels containing inter-category relationships:

[0062] 2.1.1) Set the loss function of the teacher model to standard cross-entropy loss. Where C represents the total number of categories in the dataset, q i y represents the teacher model's predicted probability for category i. i This represents the one-hot encoded value of the sample's true label in category i, and its value can be 0 or 1.

[0063] 2.1.2) Input the local dataset into the teacher model to calculate the parameter gradient of the model, perform L2 norm clipping, and then use the stochastic gradient descent (SGD) optimizer to optimize the teacher model;

[0064] 2.1.3) The optimized teacher model receives local data samples, outputs a logits vector (i.e., the unnormalized class prediction score), and generates soft labels through temperature scaling. i = 1, 2, ..., C, where p i The probability of the i-th class soft label output by the teacher model. Let τ be the original logits output of the teacher model for the i-th class, τ be the temperature coefficient, and exp(·) be the exponential function used to convert logits into non-negative values.

[0065] 2.2) The student model learns the soft label distribution output by the teacher model by jointly optimizing the cross-entropy loss and KL divergence loss:

[0066] 2.2.1) Input the local dataset and the soft labels output by the teacher model into the student model to jointly optimize the objective function. in, Let C be the cross-entropy loss function, C be the total number of classes in the dataset, and y be the cross-entropy loss function. i The one-hot encoding of the real label, q i =softmax(z S ) outputs the probability for the student model; p is the distillation loss function. i =softmax(z T / τ) represents the softening probability of the teacher model, τ is the temperature coefficient, and α is the balancing weight. This KL divergence loss makes the output distribution of the student model approximate the soft label distribution of the teacher model.

[0067] The dual supervision mechanism used in the objective function not only preserves the true information of the original data, but also makes full use of the high-order knowledge features extracted by the teacher model, thereby significantly improving the model's generalization ability in the Non-IID data environment.

[0068] 2.2.2) The parameter gradient of the student model is calculated using the objective function and then L2 norm clipped. The stochastic gradient descent (SGD) optimizer is then used to optimize the model. The soft label distribution output by the teacher model is matched with the KL divergence loss, so that the student model can learn the deep semantic features extracted by the teacher model and complete this round of training.

[0069] Step 3, privacy protection.

[0070] Reference Figure 4 After completing local training, the client uses a bounded, unbiased privacy mechanism to perturb the gradient of the uploaded student model before uploading it to the server. The specific implementation is as follows:

[0071] 3.1) Construct a perturbation function that includes a baseline function and a dome function:

[0072] 3.1.1) Define the baseline function: G(x) = y, if x ∈ [-L1, L2];

[0073] 3.1.2) Define a quadratic function as the dome function H(x):

[0074]

[0075] Among them, near the center point p Within the specified range, a parabolic adjustment window is formed, while in other regions, a zero-value output is maintained to achieve fine-tuning of the expected value of the disturbance.

[0076] 3.1.3) Add the baseline function to the dome function H(x) to obtain the perturbation function P(x):

[0077]

[0078] like Figure 5 As shown, this perturbation function inherits the stability of the baseline function and the precise adjustment capability of the vault function, particularly in the central region. Within the core, the function exhibits a quadratic perturbation enhancement characteristic; while in the boundary region, it maintains the uniform perturbation intensity of the baseline function. This structural design ensures that the final perturbation result maintains overall stability while allowing for fine-tuning in key regions, perfectly balancing the contradiction between privacy protection strength and data availability. Furthermore, this composite structure achieves several important characteristics through parameterization: First, the boundedness of the function is strictly guaranteed by the boundary parameters L1 and L2, ensuring that the perturbed data always remains within a preset safety range; second, by precisely adjusting the parameters p and q of the dome function, fine-grained control of the expected value of the perturbation distribution can be achieved, thus satisfying the core requirement of unbiasedness; finally, the overall shape of the function can adapt to different levels of privacy protection requirements through a systematic parameter adjustment mechanism.

[0079] 3.2) Set constraints:

[0080] Differential privacy constraints: Where ε is the privacy budget symbol, q is the parameter in the dome function, and y is the parameter in the baseline function;

[0081] Unbiasedness constraint: Where ξ is the input raw data value, p is the parameter in the arch function, -L1 is the left boundary of the baseline function, arch function and perturbation function, and L2 is the right boundary of the baseline function, arch function and perturbation function;

[0082] Normalization constraints:

[0083] Nonnegativity constraint: P(x)≥0,

[0084] Maximum value constraint:

[0085] vault boundary constraints:

[0086] Interval constraints: L1, L2 ∈ [0, 2M], where M is a constant that determines the range of L1 and L2.

[0087] 3.3) A uniform perturbation basis is constructed using the baseline function. Quadratic adjustment is introduced in key regions using the dome function. The differential evolution algorithm is used to find parameter combinations (p,q,y,L1,L2) that strictly satisfy the constraints of the differential privacy definition, unbiasedness, and probability density function definition, and input them into the perturbation function. Here, p and q are two different parameters in the dome function. p is used to adjust the center position of the dome function, and q is used to adjust the height of the dome function. y is a parameter of the baseline function, which provides the basic probability distribution. -L1 is the left boundary of the baseline function, the dome function, and the perturbation function, and L2 is the right boundary of the baseline function, the dome function, and the perturbation function.

[0088] 3.4) Flatten the gradient tensor of the student model that is to be uploaded to the server for aggregation into a one-dimensional vector gradient tensor;

[0089] 3.5) For each gradient value in the flattened gradient vector, it is converted into a bounded and unbiased random value by the perturbation function with its corresponding optimized parameters, which is the perturbed gradient.

[0090] 3.5.1) The differential evolution algorithm is used to optimize the parameter combination of the perturbation function so that it satisfies all the set constraints, and then the optimized parameters are passed into the perturbation function.

[0091] 3.5.2) Uniformly sample 1000 points on the interval [-L1,L2] and calculate the probability density of each point; then generate the discrete cumulative distribution function (CDF) by cumulative summation and normalization.

[0092] 3.5.3) Generate uniform random numbers u ~ U(0,1), and use binary search to determine Δ = F. -1 (u), directly output Δ as the gradient value after perturbation;

[0093] 3.6) The gradient of the clipped model will be directly mapped to a bounded and unbiased random value through inverse transformation sampling. The perturbed gradient is then reassembled into the original tensor structure, that is, the perturbed gradient is expanded to restore its original shape, thus completing the entire perturbation process.

[0094] Compared to traditional Laplace or Gaussian mechanisms, this step strictly limits the perturbation to a preset range and dynamically adapts to different data characteristics through parameter optimization algorithms, completely avoiding extreme noise and achieving lower variance and higher data utility under the same privacy budget.

[0095] Step 4: Aggregate the gradients of the student model.

[0096] The server-side employs a weighted federated average algorithm to aggregate and optimize the perturbed gradients of the student models uploaded by each client. This model aggregation focuses on knowledge fusion and global updates of the student models, and its implementation includes:

[0097] 4.1) In each round of training, K student models are randomly and uniformly selected from N clients to participate in the aggregation;

[0098] 4.2) For the selected K clients, aggregate the student model gradient using a weighted average based on data volume:

[0099]

[0100] Where n i Let i be the amount of data for the i-th client;

[0101] 4.3) Update the global student model parameters using the backpropagation algorithm after the aggregated global gradients, and evaluate the accuracy on the test set after each training round;

[0102] 4.4) The final generated global student model is then broadcast to each client to start a new training cycle until the model converges or reaches the preset number of training rounds to complete the federated learning process.

[0103] Example 2: A Federated Learning System Based on Bounded Unbiased Discrete Privacy Protection

[0104] This system employs a hierarchical federated architecture to achieve bounded, unbiased, and privacy-preserving features. A bidirectional communication network is formed between a central server-side coordinator and distributed client nodes. During system operation, the server broadcasts the initialized global teacher model and lightweight student model to the clients. Client nodes deploy a local training engine, which includes: a dual-model collaborative training module, a privacy perturbation module, and a server-side aggregation module.

[0105] The dual-model collaborative training module is used to train the global teacher model and the student model. The teacher model is trained on local private data and generates soft labels containing inter-category relationships through temperature scaling. The student model learns by jointly optimizing KL divergence loss and cross-entropy loss based on these soft labels and the real labels.

[0106] The privacy perturbation module is used to implement bounded and unbiased privacy protection for the student model using a hierarchical federated architecture. Specifically, in the Non-IID data scenario, gradient perturbation is achieved through a perturbation function: first, the parameter combination (p,q,y,L1,L2) is optimized using a differential evolution algorithm to satisfy all constraints, including ε-differential privacy constraints and unbiased constraints; then, the gradient is mapped to the preset boundary [-L1,L2] using an inverse transformation sampling technique.

[0107] The server-side aggregation module is used to perform weighted average aggregation optimization on the student model after gradient perturbation to complete the federated learning process.

[0108] The teacher model in this system adopts a fully localized design, which avoids the risk of high-order feature leakage and allows for personalized training based on the local data distribution characteristics of each client, effectively improving the model's adaptability in Non-IID scenarios. The final output is a lightweight student model that balances privacy protection and model performance, effectively solving the client drift problem caused by Non-IID data distribution.

[0109] It should be noted that embodiments of the present invention can be implemented using hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic, including but not limited to IoT terminal devices, edge computing nodes, application-specific integrated circuits (ASICs), or programmable logic devices. The software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor, an IoT device processor, or dedicated hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), local storage of the IoT device, or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented using hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips or transistors, or programmable hardware devices such as field-programmable gate arrays or programmable logic devices. They can also be implemented using software executed by various types of processors (including embedded processors for IoT devices), or by a combination of the above-described hardware circuitry and software, such as firmware.

[0110] The effectiveness of this invention can be further illustrated by the following simulation results:

[0111] I. Simulation Experiment Conditions

[0112] The experimental hardware platform uses an Intel Xeon Silver 4210R processor (10 cores and 20 threads, base frequency 2.4GHz, turbo frequency 3.2GHz) and is equipped with 32GB of DDR4 memory.

[0113] The software environment is based on Python 3.8 and the PyTorch 1.10 deep learning framework to build the federated learning system.

[0114] To simulate real-world application scenarios, the experiment constructed a distributed training environment with 100 clients, each running an independent model training process.

[0115] The dataset uses FashionMNIST data with non-IID partitioning, totaling 60,000 samples. It is non-uniformly partitioned into 2 classes / 100 subsets per client, with 100 clients and each client receiving 600-800 samples. The privacy budget is set to 0.5. This dataset is designed for application scenarios with data skew and containing highly sensitive data.

[0116] II. Experimental Content and Results Analysis

[0117] Under the aforementioned extreme Non-IID data distribution and strict privacy budget constraints, the effectiveness of this method is verified through comparative experiments, and the results are as follows: Figure 6 As shown, where:

[0118] Figure 6 (a) shows the accuracy change curves of the method of the present invention and the baseline method during the training process;

[0119] Figure 6 (b) shows the model loss change curves of the method of the present invention and the baseline method during the training process.

[0120] from Figure 6 As can be seen, the federated distillation learning combined with the bounded unbiased perturbation scheme of this invention exhibits significant advantages. The initial accuracy of the student model reaches 32.51%, and after 50 rounds of training, it stabilizes at 59.81%, with a maximum accuracy of 72.59%. In contrast, the ordinary federated learning scheme using traditional Laplace perturbation performs significantly worse under strict privacy budget conditions, with an initial accuracy of only 10.03%, a final accuracy of only 15.96%, and a maximum accuracy of only 26.76%, failing to achieve effective model learning. The significant difference between this invention and traditional methods stems primarily from two key factors: First, under strong privacy constraints, the traditional Laplace mechanism is forced to inject substantial noise, resulting in the complete submersion of gradient information and hindering effective model learning. Second, this invention employs a dual approach of federated distillation learning and bounded unbiased perturbation to achieve a synergistic breakthrough in knowledge distillation and bounded unbiased perturbation: the local teacher model transforms prediction results into semantically rich soft labels through the knowledge distillation mechanism, providing noise resistance for the student model; simultaneously, the innovative bounded unbiased perturbation mechanism, through a carefully designed perturbation degree function system, reduces the noise amplitude by an order of magnitude under the same privacy budget, maximizing the preservation of gradient information's effectiveness while ensuring differential privacy security.

[0121] The above description is merely two specific examples of the present invention and does not constitute any limitation on the present invention. Obviously, those skilled in the art, after understanding the content and principles of the present invention, may make various modifications and changes in form and details without departing from the principles and structure of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

[0122] It should be noted that the step numbers in the specification and claims of this invention are only for the purpose of clearly and completely describing the embodiments of this invention and for ease of understanding, and their order is not limited.

Claims

1. A federated learning method based on bounded unbiased privacy protection, characterized in that, include: Obtain the dataset, divide it into data categories, and assign one to two data shards to each client to make it a Non-IID (Non-Independent Identical Distribution) case; The server constructs a global teacher model and a global student model. The teacher model adopts a fully localized design with a deep network structure, while the student model adopts a lightweight architecture. Both models and their initialization parameters are broadcast to all participating clients. Each client initializes local teacher and student model copies according to the parameters sent, and deploys the teacher-student distillation model architecture locally. The teacher model is trained on local private data and generates soft labels containing inter-category similarity information through a temperature scaling mechanism to guide the training of the student model. The student model fits the real labels through cross-entropy loss on the one hand, and matches the distribution of soft labels output by the teacher model through KL divergence loss on the other hand, so as to learn the deep semantic features extracted by the teacher model. The client uses a bounded, unbiased privacy mechanism to perturb the gradient of the uploaded student model before uploading it to the server. The client employs a bounded, unbiased, privacy-partial mechanism to perturb the gradient of the uploaded student model, including: 5a) Construct a perturbation function that includes a baseline function and a dome function, including: 5a1) Define the baseline function: ; 5a2) Define a quadratic function as the apex function. : ， Among them, near the center point p, [ Within the specified range, a parabolic adjustment window is formed, while in other regions, a zero-value output is maintained to achieve fine-tuning of the expected value of the disturbance; 5a3) Compare the baseline function with the vault function Adding them together, we get the disturbance function P. : ； 5b) Flatten the gradient tensor of the student model that is to be uploaded to the server for aggregation into a one-dimensional vector gradient tensor; 5c) For each gradient, a parameter combination (p, q, y, L1, L2) that meets the set constraints is found using the differential evolution algorithm and passed to the perturbation function. Here, p and q are two different parameters in the arch function, p is used to adjust the center position of the arch function, and q is used to adjust the height of the arch function; y is the parameter of the baseline function, which provides the basic probability distribution; -L1 is the left boundary of the baseline function, the arch function, and the perturbation function, and L2 is the right boundary of the baseline function, the arch function, and the perturbation function. 5d) For each gradient value in the flattened gradient vector, convert it into a bounded and unbiased random value through the perturbation function with its corresponding optimized parameters, which is the perturbed gradient. 5e) Reassemble the perturbed gradients into the original tensor structure; The server only aggregates the student models and calculates a weighted average of the perturbed student model gradients uploaded by each client. After updating the global student model, a new training cycle is started until the model converges or reaches the preset number of cycles to complete the federated learning process.

2. The method as described in claim 1, characterized in that, The server constructs a global teacher model and a global student model, including: 2a) Select the model architecture based on the data type: For the simple MNIST dataset and the FashionMNIST dataset, the teacher model uses a 3-layer convolutional structure, and the student model uses a 2-layer convolutional structure. For the more complex CIFAR10 dataset, the teacher model uses an enhanced convolutional structure with 128-256 channels, while the student model uses a lightweight structure with 64-128 channels. 2b) Initialize model parameters: For the teacher model, its convolutional layer parameters are initialized using Kaiming_normal_ and follow the rules. Distribution, among which Input the number of channels. This is the weight parameter matrix of the convolutional layer. It follows a normal distribution; For the student model, its parameters are initialized using PyTorch's default settings.

3. The method as described in claim 1, characterized in that, The teacher model is trained on local private data and generates soft labels containing inter-category relationships, including: 3a) Set the loss function of the teacher model to standard cross-entropy loss. : Where C represents the total number of categories in the dataset. This represents the teacher model's predicted probability for category i. This represents the one-hot encoded value of the sample's true label in category i, and its value can be 0 or 1. 3b) Input the local dataset into the teacher model to calculate the parameter gradient of the model, perform L2 norm clipping, and then use the stochastic gradient descent (SGD) optimizer to optimize the teacher model; 3c) The optimized teacher model receives local data samples, outputs a logits vector (i.e., the unnormalized class prediction score), and generates soft labels through temperature scaling. ,in, The probability of the i-th class soft label output by the teacher model. This is the original logits output of the teacher model for the i-th class. For temperature coefficient, This is an exponential function used to convert logits to non-negative values.

4. The method as described in claim 1, characterized in that, The student model learns the soft label distribution output by the teacher model by jointly optimizing the cross-entropy loss and KL divergence loss, including: 4a) inputting the local dataset and the soft labels output by the teacher model into the student model, and jointly optimizing the objective function. : ,in, Here, C is the cross-entropy loss function, and C is the total number of classes in the dataset. One-hot encoding of the real label. Output probabilities for the student model; Let be the distillation loss function. Soften the probability for the teacher model. Temperature coefficient; To balance the weights; 4b) Calculate the parameter gradient of the student model using the objective function, perform L2 norm clipping, and then optimize it using the stochastic gradient descent (SGD) optimizer to complete this round of training.

5. The method as described in claim 1, characterized in that, In step 5d), for each gradient value in the flattened gradient vector, it is converted into a bounded and unbiased random value through its corresponding parameter-optimized perturbation function, including: 5d1) The differential evolution algorithm is used to optimize the parameter combination of the perturbation function so that it meets all the set constraints, and then the optimized parameters are input into the perturbation function; 5d2) Uniformly sample 1000 points on the interval [-L1, L2] and calculate the probability density of each point; then generate the discrete cumulative distribution function (CDF) by cumulative summation and normalization. 5d3) Generate uniform random numbers Use binary search to determine Direct output As the gradient value after perturbation.

6. The method as described in claim 1, characterized in that, The fully localized design of the teacher model includes: 8a) The local teacher model is initially distributed by the server only once and does not participate in any parameter aggregation on the server side; 8b) The teacher model is trained on local data and extracts deep features only on the client side, generating soft labels containing inter-category relationships to guide the training of the local student model. 8c) Teacher model parameters and intermediate calculation results are always kept locally on the client and are not transmitted or shared in any form.

7. The method as described in claim 1, characterized in that, The server only aggregates student models and performs a weighted average calculation of the gradients of the perturbed student models uploaded by each client, including: 9a) In each round of training, K students are randomly and uniformly selected from N clients to participate in student model aggregation; 9b) For the selected K clients, aggregate the student model gradients using a weighted average based on data volume: ， in, Let i be the amount of data from the i-th client. Let K be the global model parameters after aggregation in round t+1, and K be the number of clients participating in aggregation in each round. Let i be the amount of local data for the i-th client. These are the local model parameters for the i-th client after the t-th round of training.

8. The method as described in claim 5, characterized in that, All constraints set in 5d1) include: Differential privacy constraints: , where ε is the privacy budget symbol, q is the parameter in the dome function, and y is the parameter in the baseline function; Unbiasedness constraint: ,in The input is the original data value, p is the parameter in the arch function, -L1 is the left boundary of the baseline function, arch function and perturbation function, and L2 is the right boundary of the baseline function, arch function and perturbation function. Normalization constraints: ; Nonnegativity constraint: ; Maximum value constraint: ; vault boundary constraints: ; Interval constraints: , where M is a constant that determines the range of L1 and L2.

9. A federated learning system for implementing the method of claim 1, characterized in that, A bidirectional communication network is formed through a server-side central coordinator and distributed client nodes. During system operation, the server broadcasts the initialized global teacher model and lightweight student model to the clients, and the client nodes deploy their local training engines, which include: The dual-model collaborative training module is used to train the global teacher model and output soft labels, which are then used to train the lightweight student model. The privacy perturbation module is used to implement bounded, unbiased privacy protection for student models using a hierarchical federated architecture. The server-side aggregation module is used to perform weighted average aggregation of the gradients of student models after they have been uploaded and perturbed, and the federated learning process is completed when the model converges or reaches a preset number of rounds.

Images