Industrial federated learning-oriented adaptive differential privacy protection and communication compression joint optimization method and system

Abstract

The invention discloses an adaptive differential privacy protection and communication compression joint optimization method and system for industrial federated learning, and belongs to the technical field of communication compression. Collecting equipment data to construct a local data set; performing local training and gradient calculation by using the local data set to obtain a local gradient; differential privacy protection is carried out to obtain a privacy protection gradient after noise addition; performing compression processing on the privacy protection gradient to obtain compressed data; the client uploads the compressed data to a server; the server performs inverse quantization reconstruction on the gradient of each client according to the received masks and quantization parameters to obtain a reconstruction ladder; and carrying out gradient aggregation, privacy budget allocation and accounting on the reconstruction ladder. According to the method, higher model precision and communication efficiency can be realized, and the signal-to-noise ratio is further optimized.

Description

An Adaptive Differential Privacy-Preserving and Communication Compression Joint Optimization Method and System for Industrial Federated Learning Technical Field

[0001] This invention relates to the fields of federated learning technology, differential privacy protection technology, and gradient compression technology, specifically to an adaptive differential privacy protection and communication compression joint optimization method for industrial federated learning. Background Technology

[0002] Federated learning is a distributed machine learning framework that allows multiple clients to train models locally and only upload model updates to a server for aggregation, thus avoiding the sharing of raw data. However, industrial federated learning faces two key challenges. First, model updates may leak sensitive information; directly adding differential privacy noise can interfere with model convergence, leading to decreased accuracy. Traditional fixed noise injection mechanisms cannot adapt to training dynamics, introducing excessive noise in the later stages of training and causing unnecessary performance loss. Second, the computing and communication capabilities of industrial field equipment are limited. Frequent uploading of large amounts of model gradients during multiple iterations of federated learning generates significant communication overhead, becoming a bottleneck for practical applications. In existing technologies, gradient compression techniques can improve communication efficiency, but simple compression can compromise differential privacy protection. Furthermore, the approach of compressing first and then adding noise results in noise being wasted on secondary gradient dimensions, reducing the utilization of the privacy budget. Therefore, a method that can synergistically optimize privacy protection and communication efficiency is urgently needed. Summary of the Invention

[0003] Purpose of the invention: This invention provides an adaptive differential privacy protection and communication compression joint optimization method for industrial federated learning to solve the above-mentioned problems.

[0004] Technical Solution: To achieve the above objectives, the technical solution adopted in this invention is: an adaptive differential privacy protection and communication compression joint optimization method for industrial federated learning, comprising the following steps: Step 1, Local Training and Gradient Calculation: Collecting device data to construct a local dataset; the server initializes the global model. And related parameters, and then use the local dataset to perform local training and gradient calculation to obtain the local gradient. Step 2, Differential Privacy Protection: The client processes local gradients. Perform L2 norm clipping to obtain the clipped gradient. The following is the gradient after clipping. Adding adaptive Gaussian noise yields the privacy-preserving gradient with added noise. Step 3, Gradient Compression: Client-side privacy-preserving gradient compression. The data is compressed to obtain the compressed data; Step 4, Upload and Dequantization Reconstruction: The client uploads the compressed data to the server; the server performs dequantization reconstruction on the gradient of each client based on the received mask and quantization parameters to obtain the reconstructed gradient. Step 5, Gradient Aggregation: The server reconstructs gradients for all clients. Perform a weighted average to obtain the global gradient update. And update the global model using global gradients. ,in For the learning rate; Step 6, the server performs privacy budget allocation and accounting.

[0005] Preferred method: The method for client-side compression of privacy-preserving gradients in step 3 includes: Step 31, Top-k sparsity: preserving... The sparse gradient is obtained by selecting the top k% of elements with the largest absolute values. and its location mask Step 32, Quantification: [This refers to a specific step or process, likely related to quantification or quantification. The non-zero values ​​in the data are quantized using b-bit quantization to obtain the quantized gradient. and quantization parameter scaling factor and minimum value Step 32: Obtain the location mask Quantized gradient Minimum value Build compressed data .

[0006] Preferably, in step 1, local training and gradient calculation include: in the t-th round of training, the server randomly selects a set of clients. and the current global model Distributed to selected clients; each client Using local datasets For the model Training is performed, and the local gradient is calculated. .

[0007] Preferably, the L2 norm clipping in step 2 restricts the L2 norm of the gradient vector to within a threshold C, providing a defined sensitivity boundary for Gaussian noise injection. .

[0008] Preferred: The cutting formula in step 2 is as follows C is the threshold; the privacy-preserving gradient after adding noise. , express, express.

[0009] Preferably, the noise standard deviation of the adaptive noise injection mechanism described in step 2. From the formula Confirmed, among which Let L2 norm be the gradient after clipping. The privacy budget allocated for the current round.

[0010] Preferably, step 4 also includes an error feedback mechanism: the server records and accumulates the gradient residuals discarded by each client i due to sparsity. The residuals will be accumulated during the next training round on the client. It is added to its local gradient calculation.

[0011] Preferred method: Step 6, the server's method for allocating and accounting for the privacy budget, includes: the server using a privacy accounting method to calculate the privacy budget consumed in this round. This is accumulated into the total consumption of each client; when the client's accumulated privacy budget approaches... When this occurs, their participation in training is suspended; privacy accounting methods include moment accounting or Gaussian difference privacy methods.

[0012] Another objective of this invention is to provide an adaptive differential privacy-preserving and communication compression joint optimization system for industrial federated learning, comprising an input unit, a local training and gradient calculation unit, a differential privacy-preserving unit, a gradient compression unit, an upload and dequantization reconstruction unit, a gradient aggregation unit, a privacy budget allocation and accounting unit, and an output unit, wherein: the input unit is used to construct a local dataset from input device data; the local training and gradient calculation unit is used for server initialization of the global model. And related parameters, and then use the local dataset to perform local training and gradient calculation to obtain the local gradient. The differential privacy protection unit is used by the client to monitor local gradients. Perform L2 norm clipping to obtain the clipped gradient. The following is the gradient after clipping. Adding adaptive Gaussian noise yields the privacy-preserving gradient with added noise. The gradient compression unit is used by the client to compress the privacy-preserving gradient. The data is compressed to obtain compressed data; the upload and dequantization reconstruction unit is used by the client to upload the compressed data to the server; the server performs dequantization reconstruction on the gradient of each client according to the received mask and quantization parameters to obtain the reconstructed gradient. The gradient aggregation unit is used by the server to reconstruct gradients for all clients. Perform a weighted average to obtain the global gradient update. And update the global model using global gradients. ,in The learning rate is used; the privacy budget allocation and accounting unit is used by the server to allocate and account for the privacy budget.

[0013] Another object of the present invention is to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the aforementioned adaptive differential privacy protection and communication compression joint optimization method for industrial federated learning.

[0014] Compared with the prior art, the present invention has the following advantages: The present invention avoids wasting noise budget on the discarded dimension by "protecting first and then compressing", reduces communication cost and model noise under the same total privacy budget, and maintains high model accuracy and stable convergence. Attached Figure Description

[0015] Figure 1 is a schematic diagram of the overall process of the present invention.

[0016] Figure 2 is a schematic diagram of the adaptive privacy budget allocation and noise injection mechanism.

[0017] Figure 3 is a schematic diagram of the gradient compression and quantization process based on Top-k.

[0018] Figure 4 is a schematic diagram of the server aggregation and model update process. Detailed Implementation

[0019] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these examples are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0020] An adaptive differential privacy-preserving and communication compression joint optimization method for industrial federated learning, as shown in Figure 1-4, includes the following steps: Step 1, Local training and gradient calculation: The server initializes the global model. And set parameters, including the total number of clients N and the local dataset. Local training rounds E, total privacy budget , The target pruning threshold C, the sparsity ratio k%, and the number of quantization bits b are defined. In the t-th round of training, the server randomly selects a set of clients. Distribution model To the selected client. Each client Using local datasets Perform stochastic gradient descent training for E epochs and compute the local gradient. The training process employs mini-batch gradient descent to improve efficiency.

[0021] Step 2, Differential Privacy Protection: Client-side local gradient Perform L2 norm clipping, the formula is as follows: This ensures that the gradient norm does not exceed the threshold C, thereby limiting the sensitivity boundary.

[0022] Figure 2, illustrating the adaptive noise injection mechanism, demonstrates the relationship between the noise standard deviation and the gradient norm. The client side represents the clipped gradient. Add adaptive Gaussian noise, noise standard deviation The calculation formula is ,in To reduce the total privacy budget The current round budget allocated in the middle, Take the preset constant The privacy-preserving gradient is obtained after adding noise. This mechanism dynamically adjusts the noise level to optimize the signal-to-noise ratio.

[0023] Step 3, Gradient Compression: Client-side privacy-preserving gradient compression Compression processing is then performed; Figure 3 illustrates the gradient compression process, showing the specific operations of Top-k sparsity and quantization. First, Top-k sparsity is performed, retaining the top k% of elements with the largest absolute values, resulting in the sparse gradient. and location mask Sparsity reduction employs a fast selection algorithm to improve efficiency. Then, non-zero values ​​are quantized using b-bit quantization, with the quantization formula as follows: ,in Record the parameter scaling factor after quantization. and minimum value Compression effectively reduces the amount of communication data.

[0024] Then perform the following steps on non-zero values: Bit quantization: To avoid scale jitter caused by traditional min-max quantization under sparse gradient and noise injection conditions, this invention employs symmetric saturation quantization. Let... The quantization scaling factor is defined as follows:

[0025] in It is a very small constant used to avoid Division by zero occurs. The quantization formula is: in This indicates rounding. This indicates saturation truncation to a given, representable range. The scaling factor is recorded after quantization. And set the zero point offset This is to enable server-side dequantization and reconstruction.

[0026] Step 4, Upload and Dequantization Reconstruction: The client will compress the data. The data is uploaded to the server via encrypted communication; the server performs inverse quantization reconstruction based on the mask and quantization parameters, using the following formula: And combine with a mask to recover the gradient sparse structure.

[0027] Step 5, Gradient Aggregation: The server reconstructs gradients for all clients. Perform a weighted average to obtain the global gradient update. ,in The number of selected clients; then the learning rate is used. Update global model The learning rate can be dynamically adjusted according to the training progress.

[0028] Referring to Figure 4, the error feedback mechanism diagram illustrates how the server accumulates and compensates for gradient residuals. The server implements the error feedback mechanism by recording and accumulating the gradient residuals lost by each client i due to sparsity. The residuals will be accumulated during the next training round on this client. Local gradient calculations are incorporated to compensate for biases and improve convergence.

[0029] Step 6, Privacy Budget Allocation and Accounting: The server uses privacy accounting methods such as moment accounting or Gaussian difference privacy to calculate the privacy budget consumed in this round. And accumulate it into the total consumption of each client; when the accumulated budget approaches At that time, stop the client from participating in training to ensure that the overall privacy budget is not overused.

[0030] This invention is applicable to model training tasks under the conditions of industrial internet and edge collaborative learning, and can cope with practical constraints such as the difficulty of data leaving the domain, limited network bandwidth, and strict privacy compliance requirements. For example, in smart manufacturing production lines, vibration, current, temperature, and image data collected at each workstation or equipment can be used for bearing fault diagnosis, predictive maintenance, and visual quality inspection; in energy and park management, load, power, and environmental data collected by each edge node can be used for energy consumption prediction, anomaly detection, and equipment health assessment. Since such scenarios often face the problems of limited communication resources and high data sensitivity, this invention, under the premise of controlled privacy budget, coordinates differential privacy protection and gradient compression design, thereby effectively reducing communication overhead and improving the deployment feasibility and convergence stability of federated training.

[0031] This invention, after training is performed locally on the client side, first performs L2 norm clipping on the gradient to limit sensitivity, then injects adaptive Gaussian noise related to the gradient norm and privacy budget to achieve differential privacy protection of the gradient. Next, the noisy gradient is compressed using Top-k sparsity and low-bit quantization, and then the compressed data is uploaded to the server. The server performs dequantization, gradient aggregation, and optional error feedback, and tracks the privacy budget consumption of each client. The core of this invention lies in the joint design framework of "protection first, compression later," which concentrates noise on important gradient dimensions, avoiding noise waste, thereby achieving higher model accuracy and communication efficiency with the same privacy budget. The adaptive noise mechanism dynamically adjusts the amount of noise according to training, further optimizing the signal-to-noise ratio.

[0032] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An adaptive differential privacy-preserving and communication compression joint optimization method for industrial federated learning, characterized in that, The process includes the following steps: Step 1, Local Training and Gradient Calculation: Collect device data to construct a local dataset; Initialize the global model on the server. And related parameters, and then use the local dataset to perform local training and gradient calculation to obtain the local gradient. Step 2, Differential Privacy Protection: The client processes local gradients. Perform L2 norm clipping to obtain the clipped gradient. The following is the gradient after clipping. Adding adaptive Gaussian noise yields the privacy-preserving gradient with added noise. Step 3, Gradient Compression: Client-side privacy-preserving gradient compression. The data is compressed to obtain the compressed data; Step 4, Upload and Dequantization Reconstruction: The client uploads the compressed data to the server; The server dequantizes and reconstructs the gradient of each client based on the received mask and quantization parameters, thus obtaining the reconstructed gradient. Step 5, Gradient Aggregation: The server reconstructs gradients for all clients. Perform a weighted average to obtain the global gradient update. And update the global model using global gradients. ,in For the learning rate; Step 6, the server performs privacy budget allocation and accounting.

2. The adaptive differential privacy protection and communication compression joint optimization method for industrial federated learning according to claim 1, characterized in that: Step 3 involves the client compressing the privacy-preserving gradient, including: Step 31, Top-k sparsity: preserving... The sparse gradient is obtained by selecting the top k% of elements with the largest absolute values. and its location mask Step 32, Quantification: [This refers to a specific step or process, likely related to quantification or quantification. The non-zero values ​​in the data are quantized using b-bit quantization to obtain the quantized gradient. and quantization parameter scaling factor and minimum value Step 32: Obtain the location mask Quantized gradient Minimum value Build compressed data 。 3. The adaptive differential privacy protection and communication compression joint optimization method for industrial federated learning according to claim 1, characterized in that: Step 1, local training and gradient calculation, includes: In the t-th round of training, the server randomly selects a set of clients. and the current global model Distributed to selected clients; each client Using local datasets For the model Training is performed, and the local gradient is calculated. 。 4. The adaptive differential privacy protection and communication compression joint optimization method for industrial federated learning according to claim 1, characterized in that: The L2 norm clipping described in step 2 restricts the L2 norm of the gradient vector to within a threshold C, providing a defined sensitivity boundary for Gaussian noise injection. 。 5. The adaptive differential privacy protection and communication compression joint optimization method for industrial federated learning according to claim 1, characterized in that: The cutting formula in step 2 is: C is the threshold; the privacy-preserving gradient after adding noise. ， express, express.

6. The adaptive differential privacy protection and communication compression joint optimization method for industrial federated learning according to claim 1, characterized in that: The noise standard deviation of the adaptive noise injection mechanism described in step 2 From the formula Confirmed, among which Let L2 norm be the gradient after clipping. The privacy budget allocated for the current round.

7. The adaptive differential privacy protection and communication compression joint optimization method for industrial federated learning according to claim 1, characterized in that: Step 4 also includes an error feedback mechanism: the server records and accumulates the gradient residuals discarded by each client i due to sparsification. The residuals will be accumulated during the next training round on the client. It is added to its local gradient calculation.

8. The adaptive differential privacy protection and communication compression joint optimization method for industrial federated learning according to claim 1, characterized in that: Step 6, which involves the server allocating and accounting for the privacy budget, includes: the server using privacy accounting methods to calculate the privacy budget consumed in this round. This is accumulated into the total consumption of each client; when the client's accumulated privacy budget approaches... When this occurs, their participation in training is suspended; privacy accounting methods include moment accounting or Gaussian difference privacy methods.

9. A joint optimization system based on the adaptive differential privacy protection and communication compression joint optimization method for industrial federated learning as described in claim 1, characterized in that: The system includes an input unit, a local training and gradient computation unit, a differential privacy protection unit, a gradient compression unit, an upload and dequantization reconstruction unit, a gradient aggregation unit, a privacy budget allocation and accounting unit, and an output unit. The input unit is used to construct a local dataset from input device data; the local training and gradient computation unit is used for server-side global model initialization. And related parameters, and then use the local dataset to perform local training and gradient calculation to obtain the local gradient. The differential privacy protection unit is used by the client to monitor local gradients. Perform L2 norm clipping to obtain the clipped gradient. The following is the gradient after clipping. Adding adaptive Gaussian noise yields the privacy-preserving gradient with added noise. The gradient compression unit is used by the client to compress the privacy-preserving gradient. The data is compressed to obtain compressed data; the upload and dequantization reconstruction unit is used by the client to upload the compressed data to the server; the server performs dequantization reconstruction on the gradient of each client according to the received mask and quantization parameters to obtain the reconstructed gradient. The gradient aggregation unit is used by the server to reconstruct gradients for all clients. Perform a weighted average to obtain the global gradient update. And update the global model using global gradients. ,in The learning rate is used; the privacy budget allocation and accounting unit is used by the server to allocate and account for the privacy budget.

10. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that... When the processor executes the program, it implements the adaptive differential privacy protection and communication compression joint optimization method for industrial federated learning as described in any one of claims 1 to 8.

Images