Methods and systems for federated learning

Abstract

Methods, systems, and apparatuses for federated learning employing differential privacy (DP)-inherent quantization are disclosed, comprising computer programs encoded on a computer storage medium. An exemplary method may include: obtaining a parameter vector of a local model; updating the parameter vector of the local model by adding a plurality of noise vectors to the parameter vector of the local model; quantizing the updated parameter vector to obtain a quantized parameter vector, wherein the quantization includes mapping coordinates in the updated parameter vector to a set of discrete finite values; and sending the quantized parameter vector and at least one of the plurality of noise vectors to a server for the server to update a global model.

Description

Technical Field

[0001] This disclosure generally relates to systems and methods for maximizing risk detection coverage under constraints, and in particular to the inherent quantization of Differentially Private (DP) for federated learning. Background Technology

[0002] Federated learning (also known as collaborative learning) is a machine learning technique that trains algorithms on multiple decentralized edge devices (e.g., devices providing access to the core network of an enterprise or service provider) or on servers holding local data samples, without exchanging these data between edge devices. During training, many clients train customized machine learning models individually on independent devices and then send their local updates (e.g., model gradients or model parameters) to a trusted server. The server then aggregates these local updates to compute a global update and sends the global update back to the clients for model synchronization. However, existing federated learning faces several challenges. In an example of the first challenge, even if the server is secure and trusted, global updates sent from the server to the clients can leak information about other clients' data (also known as differential attacks). A client could potentially reconstruct other clients' training data accurately at high resolution with only the model gradients sent from the server. As another example of the second challenge, transmitting local updates to the server in each round can have high communication costs as the model size increases. Since local updates can be large in volume, their transmission consumes significant communication bandwidth. If insufficient bandwidth is provided, time delays occur, extending the overall training time of the model.

[0003] For the first challenge, existing solutions typically introduce differential privacy protection on the server side for global updates to prevent individual clients from reconstructing user data for other clients based on received global updates. For the second challenge, existing solutions typically apply compression on the client side to compress local updates, reducing the amount of data in local updates before transmitting them to the server. However, compression alone does not solve the first challenge, nor does differential privacy protection (DP protection). Furthermore, combining DP protection and compression is technically challenging because embedding known DP protection mechanisms into known quantization processes is almost impractical. For example, to quantize the degree of DP protection, common DP protection is achieved by introducing noise extracted from Gaussian or Laplacian distributions. The noise generated by these distributions ranges from negative infinity to positive infinity, which conflicts with quantization processes that require a finite range of discrete values. Furthermore, combining compression and DP protection may introduce a degree of randomness / noise into the data, which could significantly reduce the accuracy of federated learning models. Summary of the Invention

[0004] Various embodiments of this specification may include systems, methods, and non-transitory computer-readable media for federated learning (FL) employing differential privacy (DP) inherent quantization.

[0005] On one hand, the FL method using DP intrinsic quantization may include: obtaining the parameter vector of a local model by a terminal device; updating the parameter vector of the local model by adding multiple noise vectors to the parameter vector of the local model by the terminal device; quantizing the updated parameter vector by the terminal device to obtain a quantized parameter vector, wherein the quantization includes mapping the coordinates in the updated parameter vector to a set of discrete finite values; and sending the quantized parameter vector and at least one noise vector from the multiple noise vectors to a server for the server to update the global model.

[0006] In some embodiments, the server is coupled to each of a plurality of terminal devices; the plurality of terminal devices includes the terminal devices; and the server and the plurality of terminal devices form a federated learning network.

[0007] In some embodiments, obtaining the parameter vector of the local model includes: receiving the parameter vector of the global model from the server by the terminal device; obtaining a gradient for each data record in the multiple data records of the terminal device based on the loss function of the local model, the parameter vector of the global model, and each data record of the multiple data records; and obtaining the parameter vector of the local model by aggregating multiple gradients corresponding to the multiple data records.

[0008] In some embodiments, obtaining the parameter vector of the local model further includes: for each of the plurality of data records, using a clipping parameter to clip the gradient, so as to map the gradient to the range corresponding to the clipping parameter.

[0009] In some embodiments, obtaining the parameter vector by aggregating multiple gradients includes: determining the parameter vector of the local model based on the number of the multiple data records and the aggregation of the multiple gradients.

[0010] In some embodiments, the plurality of noise vectors includes a first noise vector and a second noise vector; the first noise vector includes the sum of a plurality of independent random variable vectors, each of which follows a uniform distribution; the second noise vector includes a jitter vector that randomizes the quantization error of the quantization.

[0011] In some embodiments, the second noise vector follows a uniform distribution and is independent of the first noise vector and the parameter vector of the local model.

[0012] In some embodiments, sending the quantized parameter vector and at least one noise vector among the plurality of noise vectors includes: sending the quantized parameter vector and the second noise vector to the server.

[0013] In some embodiments, quantizing the updated parameter vector includes: determining a global quantization hyperparameter for the terminal device; determining an increment based on the global quantization hyperparameter and the number of the multiple data records; and performing the quantization based on the space determined according to the increment.

[0014] In some embodiments, the quantization based on the space determined according to the increment includes: determining the number of quantization levels according to the increment; and performing the quantization according to the range and the number of levels.

[0015] In some embodiments, obtaining the parameter vector of the local model based on multiple data records of the terminal device includes: sampling a subset of the multiple data records collected by the terminal device; and training the local model based on the subset of the multiple data records to obtain the parameter vector of the local model.

[0016] In some embodiments, sampling a subset of multiple data records includes sampling the subset based on a Poisson sampling rate.

[0017] According to another aspect, a system for FL employing DP intrinsic quantization may include: one or more processors and one or more non-transitory computer-readable storage devices coupled to the one or more processors and configured with instructions executable by the one or more processors to cause the system to perform operations, the operations including: obtaining a parameter vector of a local model; updating the parameter vector of the local model by adding a plurality of noise vectors to the parameter vector of the local model; quantizing the updated parameter vector to obtain a quantized parameter vector, the quantization including mapping coordinates in the updated parameter vector to a set of discrete finite values; and sending the quantized parameter vector and at least one of the plurality of noise vectors to a server for the server to update a global model.

[0018] According to another aspect, a non-transitory computer storage medium for FL employing DP intrinsic quantization can be configured with instructions executable by one or more processors to cause the one or more processors to perform operations, the operations including: obtaining a parameter vector of a local model; updating the parameter vector of the local model by adding a plurality of noise vectors to the parameter vector of the local model; quantizing the updated parameter vector to obtain a quantized parameter vector, the quantization including mapping coordinates in the updated parameter vector to a set of discrete finite values; and sending the quantized parameter vector and at least one of the plurality of noise vectors to a server for the server to update a global model.

[0019] The embodiments disclosed in this specification have one or more technical effects. In some embodiments, DP protection is implemented as part of a quantization process on the client side. This design may also be referred to as DP-inherent quantization. DP-inherent quantization offloads the process of providing DP protection from the server to the client. This effectively reduces the workload on the server side and distributes that workload to multiple clients. Furthermore, DP-inherent quantization provides stronger privacy protection because data is DP protected before being transmitted out of the client. It can reduce the likelihood of data privacy breaches even if the server is configured in a certain way. Some embodiments in this application describe a quantization process that inherently provides DP protection. In some embodiments, the DP-inherent quantization described herein includes a way of combining DP protection with quantization and makes it possible to provide a mathematical way to quantize the level of DP protection. In some embodiments, the client-side quantization process can effectively reduce the amount of locally updated data representation to be transmitted from the client to the server for aggregation. In some embodiments, avoiding the step of implementing DP protection on the server side can effectively reduce the additional noise introduced into the federated learning (FL) model. DP-inherent quantization on the client side can be explicitly represented as the sum of all local quantization noise on the server side, which leads to a novel, inherent generalized Irwin-Hall (GIH) mechanism for DP protection at the data sample level. By adjusting some hyperparameters, the described embodiments can tune the quantization bit depth and the strength of DP protection. To fully utilize the inherent noise of quantization, some embodiments described herein propose a DP analysis of the GIH mechanism for FL training and derive tractable expressions for privacy guarantees for the T-iterations used in distributed stochastic gradient descent. In some embodiments, DP-intrinsic quantization offers the following advantages: (i) providing an unbiased estimate of the original local update; (ii) fully utilizing the corresponding estimation error for DP protection; and (iii) integrating DP amplification into the GIH-DP analysis through subsampling of the training data. Therefore, DP-intrinsic quantization can achieve significant performance improvements over separate quantization and DP protection while maintaining the same privacy guarantees and communication efficiency.

[0020] These and other features of the systems, methods, and non-transitory computer-readable media disclosed herein, as well as the functionality of the elements of the operating methods and related structures, and the economics of the combination of components and manufacture, will become more apparent upon consideration of the following description and appended claims with reference to the accompanying drawings. All these drawings form a part of this specification, wherein like reference numerals denote corresponding portions in the respective drawings. However, it should be clearly understood that the drawings are for illustration and description only and are not intended to limit the invention. Attached Figure Description

[0021] Figure 1 An exemplary framework for federated learning (FL) according to some embodiments is shown.

[0022] Figure 2 An exemplary setup of FL employing DP-inherent quantization is shown according to some embodiments.

[0023] Figure 3 An exemplary flowchart of FL employing DP-inherent quantization is shown according to some embodiments.

[0024] Figure 4 An exemplary method for employing FL with inherent quantization of DP is shown according to some embodiments.

[0025] Figure 5 Another exemplary method for employing FL with inherent quantization of DP, according to some embodiments, is shown.

[0026] Figure 6 A block diagram of a computer system for FL employing DP-inherent quantization is shown according to some embodiments.

[0027] Figure 7 An example computing device is shown that can implement any of the embodiments described herein. Detailed Implementation

[0028] Machine learning models that unleash their full potential using massive amounts of training data have achieved success in various applications, such as computer vision, speech recognition, and disease diagnosis. In traditional machine learning operations, training data from clients (such as individual customers or business entities like technology companies, banks, and hospitals) is aggregated on a central server used for model training. However, limited communication bandwidth restricts the efficiency of data transmission. More importantly, in many cases, collecting sensitive data (typing history, location tracking, personal photos, etc.) is inappropriate due to privacy and security requirements.

[0029] To address these issues, Federated Learning (FL) has been proposed, whereby multiple clients train customized machine learning models independently on separate devices and then send their local updates (e.g., model gradients or model parameters, collectively referred to as parameter vectors) to a trusted server. The server then aggregates these updates to compute a global update. Federated Learning enables edge devices, such as mobile phones, to collaboratively learn shared predictive models while keeping the training data on that device, decoupling the ability to perform machine learning from the need to store data in the cloud.

[0030] However, although raw data is not directly sent to the server in FL, the protocol remains vulnerable to differential attacks, which could originate from any party contributing during federated optimization. In such attacks, information about a client's contributions during training (e.g., local updates) and the data they used for the training set can be revealed by analyzing the distributed model. Therefore, when the server directly broadcasts global updates, a client can glean information about other clients' data from the received global updates. A practical method to prevent this data leakage is to randomly perturb the global updates on the server side; this is known as Differential Privacy (DP). Due to provable privacy protection, DP has become the de facto standard for private data publishing, regardless of the adversary's background knowledge and computational power. However, while adding noise to the data can prevent privacy leaks, a significant amount of additional noise can significantly reduce the model's accuracy.

[0031] Furthermore, in FL, exchanging model updates in each round incurs high communication costs as the model size increases, especially for clients uploading local updates to the server. For example, for medium-sized models... PennTreeBank The language model has 38MB of parameters (assuming 32-bit floating-point); generally... ResNet-50 The model has 97MB of parameters and 24 layers for Natural Language Processing (NLP) tasks. BERT The model size reaches 340MB. Communication costs may increase further due to the need for more updates and iterations before the training model converges. To overcome the communication bandwidth challenge, compression techniques such as quantization have been extensively studied for peer-to-peer communication and distributed learning settings.

[0032] However, both server-side DP protection and client-side quantization can introduce noise or errors that significantly degrade statistical performance. In various embodiments of this disclosure, DP-inherent quantization is described to improve data transmission efficiency and training accuracy in federated learning.

[0033] Figure 1 An exemplary framework for federated learning (FL) according to some embodiments is shown. Federated learning (FL) enables multiple participants to build general, robust machine learning models without sharing data.

[0034] like Figure 1As shown, in some embodiments, the FL framework may include a server 100 and multiple clients 102 or terminal devices, communicating via a communication channel between the clients 102 and the server 100. The server 100 may be coupled to each of the multiple clients 102 to form a federated learning network. In some embodiments, the server 100 may include a computing system or computing device. It should be understood that, although... Figure 1 The diagram illustrates a server 100, and any number of computing devices can work together and be considered as server 100. Server 100 can be implemented in one or more networks (e.g., an enterprise network), one or more endpoints, one or more data centers, or one or more clouds. Server 100 may include hardware or software that manages access to centralized resources or services within the network. A cloud may include clusters of servers and other devices distributed across a network.

[0035] In some embodiments, client 102 may include various types of terminal devices, such as mobile phones, tablets, servers, desktop computers, laptops, etc. Server 100 can communicate with client 102 and other computing devices. Communication between devices can be achieved via the Internet, local area networks (e.g., Local Area Networks (LANs)), or direct communication (e.g., Bluetooth). TM (and other methods such as radio frequency and infrared) are used.

[0036] FL relies on an iterative process that can be decomposed into a set of atomic interactions between clients and servers, called federated learning rounds. Each round involves transmitting the current global model state to the participating clients 102, training local models on these clients 102 to produce a set of potential model updates, then aggregating these local updates and processing them into a single global update, which is then applied to update the global model in server 100. Figure 1 As shown, the central server 100 aggregates and processes local updates, while the local client 102 performs local training according to the commands of the central server 100.

[0037] In some embodiments, server 100 can be configured to perform global machine learning model initialization, local update aggregation and model synchronization, and differential privacy (DP) protection. Global machine learning model initialization can be considered the starting point of a federated learning (FL) process. For example, a global machine learning model trained on server 100 (e.g., linear regression, neural network, boosting) can be selected, and some initial parameters (also called parameter vectors) 103 of the global machine learning model can be broadcast to client 102 for the first round of FL training. The parameter vectors of the global machine learning model can include global parameters or gradients 103 of the global machine learning model. After client 102 performs local training based on global parameters 103 and locally collected training data, client 102 can send local updates 104 back to server 100 for aggregation and model synchronization.

[0038] After receiving local updates from client 102, server 100 can aggregate these updates to perform a global update of the global machine learning model. The global update can include the aggregation of local updates from client 102 and model synchronization based on the global machine learning model and local updates 104. This process can generate a new set of global parameters, which improves the performance of the global machine learning model.

[0039] On the client 102 side, each client 102 can obtain a local update after local training based on global parameters or gradients 103 and local training data. The local update can include the gradients or model parameters of the locally trained model. In this disclosure, "parameters or gradients of the local / global model" can also be referred to as "parameter vector of the local / global model." The size of the local update can increase with the size of the locally trained model. To reduce the data transfer cost of local updates from client to server, the local update may undergo a quantization process to map continuous infinite values ​​to a smaller set of discrete finite values. For example, each floating parameter in the local update can be represented using 32 bits. After quantization, the floating parameter can be mapped to a smaller set of values ​​that can be represented using 4 bits. That is, quantization can improve transmission efficiency by 8 times, thereby reducing the requirement for communication bandwidth.

[0040] In some embodiments, server 100 and client 102 may be further configured to perform other suitable operations. For example, server 100 may perform client selection to determine which clients 102 participate in the current round of FL learning. Server 100 and / or client 102 may also perform data encryption / decryption, training data sampling (e.g., subsampling mini-batch), fault recovery (e.g., handling disconnected clients or lost model updates), flow control (e.g., starting and terminating FL learning), other suitable tasks, or any combination thereof. For example, different clients (e.g., terminal devices) may be selected to participate in each round of FL.

[0041] Figure 2 An exemplary setup for a flow graph (FL) employing DP-intrinsic quantization according to some embodiments is shown. In some embodiments, quantization on client 202 during FL learning can intrinsically provide DP protection for local data before it is transmitted to server 200. This DP-intrinsic quantization can maintain the same amount of noise added by a vanilla quantization process while providing DP protection. In this way, server 200 does not need to introduce additional noise during FL learning, thus providing DP protection in a centralized manner.

[0042] In some embodiments, at the start of a round of FL training, server 200 can send the weights 204 of the global machine learning model to multiple selected clients 202 for local training. Each client 202 can obtain the gradient vector (also called the parameter vector) of its local machine learning model and train the local machine learning model based on multiple data records from the terminal device and the weights 204 of the global machine learning model. Before sending the gradient vector back to server 200, client 202 can obtain an updated gradient vector by adding multiple noise vectors to it. In some embodiments, each of the multiple noise vectors can follow a uniform distribution. Subsequently, client 202 can quantize the updated gradient vector to obtain a quantized gradient vector 203. The quantization operation maps the coordinates in the updated gradient vector to a set of discrete finite values, so that the values ​​in the updated gradient vector can be represented with fewer bits, thus reducing the amount of data to be transmitted to the server.

[0043] In some embodiments, after quantization, client 202 can transmit the quantized gradient vector 203 and at least one of multiple noise vectors to server 200 for further training of the global machine learning model. For example, server 200 can securely aggregate multiple quantized gradient vectors 203 received from multiple participating clients 202 and synchronize the model to update the weights / parameters of the global machine learning model. Here, "secure aggregation" means that server 200 requires a weighted average of the local updates 203 from the clients for a global update, without learning any individual updates. For example, to perform secure aggregation at server 200, a Diffie-Hellman key exchange protocol can be implemented to create a one-time pad between each pair of client updates 203. Client dropouts during the FL process can also be addressed by secretly sharing the aforementioned one-time pad between clients. Other encryption techniques, such as homomorphic encryption and function encryption, can also be implemented for secure aggregation. As another example, a Trusted Execution Environment (TEE) such as SGX (Intel Software Guard Extensions) can provide another secure and efficient solution, allowing server 200 to perform weighted average calculations in SGX without accessing updates 203 on each client.

[0044] In some embodiments, training the global machine learning model on the server 200 side can aim to minimize the loss function.

[0045]

[0046] Where, w∈R d This refers to local model parameters (e.g., real gradients received from client 202). It refers to the first k The client's loss function It refers to the first k The client's sample size, where N refers to the total number of training samples, i.e., The sample size here is in the first... k The number of training records sampled on the client side. Without loss of generality, the following description assumes the following: The 202nd client participated in the... t FL training rounds. Each client. Holding distributed datasets Local data records .

[0047] In some embodiments, in the t There are two steps in round FL training: (1) on the client side It can be based on quantitative operations Compressed local computation of stochastic gradients (For example, a vector of scalar gradients), and then the resulting quantized gradients can be obtained. 203 is transmitted to the server; and (2) the server can aggregate. To perform model synchronization, thereby obtaining the updated parameters / weights of the global model. 204, and will A 204 broadcast is sent to all clients, resulting in a 202 error.

[0048] If for any pair of adjacent datasets And for the output range All events The conditions are met. ,in, Representing probability, the random algorithm yes - Differential privacy (DP). When hour, It can be called . This can be used to represent the strength of DP protection. Figure 3 Describes the intrinsic quantization of DP and the corresponding -DP analysis.

[0049] Figure 3 An exemplary flowchart of FL employing DP-inherent quantization is shown according to some embodiments. Figure 3 The flowcharts in the document include operations performed on the client side 310 and operations performed on the server side 320. These operations are for illustrative purposes. Depending on the implementation, the client side 310 and / or the server side 320 may perform more, fewer, or alternative operations in various orders or in parallel. For example, the server side 320 may select clients to participate in a round of FL training. In the following description, the terms "client" and "terminal device" are used interchangeably to refer to a computing device performing local training within the FL framework. Example devices may include mobile phones, tablets, servers, desktop computers, laptops, etc.

[0050] In some embodiments, client 310 may receive the weights of the global machine learning model from server 320 in step 302. If this round of FL training is the first round of training, the weights may be initial weights, or if some rounds of FL training have already occurred, the weights may be partially trained weights.

[0051] In some embodiments, in step 303, client 310 may acquire the gradient vector of a local machine learning model trained based on multiple data records from client 310 and received weights. The multiple data records may include user data collected by client 310 and not shared with other clients or servers 320. That is, the training data comes only from data records collected from client 310. The training process performed by client 310 may include: receiving the parameter vector of the global model (e.g., multiple weights); obtaining the gradient for each of the multiple data records from client 310, based on the loss function of the local model, the parameter vector of the global model, and each of the multiple data records; and obtaining a gradient vector by aggregating the multiple gradients corresponding to the multiple data records. In some embodiments, the gradients in the gradient vector may undergo a pruning operation, which maps the gradients to a range corresponding to pruning parameters. This step can suppress the influence of locally trained gradient vectors on the global model, thereby making the learning process more robust.

[0052] In some embodiments, in step 304, client 310 may perform DP intrinsic quantization on the updated gradient vector to obtain a quantized gradient vector. DP intrinsic quantization includes mapping the coordinates in the updated gradient vector to a set of discrete finite values. Furthermore, to provide a controllable DP protection protocol, DP intrinsic quantization is designed to adapt to the local subsampling size of the data record on each client in each round of FL training. In some embodiments, DP intrinsic quantization includes: determining a global quantization hyperparameter for client 310; determining an increment (delta) based on the global quantization hyperparameter and the number of multiple data records; and quantizing based on a space determined according to the increment. The increment can here define a quantization space with multiple quantization steps. For example, if the quantization range is -5 to 5 and the increment is 2, the quantization space can be defined using the following quantization steps: (-5, -3), (-3, -1), (-1, 1), (1, 3), and (3, 5). During quantization, floating-point numbers (e.g., 32 bits) within a quantization step can be mapped to corresponding integers (e.g., 5 quantization steps can correspond to 5 integers, which can be represented with as few as 3 bits). In some embodiments, the quantization process includes: determining the number of quantization levels based on an increment; and performing quantization based on the range and the number of levels.

[0053] This paper describes an example design of DP intrinsic quantization, which has the following representation: global quantization hyperparameters Used for control In the round of training on the client The local quantization step size (also known as the client-specific increment). , Indicates the first t In the round of training on the client The size of the subsampled data record Representation and quantization step size The number of levels of the relevant uniform quantizer, and Indicates having DP's inherent quantization. That is, quantization is based on the client-specific increment. Determined range and number of levels of the uniform quantizer This is executed. In some embodiments, using the above representation, it can be based on the global quantization hyperparameters. and multiple data records Quantity determined For example, through . The first in i The coordinates satisfy the following inequalities.

[0054]

[0055] in, (Integer) (Real numbers) are The i Coordinates. To simplify the notation, Q can be used instead where there is no confusion. Thus, this quantification can be expressed as:

[0056]

[0057] However, directly applying the above quantization will lead to irreversible quantization errors / noise. In some embodiments, the jitter noise vector can be... Added to quantization to randomize the quantization effect. Jitter noise. Can be independent of And it follows a uniform distribution. In some embodiments, the jitter noise vector It can include multiple scalar values, which are respectively added to Each coordinate in the array.

[0058] In some embodiments, in addition to jitter noise In addition, an additional noise vector can be added. Added to quantized inputs to further enhance privacy protection. In some embodiments, It can be the sum of Z independent random variables, each with a uniform distribution, i.e. Where Z is a hyperparameter used to control the strength of privacy. When jitter noise... When there is enough, it can Set to 0. (This is the correct setting.) The quantized input can be represented as .

[0059] In quantization using a subtractive dither quantizer, dither noise needs to be subtracted from the quantizer output to produce the system output, i.e. For this reason, in some embodiments, the client 310 may, in step 305, transfer the gradient vector (e.g., ) and jitter noise All data is sent to server 320 so that server 320 can determine the system output (quantization result).

[0060] Using the above symbols, the quantization error is... , defined as the difference between the quantizer output and input, can be expressed as The noise perturbation added to the gradient vector of 310 on the client side is This means that the local DP inherent quantization on the client side is 310. Equivalent to The linear mapping of DP. Therefore, DP's inherent quantization is fully compatible with the one-time padding protocol used for secure aggregation implemented on the server side 320.

[0061] In some embodiments, jitter noise Can be designed to limit , making and They are statistically independent. To achieve this goal, in some embodiments, The scope of operation is represented as [ ], can be based on It is confirmed that, among them, Z is a constant parameter and is an additional noise vector. The number of independent random variables in the system. In some embodiments, once determined... It can be based on incremental and range Determine the number of quantification levels, for example Using this DP-inherent quantization, the representation of floating-point values ​​(32 bits) can be reduced to [amount missing]. Bit.

[0062] In some embodiments, after the server receives the gradient vector and jitter noise from multiple clients in step 306, the server may first determine the quantization result by subtracting the jitter noise from the gradient vector in step 307, and then perform secure aggregation and update the weights of the global machine learning model in step 308. An example of secure aggregation can be represented as follows: The weights of a global machine learning model can be determined through... Update, in which, This refers to the learning rate. Subsequently, server 320 can broadcast the updated weights to the client in step 309. This will begin a new round of local training. The client that participated in the previous round can be the same as or different from the client that participates in the new round.

[0063] Figure 4 An exemplary method 400 for learning FL with DP-intrinsic quantization according to some embodiments is illustrated. Method 400 is described in pseudocode format for learning FL with DP-intrinsic quantization. The steps of method 400 are for illustrative purposes. Depending on the implementation, method 400 may include more, fewer, or alternative steps performed in various orders.

[0064] exist Figure 4 First, let's define some representations: This refers to a clipping operation with a boundary parameter C. This refers to the client side. The t Gradient vectors in round learning; Refers to the client The t In round learning, uniform noise is added before quantization; Q[] refers to the quantization operation; PS() refers to the subsampling operation; Refers to the client The t jitter noise vector in round learning; This refers to the global machine learning model trained through the FL learning process; Z refers to the amount of uniform noise added before quantization. This refers to the global parameters that control the quantization level; and This refers to the client side. The t The quantization step size used in round-robin learning.

[0065] In some embodiments, FL learning can begin by initializing the weights of a global machine learning model, which are represented as follows: This initialization step can be performed on the server. FL learning can involve multiple rounds of learning, represented as... Each round involves multiple clients. In the first round... t During round-robin learning, client k can be randomly selected to compute mini-batches obtained from subsampling of multiple data records collected from client k. In some embodiments, an independent but identical Poisson sampler can be used on different clients. This has the advantage of being equivalent to sampling the entire dataset across all clients using the same Poisson sampler. The Poisson sampler follows a Poisson sampling rate. For example, sampling... By targeting Independent sampling To output a subset of data On the client side The t The sample size in round learning can be determined by As shown in line 3 of method 400.

[0066] Line 4 of method 400 includes local gradient computation at client k. This computation process may include: for each of multiple subsampled data records (e.g., ), based on the client's loss function (e.g., ), the parameter vector of the global machine learning model (e.g., multiple weights) (e.g., ) and data records Obtaining the gradient; and obtaining a gradient vector by aggregating multiple gradients. In some embodiments, the gradient may be subjected to a clipping operation with a boundary parameter C (e.g., The boundary parameter C defines the clipping range, which the clipping operation can map gradients outside the clipping range to the clipping range. As shown in line 4 of method 400, this can be based on the aggregation of multiple gradients (e.g., in line 4). ) and the number of multiple data records (e.g., in row 4) Calculate the local gradient vector .

[0067] In some embodiments, multiple noise vectors can be added to the local gradient. Multiple noise vectors may include the first noise vector. Second noise vector First noise vector It can include the sum of multiple independent random variable vectors, each of which follows a uniform distribution. Second noise vector. It can include a dither vector, used for the quantization error of randomized quantization.

[0068] After adding multiple noise vectors, the resulting value ( The gradient vector is fed into the quantizer Q[] to obtain the quantized gradient vector. The client can then use this quantized gradient vector... With the second noise vector They are sent together to the server for aggregation and model synchronization.

[0069] Once the server receives the quantized gradient vector Second noise vector The server can first subtract the second noise vector from the quantized gradient vector. Determine the quantification result, for example, through The result of the subtraction operation can be called the quantized output (from the client). The server can then perform secure aggregation and model synchronization to update the parameters of the global machine learning model. If there is a next round of FL learning, the updated parameters can be broadcast to the clients participating in the next round.

[0070] Figure 5 Example methods for FL employing DP-inherent quantization are illustrated according to various embodiments. Method 500 can be performed by a device, apparatus, or system for FL. Method 500 can be... Figures 1 to 4 To execute, for example, one or more modules / components in the environment or system shown, such as Figure 3 The client side 310 or server side 320 are described below. The operation of method 500 is intended to illustrate the concept. According to embodiments, method 500 may include additional, fewer, or alternative steps performed in various orders or in parallel.

[0071] Block 510 includes obtaining the parameter vector of a local model trained based on multiple data records of the terminal device by the terminal device. In some embodiments, obtaining the parameter vector of the local model includes: the terminal device receiving the parameter vector of the global model from the server; for each of the multiple data records collected by the terminal device, obtaining a gradient based on the loss function of the local model, the parameter vector of the global model, and each of the multiple data records; and obtaining the parameter vector by aggregating multiple gradients corresponding to the multiple data records. In some embodiments, obtaining the parameter vector of the local model further includes: for each of the multiple data records, performing a pruning operation on the gradient using a pruning parameter to map the gradient to the range corresponding to the pruning parameter. In some embodiments, obtaining the parameter vector by aggregating multiple gradients includes: determining the parameter vector of the local model based on the number of multiple data records and the aggregation of multiple gradients. In some embodiments, obtaining the parameter vector of the local model based on multiple data records of the terminal device includes: sampling a subset of the multiple data records collected by the terminal device; and training the local model based on the subset of multiple data records to obtain the parameter vector of the local model. In some embodiments, sampling a subset of multiple data records includes sampling the subset based on a Poisson sampling rate.

[0072] Block 520 includes updating the parameter vector of a local model by the terminal device by adding a plurality of noise vectors to the parameter vector of the local model, wherein each of the plurality of noise vectors follows a uniform distribution. In some embodiments, the plurality of noise vectors includes a first noise vector and a second noise vector; the first noise vector includes the sum of a plurality of independent random variable vectors, each of the plurality of independent random variable vectors following a uniform distribution; and the second noise vector includes a jitter vector of quantization error from randomized quantization. In some embodiments, the second noise vector follows a uniform distribution and is independent of the first noise vector and the parameter vector of the local model.

[0073] Block 530 includes quantizing the updated parameter vector by the terminal device to obtain a quantized parameter vector, wherein the quantization includes mapping the coordinates in the updated parameter vector to a set of discrete finite values. In some embodiments, quantizing the updated parameter vector includes: determining a global quantization hyperparameter for the terminal device; determining an increment based on the global quantization hyperparameter and the number of multiple data records; and performing quantization based on the space determined according to the increment. In some embodiments, performing quantization based on the space determined according to the increment includes: determining the number of quantization levels according to the increment; and performing quantization according to the range and the number of levels.

[0074] Block 540 includes sending a quantized parameter vector and at least one noise vector from a plurality of noise vectors to a server for the server to train a global model. In some embodiments, sending the quantized parameter vector and at least one noise vector from a plurality of noise vectors includes: sending the quantized parameter vector and a second noise vector to the server.

[0075] In some embodiments, the server is coupled to each of a plurality of terminal devices; the plurality of terminal devices includes the terminal device; and the server and the plurality of terminal devices form a federated learning network.

[0076] Figure 6 A block diagram of a computer system for FL employing DP-inherent quantization is shown according to some embodiments. The computer system 600 may be... Figures 2 to 3 One or more modules in a computing system, or, Figures 1 to 5 Examples of implementations of one or more other components are shown. Figure 5Method 500 can be implemented by computer system 600. Computer system 600 may include one or more processors and one or more non-transitory computer-readable storage media (e.g., one or more memories), said storage media being coupled to the one or more processors and configured with instructions executable by the one or more processors to cause a system or device (e.g., a processor) to perform the methods described above (e.g., method 500). Computer system 600 may include various units / modules corresponding to said instructions (e.g., software instructions).

[0077] In some embodiments, the computer system 600 may be referred to as an apparatus for FL employing DP intrinsic quantization. The apparatus may include an acquisition module 610 for acquiring an initialized global model from a server by a terminal device; a local training module 620 for acquiring parameter vectors of a local model trained based on multiple data records from the terminal device and the initialized global model; a DP intrinsic quantization module 630 for updating the parameter vector by adding multiple noise vectors (each of the noise vectors following a uniform distribution) to the parameter vector, and quantizing the updated parameter vector to obtain a quantized parameter vector (the quantization includes mapping the coordinates in the updated parameter vector to a set of discrete finite values); and a sending module 640 for sending the quantized parameter vector and at least one noise vector from the multiple noise vectors to the server for the server to train the global model. In some embodiments, the computer system 600 may further include a continued training module 650 for receiving the updated global model from the server and performing a new round of local training using the above modules. An updated global model can be obtained by server aggregation (e.g., using secure aggregation) of multiple quantized parameter vectors from multiple terminal devices and model synchronization.

[0078] The techniques described herein are implemented by one or more dedicated computing devices. A dedicated computing device can be a desktop computer system, a server computer system, a portable computer system, a handheld device, a network device, or any other device or combination of devices that combines hardwiring and / or program logic to implement the techniques. A dedicated computing device can be implemented as a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or a combination thereof. Computing devices are typically controlled and coordinated by operating system software. Traditional operating systems control and schedule computer processes for execution, perform memory management, provide file systems, networking, I / O services, and provide user interface functions such as graphical user interfaces (“GUIs”). The various systems, devices, storage media, modules, and units described herein can be implemented in a dedicated computing device or one or more computing chips within one or more dedicated computing devices. In some embodiments, the instructions described herein can be implemented in a virtual machine on a dedicated computing device. When executed, the instructions can cause the dedicated computing device to perform the various methods described herein. A virtual machine can include software, hardware, or a combination thereof.

[0079] Figure 7 An example computing device is shown that can implement any of the embodiments described herein. The computing device can be used to implement... Figures 1 to 6 One or more components of the system and method shown. The computing device 700 may include a bus 702 or other communication mechanism for communicating information, and one or more hardware processors 704 coupled to the bus 702 for processing information. The hardware processor 704 may, for example, be one or more general-purpose microprocessors.

[0080] The computing device 700 may also include a main memory 707 coupled to a bus 702 for storing information and instructions executable by a processor 704, such as random access memory (RAM), cache, and / or other dynamic storage devices. The main memory 707 may also be used to store temporary variables or other intermediate information during the execution of instructions executable by the processor(s) 704. When stored in a storage medium accessible by the processor(s) 704, such instructions present the computing device 700 as a dedicated machine tailored to perform the operations specified in the instructions. The main memory 707 may include non-volatile media and / or volatile media. Non-volatile media may include, for example, optical discs or magnetic disks. Volatile media may include dynamic memory. Common forms of media may include, for example, floppy disks, floppy disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM and EPROM, FLASH-EPROM, NVRAM, any other memory chips or memory cartridges, and their network versions.

[0081] Computing device 700 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware and / or program logic, which, when combined with a computer system, enables computing device 700 to be a dedicated machine or to be programmed as such. According to one embodiment, computing device 700 performs the techniques described herein in response to processor(s) 704 executing one or more sequences of one or more instructions contained in main memory 707. These instructions may be read into main memory 707 from another storage medium, such as storage device 709. Executing the sequence of instructions contained in main memory 707 causes processor(s) 704 to perform the processing steps described herein. For example, the processes / methods disclosed herein may be implemented by computer program instructions stored in main memory 707. When these instructions are executed by processor(s) 704, they may perform the steps as shown in the corresponding figures and described above. In alternative embodiments, hardwired circuitry may be used instead of or in combination with software instructions.

[0082] The computing device 700 also includes a communication interface 710 coupled to the bus 702. The communication interface 710 provides bidirectional data communication coupling for one or more network links connected to one or more local networks. As another example, the communication interface 710 may be a Local Area Network (LAN) card to provide data communication connectivity to a compatible LAN (or a WAN component communicating with a WAN). Wireless links may also be implemented.

[0083] The performance of certain operations can be distributed across processors, residing not only within a single machine but also deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., within a home environment, office environment, or server cluster). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0084] Each process, method, and algorithm described in the foregoing sections can be implemented in a code module executed by one or more computer systems or a computer processor including computer hardware, and can be fully or partially automated. The processes and algorithms can be implemented partially or fully in dedicated circuitry.

[0085] When the functions disclosed herein are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, can be embodied in the form of a software product. The software product can be stored in a storage medium and includes numerous instructions to cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or part of the steps of the methods in the embodiments of this application. The storage medium may include a flash drive, portable hard disk drive, ROM, RAM, magnetic disk, optical disk, another medium that can be used to store program code, or any combination thereof.

[0086] Specific embodiments also provide a system including a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the system to perform operations corresponding to the steps in any of the methods of the above embodiments. Specific embodiments also provide a non-transitory computer-readable storage medium configured with instructions executable by one or more processors to cause one or more processors to perform operations corresponding to the steps in any of the methods of the above embodiments.

[0087] The embodiments disclosed herein can be implemented through a cloud platform, server, or group of servers (collectively referred to as the "service system") that interacts with a client. The client can be a terminal device or a client registered by a user on the platform. The terminal device can be a mobile terminal, a personal computer (PC), or any device capable of installing the platform's applications.

[0088] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Furthermore, certain method or process blocks may be omitted in some implementations. The methods and processes described herein are not limited to any particular order, and the blocks or states associated with them may be executed in other suitable orders. For example, the described blocks or states may be executed in an order different from that specifically disclosed, or multiple blocks or states may be combined in a single block or state. Example blocks or states may be executed serially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently from those described. For example, elements may be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0089] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. This algorithm may include program code or instructions stored in memory (e.g., the aforementioned non-transitory computer-readable storage medium). Such an algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not be explicitly programmed into the computer to perform the function, but may learn from training data to create a predictive model that performs the function.

[0090] The various operations of the exemplary methods described herein can be performed at least in part by one or more processors, which are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute the engine of a processor implementation that runs to perform one or more of the operations or functions described herein.

[0091] Similarly, the methods described herein can be implemented at least in part by a processor, where one or more specific processors are examples of hardware. For example, at least some operations of a method can be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors can also operate to support the performance of the relevant operations in a “cloud computing” environment or as “Software as a Service” (SaaS). For example, at least some operations can be performed by a set of computers (as an example of a machine including processors) that are accessible via a network (e.g., the Internet) and one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0092] The performance of certain operations can be distributed across processors, residing not only within a single machine but also deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., within a home environment, office environment, or server cluster). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0093] Throughout this specification, multiple instances can implement components, operations, or structures described as single instances. While individual operations of one or more methods are shown and described as separate operations, one or more individual operations may be performed simultaneously, and they need not be performed in the order shown. Structures and functionalities presented as separate components in the exemplary configuration can be implemented as composite structures or components. Similarly, structures and functionalities presented as single components can be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of this document.

[0094] Although an overview of the subject matter has been described with reference to specific example embodiments, various modifications and changes may be made to these embodiments without departing from the broader scope of embodiments of this disclosure. Where more than one disclosure or concept is actually disclosed, these embodiments of the subject matter may be referred to herein individually or collectively by the term "invention," merely for convenience and not intended to actively limit the scope of this application to any single disclosure or concept.

[0095] The embodiments described herein have been described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, allowing for structural and logical substitutions and changes without departing from the scope of this disclosure. Therefore, the specific details should not be construed as limiting, and the scope of the various embodiments is defined only by the appended claims together with the full scope of their equivalents.

[0096] Any process description, element, or block described herein and / or depicted in the flowcharts in the accompanying drawings should be understood as potentially representing modules, segments, or code portions that include one or more executable instructions for implementing a particular logical function or step in a process. Alternative implementations are included within the scope of the embodiments described herein, in which elements or functions may be omitted, or performed not in the order shown or discussed (including substantially simultaneously or in reverse order), depending on the functionality involved, as understood by those skilled in the art.

[0097] As used herein, “or” is inclusive rather than exclusive unless explicitly stated otherwise or indicated by context. Therefore, here, “A, B, or C” means “A, B, A and B, A and C, B and C, or A, B and C” unless explicitly stated otherwise or indicated by context. Furthermore, “and” is both connective and quantitative unless explicitly stated otherwise or indicated by context. Therefore, in this document, “A and B” refers to “A and B” either connectively or quantitatively, unless explicitly stated otherwise or indicated by context. Moreover, multiple instances may be provided for a resource, operation, or structure described herein as a single instance. Furthermore, the boundaries between various resources, operations, engines, and data stores are somewhat arbitrary, and a particular operation is described in the context of a particular illustrative configuration. The allocation of other functionalities is conceivable and may fall within the scope of various embodiments of this disclosure. Generally, structures and functionalities presented as distributed resources in the example configuration can be implemented as composite structures or resources. Similarly, structures and functionalities presented as single resources can be implemented as distributed resources. These and other variations, modifications, additions, and improvements are all within the scope of the embodiments represented by the appended claims. Therefore, this specification and drawings should be considered illustrative rather than restrictive.

[0098] The terms “comprising” or “including” are used to indicate the presence of a feature subsequently stated, but do not preclude the addition of other features. Conditional language, such as “may,” “can,” “may,” or “may,” unless explicitly stated otherwise or otherwise understood in the context in which they are used, is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while others do not, and that certain features, elements, and / or steps are not included. Therefore, such conditional language is generally not intended to imply that features, elements, and / or steps are necessary in any way for one or more embodiments, or that one or more embodiments must include the ability to determine, with or without user input or prompting, whether such features, elements, and / or steps are included in any particular embodiment, or will be performed in any particular embodiment.

Claims

1. A method for federated learning, comprising: obtaining, by a terminal device, a parameter vector of a local model; updating, by the terminal device, the parameter vector of the local model by adding a plurality of noise vectors to the parameter vector of the local model; the plurality of noise vectors comprising: a first noise vector; the first noise vector comprising a sum of a plurality of independent random variable vectors, each of the plurality of independent random variable vectors following a uniform distribution; quantizing, by the terminal device, the updated parameter vector to obtain a quantized parameter vector, the quantizing comprising mapping coordinates in the updated parameter vector to a set of discrete finite values; and sending, by the terminal device, the quantized parameter vector and at least one noise vector of the plurality of noise vectors to a server for updating a global model by the server.

2. The method of claim 1, wherein: the server is coupled to each of a plurality of terminal devices; the plurality of terminal devices comprises the terminal device; and the server and the plurality of terminal devices form a federated learning network.

3. The method of claim 1, wherein, obtaining the parameter vector of the local model comprises: receiving, by the terminal device, a parameter vector of the global model from the server; for each data record of a plurality of data records of the terminal device, obtaining a gradient based on a loss function of the local model, the parameter vector of the global model, and each data record of the plurality of data records; and obtaining the parameter vector of the local model by aggregating a plurality of gradients corresponding to the plurality of data records.

4. The method of claim 3, wherein, obtaining the parameter vector of the local model further comprises: for each data record of the plurality of data records, performing a clipping operation on the corresponding gradient with a clipping parameter to map the corresponding gradient to a range corresponding to the clipping parameter.

5. The method of claim 3, wherein, obtaining the parameter vector of the local model by aggregating the plurality of gradients comprises: determining the parameter vector of the local model based on a number of the plurality of data records and an aggregation of the plurality of gradients.

6. The method of claim 4, wherein, quantizing the updated parameter vector comprises: determining a global quantization hyperparameter; determining an increment based on the global quantization hyperparameter and the number of the plurality of data records; and performing the quantization based on a space determined according to the increment.

7. The method of claim 6, wherein, performing the quantization based on the space determined according to the increment comprises: determining a number of levels of the quantization according to the increment; and performing the quantization according to the range and the number of levels.

8. The method of claim 1, wherein: the plurality of noise vectors further comprises a second noise vector; the second noise vector comprises a dithering vector that randomizes quantization errors of the quantization.

9. The method of claim 8, wherein, the second noise vector follows a uniform distribution and is independent of the first noise vector and the parameter vector of the local model.

10. The method of claim 8, wherein, sending the quantized parameter vector and at least one noise vector of the plurality of noise vectors comprises: sending the quantized parameter vector and the second noise vector to the server.

11. The method of claim 1, wherein, obtaining the parameter vector of the local model comprises: sampling a subset of a plurality of data records collected by the terminal device; and The parameter vector of the local model is obtained by training the local model based on the subset of the plurality of data records.

12. The method of claim 11, wherein, The sampling of the subset of the plurality of data records collected by the terminal device comprises: The subset of the plurality of data records collected by the terminal device is sampled based on a Poisson sampling rate.

13. A system for federated learning, comprising: one or more processors, and one or more non-transitory computer-readable memories coupled to the one or more processors and configured with instructions executable by the one or more processors to cause the system to perform operations comprising: obtaining a parameter vector of a local model; updating the parameter vector of the local model by adding a plurality of noise vectors to the parameter vector of the local model; the plurality of noise vectors comprising: a first noise vector; the first noise vector comprising a sum of a plurality of independent random variable vectors, each of the plurality of independent random variable vectors following a uniform distribution; quantizing the updated parameter vector to obtain a quantized parameter vector, the quantizing comprising mapping coordinates in the updated parameter vector to a set of discrete finite values; and sending the quantized parameter vector and at least one noise vector of the plurality of noise vectors to a server for updating a global model by the server.

14. The system of claim 13, wherein, The obtaining of the parameter vector of the local model comprises: receiving, from the server, a parameter vector of the global model; for each data record of a plurality of data records collected by the system, obtaining a gradient based on a loss function of the local model, the parameter vector of the global model, and each data record of the plurality of data records; and obtaining the parameter vector of the local model by aggregating a plurality of gradients corresponding to the plurality of data records.

15. The system of claim 14, wherein, The obtaining of the parameter vector of the local model further comprises: for each data record of the plurality of data records, performing a clipping operation on the gradient with a clipping parameter to map the gradient to a range corresponding to the clipping parameter.

16. The system of claim 13, wherein: the plurality of noise vectors further comprises a second noise vector; the second noise vector comprises a dithering vector that randomizes quantization errors of the quantizing.

17. The system of claim 15, wherein, The quantizing of the updated parameter vector comprises: determining a global quantization hyperparameter; determining an increment based on the global quantization hyperparameter and a number of the plurality of data records; and performing the quantizing based on a space determined according to the increment.

18. A non-transitory computer-readable storage medium configured with instructions executable by one or more processors to cause the one or more processors to perform operations comprising: obtaining a parameter vector of a local model; updating the parameter vector of the local model by adding a plurality of noise vectors to the parameter vector of the local model; the plurality of noise vectors comprising: a first noise vector; the first noise vector comprising a sum of a plurality of independent random variable vectors, each of the plurality of independent random variable vectors following a uniform distribution; quantizing the updated parameter vector to obtain a quantized parameter vector, the quantizing including mapping coordinates in the updated parameter vector to a set of discrete finite values; and sending the quantized parameter vector and at least one noise vector of the plurality of noise vectors to a server for the server to update a global model.

19. The non-transitory computer-readable storage medium of claim 18, wherein: the plurality of noise vectors further includes a second noise vector; the second noise vector includes a dither vector that randomizes quantization errors of the quantizing.

20. The non-transitory computer-readable storage medium of claim 18, wherein, obtaining the parameter vector of the local model includes: receiving, by the terminal device, the parameter vector of the global model from the server; for each data record of a plurality of data records of the terminal device, obtaining a gradient based on a loss function of the local model, the parameter vector of the global model, and each data record of the plurality of data records; and obtaining the parameter vector of the local model by aggregating a plurality of gradients corresponding to the plurality of data records; quantizing the updated parameter vector includes: determining a global quantization hyperparameter; determining an increment based on the global quantization hyperparameter and a number of the plurality of data records; and performing the quantizing based on a space determined according to the increment.

Images