Bidirectional compression and privacy for efficient communication in federative learning

Abstract

Some aspects of the present disclosure provide techniques for performing federated learning, including receiving a global model from a federated learning server, determining an updated model based on the global model and local data, and transmitting the updated model to the federated learning server using relative entropy coding.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to PCT application PCT / US2022 / 072599 filed on 26 May 2022, and also claims the benefit and priority of Greek patent application 20210100355 filed on 28 May 2021, the entire contents of each of these applications being incorporated herein by reference.

[0002] The nature of this disclosure relates to machine learning. [Background technology]

[0003] Machine learning is generally the process of creating a trained model (e.g., an artificial neural network, tree, or other structure) that demonstrates a generalized fit to a priori known set of training data. Applying the trained model to new data creates inference information, which can be used to gain insights into the new data.

[0004] As machine learning becomes more widespread across various technological domains, sometimes referred to as artificial intelligence tasks, the need for more efficient processing of machine learning model data arises. For example, "edge processing" devices such as mobile devices, always-on devices, and Internet of Things (IoT) devices must balance the implementation of advanced machine learning capabilities with various interrelated design constraints, including packaging size, native computing power, power storage and usage, data communication capabilities and costs, memory size, and heat dissipation.

[0005] Federated learning is a distributed machine learning framework that allows several clients, such as edge processing devices, to collaboratively train a shared global model without transferring local data to a remote server. Generally, a central server coordinates the federated learning process, and each participating client communicates only model parameter information with the central server, while keeping its local data private. This distributed approach helps address the limitations of client device capabilities (because training is federated) and often mitigates data privacy concerns.

[0006] While federative learning generally limits the amount of model data in any single transmission between the server and client (or vice versa), the iterative nature of federative learning still generates a significant amount of data transmission traffic during training, which can be quite costly depending on the device and connection type. Therefore, it is generally desirable to attempt to reduce the size of data exchange between the server and client during federative learning. However, conventional methods to reduce data exchange, such as using lossy compression of model data to limit the amount of data exchanged between the server and client, result in inadequate models. Furthermore, conventional federative learning has been shown to be unprivileged. [Overview of the project] [Problems that the invention aims to solve]

[0007] Therefore, there is a need for an improved method of performing federative learning that supports communication efficiency without compromising model performance, while also improving privacy. [Means for solving the problem]

[0008] Some aspects provide a method for performing federated learning, the method including receiving a global model from a federated learning server, determining an updated model based on the global model and local data, and transmitting the updated model to the federated learning server using relative entropy coding.

[0009] Further aspects provide a method for performing federated learning, the method including transmitting a global model to a client device, determining a random seed, receiving an updated model from the client device using relative entropy coding, and determining an updated global model based on the updated model from the client device.

[0010] Other aspects provide a processing system configured to perform the methods described above and herein, a non - transient computer - readable medium comprising instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the methods described above and herein, a computer program product embodied on a computer - readable storage medium comprising code for performing the methods described above and further herein, and a processing system comprising means for performing the methods described above and further herein.

[0011] The following description and related drawings detail some exemplary features of one or more aspects.

[0012] The accompanying drawings illustrate some aspects of one or more aspects and, thus, should not be considered as limiting the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] [Figure 1] A diagram showing an exemplary federated learning architecture. [Figure 2A]FIG. showing an exemplary algorithm 1 for the sender-side implementation of non-invertible relative entropy coding. [Figure 2B] FIG. showing an exemplary algorithm for the receiver-side implementation of non-invertible relative entropy coding. [Figure 3] Schematic diagram of performing relative entropy coding for federated learning updates. [Figure 4] FIG. showing an exemplary server-side algorithm for applying relative entropy coding to federated learning. [Figure 5] FIG. showing an exemplary client-side algorithm for applying relative entropy coding to federated learning. [Figure 6A] FIG. showing an exemplary client-side algorithm for applying differential private relative entropy coding to federated learning. [Figure 6B] FIG. showing an exemplary server-side algorithm for applying differential private relative entropy coding to federated learning. [Figure 7] Schematic diagram of performing differential private relative entropy coding for federated learning updates. [Figure 8] FIG. showing an exemplary method for performing federated learning according to the aspects described herein. [Figure 9] FIG. showing another exemplary method for performing federated learning according to the aspects described herein. [Figure 10A] FIG. showing an exemplary processing system configured to execute the methods described herein. [Figure 10B] FIG. showing an exemplary processing system configured to execute the methods described herein.

DETAILED DESCRIPTION OF THE INVENTION

[0014] For ease of understanding, the same reference numerals are used to designate identical elements common to the drawings where possible. It is intended that elements and features of one embodiment may be usefully incorporated into other embodiments without further description.

[0015] Aspects of this disclosure provide apparatus, methods, processing systems, and non-temporary computer-readable media for bidirectional compression for efficient private communication in machine learning, particularly in federative learning.

[0016] The performance of modern neural network-based machine learning models is very proportional to the amount of data they are trained on. At the same time, industry, legislators, and consumers are becoming more aware of the need to protect the privacy of the data that may be used when training such models. Federated learning describes a machine learning principle that aims to enable learning on distributed data by computing updates on devices. Instead of sending its data to a central location, a “client” sends model updates computed on that data to a central server. Such an approach to learning from distributed data, due to the inherently more private nature of the approach, promises to unlock the computing power of billions of “edge” devices, enabling personalized models and new applications, for example, in healthcare.

[0017] On the other hand, the federative learning paradigm presents challenges along many dimensions, including learning from non-independent and identically distributed data, resource-constrained devices, heterogeneous computing and communication capabilities, issues of fairness and representation, and communication overhead. In particular, neural networks require many paths on the data, necessitating repeated communication of the latest server-side model to the client and its updates back to the server, which significantly increases communication overhead. Therefore, compressing updates in federative learning is a crucial step in reducing such overhead, for example, when "untethering" edge devices from Wi-Fi.

[0018] Traditional approaches to mitigate this problem include compressing uplink and / or downlink messages through either quantization or pruning of the messages to be transmitted. However, these techniques can lead to data loss and degraded performance of trained global models.

[0019] To overcome these technical problems with conventional approaches, the embodiments described herein implement a compression scheme, relative entropy coding, for transmitting models and model updates between the client and the server, and vice versa, which is independent of quantization or pruning and adapted to work for associative learning settings.

[0020] In the embodiments described herein, communication from a client to a server may be achieved by a series of steps. In one example, the server and the client first agree on a specific random seed R and a prior distribution p (for example, the last model the server sent to the client). The client then forms a probability distribution q centered on the model update it wants to send to the server. The client then draws K random samples from p according to the random seed R. In particular, K can be determined by the data by measuring the discrepancy between p and q. The client then draws a ratio for each of these K samples.

[0021]

number

[0022] A probability proportional to π k Assign π1,…,π K A random sample is selected according to the formula, and its index k is recorded. The client then communicates the index k to the server in log2K bits. The server can decode the message by drawing random samples from p using a random seed R until the k-th random sample is recovered.

[0023] In particular, this procedure can be implemented flexibly. For example, the procedure can be performed per parameter (e.g., communicating log2K bits per parameter), per layer (e.g., communicating log2K bits per layer in the network), or per network (e.g., communicating log2K bits overall). Any intermediate vector size is possible.

[0024] In the embodiments described herein, communication from the server to the client is also compressible. First, the server tracks the last time each client was selected to participate in training, along with all model updates it has received from the client. Then, whenever a client is selected to participate in a round (or instance) of federated learning, the server communicates all the model updates necessary to update the old local global model copy in the client to the current one, instead of sending the current state of the global model. Since each model update can be generated by a specific random seed R and log2K bits, the overall message length can be dramatically reduced compared to sending the entire floating-point model, especially when aggressive compression is used for model updates from the client to the server, as described above. In particular, it is always possible to compare the two message sizes (communicating the model at floating points and communicating compressed past model updates) and choose the lower-cost format.

[0025] As described above, the embodiments described herein function beneficially without imposing quantization / prunning on messages transmitted between the client and the server. Furthermore, the compression ratio when using the embodiments described herein can be much higher than that of conventional scalar quantization, especially when the scheme is implemented layer by layer. In addition, the bit width of the message can be advantageously determined / adapted on the fly.

[0026] Accordingly, the embodiments described herein provide technical solutions to the technical problems described above regarding communication overhead. The embodiments described herein beneficially improve the performance of any device participating in federated learning by reducing the total communication cost, such as the number of units (e.g., GB) of data communicated between the client and the server during federated learning. The communication cost can be dramatically reduced compared to conventional scalar compression methods, especially when layer-by-layer compression is used according to the methods described herein.

[0027] Further aspects described herein relate to improving privacy while performing the aforementioned communication-efficient federated learning. While federated learning provides an intuitive and practical concept of privacy by retaining data on devices, client updates have nevertheless been shown to reveal sensitive information while allowing for the reconstruction of the client's training data. Conventional approaches have generally involved a trade-off between limiting compression and enhancing privacy, or vice versa, but conventional approaches have failed to achieve both simultaneously.

[0028] On the other hand, the embodiments described herein can be made differentially private by implementing modified relative entropy coding in a federative learning context. In doing so, the embodiments described herein provide a differentially private federative learning algorithm that achieves extreme compression of client-to-server updates (e.g., up to 7 bits per tensor) with minimal impact on model performance at a privacy level (ε<1, where ε quantifies how private the learning algorithm is, meaning how easily a hypothetical adversary can identify where individuals and their data participated in training the model).

[0029] Therefore, the embodiments described herein provide a simultaneous solution for privacy and communication efficiency by using differential privacy and coding-efficient compression of messages communicated during federated learning.

[0030] Examples of associative learning architectures Figure 1 shows an exemplary associative learning architecture 100.

[0031] In this example, mobile devices 102A-102C are examples of edge processing devices, each having its own local data store 104A-104C and each having its own local machine learning model instance 106A-106C. For example, mobile device 102A is equipped with an initial machine learning model instance 106A (or receives an initial machine learning model instance 106A from, for example, a global machine learning model coordinator 108), which may be a software provider in some examples. Each of the mobile devices 102A-102C may use its respective machine learning model instance (106A-106C) for some useful task, such as processing local data 104A-104C, and may further perform local training and optimization of its respective machine learning model instance.

[0032] For example, mobile device 102A may use its machine learning model 106A to perform face recognition on photographs stored as data 102B on mobile device 102A. Since such photographs may be considered personal, mobile device 102A may not want to share its photographic data with the global model coordinator 108, or may be prevented from doing so. However, mobile device 102A may want to share local model updates, such as updates to the model weights and parameters, with the global model coordinator 108, or may be permitted to do so. Similarly, mobile devices 102B and 102C may use their respective local machine learning model instances 106B and 106C, and may also share local model updates with the global model coordinator 108 without sharing the underlying data used to generate the local model updates.

[0033] The global model coordinator 108 (alternatively called the federated learning server) may use all local model updates to decide on a global (or consensus) model update, which may then be delivered to mobile devices 102A-102C. In this way, machine learning can leverage mobile devices 102A-102C without centralizing training data and processing.

[0034] Therefore, the federative learning architecture 100 enables decentralized deployment and training of machine learning models, thereby advantageously reducing latency, network utilization, and power consumption, while maintaining data confidentiality and security. Furthermore, the federative learning architecture 100 allows models to evolve differently on different devices, but ultimately allows their distributed learned knowledge to be combined and returned to a global model.

[0035] In particular, the local data stored on mobile devices 102A-102C and used by machine learning models 106A-106C are sometimes referred to as individual data shards (e.g., data 104A-104C) and / or federated data. Since these data shards are generated by different users on different devices and are not mixed, they cannot be assumed to be independently and identically distributed (IID) with respect to one another. This is more generally true for any type of device-specific data that is not combined to train machine learning models. Only by combining the individual datasets 104A-104C from each of the mobile devices 102A-102C can a global dataset be generated where the IID assumption is valid.

[0036] Associative learning, general Federative learning is described in the form of the FedAvg algorithm, which is explained as follows: In each communication round t, the server (e.g., 108 in Figure 1) communicates the current model parameters w(t) This is sent to a subset S' of all S clients participating in the training (for example, mobile devices 102A, 102B, and / or 102C in Figure 1). Each selected client s receives the server-provided model w (t) For example, using stochastic gradient descent, with a given loss function as follows, size N s The local dataset D s Update the data (for example, data 104A, 104B, and / or 104C in Figure 1, respectively) to better match.

[0037]

number

[0038] After the E epoch of optimization on the local dataset, the updated model is performed by the client-side optimization procedure.

[0039]

number

[0040] This is obtained, and based on this, the client,

number

[0041]

number

[0042] The generalization of this server-side averaging method is:

[0043]

number

[0044] Interpret it as the "gradient" for the server - side model and introduce more advanced update methods such as adaptive momentum (e.g., Adam algorithm).

[0045] Federated training involves repeated communication of model updates from the client to the server and vice versa. The total communication cost of this procedure can be quite high, and thus, typically, federated learning is restricted to the use of non - metered channels such as Wi - Fi networks. Therefore, compression of the communicated messages plays an important role in moving federated learning to true mobile use cases. For this purpose, the aspects described herein extend the non - reversible version of relative entropy coding (REC) to the federated environment and, for example

[0046] [Number]

[0047] compress the model updates from the client to the server, such as

[0048] Relative entropy coding By using the information "shared" between the sender and the receiver, a random sample w from the distribution q φ (w) parameterized by φ, i.e., w ~ q φ (w), the non - reversible relative entropy coding and its preceding minimum random code learning were first proposed as a way to compress this information. This information is given by a shared prior distribution p θ (w) with parameters θ and a shared random seed R.

[0049] The sender generates K independent random samples w1,..., w θ from the prior distribution p K according to the random seed R. Then, for the K samples, the probability of each sample is the likelihood ratio π[[ID=4I]] k ∝qφ (w=w k ) / p θ (w=w k ) a categorical distribution proportional to

[0050]

number

[0051] It forms a random sample w k* of

[0052]

number

[0053] Extracted from this, this corresponds to the k*-th sample extracted from the shared prior distribution. The sender can then communicate the index k* to the receiver in log2K bits. Figure 2A shows an exemplary algorithm 1 of the sender-side implementation of irreversible relative entropy coding.

[0054] On the recipient's side, w k* This initializes a random number generator in R and runs p up to the k*th sample. θ (w) can be reconstructed by sampling. Figure 2B shows an exemplary algorithm 2 of the receiver-side implementation of irreversible relative entropy coding.

[0055] In some cases, K is q φ (w) from the prior distribution p θ It may also be set to be equal to the exponent obtained by adding a constant t to the Kullback-Leibler (KL) divergence to (w), that is,

[0056]

number

[0057] In this case, the message length is at least O(KL(q φ (w)||pθ (w))) Therefore, when the sender and receiver share a source of randomness, under some assumptions, this KL divergence is a lower bound on the expected length of the message being communicated.

[0058] This is the compression ratio, p θ (w) information for q φ This brings us to the intuitive concept of compression, which is associated with the amount of additional information encoded in (w). Therefore, the less extra information there is, the shorter the message length becomes, and q φ (w=p) θ In the extreme case of (w), the message length becomes O(1). Of course, if the bias of this procedure is high, it is not worthwhile to achieve this efficiency, and fortunately, under appropriate values ​​of t and mild assumptions, for any function f, the bias, i.e.

[0059]

number

[0060] It can become sufficiently small. Therefore, in some embodiments, K can be parameterized as a function of the binary bit width b, for example, K=2 b Therefore, b can be treated as a hyperparameter.

[0061] Relative entropy coding for efficient communication in associative learning The embodiments described herein include messages from a client to a server, for example, model updates.

[0062]

number

[0063] The distribution over time, prior distribution in each round t.

[0064]

number

[0065] By appropriately selecting these elements, irreversible relative entropy coding is adapted to the associative environment. These can be defined as follows:

[0066]

number

[0067] In other words, a Gaussian distribution centered on zero is used as the prior distribution along with a appropriately selected σ, and a Gaussian distribution with the same standard deviation centered on the model update is used for the message distribution. The form of q is chosen so that it can be implemented on resource-constrained devices and satisfies the differential privacy constraints discussed below. Here, in contrast to the FedAvg client update definition in equation (2),

number

[0068] Therefore, the length of the federative learning message from the client to the server is the local dataset D s How much "extra" information about is measured via KL divergence?

[0069]

number

[0070] This becomes a function of what is encoded. This has good interaction with differential privacy (DP) because the differential privacy constraint bounds the amount of information encoded in each update, resulting in a highly compressible message. In particular, this procedure can be done per parameter (e.g., communicating log2K bits per parameter), per layer (e.g., communicating log2K bits per layer in the global model), or per network (e.g., communicating a total of log2K bits). Arbitrary intermediate vector sizes are also possible. This is,

[0071]

number

[0072] This is achieved by dividing the vector into M independent groups (which is easy due to the assumption of a factor distribution across the vector dimensions) and applying a compression mechanism independently to each group.

[0073] Figure 3 schematically shows an example 300 of communication from client 302 to server 304. In the illustrated example, client 302 uses the distribution q (explained above). φ and shared prior distribution p θ Samples 1-k are generated based on the ratio 306. Then, index k is sent from client 302 to server 304, and server 304 then receives the shared prior distribution p θ Model update 308 can be restored by decoding the index using shared information such as the random seed R.

[0074] The compression procedure described for client-to-server federated learning messaging is a specific example of (probabilistic) vector quantization, where the shared codebook is determined by a shared random seed R. Beneficially, the principle of communicating indices to such a shared codebook further enables compression of server-to-client communication.

[0075] For example, instead of sending a complete server-side model to a specific client, the server could choose to collect all updates to the global model between two subsequent rounds in which the client participates. Based on this history in the codebook index, the client can deterministically reconstruct the current state of the server model before initiating local optimizations.

[0076] Clearly, the expected length of the history is proportional to the total number of clients and the amount of client subsampling performed during training. Therefore, at the start of any round, the server compares the bit size of the client history and instead uses the full-precision model w (t) You can choose to send this. Taking a model with 1k parameters as an example, when using 8-bit codebook compression for the entire model, a single uncompressed model update is approximately equivalent to 4k communication indices. Importantly, compressing messages from the server to the client in this way does not affect the differentially private nature of the aspects described below, as any information released from the client is private according to these aspects.

[0077] For clients participating in the first round of training, the first seed without an index can be understood as the seed for the random initialization of the server-side model. Algorithms 3 and 4, shown in Figures 4 and 5 respectively, illustrate examples of server-side and client-side procedures. Note that the client-side update rule must be equal to the server-side update rule (*). In other words, in generalized FedAvg, the current global model w (t) When sending, it may be necessary to send the optimizer state in addition.

[0078] Differential private relative entropy coding for private and efficient communication in associative learning The relative entropy coding learning compression method described above can beneficially reduce communication costs by several orders of magnitude compared to conventional methods. However, model updates can still reveal sensitive information about the client's local dataset, and at least from a theoretical standpoint, compressed model updates leak as much information as full-accuracy updates.

[0079] To mitigate privacy risks, differential privacy may be employed during training. Traditional differential privacy mechanisms for federated learning involve each client clipping the norm of full-accuracy model updates before sending them to the server. The server then averages the clipped model updates, sometimes using a secure aggregation protocol, and adds Gaussian noise with a specific variance. However, traditional applications of differential privacy do not work with compression.

[0080] Therefore, various modifications can be made to the relative entropy coding learning compression scheme described above in order to ensure privacy. Specifically, in order to ensure the differential privacy of the relative entropy coding described above, it is necessary to bound the sensitivity in order to quantify the intrinsic noise. Sensitivity bounding is performed on the client update

[0081]

number

[0082] This consists of clipping the norm of the client message distribution. In the context of relative entropy coding, this is the client message distribution.

[0083]

number

[0084] However, in any round t, the prior distribution of the servers

[0085]

number

[0086] This means that it cannot be significantly different. Note that, unlike conventional methods, since the procedure itself is probabilistic, explicit injection of additional noise into the update is not required. In each round t, two sources of randomness play a role: (1) a prior distribution of a set of K samples

[0087]

number

[0088] Extracted from, (2) Important sampling distribution

[0089]

number

[0090] Extract updates from the source.

[0091] Therefore, differential private relative entropy coding (DP-REC) can generally be achieved in two steps. First, each client may clip the norm of its model update before forming a probability distribution q centered on this clipped update. In one example, the clipping threshold is σ p It is calibrated according to [the specified formula]. The purpose of this step is to ensure that the Renyi divergence between the posterior distribution q and the server's prior distribution p is bounded. Bounding is necessary in order to be sufficient to compute the privacy guarantee.

[0092] Note that the Renyi divergence or alpha divergence of order α from distribution Q to distribution P is defined as follows:

[0093]

number

[0094] Secondly, the server records events that leak information about client data. For example, a sample of a specific client from the entire population, along with its probability in each round, or the importance distribution π. q These are samples from the data. These events define a probability distribution across all possible model updates for all clients. The privacy accounting component uses this information, in combination with clipping limits, to determine the maximum Renyi divergence between the update distributions of any two clients during training, and then computes the ε and δ parameters of differential privacy by using the Chernov limit. In probability theory, the Chernov limit gives an exponentially decreasing limit to the tail distribution of the sum of independent random variables. Furthermore, ε declares the degree of "privacy" of a particular algorithm, and δ (usually considered sufficiently small) is the probability that differential privacy fails (and therefore does not yield a private output).

[0095] Figures 6A and 6B show algorithms 5 and 6 for performing differential private relative entropy coding (DP-REC) on the client and server sides, respectively.

[0096] Figure 7 schematically shows an example 700 of communication from client 702 to server 704. In the illustrated example, client 702 is

[0097]

number

[0098] Samples 1-k are generated based on the ratio 706 (as explained above). However, in this example, the norm is clipped before the ratio is generated, and a clipped model update is produced.

[0099] For example, m q This is a model update,

[0100]

number

[0101] This is a clipped model update calculated according to the following:

[0102]

number

[0103] Δ is the amount of clipping performed.

[0104] Next, the index k is sent from client 702 to server 704, and server 704 then sends the shared prior distribution p θ Model update 708 can be restored by decrypting the index using shared information such as the random seed R.

[0105] In particular, compared to conventional differential privacy techniques, the embodiments described herein do not require the injection of additional noise into updates on either the client or the server. Rather, randomness is used in the relative entropy coding procedure for federated learning updates. In this case, beneficially, communication-efficient federated learning using relative entropy coding can be combined with a privacy-preserving embodiment of differential privacy for a unified approach.

[0106] Exemplary Method Figure 8 shows an exemplary method 800 for performing federated learning according to embodiments described herein. Method 800 can generally be performed by a client in the federated learning scheme, such as the mobile device 102 in Figure 1.

[0107] Method 800 begins in step 802 with receiving a global model from a federated learning server, such as the global model coordinator 108 in Figure 1.

[0108] Method 800 then proceeds to step 804, in which it determines an updated model based on the global model and local data. For example, a local machine learning model like 106A in Figure 1 may be trained on local data 104A to generate an updated model. Determining an updated model may involve generating updated model parameters such as weights and biases, which may be determined as direct values ​​or relative values ​​(e.g., delta). In some embodiments, determining an updated model based on the global model and local data may involve performing gradient descent on the global model using the local data.

[0109] Method 800 then proceeds to step 806, which involves sending the updated model to the federated learning server using relative entropy coding. In some embodiments, sending the updated model to the federated learning server using relative entropy coding is performed according to the algorithm depicted and described with respect to Figure 5 or Figure 6A.

[0110] In some embodiments, sending an updated model to a federated learning server using relative entropy coding includes determining a random seed. In some embodiments, determining a random seed includes receiving a random seed from a federated learning server. In other embodiments, the client may determine a random seed and send it to the federated learning server, which may prevent arbitrary manipulation of the random seed by the federated learning server and improve privacy.

[0111] In some embodiments, sending the updated model to a federated learning server using relative entropy coding further includes determining a first probability distribution based on the global model and a second probability distribution centered on the updated model.

[0112] In some embodiments, sending an updated model to a federated learning server using relative entropy coding further includes determining a plurality of random samples from a first probability distribution according to a random seed, and assigning a probability to each of the plurality of random samples based on the ratio of the likelihood of each random sample given a second probability distribution to the likelihood of each random sample given a first probability distribution.

[0113] In some embodiments, determining multiple random samples from a first probability distribution according to a random seed is performed based on the difference between the first and second probability distributions. In some cases, the number of random samples (K) is calculated as K = exp(KL(q||p)+t), where KL is the Kulback-Leibler divergence between q and p, and t is the adjustment coefficient. In other cases, K = 2 b This can be calculated as follows, where b is the number of bits allowed for a client-to-server message, such as the updated local model shown and explained with respect to Figure 1.

[0114] In particular, the ratio of the likelihood of each random sample given a second probability distribution to the likelihood of each random sample given a first probability distribution is: The ratio can be determined for each parameter, such as q(w1) / p(w1), q(w2) / p(w2), etc. The ratio is, for example, (q(w1) × q(w2) × ... × q(w k )) / (p(w1)×p(w2)×…×p(w kIt is also possible to determine a given number of elements that may represent layers of the model being updated, such as )). In other words, parameters 1 through k may represent layers, or any chunk of the entire neural network model, or the entire set of parameters of the neural network model. Thus, in some embodiments, multiple random samples are associated with multiple parameters of the global model. In some embodiments, multiple random samples are associated with layers of the global model. In some embodiments, multiple random samples are associated with a subset of parameters of the global model.

[0115] In some embodiments, sending the updated model to a federated learning server using relative entropy coding further includes selecting one random sample from a plurality of random samples according to the probability of each of the plurality of random samples.

[0116] In some embodiments, sending the updated model to a federated learning server using relative entropy coding further includes determining the index associated with a selected random sample and sending the index to the federated learning server.

[0117] For example, suppose we have eight samples, then a probability distribution over these eight samples, and then a random sample representing an index for one of the eight samples can be drawn from this distribution.

[0118] In some cases, the index is transmitted using log2K bits, where K is the number of multiple random samples from a first probability distribution.

[0119] In some embodiments, Method 800 further includes clipping the updated model before determining a second probability distribution centered on the updated model, where the clipping is based on the standard deviation (σ) of the global model, and the second probability distribution is based on the clipped updated model. In one embodiment, the clipping value is calculated as C × σ, where σ is the prior standard deviation of the global model. The complete paper contains these details, which can be found at this time in line 4 of Algorithm 5.

[0120] In some embodiments, clipping an updated model includes clipping the norm of the updated model.

[0121] Figure 9 shows an exemplary method 900 for performing federated learning according to the embodiments described herein. Method 900 can generally be performed by a server in the federated learning scheme, such as the global model coordinator 108 in Figure 1.

[0122] Method 900 begins in step 902 with sending the global model to the client device.

[0123] Method 900 then proceeds to step 904 to determine a random seed.

[0124] Next, method 900 proceeds to step 906, where it receives the updated model from the client device using relative entropy coding.

[0125] In some embodiments, receiving the updated model from the client device using relative entropy coding is performed according to the algorithm depicted and described with respect to Figure 4 or Figure 6B.

[0126] Method 900 then proceeds to step 908, where the updated global model is determined based on the updated model from the client device.

[0127] In some embodiments, receiving an updated model from a client device using relative entropy coding includes receiving an index from the client device, determining a sample from a probability distribution based on the global model, a random seed, and the index, and using the determined sample to determine the updated global model.

[0128] In some embodiments, the index is received using log2K bits, where K is the number of random samples determined from a probability distribution based on a global model.

[0129] In some aspects, the determined samples are used to update the parameters of the updated global model.

[0130] In some embodiments, the determined samples are used to update the layers of the updated global model.

[0131] In some embodiments, determining a random seed involves receiving a random seed from a client device. In other embodiments, determining a random seed is performed by a federated learning server, which then sends the random seed to the client device.

[0132] An exemplary processing system for performing sparsity-aware compute-in-memory. Figure 10A shows an exemplary processing system 1000 for performing associative learning, as described herein, for example, with respect to Figures 1 to 8. The processing system 1000 may be an example of a client device, such as the client devices 102A to C in Figure 1.

[0133] The processing system 1000 includes a central processing unit (CPU) 1002, which in some examples may be a multi-core CPU. Instructions executed in the CPU 1002 may be loaded, for example, from program memory associated with the CPU 1002, or from a memory partition 1024.

[0134] The processing system 1000 also includes additional processing components organized for specific functions, such as a graphics processing unit (GPU) 1004, a digital signal processor (DSP) 1006, a neural processing unit (NPU) 1008, a multimedia processing unit 1010, and a wireless connectivity component 1012.

[0135] NPUs, such as the 1008, are generally specialized circuits configured to perform all the necessary control and computational logic for executing machine learning algorithms, including algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), and random forests (RFs). NPUs are sometimes also called neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligence processing units (IPUs), vision processing units (VPUs), or graph processing units.

[0136] NPUs such as the 1008 are configured to accelerate the execution of common machine learning tasks such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system-on-a-chip (SoC), while in others they may be part of a dedicated neural network accelerator.

[0137] An NPU may be optimized for training or inference, or in some cases, configured to balance performance between both. In an NPU capable of performing both training and inference, the two tasks can generally be performed independently.

[0138] NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is a highly computationally intensive operation involving inputting existing datasets (often labeled or tagged), iterating through the datasets, and then tuning model parameters such as weights and biases to improve model performance. Generally, optimization based on incorrect predictions involves propagating backward through the model layers and determining gradients to reduce prediction errors. In some cases, NPUs may be configured to perform the federative learning methods described herein.

[0139] NPUs designed to accelerate inference are generally configured to operate on complete models. Therefore, such NPUs may be configured to process new data at high speed through a model that has already been trained to generate model outputs (e.g., inferences).

[0140] In one implementation configuration, the NPU1008 is one or more parts of the CPU1002, GPU1004, and / or DSP1006.

[0141] In some examples, the wireless connectivity component 1012 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G LTE), fifth-generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity processing component 1012 is further connected to one or more antennas 1014. In some examples, the wireless connectivity component 1012 enables federated learning to be performed via various wireless data connections, including cellular connectivity, in accordance with the methods described herein.

[0142] The processing system 1000 may also include one or more sensor processing units 1016 related to any type of sensor, one or more image signal processors (ISPs) 1018 related to any type of image sensor, and / or a navigation processor 1020 which may include satellite-based positioning system components (e.g., GPS or GLONASS), and inertial positioning system components.

[0143] The processing system 1000 may also include one or more input and / or output devices 1022, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, or a microphone.

[0144] In some examples, one or more of the processors in processing system 1000 may be based on an ARM or RISC-V instruction set.

[0145] The processing system 1000 also includes memory 1024 representing one or more static and / or dynamic memories, such as dynamic random access memory and flash-based static memory. In this example, memory 1024 includes computer executable components that can be executed by one or more of the processors of the processing system 1000 described above.

[0146] In particular, in this example, memory 1024 includes a receiving component 1024A, a model update component 1024B, a transmitting component 1024C, and a model parameter 1024D. Illustrated components and other components not illustrated may be configured to perform various aspects of the methods described herein.

[0147] In general, the processing system 1000 and / or its components may be configured to perform the methods described herein.

[0148] In particular, in other cases, embodiments of the processing system 1000 may be omitted or added. For example, the multimedia component 1010, wireless connectivity 1012, sensor 1016, ISP 1018, and / or navigation component 1020 may be omitted in other embodiments. Furthermore, embodiments of the processing system 1000 may be distributed among multiple devices.

[0149] Figure 10B shows another exemplary processing system 1050 for performing federative learning, as described herein, for example, with respect to Figures 1-7 and 9. Processing system 1050 may be an example of a federative learning server, such as the global model coordinator 108 in Figure 1.

[0150] In general, the CPU 1052, GPU 1054, NPU 1058, and I / O 1072 are as described above with respect to the similar elements in Figure 10A.

[0151] The processing system 1050 also includes memory 1074 representing one or more static and / or dynamic memories, such as dynamic random access memory and flash-based static memory. In this example, memory 1074 includes computer executable components that can be executed by one or more of the processors of the processing system 1050 described above.

[0152] In particular, in this example, memory 1074 includes a receiving component 1074A, a model update component 1074B, a transmitting component 1074C, and a model parameter 1074D. Illustrated components and other components not illustrated may be configured to perform various aspects of the methods described herein.

[0153] In general, the processing system 1050 and / or its components may be configured to perform the methods described herein.

[0154] In particular, in other cases, embodiments of the processing system 1050 may be omitted or added. Furthermore, embodiments of the processing system 1050 may be distributed across multiple devices, such as a cloud-based service. The components shown in the illustration are limited for clarity and brevity.

[0155] Exemplary clause Examples of implementation forms are described in the following numbered clauses.

[0156] Clause 1: A method comprising the steps of receiving a global model from a federated learning server, determining an updated model based on the global model and local data, and sending the updated model to the federated learning server using relative entropy coding.

[0157] Clause 2: The method according to Clause 1, wherein the step of sending an updated model to a federated learning server using relative entropy coding includes the steps of determining a random seed, determining a first probability distribution based on a global model, determining a second probability distribution centered on the updated model, determining a plurality of random samples from the first probability distribution according to the random seed, assigning a probability to each of the plurality of random samples based on the ratio of the likelihood of each random sample given the second probability distribution to the likelihood of each random sample given the first probability distribution, selecting one random sample from the plurality of random samples according to the probability of each of the plurality of random samples, determining an index associated with the selected random sample, and sending the index to a federated learning server.

[0158] Clause 3: The method according to Clause 2, wherein the step of determining a number of random samples from a first probability distribution according to a random seed is performed based on the difference between the first probability distribution and the second probability distribution.

[0159] Clause 4: The index is transmitted using log2K bits, where K is the number of multiple random samples from a first probability distribution, as described in any one of Clauses 2-3.

[0160] Clause 5: The method described in any one of Clauses 2-4, wherein multiple random samples are associated with multiple parameters of a global model.

[0161] Clause 6: The method described in any one of Clauses 2-4, wherein multiple random samples are associated with a layer of the global model.

[0162] Clause 7: The method described in any one of Clauses 2-4, wherein multiple random samples are associated with a subset of the parameters of a global model.

[0163] Clause 8: The method according to any one of Clauses 2-7, further comprising the step of clipping the updated model before determining a second probability distribution centered on the updated model, wherein the clipping is based on the standard deviation of the global model and the second probability distribution is based on the clipped updated model.

[0164] Clause 9: The method according to Clause 8, wherein the step of clipping the updated model includes the step of clipping the norm of the updated model.

[0165] Clause 10: The method according to any one of Clauses 1 to 9, wherein the step of determining an updated model based on a global model and local data includes the step of performing a gradient descent on the global model using the local data.

[0166] Clause 11: The method described in any one of Clauses 2 to 10, wherein the step of determining a random seed includes the step of receiving a random seed from a federated learning server.

[0167] Clause 12: A method comprising the steps of sending a global model to a client device, determining a random seed, receiving an updated model from the client device using relative entropy coding, and determining an updated global model based on the updated model from the client device.

[0168] Clause 13: The method according to Clause 12, wherein the step of receiving an updated model from a client device using relative entropy coding includes the steps of receiving an index from the client device, determining a sample from a probability distribution based on a global model, a random seed, and an index, and using the determined sample to determine the updated global model.

[0169] Clause 14: The method according to Clause 13, wherein the index is received using log2K bits, where K is the number of random samples determined from a probability distribution based on a global model.

[0170] Clause 15: The method described in any one of Clauses 13-14, wherein the determined sample is used to update the parameters of the updated global model.

[0171] Clause 16: The method described in any one of Clauses 13-15, wherein the determined sample is used to update the layer of the updated global model.

[0172] Clause 17: The method described in any one of Clauses 12 to 16, wherein the step of determining a random seed includes the step of receiving a random seed from a client device.

[0173] Clause 18: A processing system comprising memory containing computer executable instructions and one or more processors configured to execute computer executable instructions and cause the processing system to perform the method described in any one of Clauses 1 to 17.

[0174] Clause 19: A processing system comprising means for performing the method described in any one of Clauses 1 to 17.

[0175] Clause 20: Non-temporary computer-readable media comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method described in any one of Clauses 1 to 17.

[0176] Clause 21: A computer program embodied on a computer-readable storage medium that includes code for performing the method described in any one of Clauses 1 through 17.

[0177] Additional considerations The preceding explanation is provided to enable any person skilled in the art to practice the various embodiments described herein. The examples considered herein do not limit the scope, applicability, or embodiments described in the claims. Various modifications of these embodiments will be readily apparent to a person skilled in the art, and the general principles defined herein may be applied to other embodiments. For example, changes may be made to the function and configuration of the elements considered without departing the scope of this disclosure. Various examples may, as appropriate, omit, replace, or add various procedures or components. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Also, features described in some examples may be combined in some other examples. For example, an apparatus may be implemented or a method may be practiced using any number of embodiments described herein. In addition, the scope of this disclosure is intended to encompass, in addition to or in addition to the various embodiments of this disclosure described herein, such apparatuses or methods practiced using other structures, functions, or structures and functions. It should be understood that any embodiment of this disclosure disclosed herein may be embodied by one or more elements of the claims.

[0178] As used herein, the term “exemplary” means “to serve as an example, case, or illustration.” No embodiment described herein as “exemplary” should necessarily be construed as being preferable or more advantageous than any other embodiment.

[0179] The phrase “at least one of” the list of items used herein refers to any combination of those items that contain a single member. For example, “at least one of a, b, or c” shall cover a, b, c, ab, ac, bc, and abc, as well as any combination having multiple identical elements (for example, aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other order of a, b, and c).

[0180] As used herein, the term “deciding” encompasses a wide variety of actions. For example, “deciding” may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or other data structure), and confirming. It may also include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and resolving, selecting, choosing, and establishing.

[0181] The methods disclosed herein comprise one or more steps or actions for achieving the method. The steps and / or actions of the method may be interchangeable with one another without departing from the claims. In other words, unless a particular order of steps or actions is specified, the order and / or use of any particular steps and / or actions may be modified without departing from the claims. Furthermore, the various operations of the methods described above may be performed by any preferred means capable of performing the corresponding function. The means may include, but are not limited to, various hardware and / or software components and / or modules, including circuits, application-specific integrated circuits (ASICs), or processors. Generally, where there are operations shown in the figures, those operations may have corresponding relative means-plus-function components with similar numbering.

[0182] The following claims are not limited to the embodiments shown herein and should be given the full scope consistent with the language of the claims. In the claims, a singular reference to an element means "one or more" and not "one unique" unless otherwise explicitly stated. Unless otherwise explicitly stated, the term "several" means one or more. The elements of the claims should not be construed under Section 112(f) of the United States Patent Act unless the element is expressly described using the phrase "means for" or, in the case of a method claim, the element is described using the phrase "steps for". All structural and functional equivalents of the elements of the various embodiments described throughout this disclosure, which are known to those skilled in the art or will be known thereafter, are expressly incorporated by reference herein and are encompassed by the claims. Furthermore, nothing disclosed herein is intended to be made public, whether such disclosure is expressly enumerated in the claims or not. [Explanation of Symbols]

[0183] 100 Associative Learning Architectures 102A~102C Mobile Devices 102B data 104A~104C Local data store 106A~106C Local Machine Learning Model Instances 108 Global Machine Learning Model Coordinators 302 Clients 304 Server 306 ratio 308 Model Update 702 Client 704 Server 706 ratio 708 Model Update 1000 processing systems 1002 Central Processing Unit (CPU) 1004 Graphics Processing Unit (GPU) 1006 Digital Signal Processor (DSP) 1008 Neural Processing Unit (NPU) 1010 Multimedia Processing Unit 1012 Wireless connectivity components 1016 Sensor Processing Unit 1018 Image Signal Processor (ISP) 1020 Navigation Processor 1022 Input and / or Output Devices 1024 memory 1024A Receiving component 1024B Model Update Components 1024C Transmitting Components 1024D Model Parameters 1050 Processing System 1052 CPU 1054 GPU 1058 NPU 1072 inputs / outputs 1074 memory 1074A Receiving component 1074B Model Update Components 1074C Transmitting Components 1074D Model Parameters

Claims

1. A computer implementation method, Steps include receiving a global model from a federated learning server, The steps include determining an updated model based on the global model and local data, The steps include sending the updated model to the federated learning server using relative entropy coding, and Includes, The step of sending the updated model to the federated learning server using relative entropy coding is: A step of determining a random seed, which includes the step of receiving the random seed from the federated learning server, The steps include determining a first probability distribution based on the aforementioned global model, The steps include determining a second probability distribution centered on the updated model, A step of determining a plurality of random samples from the first probability distribution according to the random seed, based on the difference between the first probability distribution and the second probability distribution, wherein the plurality of random samples are associated with a plurality of parameters of the global model. A step of assigning a probability to each of the plurality of random samples based on the ratio of the likelihood of each random sample given a second probability distribution to the likelihood of each random sample given a first probability distribution, The steps include selecting one random sample from the plurality of random samples according to the probability of each of the plurality of random samples, The steps include determining the index associated with the selected random sample, The steps include sending the aforementioned index to the federated learning server and Methods that include...

2. The aforementioned index, log 2 Sent using Kbits, K is the number of the plurality of random samples from the first probability distribution. The method according to claim 1.

3. The method according to claim 1, wherein the plurality of random samples are associated with layers of the global model.

4. The method according to claim 1, wherein the plurality of random samples are associated with a subset of the parameters of the global model.

5. The process further includes a step of clipping the updated model before determining the second probability distribution centered on the updated model, The clipping is based on the standard deviation of the global model, The second probability distribution described above is based on the clipped update model, The method according to claim 1.

6. The method according to claim 5, wherein the step of clipping the updated model includes the step of clipping the norm of the updated model.

7. The method according to claim 1, wherein the step of determining the updated model based on the global model and local data includes the step of performing a gradient descent on the global model using the local data.

8. A computer implementation method, Steps to send the global model to the client device, A step of determining a random seed, which includes a step of receiving the random seed from the client device, The steps include receiving an updated model from the client device using relative entropy coding, A step of determining an updated global model based on the updated model from the client device. Includes, The step of receiving the updated model from the client device using relative entropy coding is: The steps include receiving an index from the client device, A step of determining a sample from a probability distribution based on the global model, the random seed, and the index, A step of using the determined sample to determine the updated global model, wherein the determined sample is used to update the parameters of the updated global model. Methods that include...

9. The aforementioned index, log 2 Received using K bits, K is the number of random samples determined from the probability distribution based on the global model. The method according to claim 8.

10. The method according to claim 8, wherein the determined sample is used to update the layer of the updated global model.

11. A processing system comprising: a memory containing computer executable instructions; and one or more processors configured to execute computer executable instructions and cause the processing system to perform the method according to any one of claims 1 to 10.

12. A processing system comprising means for performing the method described in any one of claims 1 to 10.

13. A non-temporary computer-readable medium comprising a computer-executable instruction that, when executed by one or more processors of a processing system, causes the processing system to perform the method according to any one of claims 1 to 10.

14. A computer program embodied on a computer-readable storage medium, comprising code for performing the method described in any one of claims 1 to 10.

Images