Communication Efficient Federated Learning

Abstract

The present disclosure relates to communication efficient federated learning. The present disclosure provides an efficient communication technique for transmitting model updates within a machine learning framework, such as, for example, a federated learning framework, in which a high-quality centralized model is trained on training data distributed across a large number of clients, each of which has unreliable network connections and low computing power. In an example federated learning setting, in each of multiple rounds, each client independently updates a model based on its local data and transmits the updated model back to a server, where all client-side updates are used to update a global model. The present disclosure provides systems and methods for reducing communication costs. In particular, the present disclosure provides at least: a structured update scheme in which model updates are constrained to be small and approximate, and in which model updates are compressed before being sent to a server.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201710874932.2, filed on September 25, 2017. Technical Field

[0003] The present disclosure relates generally to machine learning and more particularly to communication-efficient federated learning. Background Art

[0004] As data sets become larger and models more complex, machine learning increasingly requires distributing the optimization of model parameters across multiple machines. Existing machine learning algorithms are typically only applicable to controlled environments (such as data centers) where data is properly distributed across machines and high-throughput fiber networks are available. Summary of the invention

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the description which follows, or may be learned from the description, or may be learned through practice of the embodiments.

[0006] One example aspect of the present disclosure is directed to a computer-implemented method for communication-efficient machine learning. The method includes: obtaining, by a client computing device, a global value of a parameter set of a machine learning model. The method includes: training, by the client computing device, the machine learning model based at least in part on a local data set to obtain an update matrix, the update matrix describing updated values ​​of the parameter set of the machine learning model. The update matrix is ​​constrained to have a predefined structure. The local data set is stored locally by the client computing device. The method includes: transmitting, by the client computing device, information describing the update matrix to a server computing device.

[0007] Another example aspect of the present disclosure relates to a client computing device. The client device includes at least one processor; and at least one non-transitory computer-readable medium storing instructions that, when executed by the at least one processor, cause the client computing device to perform operations. The operations include obtaining global values ​​for a set of parameters of a machine learning model. The operations include: training the machine learning model based at least in part on a local data set to obtain an update matrix that describes updated values ​​for the set of parameters of the machine learning model. The local data set is stored locally by the client computing device. The operations include encoding the update matrix to obtain an encoded update. The operations include transmitting the encoded update to a server computing device.

[0008] Another example aspect of the present disclosure relates to at least one non-transitory computer-readable medium storing instructions that, when executed by a client computing device, cause the client computing device to implement operations. The operations include obtaining global values ​​for a set of parameters of a machine learning model. The operations include training the machine learning model based at least in part on a local data set to obtain an update matrix that describes updated values ​​for the set of parameters of the machine learning model. The local data set is stored locally by the client computing device. The update matrix is ​​constrained to be at least one of a low-rank matrix and a sparse matrix. The operations include encoding the update matrix to obtain an encoded update. The operations include transmitting the encoded update to a server computing device.

[0009] Other aspects of the disclosure relate to various systems, apparatus, non-transitory computer-readable media, user interfaces, and electronic devices.

[0010] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following technical effects. For example, in various embodiments, efficient processing can be achieved by training machine learning models locally on multiple client computing devices (e.g., user mobile devices), thereby making full use of the computing power of multiple devices. In addition, training machine learning models based on local data sets by client computing devices can improve the security of the training process. This is because, for example, information about model updates is less sensitive than the data itself. Privacy-sensitive user data is retained on the user's computing device and is not uploaded to the server. Instead, only less sensitive model updates are transmitted.

[0011] Furthermore, due to the asymmetric nature of Internet connections, the uplink from the client to the server may be slower than the downlink from the server to the client, such that it may be inefficient for each client device to transmit a complete, unrestricted model. However, by constraining the update matrix to have a predefined structure according to various embodiments (e.g., by constraining the update matrix to be at least one of a low-rank matrix and a sparse matrix), more efficient communication within the machine learning framework is achieved, e.g., from a network perspective, from a client device workload / resource perspective, and / or from an attempt to perform as many rounds of learning as possible / update as quickly as possible.

[0012] These and other features, aspects, and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] A detailed description of embodiments for those of ordinary skill in the art is set forth in the specification with reference to the accompanying drawings, in which:

[0014] Figure 1 A block diagram of an example computing system is depicted according to an example embodiment of the present disclosure.

[0015] Figure 2 A block diagram of an example computing system is depicted according to an example embodiment of the present disclosure.

[0016] Figure 3 Depicted is a flow chart of an example method for performing communication-efficient machine learning according to an example embodiment of the present disclosure.

[0017] Figure 4 Depicted is a flow chart of an example method for performing communication-efficient machine learning according to an example embodiment of the present disclosure.

[0018] Figure 5 Depicted is a flow chart of an example method for performing communication-efficient machine learning according to an example embodiment of the present disclosure.

[0019] Figure 6 A graph depicting example experimental results according to example embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] introduce

[0021] In general, the present disclosure provides systems and methods for performing efficient communication within a machine learning framework. For example, the efficient communication techniques provided herein can be implemented in the context of a joint learning framework. However, the efficient communication techniques provided herein can also be implemented in other machine learning frameworks. For example, the efficient communication techniques provided herein can be implemented in any instance where a first computing device is responsible for calculating updates to a model and / or transmitting updates to the model to a second computing device.

[0022] More specifically, federated learning is a machine learning framework that enables training of high-quality centralized models based on training data distributed across a large number of client computing devices. Clients typically have low computing power and / or slow / unstable connections to the network. In some instances, federated learning may also be referred to as "federated optimization."

[0023] An example that motivates federated learning occurs when training data is kept locally on a user's mobile computing device and such mobile computing device is used as a node for which computations are performed on local data to update a global model. Thus, the federated learning framework differs from conventional distributed machine learning because the number of clients is large, the data is highly unbalanced and not independently and identically distributed ("IID"), and network connections are unreliable.

[0024] Compared to performing learning on a centralized server, federated learning offers several distinct advantages. For example, the information updated by the model is less sensitive than the data itself. Therefore, privacy-sensitive user data remains on the user's computing device and is not uploaded to the server. Instead, only less sensitive model updates are transmitted. As another advantage, federated learning takes full advantage of the computing power of a large number of computing devices (e.g., user mobile devices). In addition, slight modifications to the above framework can result in the creation and use of personalized models for each user at their respective devices.

[0025] In some embodiments, a system implementing federated learning may perform the following actions in each of multiple rounds of model optimization: a subset of clients is selected; each client in the subset updates the model based on its local data; the updated model or model updates are sent by each client to the server; the server aggregates the updates (e.g., by averaging the updates) and improves the global model; and the server redistributes the global model to all clients. Performing multiple rounds of the above actions iteratively improves the global model based on the training data stored at the client devices.

[0026] In a basic implementation of the above framework, each client device sends a complete model back to the server in each round. However, the transmission of the complete model to each client device is obviously "expensive" (e.g., from the perspective of the network, from the perspective of the workload / resources of the client device, and / or from the perspective of trying to perform as many rounds of learning / updates as quickly as possible). In particular, due to the asymmetric nature of Internet connections, the uplink from the client to the server is typically much slower than the downlink from the server to the client.

[0027] In view of such significant upload costs, the present disclosure provides systems and methods for performing efficient communication within a machine learning framework (e.g., within the federated learning framework discussed above). In particular, the present disclosure provides efficient communication techniques that reduce the cost of transmitting updated models or model updates from client devices to servers.

[0028] More specifically, the present disclosure provides communication efficient schemes including: a structured update scheme in which model updates are constrained to be small and sketched updates, wherein the model updates are compressed before being sent to the server. These schemes can be combined, for example by first learning a structured update and then sketching it.

[0029] Example Model Update Symbols

[0030] Before discussing the communication efficient scheme provided by the present disclosure, the basic implementation of model updating in a federated learning environment will be discussed and the notation will be introduced.

[0031] In an example joint learning framework, the goal is to learn As an example, the model may include one or more neural networks (e.g., deep neural networks, recursive neural networks, convolutional neural networks, etc.) or other machine learning models.

[0032] In round t ≥ 0, the server will use the current model W t Distribute to n t A subset of clients S t (e.g., to a selected subset of clients whose devices are plugged into a power source, have access to broadband, and are idle). Some or all of these clients independently update the model based on their local data. The updated local model is Make the update:

[0033]

[0034] Each client then sends updates back to the server, where a global update is computed by aggregating all client-side updates. For example, an aggregation scheme could be as follows:

[0035]

[0036] In addition, in some embodiments, a weighted sum based on expected performance can be used instead of the average. The server can also choose a learning rate η t For example, careful selection of the server-side learning rate can lead to faster convergence. However, in at least one example, using η t = 1 and no weighting is used.

[0037] In implementations that perform deep learning, a separate matrix W may be used to represent the parameters of each layer. Therefore, each reference to a matrix (e.g., model parameter matrix W) in this document Or update the matrix ) should be understood not to be limited to a single matrix, but instead refers to one or more matrices that can be used to represent a given model or an update to such a model. For example, a neural network with five layers may have five model parameter matrices W representing the parameters of such layers, respectively, and five update matrices W representing updates to the parameters of such layers, respectively.

[0038] In many instances, W and is a 2D matrix. W can be right-multiplied, such that d1 and d2 represent the output and input dimensions, respectively. For example, the parameters of a fully connected layer itself form a 2D matrix. However, the kernel of a convolutional layer is a 4D tensor of shape #input×width×height×#output. In such a case, W can be reshaped from the kernel to shape (#input×width×height)×#output.

[0039] According to an aspect of the present disclosure, there is provided a method for reducing the amount of data sent to a server. Specifically, the present disclosure provides a communication efficiency solution, which includes: A structured update scheme that is limited to being small or otherwise conforming to a certain structure and before being sent to the server A compressed or otherwise encoded summary update scheme.

[0040] Example Structured Update Technique

[0041] The first type of communication efficient update provided by the present disclosure is one in which Structured Updates with Predefined Structures. In particular, two types of structured update techniques will be described in more detail: low-rank techniques and random masking techniques.

[0042] In the low-rank structured update technique, each update can be constrained or forced to be a low-rank matrix (e.g., a low-rank matrix of at most rank k, where k is a fixed number). In some embodiments, the update Restricting to low-rank matrices can be done by It is implemented by expressing it as the product of two matrices.

[0043] As an example, is a low-rank matrix with rank k. Update It can be expressed as the product of two matrices: in is randomly generated and fixed, and In some embodiments, the client device only needs to send Because it can be generated or otherwise known by the server In particular, in some embodiments, both the client and the server may know the synchronization random seed. Both the client and the server may use the random seed (e.g., together with a pseudo-random number generator) to separately generate Therefore, the client device only needs to send Because the server can use the locally generated and To reproduce

[0044] In some embodiments, fixation (e.g., randomly or deterministically) is also an option.

[0045] In some implementations, low-rank structured update techniques may provide a compression ratio of k / d1. In one example, the seed may be 32 bits long.

[0046] In the random mask structured update technique, each update May be constrained or limited to be a sparse matrix. For example, each sparse matrix may follow a predefined sparse pattern (e.g., a predefined random sparse pattern, such as a random mask). In some embodiments, a different sparse pattern is generated for each round and / or each client in each round.

[0047] Similar to low-rank schemes, the sparsity pattern can be fully specified or otherwise generated based on a random seed. Therefore, the client device only needs to send For example, a server that also knows the seed may generate a sparse pattern and determine, based on such a pattern, which communicated values ​​correspond to which parameters.

[0048] Example Overview Update Technique

[0049] The second type of communication efficient update provided by the present disclosure is one in which the client updates the data in a compressed form before sending to the server. The client device can calculate the complete update And then encode the update, or the update can be calculated according to structured techniques And then encode such structured updates. The server can decode the update before aggregation.

[0050] The present disclosure contemplates many different types of encoding or compression. For example, the compression may be lossless or lossy. Two example encoding techniques are described in more detail below: a subsampling technique and a quantization technique.

[0051] One way to encode an update is to sample only a random subset of the parameters described by the update. In other words, instead of transmitting the full update Each client device can transmit matrix It only includes For example, A subset of the values ​​of can be A random subset of the scaling values ​​for .

[0052] The server then aggregates the sampled updates (e.g., by taking the average) to produce a global update Sampling in a random manner has the following property: the updated mean after sampling is an unbiased estimate of the true mean:

[0053] In some embodiments, the portion of parameters that are sampled can be selected by using, for example, a random mask as described above. In particular, a seed can be used to form a parameter mask that identifies which parameters are sampled. A different mask can be generated for each round and / or for each client in each round. The mask itself can be stored as a synchronized seed. Perturbing the expected iterations of SGD by a random matrix of zero mean - which is what some embodiments of the subsampling strategy do - does not affect this type of convergence.

[0054] Another way to encode the updates is by quantizing the weights. For example, the weights can be quantized probabilistically.

[0055] First, we will describe the algorithm for quantizing each scalar value to 1 bit. Consider the update make And make h max =max j (h j ),h min =min j (h j ). The compressed update of h is It can be generated as follows:

[0056]

[0057] Easy to show is an unbiased estimate of h. When the uncompressed scalar value is described by 4-byte floating point, the above 1-bit method provides 32 times compression.

[0058] Furthermore, the above can be generalized to more than 1 bit per scalar. For example, for b-bit quantization, [h min ,h max ] is divided into 2 b Assume h i falls into the interval defined by h′ and h″. By replacing h in the above equation with h′ and h″ respectively min and h max Can be quantified.

[0059] In some embodiments, the intervals need not be evenly spaced, but may instead be dynamically spaced. For embodiments using dynamically spaced intervals, the client may provide a table identifying the max / min values ​​for each interval to the server. The client will typically transmit the min and max to the server regardless of whether dynamic spacing is used. In some embodiments, incremental, random, and / or distributed optimization algorithms may similarly be used in a quantized update setting.

[0060] According to another aspect of the present disclosure, the above quantization techniques can be improved by using random rotations. More specifically, the above 1-bit and multi-bit quantization schemes work best when the scalars are approximately equal across different dimensions. For example, when max=100 and min=-100 and most values ​​are 0, 1-bit quantization will result in large quantization errors.

[0061] Therefore, according to one aspect of the present disclosure, performing a random rotation of h before quantization can solve this problem. For example, the update can be multiplied by a reversible matrix R. For example, the rotation matrix can be an orthogonal matrix. Randomly rotating h before quantization can distribute the scalar values ​​more evenly across the interval. In the decoding stage, the server can perform the inverse rotation before aggregating all updates.

[0062] In some embodiments, the dimension of h can be as high as d=1M, and is generated in the rotation matrix and Applications is computationally prohibitive. Therefore, according to another aspect of the present disclosure, a structured rotation matrix may be used. For example, the structured rotation matrix may not require explicit construction of the client computing device, but instead has a defined structure that enables calculation using R without explicit construction, for example, the structured rotation matrix may be the product of a Walsh-Hadamard matrix and a binary diagonal matrix. This can reduce the time required to generate and apply the matrix to and The computational complexity of .

[0063] In some embodiments, the same rotation matrix is ​​used for all client devices and / or for all rounds. In some embodiments, the server may then perform some aggregation on the updates before derotating the aggregated updates. In some embodiments, the server decodes each update before aggregation.

[0064] Example System

[0065] Figure 1An example system 100 for training one or more global machine learning models 106 using corresponding training data 108 stored locally on multiple client devices 102 is depicted. The system 100 may include a server device 104. The server 104 may be configured to access the machine learning model 106 and provide the model 106 to multiple client devices 102. For example, the model 106 may be a linear regression model, a logistic regression model, a support vector machine model, a neural network (e.g., a convolutional neural network, a recurrent neural network, etc.), or other suitable model. In some embodiments, the server 104 may be configured to communicate with the client device 102 via one or more networks.

[0066] Each of the client devices 102 may be configured to determine one or more local updates associated with the model 106 based at least in part on the training data 108. For example, the training data 108 may be data that is respectively stored locally on the client device 106. The training data 108 may include audio files, image files, video files, typing history, location history, and / or various other suitable data. In some implementations, the training data may be any data derived through user interaction with the client device 102.

[0067] In addition to the above description, the user may be provided with controls to allow the user to select whether and when the systems, programs, or features described herein can collect, store, and / or use user information (e.g., training data 108) and whether the user sends content or communications from the server. In addition, certain data may be processed in one or more ways before being stored or used so that personally identifiable information is removed. For example, the user's identifiable information may be processed so that it is impossible to determine the user's personally identifiable information, or the user's geographic location may be generalized (such as to a city, zip code, or state level) when location information is obtained so that the user's specific location cannot be determined. Therefore, the user can control what information is collected about the user, how the information is used, and what information is provided to the user.

[0068] Although the training data 108 Figure 1 Although illustrated as a single database in , the training data 108 consists of data stored separately at each device 102. Therefore, in some embodiments, the training data 108 is highly unbalanced and is not independently and identically distributed.

[0069] The client device 102 may be configured to provide local updates to the server 104. As described above, the training data 108 may be privacy sensitive. In this manner, local updates may be performed and provided to the server 104 without compromising the privacy of the training data 108. For example, in such an embodiment, the training data 108 is not provided to the server 104. The local updates do not include the training data 108. In some embodiments where the locally updated model is provided to the server 104, some privacy sensitive data may be able to be derived or inferred from the model parameters. In such an embodiment, one or more of encryption techniques, random noise techniques, and / or other security techniques may be added to the training process to obscure any inferable information.

[0070] As described above, server 104 can receive each local update from client device 102 and can aggregate the local updates to determine a global update to model 106. In some implementations, server 104 can determine an average (e.g., a weighted average) of the local updates and determine a global update based at least in part on the average.

[0071] In some embodiments, scaling or other techniques may be applied to local updates to determine global updates. For example, a local step size may be applied to each client device 102, aggregation may be performed in proportion to the size of the individual data partitions of the client devices 102, and / or one or more scaling factors may be applied to local updates and / or aggregated updates. It should be appreciated that various other techniques may be applied without departing from the scope of the present disclosure.

[0072] Figure 2 An example computing system 200 is depicted that can be used to implement the methods and systems of the present disclosure. The system 200 can be implemented using a client-server architecture that includes a server 210 communicating with one or more client devices 230 via a network 242. Thus, Figure 2 Provides the ability to implement Figure 1 An example system 200 is shown in the manner illustrated by system 100 .

[0073] System 200 includes a server 210, such as a web server. Server 210 may be implemented using any suitable computing device. Server 210 may have one or more processors 212 and one or more memory devices 214. Server 210 may be implemented using one server device or multiple server devices. In an embodiment using multiple devices, such multiple devices may operate according to a parallel computing architecture, a serial computing architecture, or a combination thereof.

[0074] Server 210 may also include a network interface for communicating with one or more client devices 230 over network 242. The network interface may include any suitable components for interfacing with one or more networks, including, for example, transmitters, receivers, ports, controllers, antennas, or other suitable components.

[0075] The one or more processors 212 may include any suitable processing device, such as a microprocessor, a microcontroller, an integrated circuit, a logic device, or other suitable processing device. The one or more memory devices 214 may include one or more computer-readable media, including but not limited to non-transitory computer-readable media, RAM, ROM, a hard drive, a flash drive, or other memory devices. The one or more memory devices 214 may store information accessible by the one or more processors 212, including computer-readable instructions 216 executable by the one or more processors 212.

[0076] The instructions 216 may be any set of instructions that, when executed by the one or more processors 212, cause the one or more processors 212 to perform operations. For example, the instructions 216 may be executed by the one or more processors 212 to implement the global updater 220. The global updater 220 may be configured to receive one or more local updates and determine a global model based at least in part on the local updates.

[0077] Instructions 216 may also include instructions to cause server 210 to implement decoder 222. Decoder 222 may decode updates that have been encoded by client device 230 (eg, according to one of the encoding techniques discussed above, such as subsampling, quantization, random rotation, etc.).

[0078] like Figure 2 As shown, one or more memory devices 214 may also store data 218, which may be retrieved, manipulated, created, or stored by one or more processors 212. Data 218 may include, for example, local updates, global parameters, and other data. Data 218 may be stored in one or more databases. One or more databases may be connected to server 210 via a high bandwidth LAN or WAN, or may also be connected to server 210 via network 242. One or more databases may be split so that they are located in multiple locations.

[0079] Server 210 may exchange data with one or more client devices 230 via network 242. Any number of client devices 230 may be connected to server 210 via network 242. Each of client devices 230 may be any suitable type of computing device, such as a general purpose computer, a special purpose computer, a laptop computer, a desktop computer, a mobile device, a navigation system, a smart phone, a tablet computer, a wearable computing device, a game console, a display with one or more processors, or other suitable computing device.

[0080] Similar to the server 210, the client device 230 may include one or more processors 232 and memory 234. The one or more processors 232 may include, for example, one or more central processing units (CPUs), graphics processing units (GPUs) dedicated to efficiently rendering images or performing other specialized calculations, and / or other processing devices. The memory 234 may include one or more computer-readable media and may store information accessible by the one or more processors 232, including instructions 236 and data 238 that may be executed by the one or more processors 232.

[0081] Instructions 236 may include instructions for implementing a local updater that is configured to determine one or more local updates according to example aspects of the present disclosure. For example, the local updater may perform one or more training techniques, such as back-propagating errors to retrain or otherwise update the model based on locally stored training data. The local updater may be configured to perform structured updates, summary updates, or other techniques. The local updater may be included in an application, or may be included in an operating system of device 230.

[0082] Instructions 236 may also include instructions for implementing an encoder. For example, the encoder may perform one or more of the above encoding techniques (eg, subsampling, quantization, random rotation, etc.).

[0083] Data 238 may include one or more training data examples for solving one or more optimization problems. The training data examples for each client device 230 may be unevenly distributed among the client devices, such that client device 230 does not include a representative sample of the overall distribution of training data examples.

[0084] Data 238 may also include updated parameters to be transmitted to server 210 .

[0085] Figure 2 The client device 230 may include various input / output devices for providing and receiving information from a user, such as a touch screen, a touch pad, data entry keys, a speaker, and / or a microphone suitable for voice recognition.

[0086] Client device 230 may also include a network interface for communicating with one or more remote computing devices (e.g., server 210) over network 242. The network interface may include any suitable components for interfacing with one or more networks, including, for example, a transmitter, receiver, port, controller, antenna, or other suitable components.

[0087] The network 242 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), a cellular network, or some combination thereof. The network 242 may also include a direct connection between the client device 230 and the server 210. In general, the communication between the server 210 and the client device 230 may be implemented via a network interface using any type of wired and / or wireless connection, using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection methods (e.g., VPN, secure HTTP, SSL).

[0088] Example Method

[0089] Figure 3 A flowchart of an example method (300) for determining a global model according to an example embodiment of the present disclosure is depicted. The method (300) may be implemented by one or more computing devices, such as Figure 1 and / or one or more of the computing devices depicted in 2. In addition, Figure 3 The steps are depicted as being performed in a specific order for purposes of illustration and discussion. One of ordinary skill in the art, using the disclosure provided herein, will understand that the steps of any method discussed herein may be adjusted, rearranged, expanded, omitted, or modified in various ways without departing from the scope of the present disclosure.

[0090] At (302), the method (300) may include determining, by the client device, a local model based on one or more local data examples. In particular, the local model may be determined for a loss function using the one or more data examples. For example, the data examples may be generated by a user interacting with the client device. In some embodiments, the model may have been pre-trained prior to the local training at (302). In some embodiments, structured updates, summary updates, or other techniques may be used at (302) to make the learned local model or local updates communication efficient.

[0091] At (304), the method (300) may include providing, by the client device, the local model to the server, and at (306), the method (300) may include receiving, by the server, the local model. In some implementations, the local model or local update may be encoded or compressed before being sent to the server.

[0092] At (308), the method (300) may include determining, by the server, a global model based at least in part on the received local models. For example, the global model may be determined based at least in part on a plurality of local models provided by a plurality of client devices, each of which has a plurality of unevenly distributed data examples. In particular, the data examples may be distributed among the client devices such that the client devices do not include a representative sample of the overall distribution of the data. Furthermore, the number of client devices may exceed the number of data examples on any one client device.

[0093] In some implementations, as part of the aggregation process, the server may decode each received local model or local update.

[0094] At (310), the method (300) may include providing the global model to each client device, and at (312), the method (300) may include receiving the global model.

[0095] At (314), the method (300) may include determining, by the client device, a local update. In a particular embodiment, the local update may be determined by retraining or otherwise updating a global model based on locally stored training data. In some embodiments, structured updates, summary updates, or other techniques may be used at (314) to make the learned local model or local update communication efficient.

[0096] In some embodiments, the local update can be determined based at least in part on using one or more random updates or iterations. For example, the client device can randomly sample partitions of data examples stored on the client device to determine the local update. In particular, the local update can be determined using a random model descent technique to determine a direction to adjust one or more parameters of the loss function.

[0097] In some embodiments, a step size associated with the local update determination may be determined based at least in part on the number of data examples stored on the client device. In further embodiments, the stochastic model may be scaled using a diagonal matrix or other scaling techniques. In other embodiments, the local update may be determined using a linear term that forces each client device to update the parameters of the loss function in the same direction.

[0098] At (316), the method (300) may include providing, by the client device, the local update to the server. In some implementations, the local model or the local update may be encoded before sending the local model or the update to the server.

[0099] At (318), the method (300) may include receiving, by the server, a local update. Specifically, the server may receive a plurality of local updates from a plurality of client devices.

[0100] At (320), the method (300) may include again determining the global model. In particular, the global model may be determined based at least in part on the received local updates. For example, the received local updates may be aggregated to determine the global model. The aggregation may be an additive aggregation and / or an average aggregation. In a particular embodiment, the aggregation of local updates may be proportional to the partition size of the data instance on the client device. In another embodiment, the aggregation of local updates may be scaled on a per-coordinate basis.

[0101] Any number of iterations of local and global updates may be performed. That is, the method (300) may be iteratively performed to update the global model based on locally stored training data over time.

[0102] Figure 4 A flow chart of an example method 400 for performing communication efficient machine learning according to an example embodiment of the present disclosure is depicted. For example, the method 400 may be performed by a client computing device.

[0103] At 402, a client computing device obtains a global value for a set of parameters of a machine learning model.

[0104] At 404, the client computing device trains the machine learning model based at least in part on the local data set to obtain an update matrix that describes updated values ​​of the parameter set of the machine learning model. The update matrix is ​​constrained to have a predefined structure. The local data set is stored locally by the client computing device. In some embodiments, the update matrix describes updated values ​​of the parameter set and / or differences between the updated values ​​and global values.

[0105] In some embodiments, the update matrix is ​​restricted to be a low rank matrix.

[0106] In some implementations, training the machine learning model at 404 may include defining, by the client computing device, an update matrix as a product of a first matrix and a second matrix. The first matrix may include fixed values, and the second matrix may include optimizable variables. The client computing device may train the machine learning model based at least in part on the local data set to obtain the second matrix.

[0107] In some such embodiments, method 400 may further include: generating a first matrix based at least in part on a seed and a pseudo-random number generator prior to training the model at 404. The client computing device and the server computing device may both know the seed so that the first matrix can be reproduced by the server computing device.

[0108] In some implementations, the update matrix is ​​restricted to be a sparse matrix.

[0109] In some implementations, training the machine learning model at 404 may include training the machine learning model based at least in part on the local data set so that updated values ​​are determined only for a preselected portion of the parameter set. In such implementations, the update matrix may describe only updated values ​​for the preselected portion of the parameter set.

[0110] In some such embodiments, method 400 may further include: prior to training the model at 404: generating a parameter mask that specifies which parameters in the parameter set are included in the preselected portion of the parameter set. For example, generating the parameter mask may include generating, by the client computing device, the parameter mask based at least in part on a seed and a pseudo-random number generator, wherein the seed is known to both the client computing device and the server computing device, such that the parameter mask is reproducible by the server computing device.

[0111] At 406 , the client computing device transmits information describing the update matrix to the server computing device.

[0112] As an example, in embodiments where training the model at 404 includes optimizing a second matrix, transmitting information describing the updated matrix at 406 may include transmitting information describing the second matrix to a server computing device.

[0113] Figure 5 A flow chart of an example method 500 for performing communication efficient machine learning according to an example embodiment of the present disclosure is depicted. For example, the method 500 may be performed by a client computing device.

[0114] At 502, a client computing device obtains a global value for a set of parameters of a machine learning model.

[0115] At 504, the client computing device trains the machine learning model based at least in part on the local data set to obtain an update matrix describing updated values ​​of a set of parameters of the machine learning model. The local data set is stored locally by the client computing device. In some embodiments, the update matrix describes updated values ​​of the set of parameters and / or differences between the updated values ​​and global values.

[0116] At 506 , the client computing device encodes the update matrix to obtain encoded updates.

[0117] In some embodiments, encoding the update matrix at 506 may include subsampling the update matrix to obtain the encoded update. In some such embodiments, subsampling the update matrix may include: generating a parameter mask that specifies a portion of a parameter set to be sampled; and subsampling the update matrix according to the parameter mask.

[0118] In some such implementations, generating the parameter mask may include generating the parameter mask based at least in part on a seed and a pseudo-random number generator, wherein the seed is known to both the client computing device and the server computing device such that the parameter mask is reproducible by the server computing device.

[0119] In some embodiments, encoding the update matrix at 506 may include probabilistically quantizing one or more values ​​included in the update matrix. In some embodiments, encoding the update matrix at 506 may include performing probabilistic binary quantization on one or more values ​​included in the update matrix to change each of the one or more values ​​to a maximum value included in the update matrix or a minimum value included in the update matrix. In some embodiments, encoding the update matrix at 506 may include: defining a plurality of intervals between a maximum value included in the update matrix and a minimum value included in the update matrix; and probabilistically changing one or more values ​​included in the update matrix to a local interval maximum value or a local interval minimum value.

[0120] In some embodiments, encoding the update matrix at 506 may include multiplying the vector of the update matrix by the rotation matrix to obtain the rotation update. In some such embodiments, encoding the update matrix may also include probabilistically quantizing one or more values ​​included in the rotation update. In some embodiments, the rotation matrix may be a structured rotation matrix that does not require the client computing device to fully generate the rotation matrix.

[0121] The client computing device transmits the encoded update to the server computing device at 508. The server computing device may decode the encoded update.

[0122] Example Experiment

[0123] Example experiments were performed using federated learning to train a deep neural network for the CIFAR-10 image classification task (see Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009). There are 50,000 training examples, which are divided into 100 clients, each containing 500 training examples. The model architecture is taken from the TensorFlow tutorial (Tensorflowconvolutional neural networks tutorial. http: / / www.tensorflow.org / tutorials / deep_cnn, 2016), which consists of two convolutional layers, followed by two fully connected layers, and then a linear transformation layer to generate logits for a total of more than 1e6 parameters. Although this model is not state-of-the-art, it is different from the best possible accuracy for achieving this task, which is sufficient for the purpose of evaluating the compression method described in this article.

[0124] A federated averaging algorithm is employed (McMahan et al. Federated Learning of deep networks using model platforms. ArXiv: 1602.05629, 160), which greatly reduces the number of communication rounds required to train a good model. However, these techniques are expected to show similar reductions in communication costs when applied to synchronous SGD. For federated averaging, 10 clients are randomly selected per round, each of which performs 10 epochs of SGD on its local dataset with a learning rate of η using a mini-batch of 50 images, for a total of 100 local updates. From this updated model, compute The delta of each layer.

[0125] Table 1 provides the low-rank and sampling parameters for the example CIFAR experiment. The sampling probability column gives the fraction of elements uploaded for each of the two convolutional layers and the two fully connected layers; these parameters are used by StructMask, SketchMask, and SketchRotMask. The low-rank column gives the rank limit k for these four layers. The final softmax layer is small, so updates to it are not compressed.

[0126] Table 1:

[0127]

[0128] Figure 6 Plots depict example non-quantized results (left and middle columns) and results including binary quantization (dashed lines SketchRotMaskBin and SketchMaskBin, right column). Note that the x-axis of the upper right plot is logarithmic in scale. Over 70% accuracy is achieved with less than 100MB of communication.

[0129] Medium and high rank / sampling parameter settings are defined so that the compression ratios of the two schemes are the same, as given in Table 1. Figure 6 The left and center columns of show the unquantized results of the test set accuracy, both as a function of the number of algorithm rounds and as a function of the total number of megabytes uploaded. For all experiments, a resolution centered at 0.15 The learning rate is tuned using a multiplicative grid of . The learning rate results are plotted with the best median accuracy over 400-800 rounds. A multiplicative learning rate decay of 0.988 is used, which is chosen by tuning only for the baseline algorithm.

[0130] For moderate subsampling, all three schemes provide large improvements in test set accuracy after fixed bandwidth usage; the bottom plot shows little loss in accuracy as a function of the number of update rounds, except for the StructLowRank scheme which performs poorly for high subsampling parameters.

[0131] Figure 6 The two plots on the right give the results for SketchMask and SketchRotMask with and without binary quantization; only the medium subsampling regime is considered, which is representative. It is observed (as expected) that introducing random rotations has essentially no effect in the absence of quantization. However, binary quantization greatly reduces the total communication cost, and further introducing random rotations significantly speeds up convergence and also allows convergence to higher accuracy levels. A reasonable model (70% accuracy) can be learned with only about 100MB of communication, two orders of magnitude smaller than the baseline.

[0132] Additional disclosure

[0133] The technology discussed herein quotes servers, databases, software applications, and other computer-based systems, as well as the measures taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows configuration, combination, and division of various possible tasks and functions between and among components. For example, a single device or component or multiple devices or components working in combination can be used to implement the process discussed herein. Databases and application programs can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0134] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of illustration rather than limitation of the present disclosure. Those skilled in the art can easily make changes, modifications, and equivalents to such embodiments after understanding the foregoing. Therefore, the present disclosure does not exclude such modifications, modifications, and / or additions to the subject matter that are obvious to those of ordinary skill in the art. For example, a feature illustrated or described as a part of an embodiment can be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover such changes, modifications, and equivalents.

Claims

1. A client computing device, comprising: at least one processor; and at least one non-transitory computer-readable medium storing instructions that, when executed by the at least one processor, cause the client computing device to perform operations comprising: Get the global value of the parameter set of the machine learning model; training the machine learning model based at least in part on a local data set to obtain an update matrix describing updated values ​​of the set of parameters of the machine learning model, wherein the local data set includes data describing input received by the client computing device from a user of the client computing device; encoding the update matrix to obtain an encoded update; and The encoded update is transmitted to a server computing device.

2. The client computing device of claim 1, wherein: Encoding the update matrix includes sub-sampling the update matrix, wherein sub-sampling the update matrix includes: generating a parameter mask specifying a portion of the parameter set to be sampled; and The update matrix is ​​sub-sampled according to the parameter mask.

3. The client computing device of claim 2, wherein: Generating the parameter mask includes generating the parameter mask based at least in part on a seed and a pseudo-random number generator, wherein the seed is known to both the client computing device and the server computing device such that the parameter mask can be reproduced by the server computing device.

4. The client computing device of claim 1, wherein: Encoding the update matrix includes probabilistically quantizing one or more values ​​included in the update matrix.

5. The client computing device of claim 1 , wherein: Encoding the update matrix includes performing probabilistic binary quantization on one or more values ​​included in the update matrix to turn each of the one or more values ​​into a maximum value included in the update matrix or a minimum value included in the update matrix.

6. The client computing device of claim 1, wherein: Encoding the update matrix comprises: defining a plurality of intervals between a maximum value included in the update matrix and a minimum value included in the update matrix; and Each of the one or more values ​​included in the update matrix is ​​probabilistically changed to a local interval maximum or a local interval maximum.

7. The client computing device of claim 1, wherein: Encoding the update matrix includes multiplying a vector of the update matrix by a rotation matrix to obtain a rotated update.

8. The client computing device of claim 7, wherein: Encoding the update matrix further comprises: One or more values ​​included in the rotated update are probabilistically quantized.

9. A client computing device according to claim 7 or claim 8, wherein: The rotation matrix is ​​a structured rotation matrix that does not require the client computing device to fully generate the rotation matrix.

10. At least one non-transitory computer-readable medium storing instructions that, when executed by a client computing device, cause the client computing device to perform operations comprising: Get the global value of the parameter set of the machine learning model; training the machine learning model based at least in part on a local data set to obtain an update matrix describing updated values ​​of the set of parameters of the machine learning model, wherein the local data set includes data describing input received by the client computing device from a user of the client computing device; encoding the update matrix to obtain an encoded update; and The encoded update is transmitted to a server computing device.

11. A computer-implemented method comprising: Obtaining, by the client computing device, a global value of a set of parameters of the machine learning model; training, by the client computing device, the machine learning model based at least in part on a local data set to obtain an update matrix describing updated values ​​of the set of parameters of the machine learning model, wherein the local data set includes data describing input received by the client computing device from a user of the client computing device; encoding the update matrix to obtain an encoded update; and The encoded update is transmitted by the client computing device to a server computing device.

12. The method according to claim 11, wherein: Encoding the update matrix includes sub-sampling the update matrix, wherein sub-sampling the update matrix includes: generating a parameter mask specifying a portion of the parameter set to be sampled; and The update matrix is ​​sub-sampled according to the parameter mask.

13. The method according to claim 12, wherein: Generating the parameter mask includes generating the parameter mask based at least in part on a seed and a pseudo-random number generator, wherein the seed is known to both the client computing device and the server computing device such that the parameter mask can be reproduced by the server computing device.

Images