Inducing sparsity in joint machine learning

Abstract

Aspects described herein provide techniques for performing federated learning of a machine learning model, including, for each respective client of a plurality of clients and for each training round of a plurality of training rounds: generating a subset of model elements for the respective client based on sampling a gate probability distribution for each model element of a set of model elements of a global machine learning model; transmitting, to the respective client: the subset of model elements; and a set of sampled gates, where each gate of the set of sampled gates is associated with one model element of the subset of model elements; receiving, from each respective client of the plurality of clients, a respective set of model updates; and updating the global machine learning model based on the respective set of model updates from each respective client of the plurality of clients.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit and priority of Greek patent application No. 20200100587, filed on September 28, 2020, the entire contents of which are incorporated herein by reference.

[0003] introduction

[0004] The various aspects of this disclosure relate to joint machine learning that induces sparsity.

[0005] Machine learning is generally the process of generating a trained model (e.g., an artificial neural network, a tree, or other structure) that represents a generalized fit to a training dataset. Applying the trained model to new data produces inferences, which can then be used to gain insights about the new data.

[0006] As the use of machine learning surges across various technological fields, sometimes referred to as artificial intelligence tasks, there is a growing need for more efficient processing of machine learning model data. For example, “edge processing” devices (such as mobile devices, always-on devices, Internet of Things (IoT) devices, etc.) must balance the implementation of advanced machine learning capabilities with a variety of interrelated design constraints (such as package size, native computing power, power storage and usage, data communication capabilities and costs, memory size, heat dissipation, etc.).

[0007] Federation learning is a distributed machine learning framework that enables several clients (such as edge processing devices) to collaboratively train a shared global model without transferring their local data to a remote server. Typically, a central server coordinates the federated learning process, and each participating client communicates its model parameters only to the central server while keeping its local data private. This distributed approach helps address the limitations of client device capabilities (because training is federated) and, in many cases, also mitigates data privacy concerns.

[0008] Even though federated learning generally limits the amount of model data in any single transmission between the server and client (or between the client and server), the iterative nature of federated learning still generates a significant amount of data transfer traffic during training, which can be costly depending on the device and connection type. Therefore, it is generally desirable to try to reduce the size of data exchanged between the server and client during federated learning. However, conventional methods for reducing data exchange have resulted in poor model performance, such as when using lossy compression of model data to limit the amount of data exchanged between the server and client.

[0009] Accordingly, improved methods for joint learning are needed, where model performance is not compromised by communication efficiency.

[0010] Brief Overview

[0011] Some aspects provide a method for performing joint learning of a machine learning model, comprising: for each of a plurality of clients and for each of a plurality of training epochs: generating a subset of model elements for that respective client based on sampling the gate probability distribution of each model element in a set of model elements of a global machine learning model; transmitting to the respective client: the subset of model elements; and a set of gate probabilities based on the sampled set, wherein each gate probability in the set of gate probabilities is associated with a model element in the subset of model elements; receiving a respective set of model updates from each of the plurality of clients; and updating the global machine learning model based on the respective set of model updates from each of the plurality of clients.

[0012] A further aspect provides a method for performing joint learning of a machine learning model, comprising: receiving from a server managing joint learning of a global machine learning model: a subset of model elements from a set of model elements of the global machine learning model; and a set of gate probabilities, wherein each gate probability in the set of gate probabilities is associated with a model element in the subset of model elements; generating a set of model updates based on training a local machine learning model according to the set of model elements and the set of gate probabilities; and transmitting the set of model updates to the server.

[0013] Other aspects include: a processing system configured to perform the foregoing methods and those methods described herein; a non-transient computer-readable medium including instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the foregoing methods and those methods described herein; a computer program product implemented on a computer-readable storage medium including code for performing the foregoing methods and those methods further described herein; and a processing system including means for performing the foregoing methods and those methods further described herein.

[0014] The following description and related figures illustrate certain illustrative features of one or more embodiments in detail. Brief description of the attached diagram

[0016] The accompanying drawings illustrate certain aspects of one or more embodiments and are therefore not intended to limit the scope of this disclosure.

[0017] Figure 1 An example training process for encouraging sparsity in federated learning is described.

[0018] Figure 2 An example method for performing joint learning that induces sparsity is described.

[0019] Figure 3 Another example method for performing joint learning that induces sparsity is described.

[0020] Figure 4 An example processing system that can be configured to perform aspects of the joint learning method described in this paper is depicted.

[0021] To facilitate understanding, the same reference numerals are used where possible to designate common elements in the various figures. Elements and features contemplated in one embodiment may be beneficially incorporated into other embodiments without further elaboration.

[0022] Detailed description

[0023] This disclosure provides apparatus, methods, processing systems, and computer-readable media for inducing joint machine learning of sparsity.

[0024] As machine learning models become increasingly complex and thus larger, training them anywhere other than on high-powered computers, such as servers, becomes increasingly difficult. Federation learning is a distributed machine learning framework that enables several clients (including lower-powered devices, such as edge processing devices) to collaboratively train a shared global model. In such settings, the general expectation is to reduce computational costs on client devices and overall communication costs. Specifically, high communication costs can make federated learning over mobile data impractical.

[0025] One approach to addressing these issues is "federated dropout," where the server selects a specific probability for choosing a sub-model from the original model before the joint training process. Subsequently, during training, the server opportunistically selects and delivers random sub-models to each client. Accordingly, instead of training updates to the entire global model locally, each client trains updates to the smaller sub-model. Since the sub-models are subsets of the global model, the local updates computed by the client are naturally interpreted as updates to the larger global model.

[0026] Another approach is to modify the message from client to server to achieve data transfer economy. For example, the client could select the k most informative elements from the message bound to the server and only convey these k most informative elements to the server. Alternatively, the client could quantize the message before conveying it to the server.

[0027] The embodiments described herein improve upon existing methods in several significant ways. First, unlike conventional joint dropout methods, the methods described herein enable each client to automatically determine the appropriate sub-model of the original model in a manner consistent with its local dataset and as efficient as possible. Second, instead of the server adhering to a specific global probability on the sub-models, the global model can be optimized using client-specific probabilities.

[0028] Joint average from the perspective of expectation maximization

[0029] As mentioned above, joint learning typically processes datasets with N data points. The problem involves learning a server model (e.g., a neural network) with parameters w (which can generally represent a vector, matrix, or tensor) on a dataset that is potentially distributed across S partitions in a non-independent and identically distributed (IID) manner. This eliminates the need for direct access to datasets that vary depending on the shards. Note that shards can typically be processing clients participating in joint learning with a central server, and can include remote computers, servers, mobile devices, smart devices, edge processing devices, etc. For simplicity but without loss of generality, it is assumed below that all shards S have the same number of data points; however, this framework can be extended to unevenly distributed data points by choosing appropriate weighting factors. This is achieved by defining a loss function on each shard. The total loss can be written as:

[0030]

[0031] Where N s It is the number of data points at slice (e.g., device) s, and This is the dataset located at the device in shard 's'. It's worth noting that this target corresponds to the joint dataset. Empirical Risk Minimization (ERM) is used, where each data point has a loss L(·).

[0032] The goal is to reduce the communication cost of joint learning. One approach to reduce communication during joint learning is to perform multiple gradient updates on w within the internal optimization objective of each partition s, thereby obtaining a result with parameter φ. s The "local" model. These multiple gradient updates are represented as "local epochs," that is, the number of times the entire local dataset is traversed, abbreviated as E. Each shard then transmits its local (or sub-model) φ to the server. s The server updates the global model in the t-th "round" by, for example, averaging the parameters of the local machine learning model according to the following formula:

[0033]

[0034] This method can be called joint averaging.

[0035] While simple to implement, joint averaging can yield suboptimal results on non-IID data, even if its convergence can be proven. In fact, if the shards S have a skewed distribution, the averaging of local machine learning model parameters may be a poor estimate for the global model. To address this, a shard-level optimized "proximal" term can be used, which encourages the local machine learning model φ to... s The model w that is "close" to the server within a certain distance. More formally, this can be defined as:

[0036]

[0037] in These are proximal terms. After each partition-specific optimization has been completed, the global model can be updated in a manner similar to joint averaging, i.e., by averaging the partition-specific parameters using Equation 2.

[0038] Connecting Joint Average and Expectation Maximization

[0039] It is worth noting that the overall joint averaging algorithm is compatible with optimization procedures based on a given objective function. For example, consider the following objective function:

[0040]

[0041] in Corresponding to having N s A dataset where each data point varies depending on the sharding. Corresponding to server parameter w The possibility, and ∑ s N s = N. Now consider the following decomposition of each possibility that varies due to the partition:

[0042]

[0043] This introduces an auxiliary latent variable φ. s The server parameter w serves as a prior hyperparameter p(φ) on the parameters that vary depending on the sharding. s |w). These latent variables are the parameters of the local machine learning model at slice s, and can be expressed in the following convenient prior form:

[0044]

[0045] Where λ acts as a stop φ s The regularization intensity is adjusted to prevent the movement from going too far away from w. Overall, this leads to the following objective function:

[0046]

[0047] In the presence of latent variable φ s One way to optimize this objective in this context is through expectation-maximization (EM). EM generally involves two steps, with the expectation step forming the posterior distribution over the latent variables:

[0048]

[0049] and the maximization step, wherein marginalization is achieved on the posterior. The probability is maximized relative to the model parameters w, such that:

[0050]

[0051] Accordingly, if a single gradient step is performed with respect to w in the maximization step, then this procedure corresponds to gradient descent on the original objective of Equation 7. To illustrate this, the gradient of Equation 7 can be taken with respect to w, where So that:

[0052]

[0053]

[0054]

[0055] In order to calculate equation 12, the local variable φ must first be obtained. s The posterior distribution of w is obtained, and the gradient of w is then estimated by marginalizing the posterior.

[0056] Hard EM is sometimes used when posterior inference is difficult to handle. In this case, the expected step can be approximated by using its most probable point. To the latent variable φ s Make a "hard" assignment, for example:

[0057]

[0058] This is generally easier to do using techniques such as stochastic gradient ascent. Given these hard assignments, the maximization step then corresponds to another simple maximization:

[0059]

[0060] As a result, the hard EM corresponds to a block coordinate ascending algorithm on the following objective function:

[0061]

[0062] Among them, φ is optimized while keeping w constant.1:s With maintaining φ 1:s Optimize w alternation under fixed conditions.

[0063] By setting λ→0 in Equation 6, it is clear that the hard assignment in the expected step simulates the process of optimizing the local machine learning model on each slice. In fact, even at a given learning rate, a specific prior can be assumed for the parameters when optimizing the model locally using stochastic gradient descent up to a fixed number of iterations. For linear regression, this prior is a Gaussian distribution centered at the initial values ​​of the parameters, while for nonlinear models, it can be shown by a proximal view of each gradient descent iteration:

[0064]

[0065] It applies a Gaussian-like prior centered on previous iterations, where the learning rate η acts as the variance of that prior. In obtaining... The maximization step then corresponds to:

[0066]

[0067] The closed-form solution to the objective can then be found by setting the derivative of the objective with respect to w to zero and solving for w according to the following formula:

[0068]

[0069] Where given The optimal solution for w is generated using joint averaging. The same average value.

[0070] The joint average will not affect the local parameter φ s Optimization is performed to converge in each round. However, the alternating EM process corresponds to block coordinate ascent on a single objective function, which is a variational lower bound of the marginal log-likelihood. More specifically, the EM iteratively performs block coordinate ascent to optimize the following objective:

[0071]

[0072] Where w s It is a posterior distribution The parameters of the variational approximation. To obtain the federated average procedure that achieves machine accuracy, φ can be used. s Deterministic distribution This leads to the following simplification of the objective:

[0073]

[0074] Where C is a fixed constant independent of the parameter to be optimized. It is worth noting that this objective is the same as the objective in Equation 15.

[0075] Encouraging sparsity in collaborative learning

[0076] The enhancement of joint averaging is achieved by encouraging sparsity through appropriate priors. Encouraging sparsity has two significant advantages: first, the model becomes smaller and thus easier to train on devices in terms of hardware; second, it reduces communication costs because there is no need to communicate pruned parameters.

[0077] The criterion for sparsity in Bayesian models is the spike and slab prior. It is a mixture of two components: an incremental spike δ(0) at zero and a continuous distribution along the solid line (i.e., a slab). More specifically, for a Gaussian slab, it can be defined as:

[0078]

[0079] Or it can be equivalently defined as a hierarchical model:

[0080] p(x)=∑ z p(z)p(x|z),p(z)=Bern(π), (22)

[0081]

[0082] Here, z acts as a "gating" variable, turning parameter w on or off. Now consider using this distribution (instead of a single Gaussian distribution) as a priori for the parameters in the joint setting. In this case, the hierarchical model would become:

[0083]

[0084] Where w is the model weight at the server and θ is the probability of the binary gate. Hard EM can be performed in a manner similar to joint averaging to optimize for a model with an approximate distribution q(φ). s |z s )q(z s The variational lower bound of the model can then be written as:

[0085]

[0086] Or equivalently written as:

[0087]

[0088] For weight distributions that vary depending on the partitioning, since they are continuous, they can be used... Where ∈≈0, this will be deterministic in terms of machine precision, while for gating variables, since they are binary, it is possible to use Where πsi Is it to activate the local gate z? si The probability of is given by , where Bern(·) indicates the Bernoulli distribution. To perform a hard EM for binary variables, we can remove from the above boundary... The entropy term, because this will encourage the approximate distribution to shift towards z. s The most likely value. Furthermore, to achieve a simple and intuitive goal at the piecewise level, the peak at zero can be broadened to a Gaussian distribution with precision λ², i.e., Taking all these factors into account and by inserting the appropriate expression into equation 26, it can be seen that the local and global targets will be respectively:

[0089]

[0090]

[0091] in Furthermore, C is a constant independent of the variable being optimized. It's worth noting that each shard locally optimizes its weights to approximate the server weights (this is determined by the prior accuracy λ and the probability π of keeping that weight locally). s Adjustment, while explaining as much as possible. Furthermore, the gate activation probability is optimized to approximate the server θ, with an additional term penalizing the sum of local activation probabilities. This is similar to the previously proposed L0 regularization objective.

[0092] It is now possible to consider optimizing φ through a certain procedure for local sharding. s and π s What happens next at the server? Since the server loss for w and θ is simply the sum of all local losses, the gradient for each parameter will be:

[0093]

[0094] Set these derivatives to zero, and fix the point at:

[0095]

[0096] That is, the weighted average of the local weights and the average of the local probabilities that preserve these weights. Therefore, due to π s The L0 penalty optimizes the architecture to be sparse, so the server probability θ will also become sparse for weights that are not used by any shard. Therefore, to obtain the final sparse architecture, weights can be pruned if the server inclusion probability θ is less than a threshold (such as 0.1, but other thresholds are also possible).

[0097] Local optimization

[0098] Although a gradient-based optimizer is used to optimize φ locallys It is direct, but π s It's not so direct, because the binary variable z in Equation 27 s The expectation is difficult to compute in closed-form, and Monte Carlo integration does not produce reparameterizable samples. To avoid these problems, the objective can be rewritten in an equivalent manner as:

[0099]

[0100] And then Bernoulli distribution Continuous relaxation (such as a hard-concrete distribution) can be used instead. Let continuous relaxation be... Where v s These are the parameters of the alternative distribution. In this case, the local objective will become:

[0101]

[0102] in It is continuous relaxation The cumulative distribution function (CDF) is obtained. Therefore, the alternative objective can now be directly optimized using gradient descent.

[0103] Reduce client-to-server communication costs

[0104] The model described above allows for the learning of sparse models for inference at the server. The same framework can be used to reduce communication costs during training time by employing two techniques, one to reduce the communication costs of client-to-server communication and the other to reduce the communication costs of server-to-client communication.

[0105] To reduce client-to-server costs, sparse samples can be communicated from the local distribution instead of the distribution itself. For example, instead of sending local weights φ to the server... s and local probability π s The client can replace it with π s To extract random binary samples z s ∈{0,1}, and then only communicate to the server the z-value. si =1 weight φ si Together with z s In this way, it's unnecessary to convey the parameter vector with zero values, resulting in meaningful savings while still keeping the server gradients unbiased. More specifically, the gradients and fixed points of the server weights can be expressed as follows:

[0106]

[0107]

[0108] The expression for the server probability is as follows:

[0109]

[0110]

[0111] As a result, the client can only access via To convey a subset of local weights In this way, the client conveys a subset of local weights along with z. s With these samples available, gradients or single-sample stochastic estimates of fixed points for w and θ can then be formed. Similar to the local implementation, the client uses hard-concrete relaxation. Operating on a smooth target, whenever the client communicates with the server, it can be done from zero temperature. Sampling to form Thus, the exact discrete sample z is obtained. s .

[0112] Note that this is a way to reduce communication without adding bias to the gradient of the original objective. Further techniques, such as quantization and selection of the first k gradients, can be used to further reduce communication in situations where introducing additional bias is acceptable.

[0113] Reduce server-to-client communication costs

[0114] The server needs to communicate the updated distribution to the client in each round. Unfortunately, for simple unstructured pruning, this doubles the communication cost because for each weight w... i There exists an associated θ that needs to be sent to the client. i To mitigate this effect, structured pruning can be employed, which introduces a single additional parameter indicating the probability of each group's weights, and is thus more efficient in terms of the number of trainable parameters compared to unstructured pruning. Even in the case of structured pruning, the normal weights and probabilities are sent to the server (except in the case of conveying sparse samples as described above, but the probability vector is significantly smaller in the case of structured pruning). Thus, for moderately sized groups (e.g., given the weight set of convolutional filters), the additional overhead is relatively small.

[0115] Communication costs can be further reduced by allowing some bias in the optimization procedure. For example, the global model can be pruned after each epoch during training, and thus only a subset of the model remaining is sent to each client. Note that this is efficient and requires no data at the server because it contains probabilities θ, and parameters with θ less than a threshold (e.g., 0.1) can be removed. This can lead to a significant reduction in communication costs, especially during the later stages of training when the model is sparse.

[0116] An additional way to reduce communication costs would be for the client to perform local pruning, and thus request only a subset of the original model parameters from the server that will remain locally.

[0117] Accordingly, when performing joint learning, joint averaging can be used to optimize sparse neural networks, which subsequently leads to significant communication savings while maintaining similar performance.

[0118] Example training process to encourage sparsity in federated learning

[0119] Figure 1 An example training process for encouraging sparsity in joint learning is described, as conceptually outlined above.

[0120] Initially, server 102 generates or maintains a global model 104 in a first state. In this example, each edge between nodes in the global model 104 is associated with parameters, including weights w and gate probabilities θ (e.g., parameter set 105). As mentioned above, the gate probability generally represents the likelihood that the associated weights will be included in the local (or sub) model used for joint training.

[0121] At 110, server 102 samples the global model weights w according to their associated gate probabilities θ to generate various subsets of weights and gate probabilities for each of the shards 106A-K, where each shard may represent a client device participating in joint learning with server 102.

[0122] Based on this information, each shard 106A-K (where K is the total number of shards participating in joint learning) generates a parameter φ based on the parameters received from server 102. s ,π s The local machine learning model 108A-K, where s is a specific shard in the shard set S. Figure 1 In the local machine learning model 108A-K, the dashed lines between nodes are turned off and are therefore not included in the weights during the training of the local machine learning model.

[0123] As depicted, the local machine learning model is generally different for each slice, based on different gate probabilities and random sampling performed by server 102. This helps to improve the comprehensiveness of joint learning.

[0124] In 112, each shard 106A-K trains its local machine learning model 108A-K and generates an updated local machine learning model 108A'-K'. Furthermore, each shard 106A-K generates weight gradients and gate gradients based on the training, for example, as described above with respect to Equations 31 and 32.

[0125] At 114, each shard 106A-K transmits model update data back to server 102. Server 102 then uses the model update data to generate an updated global model 104'. In the depicted embodiment, the model update data sent by each shard 106A-K includes the weight gradient and gate gradient of each element of the shard's local machine learning model (e.g., 108A'-K').

[0126] It is worth noting that, Figure 1 For simplicity, a single round of training is described, and the process can be repeated iteratively any number of times until, for example, the training objective is achieved (e.g., the number of iterations is completed, the weights converge, the accuracy threshold is reached, etc.).

[0127] After joint training ends (e.g., when the global model 104 converges), it is possible that one or more nodes (in the neural network model example) may be permanently and effectively turned off. Figure 1 (Not depicted in the text). More generally, the pruning rate of the global model 104 can be gradually increased during training so that the model becomes very sparse at the end of training (e.g., approximately 90% sparsity). For example, 90% sparsity of the trained global model 104' is... Figure 1 In this context, it would mean that 90% of the weights would be pruned during training based on a set threshold.

[0128] It is worth noting that in this example, sparsity is induced in the weights of the edges between nodes in the example model. However, in other examples, other aspects of the model can be associated with gate probabilities to induce alternative or additional sparsity. For example, individual nodes or layers in the model may be associated with gate probabilities and thus sampled and pruned during joint training. As another example, in the context of a convolutional neural network model, individual filter channels can be associated with gate probabilities and thus sampled and pruned to induce sparsity during training.

[0129] In addition to the sparsity induced during training based on gate probabilities, further strategies can be implemented to reduce communication costs. As mentioned above, to reduce the communication cost between shards (or clients) and the server (e.g., in step 114), during each training epoch, only the model aspects that are not turned off (e.g., those generated by...) are... Figure 1 The gradients of the weights (represented by solid lines between nodes) are propagated back to the server. Therefore, unlike conventional joint learning (where all weights are transmitted between the shard and the server in each training epoch), this approach saves communication time and cost by sending only a subset of the model data corresponding to the weights updated by each local machine training the model 108A-K during local training.

[0130] Furthermore, each slice (e.g., 106A-K) can be determined according to the gate probability π.s Elements of the local machine learning model (e.g., 108A-K) are sampled. Therefore, for example, instead of sending the entire set of weight gradients (for the parameters φ of the local machine learning model),... s ) and gate gradient (used for local gate probability π) s The fragment can send a weight update and z=1 or send nothing (corresponding to z=0), where z is a "gating" variable as described above. Thus, z is a value in {0,1}, and π is the probability of having z=1 while 1–π is the probability of having z=0.

[0131] This helps reduce the communication cost between each shard and server 102 in step 114. In this case, the server update rule can be modified from equation (30) to equations (34) and (36) to update the weight w and the probability of the binary gate, respectively.

[0132] Example methods for performing joint learning

[0133] Figure 2 An example method 200 for performing joint learning that induces sparsity is described, which can be, for example, provided by a joint learning server (such as...). Figure 1 Execute it using 102).

[0134] Method 200 begins at step 202, where a generation is performed for multiple clients (e.g., based on sampling the gate probability distribution of each model element in the set of model elements of the global machine learning model). Figure 1 A subset of model elements for each client in the 106A-K partitions.

[0135] In some embodiments of method 200, the subset of model elements includes a subset of weights associated with the edges connecting the nodes in the global machine learning model. In some embodiments of method 200, the subset of model elements includes a subset of nodes in the global machine learning model. In some embodiments of method 200, the subset of model elements includes a subset of channels in the convolutional filters of the global machine learning model.

[0136] Method 200 then proceeds to step 204, whereby to each of the plurality of clients: a subset of model elements; and a set of gate probabilities based on sampling, wherein each gate probability in the set of gate probabilities is associated with a model element in the subset of model elements (for example, such as regarding Figure 1 (as described in step 110).

[0137] Method 200 then proceeds to step 206, where a corresponding set of model updates is received from each of the plurality of clients (for example, such as regarding...). Figure 1 (as described in step 114).

[0138] Method 200 then proceeds to step 208, where the global machine learning model is updated based on the corresponding model update set from each of the plurality of clients.

[0139] In some embodiments of method 200, the corresponding model update set includes: a set of weight gradients associated with a local machine learning model trained by the corresponding client; and a set of gate probability gradients associated with a local machine learning model trained by the corresponding client.

[0140] In some embodiments of method 200, the corresponding model update set includes: a set of weight gradients associated with a local machine learning model trained by the corresponding client; and binary gate variable values ​​associated with each weight gradient in the set of weight gradients.

[0141] In some embodiments of method 200, updating the global machine learning model based on a corresponding set of model updates from each of the plurality of clients further includes: pruning the updated global machine learning model based on the updated gate probability and threshold gate probability values ​​for the global machine learning model.

[0142] It is worth noting that, Figure 2 This is merely one example of a model consistent with the disclosure herein, and further examples with additional, fewer, and / or additional steps are possible.

[0143] Figure 3 Another example method 300 for performing joint learning that induces sparsity is described, which can be, for example, by a joint learning client (such as...) Figure 1 It is executed using 106A-K).

[0144] Method 300 begins at step 302, wherein the following are received from a server that manages the joint learning of the global machine learning model: a subset of model elements from the set of model elements of the global machine learning model; and a set of gate probabilities, wherein each gate probability in the set of gate probabilities is associated with a model element in the subset of model elements.

[0145] In some embodiments of method 300, the subset of model elements includes a subset of weights associated with the edges connecting the nodes in the global machine learning model. In some embodiments of method 300, the subset of model elements includes a subset of nodes in the global machine learning model. In some embodiments of method 300, the subset of model elements includes a subset of channels in the convolutional filters of the global machine learning model.

[0146] Method 300 then proceeds to step 304, where a model update set is generated based on training a local machine learning model using the model element set and the gate probability set (for example, regarding...). Figure 1 (as described in step 112).

[0147] Method 300 then proceeds to step 306, where a set of model updates is sent to the server (for example, such as regarding...). Figure 1 (as described in step 114).

[0148] In some embodiments of method 300, the model update set includes: a set of weight gradients associated with the local machine learning model; and a set of gate probability gradients associated with the local machine learning model (e.g., Figure 1 (Local machine learning model 108A-K in the database).

[0149] In some embodiments of method 300, the model update set includes: a set of weight gradients associated with the local machine learning model; and binary gate variable values ​​associated with each weight gradient in the set of weight gradients.

[0150] In some embodiments, method 300 further includes: receiving a final set of model elements from a server, wherein the final set of model elements corresponds to a pruned global machine learning model.

[0151] It is worth noting that, Figure 3 This is merely one example of a model consistent with the disclosure herein, and further examples with additional, fewer, and / or additional steps are possible.

[0152] Example processing system

[0153] Figure 4 An example processing system 400 is depicted, which can be configured to perform the joint learning methods described herein (including, for example, Figure 2 and 3 Methods 200 and 300) in various aspects.

[0154] The processing system 400 includes a central processing unit (CPU) 402, which in some examples may be a multi-core CPU. Instructions executed at the CPU 402 may be loaded, for example, from program memory associated with the CPU 402, or from memory 424.

[0155] The processing system 400 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 404, a digital signal processor (DSP) 406, a neural processing unit (NPU) 408, a multimedia processing unit 410, and a wireless connectivity component 412.

[0156] NPUs (such as the 408) are generally configured to implement dedicated circuitry for the control and arithmetic logic used to execute machine learning algorithms, such as those for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. NPUs are sometimes alternatively referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligent processing units (IPUs), or vision processing units (VPUs).

[0157] NPUs (such as the 408) can be configured to accelerate the execution of common machine learning tasks, such as image classification, sound classification, and various other predictive models. In some examples, multiple NPUs can be instantiated on a single chip (such as a system-on-a-chip (SoC)), while in other examples, multiple NPUs can be part of a dedicated neural network accelerator.

[0158] An NPU can be optimized for training or inference, or in some cases configured to balance performance between the two. For an NPU capable of performing both training and inference, these two tasks can generally still be performed independently.

[0159] NPUs configured to accelerate training are typically used to speed up the optimization of new models. This involves taking an existing dataset (usually labeled or sublabeled), iterating over the dataset, and subsequently tuning model parameters (such as weights and biases) to improve model performance—a highly computationally intensive operation. Generally, optimization based on incorrect predictions involves backtracking through the layers of the model and determining gradients to reduce prediction errors.

[0160] NPUs designed to accelerate inference are typically configured to operate on the full model. Such NPUs can thus be configured to take new data segments as input and rapidly process them through an already trained model to generate model outputs (e.g., inference).

[0161] In one implementation, the NPU 408 is part of one or more of the CPU 402, GPU 404, and / or DSP 406.

[0162] In some examples, the wireless connectivity component 412 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G LTE), fifth-generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity processing component 412 is further connected to one or more antennas 414.

[0163] The processing system 400 may also include one or more sensor processing units 416 associated with any type of sensor, one or more image signal processors (ISPs) 418 associated with any type of image sensor, and / or a navigation processor 420, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0164] The processing system 400 may also include one or more input and / or output devices 422, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones, etc.

[0165] In some examples, one or more processors of the processing system 400 may be based on the ARM or RISC-V instruction set.

[0166] The processing system 400 also includes a memory 424, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 424 includes computer-executable components that can be executed by one or more of the aforementioned processors of the processing system 400.

[0167] In this example, memory 424 includes a transmission component 424A, a receiving component 424B, a training component 424C, an inference component 424D, a sampling component 424E, a pruning component 424F, model parameters 424G (e.g., weights and gate probabilities, as discussed above), and a model 424H. The depicted components, as well as other undepicted components, can be configured to perform various aspects of the methods described herein.

[0168] The processing system 400 is merely an example and generally performs the server and / or client / sharding operations described herein. However, in other embodiments, certain aspects may be omitted. For example, the server may omit certain features conventionally found in mobile devices, such as multimedia component 410, wireless connectivity component 412, antenna 414, sensor 416, ISP 418, and navigation component 420. The described examples are not intended to be limiting.

[0169] Example Terms

[0170] Examples of implementations are described in the following numbered clauses.

[0171] Clause 1: A method for performing joint learning of a machine learning model, comprising: for each of a plurality of clients and for each of a plurality of training epochs: generating a subset of model elements for that respective client based on sampling a gate probability distribution of each model element in a set of model elements of a global machine learning model; transmitting to the respective client: the subset of model elements; and a set of gate probabilities based on the sampled set, wherein each gate probability in the set of gate probabilities is associated with a model element in the subset of model elements; receiving a respective set of model updates from each of the plurality of clients; and updating the global machine learning model based on the respective set of model updates from each of the plurality of clients.

[0172] Clause 2: The method of Clause 1, wherein the subset of model elements includes a subset of weights associated with the edges connecting the nodes in the global machine learning model.

[0173] Clause 3: The method of Clause 2, wherein the corresponding model update set includes: a set of weight gradients associated with a local machine learning model trained by the corresponding client; and a set of gate probability gradients associated with the local machine learning model trained by the corresponding client.

[0174] Clause 4: The method of Clause 2, wherein the corresponding model update set comprises: a set of weight gradients associated with a local machine learning model trained by the corresponding client; and a binary gate variable value associated with each weight gradient in the set of weight gradients.

[0175] Clause 5: The method of any of Clauses 1-4, wherein the subset of model elements comprises a subset of nodes in the global machine learning model.

[0176] Clause 6: The method of any of Clauses 1-5, wherein the subset of model elements comprises a subset of channels in the convolutional filters of the global machine learning model.

[0177] Clause 7: The method of any of Clauses 1-6, wherein updating the global machine learning model based on a corresponding set of model updates from each of the plurality of clients further comprises: pruning the updated global machine learning model based on the updated gate probability and threshold gate probability values ​​for the global machine learning model.

[0178] Clause 8: A method for performing joint learning of a machine learning model, comprising: receiving from a server managing joint learning of a global machine learning model: a subset of model elements from a set of model elements of the global machine learning model; and a set of gate probabilities, wherein each gate probability in the set of gate probabilities is associated with a model element in the subset of model elements; generating a set of model updates based on training a local machine learning model according to the set of model elements and the set of gate probabilities; and transmitting the set of model updates to the server.

[0179] Clause 9: The method of Clause 8, wherein the subset of model elements includes a subset of weights associated with the edges connecting the nodes in the global machine learning model.

[0180] Item 10: The method of Item 9, wherein the model update set includes: a set of weight gradients associated with the local machine learning model; and a set of gate probability gradients associated with the local machine learning model.

[0181] Item 11: The method of Item 9, wherein the model update set comprises: a set of weight gradients associated with the local machine learning model; and binary gate variable values ​​associated with each weight gradient in the set of weight gradients.

[0182] Clause 12: The method of any of Clauses 8-11, wherein the subset of model elements comprises a subset of nodes in the global machine learning model.

[0183] Clause 13: The method of any of Clauses 8-11, wherein the subset of model elements comprises a subset of channels in the convolutional filters of the global machine learning model.

[0184] Clause 14: The method of any of Clauses 8-13 further includes: receiving a final set of model elements from the server, wherein the final set of model elements corresponds to a pruned global machine learning model.

[0185] Clause 15: A processing system comprising: a memory including computer-executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform a method pursuant to any of Clauses 1-14.

[0186] Clause 16: A processing system comprising means for performing a method according to any one of Clauses 1-14.

[0187] Clause 17: A non-transient computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method according to any one of Clauses 1-14.

[0188] Clause 18: A computer program product implemented on a computer-readable storage medium, comprising code for performing a method pursuant to any of Clauses 1-14.

[0189] Additional considerations

[0190] The foregoing description is provided to enable any person skilled in the art to practice the various embodiments described herein. The examples discussed herein are not intended to limit the scope, applicability, or embodiments set forth in the claims. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. For example, changes may be made to the function and arrangement of the elements discussed without departing from the scope of this disclosure. Various procedures or components may be appropriately omitted, substituted, or added to various examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Moreover, features described with reference to some examples may be combined in others. For example, any number of aspects set forth herein may be used to implement an apparatus or practice a method. Furthermore, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities that supplement or differ from the aspects of this disclosure set forth herein. It should be understood that any aspect of this disclosure disclosed herein may be implemented by one or more elements of the claims.

[0191] As used herein, the term “exemplary” means “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” should not be construed as superior to or better than the others.

[0192] As used herein, the phrase “at least one of” a list of items refers to any combination of those items, including a single member. As an example, “at least one of a, b, or c” is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0193] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, computation, processing, derivation, research, searching (e.g., looking in a table, database, or other data structure), ascertaining, and the like. Furthermore, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and the like. Moreover, "determine" can include parsing, selecting, choosing, building, and the like.

[0194] The methods disclosed herein include one or more steps or actions for implementing the method. These method steps and / or actions may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims. Furthermore, the various operations of the above methods can be performed by any suitable means capable of performing the corresponding functions. These means may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, where operations illustrated in the drawings are present, these operations may have corresponding paired means with similar numbers plus functional components.

[0195] The following claims are not intended to be limited to the embodiments shown herein, but should be granted the full scope consistent with the language of the claims. Within the claims, references to singular elements are not intended to mean “one and only one” (unless specifically stated so), but rather “one or more.” Unless specifically stated otherwise, the term “some / a” refers to one or more. No element of the claims should be interpreted in accordance with the provisions of 35 U.S.SC §112(f) unless the element is expressly stated using the phrase “means for…” or, in the case of a method claim, the element is stated using the phrase “steps for…”. Elements of all aspects described throughout this disclosure that are now or hereafter known to a person skilled in the art are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be donated to the public, whether or not such disclosure is expressly stated in the claims.

Claims

1. A method for performing joint learning of a machine learning model, comprising: Received at the device from the server that manages the joint learning of the global machine learning model: A subset of model elements from the set of model elements of the global machine learning model; and A set of gate probabilities, wherein each gate probability in the set of gate probabilities is associated with a model element in the subset of model elements, wherein the gate probability represents the probability that the associated model element is included in a local machine learning model for the corresponding device; The device generates a model update set by training the local machine learning model at the device based on the model element set and the gate probability set. The model update set includes updates to the model elements and updates to the gate probabilities. as well as The device transmits the model update set to the server to update the global machine learning model.

2. The method as described in claim 1, wherein, The subset of model elements includes a subset of weights associated with the edges connecting the nodes in the global machine learning model.

3. The method as described in claim 2, wherein, The model update set includes: The set of weight gradients associated with the local machine learning model; and The set of gate probability gradients associated with the local machine learning model.

4. The method of claim 2, wherein, The model update set includes: The set of weight gradients associated with the local machine learning model; and The binary gate variable value associated with each weight gradient in the set of weight gradients.

5. The method of claim 1, wherein, The subset of model elements includes a subset of nodes in the global machine learning model.

6. The method of claim 1, wherein, The subset of model elements includes a subset of channels in the convolutional filters of the global machine learning model.

7. The method of claim 1, further comprising: The device receives a final set of model elements from the server, wherein the final set of model elements corresponds to a pruned global machine learning model.

8. A processing system, comprising: Memory including computer-executable instructions; as well as One or more processors, the one or more processors being configured to execute the computer-executable instructions and enable the processing system to: Received from the server that manages the joint learning of the global machine learning model: A subset of model elements from the set of model elements of the global machine learning model; and A set of gate probabilities, wherein each gate probability in the set of gate probabilities is associated with a model element in the subset of model elements, wherein the gate probability represents the probability that the associated model element is included in a local machine learning model for the corresponding device; A model update set is generated by training the local machine learning model at the corresponding device based on the model element set and the gate probability set. The model update set includes updates to the model elements and updates to the gate probabilities. as well as The model update set is transmitted to the server to update the global machine learning model.

9. The processing system as described in claim 8, wherein, The subset of model elements includes a subset of weights associated with the edges connecting the nodes in the global machine learning model.

10. The processing system of claim 9, wherein, The model update set includes: The set of weight gradients associated with the local machine learning model; and The set of gate probability gradients associated with the local machine learning model.

11. The processing system of claim 9, wherein, The model update set includes: The set of weight gradients associated with the local machine learning model; and The binary gate variable value associated with each weight gradient in the set of weight gradients.

12. The processing system of claim 8, wherein, The subset of model elements includes a subset of nodes in the global machine learning model.

13. The processing system of claim 8, wherein, The subset of model elements includes a subset of channels in the convolutional filters of the global machine learning model.

14. The processing system of claim 8, wherein, The one or more processors are further configured to receive a final set of model elements from the server, wherein the final set of model elements corresponds to a pruned global machine learning model.

15. A method for performing joint learning of a machine learning model, comprising: For each corresponding client in a plurality of clients and for each training epoch in a plurality of training epochs: The server generates a subset of model elements for the corresponding client by sampling the gate probability distribution of each model element in the set of model elements of the global machine learning model, where the gate probability represents the probability that the associated model element is included in the local machine learning model for the corresponding client. Transmitted from the server to the corresponding client: The subset of model elements; and Based on the sampled set of gate probabilities, wherein each gate probability in the set of gate probabilities is associated with a model element in the subset of model elements; The server receives a corresponding model update set from each of the plurality of clients, the model update set including updates to model elements and updates to gate probabilities; as well as The server updates the global machine learning model based on the corresponding model update set from each of the plurality of clients.

16. The method of claim 15, wherein, The subset of model elements includes a subset of weights associated with the edges connecting the nodes in the global machine learning model.

17. The method of claim 16, wherein, The corresponding model update set includes: The set of weight gradients associated with the local machine learning model trained by the respective client; and The set of gate probability gradients associated with the local machine learning model trained by the corresponding client.

18. The method of claim 16, wherein, The corresponding model update set includes: The set of weight gradients associated with the local machine learning model trained by the respective client; and The binary gate variable value associated with each weight gradient in the set of weight gradients.

19. The method of claim 15, wherein, The subset of model elements includes a subset of nodes in the global machine learning model.

20. The method of claim 15, wherein, The subset of model elements includes a subset of channels in the convolutional filters of the global machine learning model.

21. The method of claim 15, wherein, The updating of the global machine learning model by the server based on the corresponding model update set from each of the plurality of clients further includes: pruning the updated global machine learning model based on the updated gate probability and threshold gate probability values ​​for the global machine learning model.

22. A processing system, comprising: Memory including computer-executable instructions; as well as One or more processors, the one or more processors being configured to execute the computer-executable instructions and enable the processing system to: For each corresponding client in a plurality of clients and for each training epoch in a plurality of training epochs: A subset of model elements for the corresponding client is generated by sampling the gate probability distribution of each model element in the set of model elements of the global machine learning model, where the gate probability represents the probability that the associated model element is included in the local machine learning model for the corresponding client. Send to the corresponding client: The subset of model elements; and Based on the sampled set of gate probabilities, wherein each gate probability in the set of gate probabilities is associated with a model element in the subset of model elements; Receive a corresponding model update set from each of the plurality of clients, the model update set including updates to model elements and updates to gate probabilities; as well as The global machine learning model is updated based on the corresponding model update set from each of the plurality of clients.

23. The processing system of claim 22, wherein, The subset of model elements includes a subset of weights associated with the edges connecting the nodes in the global machine learning model.

24. The processing system of claim 23, wherein, The corresponding model update set includes: The set of weight gradients associated with the local machine learning model trained by the respective client; and The set of gate probability gradients associated with the local machine learning model trained by the corresponding client.

25. The processing system of claim 23, wherein, The corresponding model update set includes: The set of weight gradients associated with the local machine learning model trained by the respective client; and The binary gate variable value associated with each weight gradient in the set of weight gradients.

26. The processing system of claim 22, wherein, The subset of model elements includes a subset of nodes in the global machine learning model.

27. The processing system of claim 22, wherein, The subset of model elements includes a subset of channels in the convolutional filters of the global machine learning model.

28. The processing system of claim 22, wherein, In order to update the global machine learning model based on the corresponding model update set from each of the plurality of clients, the one or more processors are further configured to prune the updated global machine learning model based on the updated gate probability and threshold gate probability values ​​for the global machine learning model.

Images