Apparatus, system and method for compressed communication of distributed machine learning

Abstract

The present invention relates to machine learning. To achieve communication-efficient distributed learning, a client is configured to obtain a prior probability distribution of parameters of a machine learning model in common with a server, and determine a codebook based on the common prior probability distribution. The client is further configured to obtain dataset and train the machine learning model using the dataset taking account of the prior probability distribution and the codebook. Through the training, an index of the codebook is obtained, which indicates a codeword in the codebook. The codeword is used as trained parameters of the machine learning model. The client sends the index to the server. In this way, communication overhead can be reduced, while precision of the learned machine learning model can be improved.

Description

[0001]APPARATUS, SYSTEM AND METHOD FOR COMPRESSED COMMUNICATION OF DISTRIBUTED MACHINE LEARNING TECHNICAL FIELDThe present disclosure relates to a field of computer technology, in particular artificial intelligence (AI). For instance, thisdisclosure provides apparatuses, methods, and a system for compressed communication applied to distributed machine learning. BACKGROUND With the grow of AI-based applications and related privacy concerns, the problem of finding alternative solutions to centralizedmachine learning gains more importance. In centralized learning, all available data is sent to a central node in order to train asuitable machine learning model (e.g., a neural network). This has several issues, including: i) it requires high bandwidth tosend all the dataset; ii) the dataset may contain private data that shall not be shared; iii) the central node performs the trainingon its own and thus cannot make use of available computational resources at remote nodes.Distributed machine learning emerges as a solution to these challenges. This approach, also referred to as remote learning, involves distributing the computational tasks of machine learning across multiple devices or nodes (also referred to as clients). By processing data locally on each node and then combining the learned results on a server, distributed machine learning can effectively handle large volumes of data while also addressing privacy concerns by reducing the need to transfer raw data. Moreover, distributed machine learning leverages the computational power of multiple clients, which can lead to faster processing times and more efficient use of resources. This is particularly relevant in scenarios where real-time data processing and decision-making are critical. SUMMARY While the advantages of the distributed machine learning are clear, there may be a major bottleneck. A machine learning modelusually comprises a large number of parameters (e.g., weights). Hence, sending learned parameters would require relativelyhigh bandwidth. This is not desirable in telecommunication, in particular due to limited bandwidth of wireless channels (e.g.,air interface).In view of the above-mentioned problems and disadvantages, this disclosure aims to improve distributed machine learning. Anobjective is to provide a transmission communication scheme that is bandwidth-efficient to communicate learned parameters obtained through distributed machine learning.The objective is achieved by the embodiments of the invention as described in the enclosed independent claims. Advantageousimplementations of the embodiments of the invention are further defined in the dependent claims.A first aspect of the present disclosure provides a machine learning method. The method is applied to a client device andcomprises the following steps:- obtaining a prior probability distribution of parameters of a machine learning model, in which the prior probabilitydistribution is in common with a server device;- determining a codebook based on the common prior probability distribution;- obtaining a dataset;- training the machine learning model using the dataset taking account of the prior probability distribution and thecodebook, to obtain an index of the codebook (also referred to as a codeword index), wherein the index of the codebook(the codeword index) indicates a codeword in the codebook as trained parameters of the machine learning model; and- sending the index to the server device.Optionally, the parameters of a machine learning model may comprise weights. In this disclosure, the parameters and theweights may be used interchangeably. Optionally, the prior probability distribution may be obtained in various possible ways, as long as the client device and the server device use the same prior probability distribution. For instance, the method may comprises receiving, by the client device, the prior probability distribution (or an indication thereof) from the server device. In this case, the prior probability distribution may be determined by the server device and is shared with the client device. Alternatively, the method may comprises determining, by the client device, the prior probability distribution; and indicating, by the client device, the determined prior probability distribution to the server device. In this case, the prior probability distribution may be determined by the client device and is shared with the server device. Alternatively, the prior probability distribution may be pre-set, such that the client device and the server device share the same prior probability distribution. In general, how to obtain the priorprobability distribution is not limited in this disclosure. Any technique known in the field that can ensure the client device andserver client having the same prior probability distribution may be used. By using the same prior probability distribution, it can be ensured that the client device and the server device share the same the codebook that is determined based on the same prior probability distribution. By training the machine learning model using the dataset taking account of the prior probability distribution and the codebook,an improved tradeoff between bandwidth usage and model accuracy can be achieved.In an implementation form of the first aspect, the prior probability distribution of parameters of a machine learning model maybe based on previously trained parameters.Optionally, the prior probability distribution may be based on previously trained parameters of a plurality of client deviceincluding said client device of the aspect. For instance, in a distributed learning scenario where the plurality of client devicesis configured to train a common machine learning model comprising a plurality of parameters using local datasets of each own:for each parameter of the common machine learning model, an average value of the learned parameter among the plurality of client devices may be determined. Optionally, the learned parameters of one or more training phases may be averaged.Accordingly, the prior probability distribution of the parameters of a may be determined.In a further implementation form of the first aspect, the prior probability distribution of the parameters of the machine learningmodel may be determined based on one or more initial statistic parameters. The method may further comprise:- sending, by the client device, the one or more initial statistic parameters to the server device; or- receiving, by the client device, the one or more initial statistic parameters from the server device.In a further implementation form of the first aspect, for determining the codebook, the method may comprise:- obtaining, by the client device, a random value that is in common with the server device; and- determining, by the client device, a sequence based on the prior probability distribution using the random value as thecodebook. Optionally, similar to the prior probability distribution mentioned above, how to obtain the random value is not limited in thisdisclosure. Any technique known in the field that can ensure the client device and server client having the same random valuemay be used in order to obtain the random value.In a further implementation form of the first aspect, for training the machine learning model, the method may comprise:determining, by the client device, a first objective function, wherein the first objective function comprises a first regularizer adapted to enforce a distribution of unencoded learned parameters close to the prior probability distribution; and training, by the client device, the machine learning model using the dataset and a first optimization algorithm in order to minimize the first objective function, to obtain the unencoded learned parameters; and determining, by the client device, the codeword from the codebook, wherein the codeword is close to the unencoded learned parametersIn this way, the distribution of the unencoded learned parameters can be as close as possible to the prior probability distribution.Since the prior probability distribution is also used to determine the codebook, the unencoded learned weights can also be closeto the distribution of codewords comprised in the codebook. Hence, the precision for communicating the learned parameterscan be improved. Optionally, the codeword may be determined such that the index of the codeword is as small as possible. In this way, the determined codeword is close enough to the unencoded learned parameters while its index is as small as possible. Hence, a tradeoff between precision and communication efficiency can be balanced.Optionally, for determining the codeword that is close to the unencoded learned parameters, various techniques may be usedfor determining the codeword, such as but not limited to importance sampling, minimal random coding (MRC), ordered randomcoding (ORC), and the like. In this way, the in-expectation closeness can be achieved. This may be useful when interactivelearning algorithm are considered, and average parameters learned over time is important. For another instance, greedysampling (e.g., a vector quantizer encoder) may be used for determining the codeword. This may be useful when instantaneous performance after one training round is of importance. In a further implementation form of the first aspect, the step of training of the machine learning model may further obtain precision information of the trained parameters. The method may further comprise sending, by the client device, the precision information to the server device. The precision information of the trained parameters may be obtained through fine-tuning, such that the fined-tuned parameters can be as close as possible to a distribution of the codeword indicated by the index. In this way, the model accuracy can be further improved.In a further implementation form of the first aspect, for obtaining the precision information, the method may comprise:determining, by the client device, a second objective function, wherein the second objective function comprises a second regularizer adapted to enforce the unencoded learned parameters close to the codeword; fine-tuning, by the client device, the unencoded learned parameters using the dataset and a second optimization algorithm in order to minimize the second objective function, to obtain fine-tuned parameters; determining, by the client device, differences between the fine-tuned parameters and the codeword; and quantizing, by the client device, the differences as the precision information. In this way, the differences between the trained parameters and the fine-tuned parameters may be compensated based on the precision information. A second aspect of this disclosure provides a client device for machine learning. The client device is configured to:- determine a prior probability distribution of parameters of a machine learning model that is shared with a server device;- determine a codebook based on the prior probability distribution;- obtain a dataset;- train the machine learning model using the dataset taking account of prior probability distribution and the codebook,to obtain an index of the codebook, wherein the index of the codebook is associated with a set of parameters comprised in the codebook; and- send the index to the server device.In an implementation form of the second aspect, the prior probability distribution of parameters of a machine learning modelmay be based on previously trained parameters.In a further implementation form of the second aspect, the prior probability distribution of parameters of a machine learningmodel may be determined based on one or more initial statistic parameters. The client device may be further configured to:- send the one or more initial statistic parameters to the server device; or- receive the one or more initial statistic parameters from the server device.In a further implementation form of the second aspect, for training the machine learning model, the client device may be configured to:- determine a first objective function, wherein the first objective function comprises a first regularizer adapted to enforcea distribution of unencoded learned parameters close to the prior probability distribution; and- train the machine learning model using the dataset and a first optimization algorithm in order to minimize the firstobjective function, to obtain the unencoded learned parameters; and- determine the codeword from the codebook, wherein the codeword is close to the unencoded learned parameters.In a further implementation form of the second aspect, for training the machine learning model, the client device may be furtherconfigured to:- obtain precision information of the trained parameters, and- send the precision information to the server device.In a further implementation form of the second aspect, for obtaining the precision information, the client device may beconfigured to:- determine a second objective function, wherein the second objective function comprises a second regularizer adaptedto enforce the unencoded learned parameters close to the codeword;- fine-tune the unencoded learned parameters using the dataset and a second optimization algorithm in order to minimizethe second objective function, to obtain fine-tuned parameters;- determine differences between the fine-tuned parameters and the codeword; and- quantize the differences as the precision information.The client device of the second aspect or any implementation form thereof may share the same optional features and advantages as the method of the first aspect or any implementation form thereof. A third aspect of this disclosure provides a system comprising one or more client devices according to the second aspect or anyimplementation form thereof and at least one server device. The server device is configured to:- determine a prior probability distribution of parameters of a machine learning model that is in common with each clientdevice;- determine, based on the prior probability distribution, a codebook that is in common with each client device;- receive an index from each client device; and- determine a codeword from the codebook in accordance with the index as trained parameters of a machine learningmodel for each client device.A fourth aspect of the present disclosure provides a computer program comprising a program code for performing the methodaccording to the first aspect or any of its implementation forms.A fifth aspect of the present disclosure provides a non-transitory storage medium storing executable program code which, whenexecuted by a processor, causes the method according to the first aspect or any of its implementation forms to be performed.A sixth aspect of the present disclosure provides a chipset comprising a memory and a processor, which are configured to storeand execute program code to perform the method according to the first aspect or any of its implementation forms.It is noted that all devices, elements, units, and means described in the present application could be implemented in the softwareor hardware elements or any kind of combination thereof. All steps which are performed by the various entities described in the present application as well as the functionalities described to be performed by the various entities are intended to mean that the respective entity is adapted to or configured to perform the respective steps and functionalities. Even if, in the following description of specific embodiments, a specific functionality or step to be performed by external entities is not reflected in thedescription of a specific detailed element of that entity, which performs that specific step or functionality, it should be clear fora skilled person that these methods and functionalities can be implemented in respective software or hardware elements, or any kind of combination thereof. BRIEF DESCRIPTION OF DRAWINGSThe above-described aspects and implementation forms will be explained in the following description in relation to the encloseddrawings, in whichFIG. 1 shows an example of a client device and a server device;FIG. 2 shows an example of a training phase;FIG. 3 shows an example of a system;FIG. 4 shows a diagram of a method applied to a client device; andFIG. 5 shows a diagram of a method applied to a server device. DETAILED DESCRIPTION OF EMBODIMENTS In FIGs.1-5, corresponding elements may share the same features and function likewise.FIG. 1 shows an example of a client device 110 and a server device 120 of the present disclosure.The client device 110 is one of a plurality of client devices that are adapted to perform distributed learning in cooperation with the server device 120. This disclosure illustrates interactions between one client device and one server device, which can be similarly applied to any other client devices.For distributed learning, the client device 110 and the server device 120 are configured to cooperate with each other in orderto achieve a trained machine learning model (e.g., a neural network). A particular architecture of the machine learning modelmay be shared or pre-set or communicated between the client device 110 and the server device 120. For instance, the to-be-trained machine learning model may be a LeNet-5 model, or Residual Network-56 (ResNet-56) model. In general, thisdisclosure may be applied to any kind of machine learning model. The client device 110 (simply referred to as the client) and the server device 120 (simply referred to as the server) are configuredto share a common prior probability distribution 101 of parameters of the machine learning model. In this disclosure, thecommon prior probability distribution 101 may be simply referred to as a “prior 101”. The prior 101 is used to determine a codebook on the client device side 110 and on the server device 120. By sharing the same prior 101, it can be ensured that the codebook 111 of the client device 110 and the codebook 121 of the server device are thesame. It is noted that since the codebook 111 of the client device 110 and the codebook 121 of the server device 120 are thesame, they are not distinguished further in this disclosure, and either one can be simply referred to as the “codebook”. Theoryof the codebook-based compression can be found in M. Havasi, 2018, “Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters”, which is not detailed herein. For training the machine learning model, the client device 111 is configured to obtain a dataset 112 and train the machinelearning model using the dataset 112. In addition, the training of the machine learning model also takes account of the prior101 and the codebook 111. That is, the prior 101 and the codebook 111 also affects the training of the machine learning model. For instance, suitable regularizers may be imposed based on the prior 101 and the codebook 111 to the optimization problem associated with the training of the machine learning model. The suitable regularizers may be adapted to: i) encourage trained parameters to be close to elements of the codebook 111; and ii) favor the elements of the codebook 111 with smaller indices. In general, a regularizer may be added to a loss function that needs to be minimized during the training phase to guarantee agood performance, in particular to avoid overfitting, and to reduce communication overhead.After the training, an index 114 of the codebook (also referred to as a codeword index) may be obtained. The index 114 indicates a codeword comprised in the codebook. The indicated codeword is used as trained parameters of the machine learning model. The client device 110 is configured to send the index 114 to the server device 120. Upon receiving the index 114, the server device may determine a codeword from the codebook 121 indicated by the index 114.The codeword is used as potential parameters of the machine learning model that are to be aggregated on the server device 120.In distributed learning, the server device 120 is configured to receive a plurality of indices from a plurality of client devices 110. Corresponding parameters learned by the plurality of client devices may be aggregated (e.g., through averaging, MAX operation, etc.) by the server device 120 as final parameters. In this way, a better tradeoff between bandwidth usage and inference accuracy can be achieved. Optionally, through training the machine learning model, the client device 110 may be further configured to obtain precision information of the trained parameters and send the precision information to the server device. In this disclosure, a common prior 101 is chosen and shared between the client device 110 and the server device 120. This prior 101 may be pre-set information, or may be transmitted from the server device 120 to the client device 110, or may be transmitted from the client device 110 to the server device 120.A proper choice of the prior 101 may reduce the communication cost (required bandwidth) and improves the learningperformance. Because the prior 101 may be seen as estimated distribution of parameters of the machine learning model. For this purpose, existing information about the distribution of the parameters of the machine learning model may be used. This includes but not limited to: information obtained in one or more previous training rounds, and / or shared by other clientdevices, and / or shared statistics of the data. For instance, the prior 101 may be determined in one or more of the followingways: i) in multi-round scenarios, by considering the trained models in previous rounds; ii) in multi-client scenarios, by considering trained parameters from other clients, and possibly their models duringprevious rounds; iii) based on an initial shared statistic from / determined by the client / server.Example 1: For a setup where a pretrained neural network ResNet-56 is used. One choice may be to choose the prior as adiagonal multi-variate Gaussian distribution for each convolutional layer of the ResNet-56. The mean of the distribution may be agreed to be the values of the pretrained model, and does not need to be exchanged. Only the variances are needed to besent as the initial shared statistic.Example 2: For the multi-round and multi-client scenario, the prior may be chosen as either (i) an empirical distribution of theparameters (models) sent by multiple clients during previous rounds, or (ii) to first fix a parametric class of distributions, andthen to estimate the parameters of such distributions using the models sent by multiple clients during previous rounds. Forexample, it is possible to choose the prior as a multivariate Gaussian distribution with diagonal covariance matrix and toestimate the mean and diagonal elements of the covariance matrix using trained parameters shared by multiple clients duringprevious rounds.Based on the prior 101, a codebook is generated at both the client 110 and the server 120. A choice is to take a sample fromthe prior. Taking the sample from the prior may be based on a pre-determined sequence that is shared between the client 110and the server 120. Alternatively, a random value may be determined and / or shared between the client 110 and the server 120.Then, taking the sample from the prior may be based on the common random value. The shared prior 101 and the optionalshared random value make it possible that both the client 110 and the server 120 draw the same sequence of values from the prior 101, and hence to agree on a common codebook.FIG. 2 shows an example of a training phase of this disclosure. The training phase of this disclosure is performed by takingaccount of both the codebook and the prior, which are shared between the client 110 and the server 120. One or moreregularizers (also referred to as regularization term) may be used in solving one or more optimization problem during thetraining phase. For instance, as shown in FIG. 2, the client 110 may be configured to determine a first optimization function F1. The first optimization function F1 may comprise a first regularizer adapted to enforce a distribution of unencoded learned parameters close to the prior probability distribution 101. An example of the first optimization function F1 may be as follows: The first optimization function F1 in this example comprises two major parts. The first part is the average of the given lossfunction over the given dataset. The loss function may determine the performance of the training algorithm. For instance, in aclassification task, this loss should be increased when the learned model cannot not correctly predict the class. The second partis a regularizer that enforces the distribution of the learned parameters to be close to the chosen prior Q. Since the codebook is generated using the prior Q, this regularizer can force the learned parameters of the model to be close to the distribution of codewords in the codebook.By training the machine learning model 113 using the dataset S and a first optimization algorithm in order to minimize the firstoptimization function F1, the distribution of the obtained unencoded learned parameters W can be close to the prior probabilitydistribution Q. For instance, the first optimization algorithm may be a Stochastic Gradient Descent (SGD) algorithm, or anAdam algorithm, or the like. In generally, any optimization algorithm that can be used to train a machine learning model may be used as the first optimization algorithm. Optionally, the first optimization algorithm may be combined with Gibbs sampling. The client 110 is configured to send the index i* to the server 120.Optionally, the client 110 may be further configured to perform fine-tuning. That is, the training phase may further comprisefine-tuning. The client 110 is configured to further obtain precision information of the trained parameters and send the precisioninformation to the server device. For obtaining the precision information, the client 110 is configured to determine a secondobjective function used for fine-tuning. The second objective function comprises a second regularizer adapted to enforce theunencoded learned parameters close to the codeword. The client 110 is configured to fine-tune the learned parameters using the dataset and a second optimization algorithm in order to minimize the second objective function, to obtain fine-tuned parameters. Then, differences between the trained parameters and the fine-tuned parameters can be determined. The client 110 may be configured to quantize the differences as the precision information. An example of the second optimization function F2 may be as follows: The second optimization function F2 in this example comprises two major parts. The first part is the same as the first part ofthe first optimization function F1. The second part is proportional to ‖^ − ^^∗‖. By obtaining the fine-tuned parameters W*parameters that can minimize the second optimization function F2, the fine-tuned parameters W* are also close the determinedcodeword ^^∗.The client 110 may be further configured to determine the differences e between the fine-tuned parameters W* and thedetermined codeword ^^∗as: (3)The client 110 may be further configured to quantize the difference e to obtain the quantized difference ê as the precisioninformation. In the following, two examples of algorithms that may be applied to the present disclosure is shown. Algorithm Example 1: Interactive learning: In this Algorithm Example 1, the first local training algorithm that depends explicitly on both prior and dataset. Step 4 corresponds to learning based on the first optimization function F1 as mentioned above in FIG.2. Step 7 corresponds to learning (fine-tuning / optimization) based on the second optimization function F2 as mentioned above in FIG.2. This algorithm induces in-expectation guarantees on the communication overhead and learning performance. Hence, this is particularly useful forinteractive communications (e.g., distributed learning). Algorithm Example 2: One-shot learning In this Algorithm Example 2, step 4 corresponds to learning based on the first optimization function F1 as mentioned above in FIG. 2, and step 6 corresponds to learning (fine-tuning / optimization) based on the second optimization function F2 as mentioned above in FIG. 2. This Algorithm Example 2 induces one-shot guarantees on the communication overhead and learning performance. Hence, this is particularly useful for one-shot learning scenarios.FIG. 3 shows an example of a system 300 of this disclosure. The system 300 comprises a plurality of client devices 310, 330and at least one server device 320. Each client device 310, 330 is built based on the client device 110 shown in FIG. 1-2. Theat least one server 320 is built based on the server device shown in FIG. 1.One aspect of this disclosure also provides a server device like the server device 320 shown in FIG. 3. The server device 320is communicable with a plurality of client devices 310, 320. The server device 320 is configured to:- determine a prior probability distribution 101 of parameters of a machine learning model that is in common with eachclient device 310, 330;- determine, based on the prior probability distribution 101, a codebook 121 that is in common with each client device310, 330;- receive an index 314, 334 from each client device 310, 330 ; and- determine a codeword from the codebook in accordance with the index as trained parameters of the machine learningmodel for each client device. Optionally, the server device 320 may be further configured to:- aggregate the codewords indicated by the plurality of client devices 310, 330, to obtain an aggregated codeword; and- determine a server index 324 indicating a final codeword that is close to the aggregated codeword; and- send the server index 324 to each client device 310, 330.Optionally, the notion of “closeness” may be similar to that of the client device 110 mentioned above in FIG.1-2. Optionally, if the server device 320 receive precision information from each client, before aggregating the codewords, the server device 320 may be further configured to adjust each codeword based on the corresponding precision information. Then, the codeword aggregation is performed based on the adjusted codewords.For aggregating the codewords or adjusted codewords, various algorithms may be used. For instance, the server device may beconfigured to use any one of the following as the aggregated codeword:- a codeword with the smallest index, which can achieve the smallest communication overhead); or- an average value the codewords, which can achieve a balanced model quality; or- a weighted average value of the codewords (e.g., where each weight can be set according to the importance of eachclient), which can also achieve a balanced model quality; or- the firstly received codeword, which can achieve a fast inferencing speed.The above are merely shown as examples. Any other method known in the field may be used to aggregate the codewords. This is not limited in this disclosure.FIG. 4 shows a diagram of a method 400 of this disclosure. The method 400 is performed by a client device for machinelearning and comprises the following steps:- step 401: obtaining a prior probability distribution of parameters of a machine learning model that is in common witha server device;- step 402: determining a codebook based on the prior probability distribution;- step 403: obtaining a dataset;- step 404: training the machine learning model using the dataset taking account of the prior probability distributionand the codebook, to obtain an index of the codebook, in which the index of the codebook indicates a codeword in the codebook as trained parameters of the machine learning model; and- step 405: sending the index to the server device.It is noted that the steps of the method 400 may share the same functions and details from the perspective of FIG. 1-3 describedabove. Therefore, the corresponding method implementations are not described again at this point.FIG. 5 shows a diagram of a method 500 of this disclosure. The method 500 is performed by a server device for machinelearning and comprises the following steps:- step 501: determining a prior probability distribution of parameters of a machine learning model that is in commonwith each of a plurality of client devices;- step 502: determining, based on the prior probability distribution, a codebook that is in common with each client device;- step 503: receiving an index from each client device; and- step 504: determining a codeword from the codebook in accordance with the index as trained parameters of the machinelearning model for each client device.It is noted that the steps of the method 500 may share the same functions and details from the perspective of FIG. 1-3 describedabove. Therefore, the corresponding method implementations are not described again at this point.In general, this disclosure provides: i) a local training algorithm at the client side, ii) a transmission scheme between the clientand the server, and iii) a decoding scheme at the server side. According to this disclosure, with a few numbers of sent bits (theindex and the optionally precision information), the server may determine a robust model (to change of data distribution) with good inference performance. Main aspect of this disclosure comprises:- sharing a common prior 101 and construct a common codebook 111, 121 between the client 110 and the server 120;- a joint local training and coding scheme that explicitly depend on both the common prior 101 and the common codebook111, 121, with one or more mechanisms below:- forcing the distribution of the learned parameters to be close to the chosen prior 101,- favouring a codeword with a smaller index in the codebook 111; and- indicating the trained machine learning model by sending the corresponding index and possibly a quantized version ofthe error (as the precision information) for the purpose of refinement.The key property of joint local training and source coding scheme is that the training and the coding explicitly depends on thecommon prior, the common codebook and the dataset. Hence, both the inference accuracy and bandwidth usage can beimproved.The client device and the server device in the present disclosure may comprise processing circuitry (not shown) configured toperform, conduct, or initiate the various operations of the client device and the server device described herein, respectively.The processing circuitry may comprise hardware and software. The hardware may comprise analog circuitry or digital circuitry,or both analog and digital circuitry. The digital circuitry may comprise components such as application-specific integratedcircuits (ASICs), field-programmable arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors. The processing circuitry comprises one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory may carry executable program code which, when executed by the one or more processors, causes the client device and the server device to perform, conduct or initiate the operations or methods described herein, respectively.The client device may be, for example but not limited to, an electronic device capable of computing, such as but not limited to:a computer, a server blade, a computer cluster, a virtual machine based on computer architectures, a tablet, a mobile terminal,a graphics processing unit, a video game console, a vehicle, a drone, a robot, an extended-reality (XR) device, an augmentedreality (AR) device, a virtual reality (VR) device, an wearable device, or the like.The server device may be, for example but not limited to, an electronic device capable of computing, such as but not limited toa computer, a server blade, a computer cluster, a virtual machine based on computer architectures, a base station, a communication center, an application server, or the like. The present invention has been described in conjunction with various embodiments as examples as well as implementations. However, other variations can be understood and effected by those persons skilled in the art and practicing the claimedinvention, from the studies of the drawings, this disclosure, and the claims. In the claims as well as in the description the word“comprising” does not exclude other elements or steps and the indefinite article “a” or “an” does not exclude a plurality. A single element or other unit may fulfill the functions of several entities or items recited in the claims. The mere fact that certain measures are recited in the mutual different dependent claims does not indicate that a combination of these measures cannot beused in an advantageous implementation.

Claims

CLAIMS1. A machine learning method (400) comprising:obtaining (401), by a client device (110), a prior probability distribution (101) of parameters of a machine learningmodel (113) that is in common with a server device;determining (402), by the client device (110), a codebook (111) based on the prior probability distribution (101);obtaining (403), by the client device (110), a dataset (112);training (404), by the client device (110), the machine learning model (113) using the dataset (112) taking account ofthe prior probability distribution (101) and the codebook (111), to obtain an index (114) of the codebook (111), whereinthe index (114) of the codebook (111) indicates a codeword in the codebook (111) as trained parameters of the machinelearning model (113); andsending (405), by the client device (110), the index (114) to the server device.

2. The method according to claim 1, wherein the prior probability distribution (101) of parameters of a machine learningmodel (113) is based on previously trained parameters.

3. The method according to claim 1, wherein the prior probability distribution (101) of parameters of a machine learningmodel (113) is determined based on one or more initial statistic parameters, and the method further comprises:sending, by the client device (110), the one or more initial statistic parameters to the server device; or receiving, by the client device (110), the one or more initial statistic parameters from the server device.

4. The method according to any one of claims 1 to 3, wherein for determining the codebook (111), the method comprises:obtaining, by the client device (110), a random value that is in common with the server device;determining, by the client device (110), a sequence based on the prior probability distribution (101) using the randomvalue as the codebook (111).

5. The method according to any one of claims 1 to 4, wherein for training the machine learning model (113), the methodcomprises: determining, by the client device (110), a first objective function, wherein the first objective function comprises a firstregularizer adapted to enforce a distribution of unencoded learned parameters close to the prior probability distribution(101); andtraining, by the client device (110), the machine learning model (113) using the dataset (112) and a first optimizationalgorithm in order to minimize the first objective function, to obtain the unencoded learned parameters; anddetermining, by the client device (110), the codeword from the codebook (111), wherein the codeword is close to theunencoded learned parameters.

6. The method according to any one of claims 1 to 5, wherein the training of the machine learning model (113) furtherobtains precision information of the trained parameters, and the method further comprises sending, by the client device (110),the precision information to the server device.

7. The method according to claim 6, wherein for obtaining the precision information, the method comprises:determining, by the client device (110), a second objective function, wherein the second objective function comprisesa second regularizer adapted to enforce the unencoded learned parameters close to the codeword;fine-tuning, by the client device (110), the unencoded learned parameters using the dataset (112) and a secondoptimization algorithm in order to minimize the second objective function, to obtain fine-tuned parameters;determining, by the client device (110), differences between the fine-tuned parameters and the codeword; andquantizing, by the client device (110), the differences as the precision information.

8. A client device (110) for machine learning, wherein the client device (110) is configured to:determine a prior probability distribution (101) of parameters of a machine learning model (113) that is shared with aserver device; determine a codebook (111) based on the prior probability distribution (101);obtain a dataset (112); train the machine learning model (113) using the dataset (112) taking account of prior probability distribution (101)and the codebook (111), to obtain an index (114) of the codebook (111), wherein the index (114) of the codebook (111)is associated with a set of parameters comprised in the codebook (111); andsend the index (114) to the server device.

9. The client device (110) according to claim 8, wherein the prior probability distribution (101) of parameters of a machinelearning model (113) is based on previously trained parameters.

10. The client device (110) according to claim 8, wherein the prior probability distribution (101) of parameters of a machinelearning model (113) is determined based on one or more initial statistic parameters, and the client device (110) is furtherconfigured to: send the one or more initial statistic parameters to the server device; orreceive the one or more initial statistic parameters from the server device.

11. The client device (110) according to any one of claims 8 to 10, wherein for training the machine learning model (113),the client device (110) is configured to:determine a first objective function, wherein the first objective function comprises a first regularizer adapted to enforce adistribution of unencoded learned parameters close to the prior probability distribution (101); andtrain the machine learning model (113) using the dataset (112) and a first optimization algorithm in order to minimizethe first objective function, to obtain the unencoded learned parameters; and determine the codeword from the codebook (111), wherein the codeword is close to the unencoded learned parameters.

12. The client device (110) according to any one of claims 8 to 11, wherein for training the machine learning model (113),the client device (110) is further configured to:obtain precision information of the trained parameters, andsend the precision information to the server device.

13. The client device (110) according to claim 12, wherein for obtaining the precision information, the client device (110)is configured to: determine a second objective function, wherein the second objective function comprises a second regularizer adaptedto enforce the unencoded learned parameters close to the codeword;fine-tune the unencoded learned parameters using the dataset (112) and a second optimization algorithm in order tominimize the second objective function, to obtain fine-tuned parameters;determine differences between the fine-tuned parameters and the codeword; andquantize the differences as the precision information.

14. A system (300) comprising one or more client devices (310, 330) according to any one of claims 8 to 13, and at leastone server device (320),wherein the server device (320) is configured to: determine a prior probability distribution (101) of parameters of a machine learning model (113) that is incommon with each client device (310, 330);determine, based on the prior probability distribution (101), a codebook (121) that is in common with each client device (310, 330);receive an index (314, 334) from each device (310, 330); anddetermine a codeword from the codebook in accordance with the index (314, 334) as trained parameters of themachine learning model (113) for each client device (310, 330).

15. A computer program comprising instructions which, when the program is executed by a computer, cause the computerto carry out the method according to any one of claims 1 to 7.

Images