System and method of training a learning model in a distributed client / server data processing system

Abstract

A method for training a learning model in a distributed client / server data processing system (202) involving a client (204), a server (206), and a model warehouse (208) over a network is provided. The model warehouse accesses multiple learning models with respective pre-trained encoders and decoders. The client (204) sends an assistance request to the server (206) including client identifiers, task identifiers, dataset size, and computational resource identifiers. The server (206) responds with a decoder (304) of a selected learning model based on the task and computational capabilities. It also provides an estimate of the encoder's output structure, based on the client's dataset size and processing capabilities. The client (204) selects a local model matching the received decoder's architecture and having an encoder which is based on the processing capability of the client, and trains the encoder using the provided structure estimate. Further training of the encoder and decoder for the local model is performed using the client's local dataset.

Description

[0001] SYSTEM AND METHOD OF TRAINING A LEARN ING MODEL IN A DISTRIBUTED CLIENT / SERVER DATA PROCESSING SYSTEM

[0002] TECHNICAL FIELD

[0003] The disclosure relates generally to training learning models in a distributed system, and more particularly, the disclosure relates to a method of training a learning model in a distributed client / server data processing system having a client interacting with a server and with a model warehouse, over a network. Moreover, the disclosure relates to a system of training the learning model in the distributed client / server data processing system having the client interacting with the server and with the model warehouse, over the network.

[0004] BACKGROUND

[0005] With the rapid growth of Artificial Intelligence (Al) applications, providing Al-related services to clients is becoming increasingly vital, likely constituting one of the major features of 6G networks. Specifically, assisting a client or clients with limited data and computational resources in learning or fine-tuning a learning model is a promising 6G capability. However, the challenge lies in enabling the network to facilitate model learning in a resource-constrained environment and determining the optimal structure for such models.

[0006] As illustrated in FIG. 1 A, in a typical scenario, a client 102 is given an inference task, S, such as classification, and has access to its own dataset for training a model capable of performing this task. However, the client 102 faces constraints of a small dataset and limited computational power, making it impractical to train its model and requires assistance. Therefore, a server 104 that has access to a set of large pre-trained models 106, W, is utilized. The client 102 needs to receive and fine-tune one of these pre-trained models 106 from the server 104 for its specific task. The server 104 may transmit these potentially large pre-trained models 106 in a bandwidth-constrained environment.

[0007] However, the existing setup as shown in FIG. 1 A presents several technical drawbacks: (i) inefficiency in transmitting a model from the server 104 to the client 102, (ii) inability to conduct local training on the client side, and (iii) inability to determine an optimal structure of the model to achieve these goals. Further, it is essential to minimize bandwidth usage while ensuring that the client 102 can learn an effective model for its task.

[0008] Several existing approaches have been developed to address the above problems. One such approach involves transmitting the entire pre-trained model 106 to the client 102 for fine-tuning. Various compression techniques may be applied to facilitate this process. However, this approach has several drawbacks, including: (i) even when compressed, transmitting the pre-trained model 106 demands substantial bandwidth, (ii) the client 102 may require powerful computational resources to load and finetune the pre-trained models 106, and (iii) the client 102 needs access to extensive datasets to effectively fine-tune the pretrained models 106.

[0009] Another approach, for example, a Teacher-Student model (Knowledge Distillation), involves transferring the "knowledge" of a learning model to a smaller model, as depicted in FIG. IB. However, this approach has significant limitations / drawbacks, which includes: (i) sharing data / knowledge may compromise privacy, (ii) it requires multiple rounds of communication between the client and the server, and (iii) despite being smaller model size, it still requires considerable bandwidth for transmission.

[0010] Therefore, there arises a need to address the aforementioned technical problems / drawbacks in training learning models in a distributed system. SUMMARY

[0011] It is an object of the disclosure to provide a method of training a learning model in a distributed client / server data processing system having a client interacting with a server and with a model warehouse, over a network. Moreover, the disclosure relates to a system of training the learning model in the distributed client / server data processing system having the client interacting with the server and with the model warehouse, over the network.

[0012] This object is achieved by the features of the independent claims. Further, implementation forms are apparent from the dependent claims, the description, and the figures.

[0013] The disclosure provides a method of training a learning model in a distributed client / server data processing system having a client interacting with a server and with a model warehouse, over a network. Moreover, the disclosure relates to a system of training the learning model in the distributed client / server data processing system having the client interacting with the server and with the model warehouse, over the network.

[0014] According to a first aspect, there is provided a method of training a learning model in a distributed client / server data processing system having a client interacting with a server and with a model warehouse, over a network. The model warehouse has access to a plurality of learning models associated with a respective plurality of pre-trained encoders and decoders. The method is performed at the client. The method includes sending, to the server, an assistance request including an identifier for the client, an identifier for a task to be performed by the client, a dataset size for the client, and a computational resource identifier for the client where the computational resource identifier identifies a processing capability of the client and where the dataset size identifies a size of a dataset local to the client for use by the client in training a learning model. The method includes receiving, from the server, a decoder of a learning model, where the server has selected the learning model based on the identifier for the task and the computational resource identifier. The method includes receiving, from the server, an estimate of the structure of the output of an encoder corresponding to the selected learning model. The server has derived the estimate based on the dataset size for the client and the processing capability for the client. The method includes selecting a local model for the client based on the processing capability of the client, where the selected model selected by the client has the same architecture as that of the decoder which has been received from the server. The method includes training an encoder by imposing the received estimate of the structure. The method includes further training the encoder and decoder for the local model using data in the dataset local to the client.

[0015] The method significantly reduces the number of bits required for data transmission compared to existing solutions, as it shares only the statistics of the encoder's output, not the encoder itself, leading to substantial bandwidth savings. The training and fine-tuning of the learning models become feasible even in resource-constrained environments, such as cellular networks, due to the efficient use of available computational resources and bandwidth. The learning model that is trained exhibits enhanced robustness to change in data distribution, as it leverages a structure learned from a large dataset to ensure more reliable performance across varying conditions.

[0016] The method efficiently selects and transmits an appropriate learning model from the server to the client, based on the client's request and characteristics (e.g., processing capability / computing power of the client and available dataset size for the client), to ensure low bandwidth usage while achieving optimal inference performance. The method involves sharing a part of the learning model (i.e., the decoder) and the structure (i.e., statistics) of the output from the encoder.

[0017] Preferably, the decoder is fixed for the client and the server. Optionally, the assistance request is in a predefined format. Preferably, the estimate structure is a mixture-distribution and the mixture-distribution may be a Gaussian mixture distribution. Optionally, the parameters of the mixture distribution are derived using an expectation-maximization approach. Preferably, the estimated structure relates to statistics of a latent variable. The dimension of the latent variable may be the same as the output of the encoder of the model selected by the server. Preferably, the structure is provided per class. The client may initialize the encoder randomly and the decoder using parameters of the decoder provided by the server.

[0018] According to a second aspect, there is provided a method of training a learning model in a distributed client / server data processing system having a client interacting with a server and with a model warehouse, over a network. The model warehouse has access to a plurality of learning models associated with a respective plurality of pre-trained encoders and decoders. The method is performed at the server. The method includes receiving, from the client, an assistance request including an identifier for the client, an identifier for a task to be performed by the client, a dataset size for the client, and a computational resource identifier for the client where the computational resource identifier identifies a processing capability of the client and where the dataset size identifies a size of a dataset local to the client for use by the client in training a learning model. The method includes sending, to the client, a decoder of a learning model, where the server has selected the learning model based on the identifier for the task and based on the computational resource identifier. The method includes sending, to the client, an estimate of the structure of the output of an encoder corresponding to the selected learning model. The server has derived the estimate based on the dataset size for the client and the processing capability for the client. The client selects a local model for the client based on the processing capability of the client, where the selected model selected by the client has the same architecture as that of the decoder which has been received from the server. The client trains an encoder by imposing the received estimate of the structure and further trains the encoder and decoder for the local model using data in the dataset local to the client.

[0019] According to a third aspect, there is provided a system comprising means adapted for carrying out all the steps of the method described above.

[0020] The system significantly reduces the number of bits required for data transmission compared to existing solutions, as it shares only the statistics of the encoder's output, not the encoder itself, leading to substantial bandwidth savings. The training and fine-tuning of the learning models become feasible even in resource-constrained environments, such as cellular networks, due to the efficient use of available computational resources and bandwidth. The learning model that is trained exhibits enhanced robustness to change in data distribution, as it leverages a structure learned from a large dataset to ensure more reliable performance across varying conditions. The system efficiently selects and transmits an appropriate learning model from the server to the client, based on the client's request and characteristics (e.g., processing capability / computing power of the client and available dataset size for the client), to ensure low bandwidth usage while achieving optimal inference performance.

[0021] According to a fourth aspect, there is provided a computer program including instructions for carrying out all the steps of the above described method, when the computer program is executed on a computer system.

[0022] Therefore, in contradistinction to the existing solutions, the method significantly reduces the number of bits required for data transmission compared to existing solutions, as it shares only the statistics of the encoder's output, not the encoder itself, leading to substantial bandwidth savings. The training and fine-tuning of the learning models become feasible even in resource- constrained environments, such as cellular networks, due to the efficient use of available computational resources and bandwidth. The learning model that is trained exhibits enhanced robustness to change in data distribution, as it leverages a structure learned from a large dataset to ensure more reliable performance across varying conditions. The method efficiently selects and transmits an appropriate learning model from the server to the client, based on the client's request and characteristics (e.g., processing capability / computing power of the client and available dataset size for the client), to ensure low bandwidth usage while achieving optimal inference performance.

[0023] These and other aspects of the disclosure will be apparent from the implementation s) described below. BRIEF DESCRIPTION OF DRAWINGS

[0024] Implementations of the disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0025] FIG. 1 A illustrates a typical setup for training a learning model in a system;

[0026] FIG. IB illustrates a typical Teacher- Student model to transfer learning models in a system;

[0027] FIG. 2 is a block diagram that illustrates a system of training a learning model in a distributed client / server data processing system in accordance with an implementation of the disclosure;

[0028] FIG. 3 illustrates an exemplary encoder and a decoder of a learning model in a distributed client / server data processing system in accordance with an implementation of the disclosure;

[0029] FIGS. 4A-4B are flow diagrams that illustrate a method of training a learning model in a distributed client / server data processing system at a client in accordance with an implementation of the disclosure;

[0030] FIG. 5 is a flow diagram that illustrates a method of training a learning model in a distributed client / server data processing system at a server in accordance with an implementation of the disclosure; and

[0031] FIG. 6 is an illustration of a computer system in which the various architectures and functionalities of the various previous implementations may be implemented.

[0032] DETAILED DESCRIPTION OF THE DRAWINGS

[0033] Implementations of the disclosure provide a method of training a learning model in a distributed client / server data processing system having a client interacting with a server and with a model warehouse, over a network. Moreover, the disclosure relates to a system of training the learning model in the distributed client / server data processing system having the client interacting with the server and with the model warehouse, over the network.

[0034] To make solutions of the disclosure more comprehensible for a person skilled in the art, the following implementations of the disclosure are described with reference to the accompanying drawings.

[0035] Terms such as "a first", "a second", "a third", and "a fourth" (if any) in the summary, claims, and foregoing accompanying drawings of the disclosure are used to distinguish between similar objects and are not necessarily used to describe a specific sequence or order. It should be understood that the terms so used are interchangeable under appropriate circumstances, so that the implementations of the disclosure described herein are, for example, capable of being implemented in sequences other than the sequences illustrated or described herein. Furthermore, the terms "include" and "have" and any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, a method, a system, a product, or a device that includes a series of steps or units, is not necessarily limited to expressly listed steps or units but may include other steps or units that are not expressly listed or that are inherent to such process, method, product, or device.

[0036] FIG. 2 is a block diagram that illustrates a system 200 of training a learning model in a distributed client / server data processing system 202 in accordance with an implementation of the disclosure. The distributed client / server data processing system 202 have a client 204 interacting with a server 206 and with a model warehouse 208, over a network. The model warehouse 208 has access to one or more learning models associated with a respective one or more pre-trained encoders and decoders. Preferably, the one or more learning models may be a large learning model. The client 204 sends an assistance request to the server 206. Optionally, the assistance request is in a predefined format. The assistance request includes an identifier for the client 204, an identifier for a task to be performed by the client 204, a dataset size for the client 204, and a computational resource identifier for the client 204. The computational resource identifier identifies a processing capability of the client 204 and the dataset size identifies a size of a dataset local to the client 204 for use by the client 204 in training a learning model.

[0037] The server 206 selects the learning model based on the identifier for the task and the computational resource identifier of the assistance request from the client 204. The client 204 receives a decoder of the selected learning model from the server 206. The client 204 receives an estimate of the structure of the output of an encoder corresponding to the selected learning model from the server 206. The server 206 has derived the estimate based on the dataset size for the client 204 and the processing capability for the client 204. Preferably, the estimate structure is a mixture-distribution and the mixture-distribution may be a Gaussian mixture distribution. Optionally, the parameters of the mixture distribution are derived using an expectationmaximization approach. The estimated structure may relate to statistics of a latent variable. Optionally, the dimension of the latent variable is the same as the output of the encoder of the model selected by the server 206. Preferably, the structure is provided per class. Optionally, the estimated structure includes a distribution identifier, a multihead flag, and a per class flag.

[0038] The client 204 selects a local model based on the processing capability of the client 204. The selected model that is selected by the client 204 has the same architecture as that of the decoder which has been received from the server 206. For example, the selected model comprises two parts: the decoder, which has the same architecture as the decoder provided by the server 206, and the encoder, which can differ from the one used by the server 206. The client 204 chooses the encoder based on its specific capabilities and dataset size, ensuring optimal performance for its resources, allowing for variation tailored to each client 204. The client 204 may initialize the encoder randomly and the decoder using parameters of the decoder provided by the server 206. The client 204 trains an encoder by imposing the received estimate of the structure. The client 204 further trains the encoder and the decoder for the local model using data in the dataset local to the client 204. Preferably, the decoder is fixed for the client 204 and the server 206.

[0039] The system 200 significantly reduces the number of bits required for data transmission compared to existing solutions, as it shares only the statistics of the encoder's output, not the encoder itself, leading to substantial bandwidth savings. The training and fine-tuning of the learning models become feasible even in resource-constrained environments, such as cellular networks, due to the efficient use of available computational resources and bandwidth. The learning model that is trained exhibits enhanced robustness to change in data distribution, as it leverages a structure learned from a large dataset to ensure more reliable performance across varying conditions. The system 200 efficiently selects and transmits an appropriate learning model from the server 206 to the client 204, based on the client's request and characteristics (e.g., processing capability / computing power of the client 204 and available dataset size for the client 204), to ensure low bandwidth usage while achieving optimal inference performance.

[0040] FIG. 3 illustrates an exemplary encoder and a decoder of a learning model in a distributed client / server data processing system in accordance with an implementation of the disclosure. The distributed client / server data processing system have a client interacting with a server and with a model warehouse, over a network. The model warehouse has access to one or more learning models associated with a respective one or more pre-trained encoders and decoders. The learning model is divided into two parts, an encoder (We) 302 and a decoder (Wd) 304, as shown in FIG. 3. This architecture is designed so that the encoder 302 is relatively large, while the decoder 304 is comparatively small.

[0041] The client sends an assistance request to the server. The assistance request includes an identifier for the client, an identifier for a task to be performed by the client, a dataset size for the client, and a computational resource identifier for the client. Optionally, the assistance request is in a predefined format of: (client_id, task_id, dataset_size, processor_id), where the client_id is the identifier for the client, the task_id is the identifier for the task to be performed by the client, the dataset_size is the dataset size for the client, and the processor_id is the computational resource identifier for the client. The computational resource identifier identifies a processing capability of the client and the dataset size identifies a size of a dataset local to the client for use by the client in training a learning model.

[0042] Upon receiving the assistance request from the client, the server selects an appropriate learning model from the existing learning models in the model warehouse based on the identifier for the task and the computational resource identifier. The server derives an estimate of the structure (i.e. related to statistics of a latent variable) of the output of the encoder 302 corresponding to the selected learning model, based on the dataset size for the client and the processing capability for the client. The server sends the decoder 304 of the learning model and the estimated structure of the latent variable (U) to the client in a format of: (server_id, task_id, Wd, Q), where the server_id is an identifier for the server, the task_id is the identifier for the task to be performed by the client, the Wd is the decoder 304 of the learning model, and the Q is the estimated structure of the latent variable. Preferably, the estimate structure, Q, of the latent variable is a mixture-distribution and the mixture-distribution may be a Gaussian mixture distribution with M mixtures (components). Each mixture / component may be described by its parameters. Optionally, the parameters of the mixture distribution are derived using an expectation-maximization approach.

[0043] The mixture distribution may effectively estimate any distribution if the number of mixtures / components is sufficiently large. The server may select an appropriate number of components, M, based on the dataset size for the client and the processing capability for the client. For example, if the available dataset size for the client is small, the server provides more accurate statistics, thus a larger value of AT is selected. If the processing capability for the client is limited, the server may not process or utilize finer statistics, thus the server selects a smaller value of M.

[0044] Preferably, the structure, Q, is provided per class or label, (e.g., image classes such as cats or dogs in classification tasks), or per subclass (e.g., breeds of cats or dogs). Optionally, the structure, Q, is provided per head in cases where the last layer of the encoder 302 is a summation of heads, H.

[0045] Preferably, the structure, Q, is described and sent to the client as: (distribution_id, multihead_flag, per class_flag, N_components, coeff_components, param_components), where (i) the distribution_id is a distribution identifier that indicates a type of distribution considered, for example, if set to Gaussian, a Gaussian-mixture distribution is used, (ii) the multihead_flag indicates whether the structure is provided per head, (iii) the per_class_flag indicates whether the structure is provided per class, which may also include other options, such as per_sub_class, (iv) the N_components denotes the number of components / mixtures (AT), (v) the coeff_components denotes the coefficients of each component (AT), which may be sum up to one, (vi) the param_components denotes the parameters of each component in the considered mixture distribution.

[0046] The client selects a local model based on the processing capability of the client. The selected model selected by the client has the same architecture as that of the decoder 304 which has been received from the server. For example, the selected model comprises two parts: the decoder 304, which has the same architecture as the decoder 304 provided by the server, and the encoder 302, which can differ from the one used by the server. The client chooses the encoder 302 based on its specific capabilities and dataset size, ensuring optimal performance for its resources, allowing for variation tailored to each client. The encoder 302 may differ and potentially be smaller, with the only requirement being that the dimension of the latent variable (U) matches that of the pre- trained model on the server. The decoder 304 must be identical to the pre- trained model on the server. The client then trains or fine-tunes the model using the shared decoder and the structure (statistics).

[0047] The client may initialize the encoder 302 randomly and the decoder 304 using parameters of the decoder 304 provided by the server. Preferably, the decoder 304 is fixed / froze for the client and the server. The client trains an encoder 302 by imposing the received estimate of the structure, Q. The client further trains the encoder 302 and the decoder 304 for the local model using data in the dataset local to the client. In an embodiment, the structure (i.e. statistics) of the latent variable space of each client, optionally for every head h and optionally every class c, is estimated as a mixture distribution, for example, as a Gaussian-Mixture distribution, which is described as, are the coefficients of the components / mixtures, M, and each component is equal to: Thus, each component represents a multivariate

[0048] Gaussian distribution with a mean vector tic h mand a covariance matrix (which may be diagonal, for instance). The parameters ot the components is described as:

[0049] In another embodiment, the client trains the encoder 302 by imposing the received estimate of the structure, Q, from the server, for instance, by employing the following regularized objective function, where the structure serves as a prior in the regularization term as follows:

[0050] FIGS. 4A-4B are flow diagrams that illustrate a method of training a learning model in a distributed client / server data processing system at a client in accordance with an implementation of the disclosure. The distributed client / server data processing system have a client interacting with a server and with a model warehouse, over a network. The model warehouse has access to one or more learning models associated with a respective one or more pre-trained encoders and decoders. The method is performed at the client. At a step 402, the client sends an assistance request to the server. The assistance request includes an identifier for the client, an identifier for a task to be performed by the client, a dataset size for the client, and a computational resource identifier for the client. The computational resource identifier identifies a processing capability of the client and the dataset size identifies a size of a dataset local to the client for use by the client in training a learning model. At a step 404, the client receives a decoder of a learning model from the server, where the server has selected the learning model based on the identifier for the task and the computational resource identifier.

[0051] At a step 406, the client receives an estimate of the structure of the output of an encoder corresponding to the selected learning model from the server. The server has derived the estimate based on the dataset size for the client and the processing capability for the client. At a step 408, the client selects a local model for the client based on the processing capability of the client. The selected model selected by the client has the same architecture as that of the decoder which has been received from the server. At a step 410, the client trains an encoder by imposing the received estimate of the structure. At a step 412, the client further trains the encoder and decoder for the local model using data in the dataset local to the client.

[0052] FIG. 5 is a flow diagram that illustrates a method of training a learning model in a distributed client / server data processing system at a server in accordance with an implementation of the disclosure. The distributed client / server data processing system have a client interacting with a server and with a model warehouse, over a network. The model warehouse has access to one or more learning models associated with a respective one or more pre-trained encoders and decoders. At a step 502, the server receives an assistance request from the client. The assistance request includes an identifier for the client, an identifier for a task to be performed by the client, a dataset size for the client, and a computational resource identifier for the client. The computational resource identifier identifies a processing capability of the client. The dataset size identifies a size of a dataset local to the client for use by the client in training a learning model. At a step 504, the server sends a decoder of a learning model to the client. The server has selected the learning model based on the identifier for the task and based on the computational resource identifier.

[0053] At a step 506, the server sends an estimate of the structure of the output of an encoder corresponding to the selected learning model to the client. The server has derived the estimate based on the dataset size for the client and the processing capability for the client. The client selects a local model for the client based on the processing capability of the client. The selected model selected by the client has the same architecture as that of the decoder which has been received from the server and has an encoder which is based on the processing capability of the client. The client trains the encoder of the selected model by imposing the received estimate of the structure and the client further trains the encoder and decoder for the local model using data in the dataset local to the client.

[0054] FIG. 6 is an illustration of a computer system in which the various architectures and functionalities of the various previous implementations may be implemented. As shown, the computer system 600 includes at least one processor 604 that is connected to a bus 602, wherein the computer system 600 may be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), Hyper Transport, or any other bus or point-to-point communication protocol (s). The computer system 600 also includes a memory 606.

[0055] Control logic (software) and data are stored in the memory 606 which may take a form of random-access memory (RAM). In the disclosure, a single semiconductor platform may refer to a sole unitary semiconductor-based integrated circuit or chip. It should be noted that the term single semiconductor platform may also refer to multi-chip modules with increased connectivity which simulate on-chip modules with increased connectivity which simulate on-chip operation and make substantial improvements over utilizing a conventional central processing unit (CPU) and bus implementation. Of course, the various modules may also be situated separately or in various combinations of semiconductor platforms per the desires of the user.

[0056] The computer system 600 may also include a secondary storage 610. The secondary storage 610 includes, for example, a hard disk drive and a removable storage drive, representing a floppy disk drive, a magnetic tape drive, a compact disk drive, digital versatile disk (DVD) drive, recording device, universal serial bus (USB) flash memory. The removable storage drive at least one of reads from and writes to a removable storage unit in a well-known manner.

[0057] Computer programs, or computer control logic algorithms, may be stored in at least one of the memory 606 and the secondary storage 610. Such computer programs, when executed, enable the computer system 600 to perform various functions as described in the foregoing. The memory 606, the secondary storage 610, and any other storage are possible examples of computer-readable media.

[0058] In an implementation, the architectures and functionalities depicted in the various previous figures may be implemented in the context of the processor 604, a graphics processor coupled to a communication interface 612, an integrated circuit (not shown) that is capable of at least a portion of the capabilities of both the processor 604 and a graphics processor, a chipset (namely, a group of integrated circuits designed to work and sold as a unit for performing related functions, and so forth).

[0059] Furthermore, the architectures and functionalities depicted in the various previous-described figures may be implemented in a context of a general computer system, a circuit board system, a game console system dedicated for entertainment purposes, an application-specific system. For example, the computer system 600 may take the form of a desktop computer, a laptop computer, a server, a workstation, a game console, an embedded system.

[0060] Furthermore, the computer system 600 may take the form of various other devices including, but not limited to a personal digital assistant (PDA) device, a mobile phone device, a smart phone, a television, and so forth. Additionally, although not shown, the computer system 600 may be coupled to a network (for example, a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, or the like) for communication purposes through an I / O interface 608.

[0061] It should be understood that the arrangement of components illustrated in the figures described are exemplary and that other arrangement may be possible. It should also be understood that the various system components (and means) defined by the claims, described below, and illustrated in the various block diagrams represent components in some systems configured according to the subject matter disclosed herein. For example, one or more of these system components (and means) may be realized, in whole or in part, by at least some of the components illustrated in the arrangements illustrated in the described figures. In addition, while at least one of these components are implemented at least partially as an electronic hardware component, and therefore constitutes a machine, the other components may be implemented in software that when included in an execution environment constitutes a machine, hardware, or a combination of software and hardware.

[0062] Although the disclosure and its advantages have been described in detail, it should be understood that various changes, substitutions, and alterations can be made herein without departing from the spirit and scope of the disclosure as defined by the appended claims.

Claims

CLAIMS1. A method of training a learning model in a distributed client / server data processing system (202) having a client (204) interacting with a server (206) and with a model warehouse (208), over a network, wherein the model warehouse has access to a plurality of learning models associated with a respective plurality of pre-trained encoders and decoders, comprising steps, performed at the client, of: sending, to the server, an assistance request including an identifier for the client, an identifier for a task to be performed by the client, a dataset size for the client, and a computational resource identifier for the client where the computational resource identifier identifies a processing capability of the client and where the dataset size identifies a size of a dataset local to the client for use by the client in training a learning model; receiving, from the server, a decoder (304) of a learning model, where the server has selected the learning model based on the identifier for the task and the computational resource identifier; receiving, from the server, an estimate of the structure of the output of an encoder (302) corresponding to the selected learning model, wherein the server has derived the estimate based on the dataset size for the client and the processing capability for the client; selecting a local model for the client based on the processing capability of the client, where the selected model selected by the client has the same architecture as that of the decoder which has been received from the server and has an encoder which is based on the processing capability of the client; and training an encoder by imposing the received estimate of the structure; and further training the encoder and decoder for the local model using data in the dataset local to the client.

2. The method of claim 1, wherein the decoder is fixed for the client and the server.

3. The method of claim 1, wherein the assistance request is in a predefined format.

4. The method of claim 1, wherein the estimate structure is a mixture-distribution.

5. The method of claim 4, wherein the mixture-distribution is a Gaussian mixture distribution.

6. The method of claim 4, wherein the parameters of the mixture distribution are derived using an expectation-maximization approach.

7. The method of claim 1, wherein the estimated structure relates to statistics of a latent variable.

8. The method of claim 7, wherein the dimension of the latent variable is the same as the output of the encoder of the model selected by the server.

9. The method of claim 1, wherein the structure is provided per class.

10. The method of claim 1, wherein the client initializes the encoder randomly.

11. The method of claim 1, wherein the client initializes the decoder using parameters of the decoder provided by the server.

12. A method of training a learning model in a distributed client / server data processing system (202) having a client (204) interacting with a server (206) and with a model warehouse (208), over a network, wherein the model warehouse has access to a plurality of learning models associated with a respective plurality of pre-trained encoders and decoders, comprising steps, performed at the server, of: receiving, from the client, an assistance request including an identifier for the client, an identifier for a task to be performed by the client, a dataset size for the client, and a computational resource identifier for the client where the computational resource identifier identifies a processing capability of the client and where the dataset size identifies a size of a dataset local to the client for use by the client in training a learning model; sending, to the client, a decoder (304) of a learning model, where the server has selected the learning model based on the identifier for the task and based on the computational resource identifier; sending, to the client, an estimate of the structure of the output of an encoder (302) corresponding to the selected learning model, wherein the server has derived the estimate based on the dataset size for the client and the processing capability for the client; wherein the client selects a local model for the client based on the processing capability of the client, where the selected model selected by the client has the same architecture as that of the decoder which has been received from the server and has an encoder which is based on the processing capability of the client; and trains an encoder by imposing the received estimate of the structure; and further trains the encoder and decoder for the local model using data in the dataset local to the client.

13. A system (200) comprising means adapted for carrying out all the steps of the method according to any preceding method claim.

14. A computer program comprising instructions for carrying out all the steps of the method according to any preceding method claim, when said computer program is executed on a computer system.

Images