Method and system of training learning models in a distributed client / server data processing system
The method and system address privacy and resource challenges in distributed learning by using shared decoders and varying encoders, enhancing privacy, reducing bandwidth, and improving inference accuracy through uniform model configurations.
Patent Information
- Application Number
- PCT/EP2024/072899
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2026-02-19
Smart Images
Figure EP2024072899_19022026_PF_FP_ABST
Abstract
Description
[0001] METHOD AND SYSTEM OF TRAINING LEARNING MODELS E A DISTRIBUTED CLIENT / SERVER DATA
[0002] PROCESSING SYSTEM
[0003] TECHNICAL FIELD
[0004] The disclosure generally relates to training learning models, and more particularly, it relates to a method of training one or more learning models in a distributed client / server data processing system having one or more clients interacting with a server over a network. Moreover, the disclosure relates to a system of training one or more learning models in the distributed client / server data processing system having one or more clients interacting with the server over the network.
[0005] BACKGROUND
[0006] In many practical scenarios, learning is performed in a distributed manner due to the inherently distributed nature of the available data, which cannot be centrally shared due to privacy and communication constraints. Distributed learning also efficiently utilizes computational resources across all clients in the network.
[0007] FIG. 1 illustrates a block diagram of a typical collaborative distributed learning setup 100. The collaborative distributed learning setup 100 includes one or more clients (e.g. client 1 to client K) 102A-K, and a server 106. The one or more clients 102A-K includes one or more learning models 104A-K denoted as w., to vvk. The one or more clients 102A-K independently train or fine-tune their learning models using their respective datasets, which may vary, and are processed in ways that align with approaches like federated learning and split learning. The server 106 functions as a hub for receiving and transmitting messages, £^,S^ ,£^,S^ , which may include parameters of the decoder (or optionally the encoder) and the structure of the outputs of the encoders or their corresponding aggregated values, to and from the one or more clients 102A-K to facilitate the training or fine-tuning of the one or more learning models 104A-K. However, due to privacy constraints, the server 106 may not have access to the actual datasets. After training or fine-tuning the one or more learning models 104A-K, the one or more clients 102A-K can independently use their trained models for inference without relying on the server 106.
[0008] However, the distributed learning setup 100 has several drawbacks as follows: (i) The raw data from the one or more clients 102A-K cannot be shared with the server 106 due to privacy concerns, (ii) The trained models can have large dimensions, making them resource-intensive to handle, and (iii) the one or more clients 102A-K struggle to update their models independently due to the limited size of their individual datasets.
[0009] Further, the distributed learning setup 100 requires a large bandwidth for both transmission and reception of the datasets between the one or more clients 102A-K and the server 106. The distributed learning setup 100 requires high computational resources to train the one or more learning models 104A-K, which are large and may encounter convergence issues. Moreover, each of the one or more clients 102A-K cannot update their learning model independently due to lack of data in their dataset.
[0010] Therefore, there arises a need to address these aforementioned technical problems / drawbacks to utilize data from all clients effectively to train the learning models without compromising privacy while maintaining communication under bandwidth constraints.
[0011] SUMMARY
[0012] It is an object of the disclosure to provide a method of training one or more learning models in a distributed client / server data processing system having one or more clients interacting with a server over a network, and a system of training one or more learning models in the distributed client / server data processing system having one or more clients interacting with the server over the network while avoiding one or more disadvantages of prior art approaches. This object is achieved by the features of the independent claims. Further, implementation forms are apparent from the dependent claims, the description, and the figures.
[0013] The disclosure provides a method of training one or more learning models in a distributed client / server data processing system having one or more clients interacting with a server over a network. Moreover, the disclosure relates to a system of training one or more learning models in the distributed client / server data processing system having one or more clients interacting with the server over the network.
[0014] According to a first aspect, a method of training one or more learning models in a distributed client / server data processing system having one or more clients interacting with a server over a network is provided. The method is performed at each of the one or more clients. The method includes selecting a learning model corresponding to the respective client, and initializing the selected learning model. The learning model includes an encoder and a decoder. The decoder is the same for each of the one or more clients and the encoder can vary for each of the one or more clients depending on specific information regarding each respective client. The method includes performing local training computations of the selected learning model using a dataset assigned to the respective client, by performing an optimization algorithm to derive a locally updated model. The method includes deriving an estimate of the structure of the output of the encoder of the respective client using the selected learning model and the dataset assigned to the respective client. The method includes sending the derived estimate of the structure of the output of the encoder, and the parameters of the respective decoder, in a message to the server. The method includes receiving, in a message from the server, an aggregated derived estimate of the structure of the output of the encoder and an aggregated set of parameters of the respective decoder, for each of the one or more clients. The method includes reinitializing the respective decoder using the received message and training the respective encoder using the received message for each respective client.
[0015] The method efficiently trains a high-performance learning model (e.g. a large learning model) by utilizing data from each of the clients effectively while maintaining privacy and providing for efficient communication. The method significantly reduces the bandwidth required compared to traditional Federated Learning algorithms by minimizing the data transmitted between the clients and the server. The method ensures uniformity in model configurations (e.g., local models, number of components per head, etc.,) across all clients through pre-shared settings, which facilitates consistent learning outcomes. Additionally, the method enhances inference accuracy by allowing each client to select its local learning model based on available computational resources. Furthermore, it reduces transmission overhead by structuring client-server communication to include decoder descriptions and learned latent variable statistics in a predefined format.
[0016] Preferably, each client sends the parameters of its respective encoder to the server. Optionally, each client receives an aggregated set of parameters of the respective encoders from the server. Optionally, each client reinitializes the respective encoder in response to receiving the aggregated set of parameters of the respective encoders. Preferably, the message from the respective clients to the server is in a predefined format.
[0017] Preferably, the steps of the above method are repeated until a convergence is reached. Optionally, the optimization algorithm is a Stochastic Gradient Descent algorithm. Preferably, the dimension of the output of the encoder for each of the respective clients is the same. Preferably, the selection of the learning model is a function of the computational power of the respective client.
[0018] According to a second aspect, there is provided a system that includes means adapted for carrying out all the steps of the above method. The system efficiently trains a high-performance learning model (e.g. a large learning model) by utilizing data from each of the clients effectively while maintaining privacy and providing for efficient communication. The system significantly reduces the bandwidth required compared to traditional Federated Learning algorithms by minimizing the data transmitted between the clients and the server. The system ensures uniformity in model configurations (e.g., local models, number of components per head, etc.,) across all clients through pre-shared settings, which facilitates consistent learning outcomes. Additionally, the system enhances inference accuracy by allowing each client to select its local learning model based on available computational resources. Furthermore, the system reduces transmission overhead by structuring client-server communication to include decoder descriptions and learned latent variable statistics in a predefined format.
[0019] According to a third aspect, there is provided a computer program includes instructions for carrying out all the steps of the method when the computer program is executed on a computer system.
[0020] Therefore, in contradistinction to existing solutions, the above method trains a high-performance learning model (e.g. a large learning model) by utilizing data from each of the clients effectively while maintaining privacy and efficient communication. The method significantly reduces the bandwidth required compared to traditional Federated Learning algorithms by minimizing the data transmitted between the clients and the server.
[0021] These and other aspects of the disclosure will be apparent from the implementations) described below.
[0022] BRIEF DESCRIPTION OF DRAWINGS
[0023] Implementations of the disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which:
[0024] FIG. 1 illustrates a block diagram of a typical collaborative distributed learning setup;
[0025] FIG. 2 illustrates a block diagram of a system of training one or more learning models in a distributed client / server data processing system having one or more clients interacting with a server over a network in accordance with an implementation of the disclosure;
[0026] FIG. 3 illustrates an exemplary diagram of a local learning model selected by each client to estimate the structure of the output of an encoder of FIG. 2 in accordance with an implementation of the disclosure;
[0027] FIG. 4 illustrates an interaction diagram between a client (k) and a server over / ? communication rounds for training one or more learning models in accordance with an implementation of the disclosure;
[0028] FIG. 5A and 5B are flow diagrams that illustrate a method of training one or more learning models in a distributed client / server data processing system having one or more clients interacting with a server over a network in accordance with an implementation of the disclosure; and
[0029] FIG. 6 is an illustration of a computer system (e.g., a distributed client / server data processing system, a server, one or more clients) in which the various architectures and functionalities of the various previous implementations may be implemented.
[0030] DETAILED DESCRIPTION OF THE DRAWINGS
[0031] The disclosure's implementation provides a method of training one or more learning models in a distributed client / server data processing system having one or more clients interacting with a server over a network. The disclosure also relates to a system of training one or more learning models in the distributed client / server data processing system having one or more clients interacting with the server over the network.
[0032] To make solutions of the disclosure more comprehensible for a person skilled in the art, the following implementations of the disclosure are described with reference to the accompanying drawings.
[0033] Terms such as “a first”, “a second”, “a third”, and “a fourth” (if any) in the summary, claims, and foregoing accompanying drawings of the disclosure are used to distinguish between similar objects and are not necessarily used to describe a specific sequence or order. It should be understood that the terms so used are interchangeable under appropriate circumstances, so that the implementations of the disclosure described herein are, for example, capable of being implemented in sequences other than the sequences illustrated or described herein. Furthermore, the terms “include” and “have” and any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, a method, a system, a product, or a device that includes a series of steps or units, is not necessarily limited to expressly listed steps or units but may include other steps or units that are not expressly listed or that are inherent to such process, method, product, or device.
[0034] FIG. 2 illustrates a block diagram of a system 200 of training one or more learning models 210A-K in a distributed client / server data processing system 202 having one or more clients (e.g. client 1 to client K) 204A-K interacting with a server 218 over a network in accordance with an implementation of the disclosure. The distributed client / server data processing system 202 includes one or more clients 204A-K. The one or more clients 204A-K include one or more dataset (Si(r>to S^r>) 212A-K and one or more learning models 210A-K. The one or more learning models (e.g. Wi(r>to W^r>) 210A-K may be large learning models. Each client (e.g. 204A) selects a learning model (e.g. 210A) and initializes the selected learning model. Optionally, the clients 204A-K select the learning model (e.g. 210A-K) based on their computational power. The learning model (e.g. 210A-K) may include a local model. The learning model (e.g. 210A-K) includes an encoder (e.g. 206A-K) and a decoder (e.g. 208A-K). For example, the learning model 210A includes an encoder 206A and a decoder 208A, and the learning model 210K includes an encoder 206K and a decoder 208K. The decoder (e.g. 208A-K) is the same for each of the clients 204A-K and the encoder (e.g. 206 A-K) can vary for each of the clients 204 A-K depending on specific information regarding each respective client.
[0035] For example, if a client 204A has a function of high computation power, the client 204A selects a large learning model 210A for their encoder 206A to handle the complexity. Optionally, the encoder 206A has a last layer that includes a summation of projections of heads, H. For example, if the encoder 206A has only one head (H=l), a function of the projection is identity, which means that the encoder 206A corresponds to a single-head case.
[0036] Each client (e.g. 204A-K) initializes the selected learning model (e.g. 210A-K) to incorporate the selected learning model (e.g. 210A-K) corresponding to the encoder (e.g. 206A-K) of the respective client. Each client (e.g. 204A-K) performs local training computations of the selected learning model (e.g. 210A-K) using the dataset (e.g. 212A-K) assigned to the respective client. The local training computations of the selected learning model (e.g. 210A-K) are performed by an optimization algorithm (e.g. Ai to An) to derive a locally updated learning model.
[0037] The server 218 is communicatively connected with the clients 204A-K through the network to receive updates on the selected local model from each client (e.g. 204A-K). The network may be a wired network or a wireless network. The network may be an internet. The server 218 derives an estimate of the structure of the output of the encoder (e.g. 206A-K) of the respective client using the selected learning model (e.g. 210A-K) and the dataset (e.g. 212A-K) assigned to the respective client. Optionally, the dimension of the output of the encoder (e.g. 206A-K) for each of the respective clients is the same. Each of the clients 204A-K sends the parameters of the respective decoder (e.g. 208A-K) and the derived estimate of the structure (e.g. statistics) of the output of the respective encoder (e.g. 206A-K) in a message to the server 218. The server 218 generates an aggregated set of parameters of the respective decoder (e.g. 208A-K) and an aggregated derived estimate of the structure of the output of the encoder (e.g. 206A-K), for each of the clients 204A-K (i.e. model aggregation). Each client (e.g. 204A-K) receives the aggregated derived estimate of the structure of the output of the encoder (e.g. 206 A-K) and the aggregated set of parameters of the respective decoder (e.g. 208A-K) in a message from the server 218. Preferably, the message can be in a predefined format.
[0038] Each client (e.g. 204A-K) re-initializes the respective decoder (e.g. 208A-K) using the received message and trains the respective encoder (e.g. 206 A-K) using the received message from the server 218 for each respective client, to make the estimate of the structure of the output of the respective encoder (e.g. 206A-K) to be close as described in the received message. Preferably, each client (e.g. 204A-K) repeats the above steps / process for R communication rounds (r) as depicted at 214 (r = 1, . . ., R) until a convergence is reached. The convergence may be reached when each client (e.g. 204A-K) reaches a stable state after R rounds of updating the respective decoder (e.g. 208A-K) based on the parameters and adjusting the respective encoder based on the structure of the output. Optionally, the decoders (e.g. 208A-K) of the clients 204A-K are trained in a manner that is similar to the Federated Learning algorithm, and the encoders (e.g. 206A-K) of the clients 204A-K learn implicitly from the structure (e.g. statistics) of the outputs.
[0039] Optionally, when a new client connects to the distributed client / server data processing system 202, the new client can access a pre-trained version of the learning models 210A-K that is already trained. The new client may train its respective learning model to align with the other clients (e.g. 204 A-K) by imposing a derived estimate of the structure of the output of the respective encoder, which is derived from the respective decoder, using a regularized objective function as follows:
[0040] FIG. 4 illustrates an interaction diagram between a client (k) 402 and a server 404 over R communication rounds for training one or more learning models in accordance with an implementation of the disclosure. At a step 408, the client (k) 402 selects a learning model (i.e., H' ■°-)) and initializes the selected learning model for round one 406. At a step 410, the client (k) 402
[0041] After at least two communication rounds (R>2), the client (k) 402 updates the learning model to reflect the aggregated derived estimate of the structure of the output of the encoder received from the server 404. The client (k) 402 sends updates related to the parameters of the respective decoder of the previous round as a message to the server 404 for a consecutive round. Optionally, the parameters including a decoder model architecture, and a size of the output of the encoders are shared and agreed between the client (k) 402 and the server 404 before initializing the round one 406.
[0042] FIGS. 5 A and 5B are flow diagrams that illustrate a method of training one or more learning models in a distributed client / server data processing system having one or more clients interacting with a server over a network in accordance with an implementation of the disclosure. The method is performed at each of the one or more clients. At a step 502, the method includes selecting a learning model corresponding to the respective client, and initializing the selected learning model. The learning model includes an encoder and a decoder. The decoder is the same for each of the one or more clients and the encoder can vary for each of the one or more clients depending on specific information regarding each respective client. At a step 504, the method includes performing local training computations of the selected learning model in each of the one or more clients using a dataset assigned to the respective client, by performing an optimization algorithm to derive a locally updated model. At a step 506, the method includes deriving an estimate of the structure of the output of the encoder of the respective client using the selected learning model and the dataset assigned to the respective client. Turning to FIG. 5B, at a step 508, the method includes sending the derived estimate of the structure of the output of the encoder, and the parameters of the respective decoder, in a message to the server. At a step 510, the method includes receiving, in a message from the server, an aggregated derived estimate of the structure of the output of the encoder and an aggregated set of parameters of the respective decoder, for each of the one or more clients. At a step 512, the method includes re-initializing the respective decoder using the received message and training the respective encoder using the received message for each respective client.
[0043] The method efficiently trains a high-performance learning model (e.g. a large learning model) by utilizing data from each of the clients effectively while maintaining the privacy and efficient communication The method significantly reduces the bandwidth required compared to traditional Federated Learning algorithms by minimizing the data transmitted between the clients and the server. The method ensures uniformity in model configurations (e.g., local models, number of components per head, etc.,) across all clients through pre-shared settings, which facilitates consistent learning outcomes. Additionally, the method enhances inference accuracy by allowing each client to select its local learning model based on available computational resources. Furthermore, it reduces transmission overhead by structuring client-server communication to include decoder descriptions and learned latent variable statistics in a predefined format.
[0044] Preferably, each client sends the parameters of its respective encoder to the server. Optionally, each client receives an aggregated set of parameters of the respective encoders from the server. Optionally, each client reinitializes the respective encoder in response to receiving the aggregated set of parameters of the respective encoders. Preferably, the message from the respective clients to the server is in a predefined format.
[0045] Preferably, the steps of the above method are repeated until a convergence is reached. Optionally, the optimization algorithm is a Stochastic Gradient Descent algorithm. Preferably, the dimension of the output of the encoder for each of the respective clients is the same. Preferably, the selection of the learning model is a function of the computational power of the respective client.
[0046] FIG. 6 is an illustration of a computer system (e.g., a distributed client / server data processing system, a server, one or more clients) in which the various architectures and functionalities of the various previous implementations may be implemented. As shown, the computer system 600 includes at least one processor 604 that is connected to a bus 602, wherein the computer system 600 may be implemented using any suitable protocol, such as Peripheral Component Interconnect, PCI-Express, Accelerated Graphics Port, AGP, Hyper Transport, or any other bus or point-to-point communication protocol. The computer system 600 also includes a memory 606.
[0047] Control logic (software) and data are stored in the memory 606 which may take a form of random-access memory, RAM. In the disclosure, a single semiconductor platform may refer to a sole unitary semiconductor-based integrated circuit or chip. It should be noted that the term single semiconductor platform may also refer to multi-chip modules with increased connectivity which simulate on-chip modules with increased connectivity which simulate on-chip operation, and make substantial improvements over utilizing a conventional central processing unit, CPU, and bus implementation. Of course, the various modules may also be situated separately or in various combinations of semiconductor platforms per the desires of the user.
[0048] The computer system 600 may also include a secondary storage 610. The secondary storage 610 includes, for example, a hard disk drive and a removable storage drive, representing a floppy disk drive, a magnetic tape drive, a compact disk drive, a digital versatile disk, a DVD, drive, a recording device, universal serial bus, USB, flash memory. The removable storage drives at least one of reads from and writes to a removable storage unit in a well-known manner.
[0049] Computer programs, or computer control logic algorithms, may be stored in at least one of the memory 606 and the secondary storage 610. Such computer programs, when executed, enable the computer system 600 to perform various functions as described in the foregoing. The memory 606, the secondary storage 610, and any other storage are possible examples of computer-readable media.
[0050] In an implementation, the architectures and functionalities depicted in the various previous figures may be implemented in the context of the processor 604, a graphics processor coupled to a communication interface 612, an integrated circuit (not shown) that is capable of at least a portion of the capabilities of both the processor 604 and a graphics processor, a chipset (namely, a group of integrated circuits designed to work and sold as a unit for performing related functions, and so forth).
[0051] Furthermore, the architectures and functionalities depicted in the various previous-described figures may be implemented in a context of a general computer system, a circuit board system, a game console system dedicated to entertainment purposes, or an application-specific system. For example, the computer system 600 may take the form of a desktop computer, a laptop computer, a server, a workstation, a game console, or an embedded system.
[0052] Furthermore, the computer system 600 may take the form of various other devices including, but not limited to a personal digital assistant, PDA, device, a mobile phone device, smartphone, a television, and so forth. Additionally, although not shown, the computer system 600 may be coupled to a network (for example, a telecommunications network, a local area network, LAN, a wireless network, a wide area network, WAN, such as the Internet, a peer-to-peer network, a cable network, or the like) for communication purposes through an input / output, I / O, interface 608.
[0053] It should be understood that the arrangement of components illustrated in the figures described is exemplary and that other arrangements may be possible. It should also be understood that the various system components (and means) defined by the claims, described below, and illustrated in the various block diagrams represent components in some systems configured according to the subject matter disclosed herein. For example, one or more of these system components (and means) may be realized, in whole or in part, by at least some of the components illustrated in the arrangements illustrated in the described figures.
[0054] In addition, while at least one of these components is implemented at least partially as an electronic hardware component, and therefore constitutes a machine, the other components may be implemented in software that when included in an execution environment constitutes a machine, hardware, or a combination of software and hardware.
[0055] Although the disclosure and its advantages have been described in detail, it should be understood that various changes, substitutions, and alterations can be made herein without departing from the spirit and scope of the disclosure as defined by the appended claims.
Claims
CLAIMS1. A method of training a plurality of learning models (210A-K) in a distributed client / server data processing system (202) having a plurality of clients (204A-K) interacting with a server (218, 404) over a network, comprising steps, performed at each of the plurality of clients (204A-K), of: selecting a learning model (210A-K) corresponding to the respective client, and initializing the selected learning model, wherein the learning model (210A-K) comprises an encoder (206A-K) and a decoder (208A-K), wherein the decoder (208A-K) is the same for each of the plurality of clients (204A-K) and the encoder (206A-K) can vary for each of the plurality of clients (204A-K) depending on specific information regarding each respective client; performing local training computations of the selected learning model (210A-K) using a dataset (212A-K) assigned to the respective client, by performing an optimization algorithm to derive a locally updated model; deriving an estimate of the structure of the output of the encoder (206A-K) of the respective client using the selected learning model (210A-K) and the dataset (212A-K) assigned to the respective client; sending the derived estimate of the structure of the output of the encoder (206A-K), and the parameters of the respective decoder (208A-K), in a message to the server (218, 404); receiving, in a message from the server (218, 404), an aggregated derived estimate of the structure of the output of the encoder (206A-K) and an aggregated set of parameters of the respective decoder (208A-K), for each of the plurality of clients (204A-K); and re-initializing the respective decoder (208A-K) using the received message and training the respective encoder (206A-K) using the received message for each respective client.
2. The method of claim 1, wherein each client also sends the parameters of its respective encoder (206A-K) to the server (218, 404).
3. The method of claim 2, wherein each client also receives, from the server (218, 404), an aggregated set of parameters of the respective encoders (206A-K).
4. The method of claim 3, wherein each client also reinitializes the respective encoder (206A-K) in response to receiving the aggregated set of parameters of the respective encoders (206A-K).
5. The method of claim 1, wherein the message from the respective clients to the server (218, 404) is in a predefined format.
6. The method of claim 1, wherein the steps of claim 1 are repeated until a convergence is reached.
7. The method of claim 1, wherein the optimization algorithm is a Stochastic Gradient Descent algorithm.
8. The method of claim 1, wherein the dimension of the output of the encoder (206A-K) for each of the respective clients is the same.
9. The method of claim 1, wherein the selection of the learning model (210A-K) is a function of the computational power of the respective client.
10. A system comprising means adapted for carrying out all the steps of the method according to any preceding method claim.
911. A computer program comprising instructions for carrying out all the steps of the method according to any preceding method claim, when said computer program is executed on a computer system.