System and method of training a plurality of learning models in a distributed client or server data processing system

Abstract

A method of training one or more learning models in a distributed client or server data processing system (200) having one or more clients (202A-K) interacting with a server (204, 304) over a network (208) is provided. Initially, each client receives initial configuration data from the server. Clients then initialize a learning model and estimate both marginal and joint structures of encoder outputs using this received initial configuration data. The client performs local computations involving drawing subsets of data, computing encoder outputs, estimating their structures, measuring contributions in a mixture-distribution, and updating the structure with the entire dataset. Each client sends a message to the server containing the updated structures and measured contributions. This method allows for efficient and privacy-preserving training of learning models across multiple clients.

Description

[0001] SYSTEM AND METHOD OF TRAINING A PLURALITY OF LEARNING MODELS IN A DISTRIBUTED CLIENT OR SERVER DATA PROCESSING SYSTEM

[0002] TECHNICAL FIELD

[0003] The disclosure relates generally to training one or more learning models, and more particularly, the disclosure relates to a method of training one or more learning models in a distributed client or server data processing system having one or more clients interacting with a server over a network. Moreover, the disclosure relates to a system of training one or more learning models in the distributed client or server data processing system having the one or more clients interacting with the server over the network.

[0004] BACKGROUND

[0005] In many practical scenarios, learning is performed in a distributed manner using multi-modal data. However, due to privacy and bandwidth constraints, the data cannot be shared directly to the distributed clients. To handle this, existing technologies propose a method for collaborative distributed training of models that takes in the multi-modal data.

[0006] FIG.l is a block diagram that illustrates a collaborative distributed multi-modal learning system according to prior art. The collaborative distributed multi-modal learning system includes one or more clients (e.g. client 1 to client K) 102A-K, a server 104, and a multi-modal dataset (W) 106. The multi-modal dataset (W) 106 can be an image, a text, or a data and involves both multi- view and multi-modal setups. The one or more clients 102A-K have access to a first neural network, which is an encoder. The one or more clients 102A-K correlate and transform the multi-modal dataset (W) 106 into encoded data. The server 104 has access to a second neural network, which is a decoder that processes the encoded data received from the one or more clients 102A-K. The one or more clients 102A-K and the server 104 collaborate to train and fine-tune the first neural network and the second neural network.

[0007] Although this existing collaborative distributed multi-modal learning system implements training of the first neural network and the second neural network, it faces several constraints and limitations such as (i) privacy concerns since the raw data cannot be shared by the server 104, (ii) resource limitations as the encoders are too large to be loaded on the one or more clients 102A- K, and (iii) bandwidth constraints as the transmission of the raw data between the multi-modal dataset (W) 106 and the one or more clients 102A-K is limited. These constraints and limitations of the collaborative distributed multi-modal learning system necessitate an algorithm for training and fine-tuning on both the client-side and the server-side, an aggregation algorithm on the server side, and an efficient transmission scheme to optimize bandwidth usage, for training and fine tuning a high- performance learning model while minimizing bandwidth usage during both training and inference phases. Minimizing bandwidth usage by transmitting small data to the one or more clients 102A-K, and training a model with high inference accuracy, and compatible with computational power of the one or more clients 102A-K remains a challenge in the collaborative distributed multi-modal learning system.

[0008] However, existing solutions such as Split-learning learns encoders in a manner similar to Federated Learning, which requires large bandwidth and is not suited for large models. Another existing solution such as Multi-view information bottleneck, considers multiple "point-to-point" versions of the information bottleneck method, requiring large bandwidth, and is not suited for large models. Another existing solution such as In-Network Learning, INL, an extension of the information-bottleneck problem to a multi-view distributed setup, considers the joint distribution of data and latent variables, is also not suited for large models and involves implicit coordination.

[0009] Therefore, there arises a need to address the aforementioned technical problems / drawbacks for training the learning models. SUMMARY

[0010] It is an object of the disclosure to provide a method of training one or more learning models in a distributed client or server data processing system having one or more clients interacting with a server over a network, and a system of training one or more learning models in the distributed client or server data processing system having the one or more clients interacting with the server over the network.

[0011] This object is achieved by the features of the independent claims. Further, implementation forms are apparent from the dependent claims, the description, and the figures.

[0012] The disclosure provides a method of training one or more learning models in a distributed client or server data processing system having one or more clients interacting with a server over a network, and a system of training the one or more learning models in the distributed client or server data processing system having the one or more clients interacting with the server over the network.

[0013] According to a first aspect, there is provided a method of training one or more learning models in a distributed client or server data processing system having one or more clients interacting with a server over a network. The method includes receiving initial configuration data from the server at each of the one or more clients. Each client has a respective encoder. The method includes initializing a learning model, and estimating a marginal structure of the output of the encoder at the respective client, and a joint structure of the outputs of the encoders for each of the one or more clients, for the initialized learning model, using the received initial configuration data. The method includes performing local computations at each of the one or more clients including (i) drawing a subset of data from a dataset assigned to the client, (ii) computing the output of the encoder using the drawn subset of data, and estimating the marginal structure of the output of the encoder using the drawn subset of data, (iii) measuring contributions of each component in a mixture-distribution of the marginal structure on each of the outputs of the respective client’s encoders for the respective drawn subset of data at each respective client, and (iv) updating the marginal structure by drawing at least a subset of the entirety of the dataset assigned to the client and inputting the at least a subset of the entirety of the dataset to the encoder. The computing of the output of the encoder is performed by the encoder of the respective client. The method includes sending a message from each of the one or more clients to the server. The message includes the updated marginal structure, latent variables corresponding to the drawn at least a subset of the entirety of the dataset assigned to the client, and the measured contributions.

[0014] The method initializes the marginal structure derived from different views that are free from redundant information, thereby enhancing communication efficiency. Additionally, the marginal structure and the joint structure facilitate efficient data transmission. The method enables each client and the server to construct a common shared codebook with the marginal structure of the output of the encoder (i.e. latent variable), to send the output of the encoder / the latent variables and model parameters efficiently during transmission and inference time. The common shared codebook may be in a predefined format with an index.

[0015] Preferably, the server receives the message from each of the one or more clients and derives an estimation of joint structure of outputs of the encoders at each of the one or more clients, and uses the derived estimation to detect redundancies for the latent variables and assigns marginal structure targets to each of the one or more clients.

[0016] Preferably, the server uses a decoder to make predictions using the received outputs of the encoders included in the message from each of the one or more clients.

[0017] Preferably, the server performs backpropagation using the predictions of the decoder to generate backpropagation vectors. Preferably, the server updates the decoder using the backpropagation vectors.

[0018] Preferably, the server sends a message to each of the one or more clients. The message includes the assigned marginal structure target specific to each of the one or more clients and the respective backpropagation vector specific to each of the one or more clients.

[0019] Preferably, each of the one or more clients receives the message from the server and updates an encoder at each client using the received backpropagation vector and the received assigned marginal structure target.

[0020] According to a second aspect, there is provided a system comprising means adapted for carrying out all the steps of the method described above.

[0021] The system initializes the marginal structure derived from different views that are free from redundant information, thereby enhancing communication efficiency. Additionally, the marginal structure and the joint structure facilitate efficient data transmission. The system enables each client and the server to construct a common shared codebook with the marginal structure of the output of the encoder (i.e. latent variable), to send the output of the encoder / the latent variables and model parameters efficiently during transmission and inference time. The common shared codebook may be in a predefined format with an index.

[0022] According to a third aspect, there is provided a computer program including instructions for carrying out all the steps of the above-described method, when the computer program is executed on a computer system.

[0023] Therefore, in contradistinction to the existing solutions, the method initializes the marginal structure derived from different views that are free from redundant information, thereby enhancing communication efficiency. Additionally, the marginal structure and the joint structure facilitate efficient transmission during the inference phase.

[0024] These and other aspects of the disclosure will be apparent from the implementation s) described below.

[0025] BRIEF DESCRIPTION OF DRAWINGS

[0026] Implementations of the disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0027] FIG.l is a block diagram that illustrates a collaborative distributed multi-modal learning system according to prior art;

[0028] FIG. 2 is a block diagram that illustrates a system of training one or more learning models in a distributed client or server data processing system in accordance with an implementation of the disclosure;

[0029] FIG. 3 illustrates an interaction diagram between a client and a server for training a learning model in a distributed client or server data processing system in accordance with an implementation of the disclosure;

[0030] FIGS. 4A & 4B are flow diagrams that illustrate a method of training one or more learning models in a distributed client or server data processing system having one or more clients interacting with a server over a network, in accordance with an implementation of the disclosure; and

[0031] FIG. 5 is an illustration of a computer system in which the various architectures and functionalities of the various previous implementations may be implemented. DETAILED DESCRIPTION OF THE DRAWINGS

[0032] Implementations of the disclosure provide a method of training one or more learning models in a distributed client or server data processing system having one or more clients interacting with a server over a network and a system of training the one or more learning models in the distributed client or server data processing system having the one or more clients interacting with the server over the network.

[0033] To make solutions of the disclosure more comprehensible for a person skilled in the art, the following implementations of the disclosure are described with reference to the accompanying drawings.

[0034] Terms such as "a first", "a second", "a third", and "a fourth" (if any) in the summary, claims, and foregoing accompanying drawings of the disclosure are used to distinguish between similar objects and are not necessarily used to describe a specific sequence or order. It should be understood that the terms so used are interchangeable under appropriate circumstances, so that the implementations of the disclosure described herein are, for example, capable of being implemented in sequences other than the sequences illustrated or described herein. Furthermore, the terms "include" and "have" and any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, a method, a system, a product, or a device that includes a series of steps or units, is not necessarily limited to expressly listed steps or units but may include other steps or units that are not expressly listed or that are inherent to such process, method, product, or device.

[0035] FIG. 2 is a block diagram that illustrates a system of training one or more learning models in a distributed client or server data processing system 200 in accordance with an implementation of the disclosure. The distributed client or server data processing system 200 have one or more clients (e.g. client 1 to client K) 202A-K interacting with a server 204 over a network 208. Each client (e.g. 202A-K) receives the initial configuration data from the server 204 through the network 208. Each client (e.g. 202A- K) includes a respective encoder. Each client (e.g. 202A-K) choses an encoder that is compatible with computational power of that client (e.g. 202A-K). The client (e.g. 202A-K) is configured to initialize a learning model and estimate a marginal structure of the output at the respective encoder, and a joint structure of the outputs of the encoders for the initialized learning model using the received initial configuration data. Optionally, the marginal structure and the joint structure are determined using the Expectation-Maximization, EM, procedure.

[0036] The client (e.g. 202A-K) is configured to perform local computations, which include (i) drawing a subset of data from a dataset assigned to the respective client, and (ii) computing the output of the encoder using the drawn subset of data, and estimating the marginal structure of the output of the encoder using the drawn subset of data. The computing of the output of the encoder may be performed by the encoder of the respective client. The client (e.g. 202A-K) is configured to measure contributions of each component in a mixture-distribution of the marginal structure on each of the outputs of the respective client’s encoders for the respective drawn subset of data at each respective client. The client (e.g. 202A-K) is configured to update the marginal structure by drawing at least a subset of the entirety of the dataset assigned to the respective client and input the at least a subset of the entirety of the dataset to the encoder of the respective client. The client (e.g. 202A-K) sends a message to the server 204. The message includes the updated marginal structure, latent variables corresponding to the drawn at least a subset of the entirety of the dataset assigned to the client, and the measured contributions.

[0037] The system initializes the marginal structure derived from different views that are free from redundant information, thereby enhancing communication efficiency. Additionally, the marginal structure and the joint structure facilitate efficient data transmission. The system enables each client and the server 204 to construct a common shared codebook with the marginal structure of the output of the encoder (i.e. latent variable), to send the output of the encoder / the latent variables and model parameters efficiently during transmission and inference time. The common shared codebook may be in a predefined format with an index. Preferably, the server 204 receives the message from the client (e.g. 202A-K) and derives an estimation of the joint structure of outputs of the encoders at the client (e.g. 202A-K), and uses the derived estimation to detect redundancies for the latent variables and assigns marginal structure targets to each of the client (e.g. 202A-K). Optionally, the server 204 uses a decoder to make predictions using the received outputs of the encoders included in the message from the clients (e.g. 202A-K). The server 204 may perform backpropagation using the predictions of the decoder to generate a backpropagation vector. The server 204 may update the decoder using the backpropagation vector.

[0038] Preferably, the server 204 sends a message to each client (e.g. 202A-K). The message may include the assigned marginal structure target specific to the client (e.g. 202A-K) and the respective backpropagation vector specific to the clients (e.g. 202A- K).

[0039] Preferably, each client (e.g. 202A-K) receives the message from the server 204 and updates an encoder at each client (e.g. 202A-K) using the received backpropagation vector and the received assigned marginal structure target.

[0040] FIG. 3 illustrates an interaction diagram 300 between a client (k) 302 (which is one of the clients 202A to 202K) and a server 304 for training a learning model in a distributed client or server data processing system in accordance with an implementation of the disclosure. At a step 306, the server 304 sends initial configuration data to the client (k) 302. The client (k) 302 includes an encoder which may be selected by the client (k) 302 based on the initial configuration data. The initial configuration data may include a number of heads (h), a number of classes (c), a number of components (M), mini-batch size (b), and a number of joint components. At a step 310A, the client (k) 302 initializes a learning model and estimates a marginal structure of the output of the encoder (e.g. latent variable) Q^’1at round one 308 using the received initial configuration data. Consecutively, at a step 310B, the server 304 initializes the learning model w0'1of the client (k) 302 and estimates a joint structure of the outputs of the encoder at round one 308 using the received initial configuration data.

[0041] For example, for every head h and optionally every class c, the marginal structure of the output of the encoder (e.g. latent variable) of each client is estimated as a Gaussian-Mixture distribution as follows:

[0042] The marginal structure of the output of the encoder are sent as pairs of:

[0043] (client

[0044] For every head h and optionally every class c, the joint structure of the output of the encoder is estimated as a T-Mixture of product of K Gaussian distributions as follows:T^P ] where, for every r, mT= mT 1,mT 2, mr K) E [M]®K.

[0045] At a step 312, the client (k) 302 draws a mini-batch / a subset of data from a dataset Skand computes the associated marginal structure of the output of the encoder Ukusing the drawn dataset Sk. The client (k) 302 measure contributions of each components in a mixture-distribution of the marginal structure on each output of the encoder (i.e. marginal latent variable) Uk l, denoted as yk.

[0046] The components in the mixture-distribution refer to distinct elements or clusters that collectively form the overall distribution. In this context, each component represents a specific pattern or feature learned by the encoder from the subset of data. For example, if the encoder is processing multi-modal data which is image data, one component might represent a specific type of edge or texture detected in the images. The marginal structure refers to an overall distribution of the encoder’s output, which reflects the probability distribution of the encoded features across the entire subset of data. Measuring contributions involves determining how much each component (in the mixture-distribution) contributes to the output of the encoder of the client (k) 302 for a drawn sample. This is performed by assessing the influence or weight of each component on the overall distribution of the encoder's outputs. For example, in a Gaussian mixture model, this may involve computing the likelihood that a given output belongs to each Gaussian component. The outputs of the encoders are the encoded representations of the data subset drawn by each client. These outputs are essentially feature vectors that capture important characteristics of the data. The marginal structure is generated from these outputs, and the mixture-distribution components describe how these outputs are distributed. Each component of the mixture-distribution affects the shape and nature of the marginal structure. By measuring the contributions, the client (k) 302 can understand which components are most significant in shaping the marginal structure, which helps in fine-tuning the encoder to focus on the most relevant features and patterns in the data.

[0047] Optionally, the client (k) 302 updates the marginal structure Qkby drawing at least a subset of the entirety of the dataset Sk.

[0048] The client (k) 302 may update marginal structure Qkbased on the whole dataset Sk.

[0049] At a step 318, the server 304 sends a message to the client (k) 302 including a new marginal structure and corresponding backpropagation vector to the client (k) 302. The message sent by the server 304 includes the marginal structure Qk. The server 304 sends the message in a pre-defmed format as follows: Message server i '1- ( server _id, task _id, time_id, Qk,backpropk). FIGS. 4A & 4B are flow diagrams that illustrate a method of training one or more learning models in a distributed client or server data processing system having one or more clients interacting with a server over a network, in accordance with an implementation of the disclosure. At a step 402, initial configuration data is received from the server at each of the one or more clients. Each client has a respective encoder. At a step 404, a learning model is initialized, and a marginal structure of the output of the encoder at the respective client, and a joint structure of the outputs of the encoders for each of the one or more clients, are estimated for the initialized model using the received initial configuration data. At a step 406, local computations are performed at each of the one or more clients including (i) drawing a subset of data from a dataset assigned to the client, (ii) computing the output of the encoder using the drawn subset of data, and estimating the marginal structure of the output of the encoder using the drawn subset of data, (iii) measuring contributions of each component in a mixture-distribution of the marginal structure on each of the outputs of the respective client’s encoders for the respective drawn subset of data at each respective client, and (iv) updating the marginal structure by drawing at least a subset of the entirety of the dataset assigned to the client and inputting the at least a subset of the entirety of the dataset to the encoder. The computing the output of the encoder is performed by the encoder of the respective client. At a step 408, a message from each of the one or more clients is sent to the server. The message includes the updated marginal structure, latent variables corresponding to the drawn at least a subset of the entirety of the dataset assigned to the client, and the measured contributions.

[0050] The method initializes the marginal structure derived from different views that are free from redundant information, thereby enhancing communication efficiency. Additionally, the marginal structure and the joint structure facilitate efficient data transmission. The method enables each client and the server to construct a common shared codebook with the marginal structure of the output of the encoder (i.e. latent variable), to send the output of the encoder / the latent variables and model parameters efficiently during transmission and inference time. The common shared codebook may be in a predefined format with an index.

[0051] Preferably, the server receives the message from each of the one or more clients and derives an estimation of joint structure of outputs of the encoders at each of the one or more clients, and uses the derived estimation to detect redundancies for the latent variables and assigns marginal structure targets to each of the one or more clients. Optionally, the server uses a decoder to make predictions using the received outputs of the encoders included in the message from each of the one or more clients. The server may perform backpropagation using the predictions of the decoder to generate backpropagation vectors. The server may update the decoder using the backpropagation vectors.

[0052] Preferably, the server sends a message to each of the one or more clients. The message may include the assigned marginal structure target specific to each of the one or more clients and the respective backpropagation vector specific to each of the one or more clients. Preferably, each of the one or more clients receives the message from the server and updates an encoder at each client using the received backpropagation vector and the received assigned marginal structure target.

[0053] FIG. 5 is an illustration of a computer system in which the various architectures and functionalities of the various previous implementations may be implemented. As shown, the computer system 500 includes at least one processor 504 that is connected to a bus 502, wherein the computer system 500 may be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), Hyper Transport, or any other bus or point-to-point communication protocol (s). The computer system 500 also includes a memory 506.

[0054] Control logic (software) and data are stored in the memory 506 which may take a form of random-access memory (RAM). In the disclosure, a single semiconductor platform may refer to a sole unitary semiconductor-based integrated circuit or chip. It should be noted that the term single semiconductor platform may also refer to multi-chip modules with increased connectivity which simulate on-chip modules with increased connectivity which simulate on-chip operation, and make substantial improvements over utilizing a conventional central processing unit (CPU) and bus implementation. Of course, the various modules may also be situated separately or in various combinations of semiconductor platforms per the desires of the user. The computer system 500 may also include a secondary storage 510. The secondary storage 510 includes, for example, a hard disk drive and a removable storage drive, representing a floppy disk drive, a magnetic tape drive, a compact disk drive, digital versatile disk (DVD) drive, recording device, universal serial bus (USB) flash memory. The removable storage drives at least one of reads from and writes to a removable storage unit in a well-known manner.

[0055] Computer programs, or computer control logic algorithms, may be stored in at least one of the memory 506 and the secondary storage 510. Such computer programs, when executed, enable the computer system 500 to perform various functions as described in the foregoing. The memory 506, the secondary storage 510, and any other storage are possible examples of computer-readable media.

[0056] In an implementation, the architectures and functionalities depicted in the various previous figures may be implemented in the context of the processor 504, a graphics processor coupled to a communication interface 512, an integrated circuit (not shown) that is capable of at least a portion of the capabilities of both the processor 504 and a graphics processor, a chipset (namely, a group of integrated circuits designed to work and sold as a unit for performing related functions, and so forth).

[0057] Furthermore, the architectures and functionalities depicted in the various previous-described figures may be implemented in a context of a general computer system, a circuit board system, a game console system dedicated for entertainment purposes, an application-specific system. For example, the computer system 500 may take the form of a desktop computer, a laptop computer, a server, a workstation, a game console, an embedded system.

[0058] Furthermore, the computer system 500 may take the form of various other devices including, but not limited to a personal digital assistant (PDA) device, a mobile phone device, a smart phone, a television, and so forth. Additionally, although not shown, the computer system 500 may be coupled to a network (for example, a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, or the like) for communication purposes through an I / O interface 508.

[0059] It should be understood that the arrangement of components illustrated in the figures described are exemplary and that other arrangement may be possible. It should also be understood that the various system components (and means) defined by the claims, described below, and illustrated in the various block diagrams represent components in some systems configured according to the subject matter disclosed herein. For example, one or more of these system components (and means) may be realized, in whole or in part, by at least some of the components illustrated in the arrangements illustrated in the described figures.

[0060] In addition, while at least one of these components are implemented at least partially as an electronic hardware component, and therefore constitutes a machine, the other components may be implemented in software that when included in an execution environment constitutes a machine, hardware, or a combination of software and hardware.

[0061] Although the disclosure and its advantages have been described in detail, it should be understood that various changes, substitutions, and alterations can be made herein without departing from the spirit and scope of the disclosure as defined by the appended claims.

Claims

CLAIMS1. A method of training a plurality of learning models in a distributed client or server data processing system (200) having a plurality of clients (202A-K) interacting with a server (204, 304) over a network (208), comprising steps of: receiving initial configuration data from the server (204, 304) at each of the plurality of clients (202A-K), where each client has a respective encoder; initializing a learning model, and estimating, for the initialized model, a marginal structure of the output of the encoder at the respective client and a joint structure of the outputs of the encoders for each of the plurality of clients (202A-K), using the received initial configuration data; performing local computations at each of the plurality of clients (202A-K) including: i) drawing a subset of data from a dataset assigned to the client; ii) computing the output of the encoder using the drawn subset of data, and estimating the marginal structure of the output of the encoder using the drawn subset of data, wherein the computing is performed by the encoder of the respective client; iii) measuring contributions of each component in a mixture-distribution of the marginal structure on each of the outputs of the respective client’s encoders for the respective drawn subset of data at each respective client; and iv) updating the marginal structure by drawing at least a subset of the entirety of the dataset assigned to the client and inputting the at least a subset of the entirety of the dataset to the encoder; and sending a message from each of the plurality of clients to the server (204, 304), the message comprising the updated marginal structure, latent variables corresponding to the drawn at least a subset of the entirety of the dataset assigned to the client, and the measured contributions.

2. The method of claim 1, wherein the server (204, 304) receives the message from each of the clients and derives an estimation of the joint structure of outputs of the encoders at each of the clients, uses the derived estimation to detect redundancies for the latent variables and assigns marginal structure targets to each of the plurality of clients (202A-K).

3. The method of claim 2, wherein the server (204, 304) uses a decoder to make predictions using the received outputs of the encoders included in the message from each of the clients (202A-K).

4. The method of claim 3, wherein the server (204, 304) performs backpropagation using the predictions of claim 3 to generate backpropagation vectors.

5. The method of claim 4, wherein the server (204, 304) updates the decoder using the backpropagation vectors.

6. The method of claim 5, wherein the server (204, 304) sends a message to each of the clients, the message comprising the assigned marginal structure target specific to each of the clients and the respective backpropagation vector specific to each of the clients.

7. The method of claim 6, wherein each client receives the message from claim 6 and updates an encoder at each client using the received backpropagation vector and the received assigned marginal structure target.

8. A system comprising means adapted for carrying out all the steps of the method according to any preceding method claim.

99. A computer program comprising instructions for carrying out all the steps of the method according to any preceding method claim, when said computer program is executed on a computer system. io

Images