Method, federated learning system, and computer program (vertical federated learning using compressed embedding)

Abstract

To solve the following problem: standard VFL approaches may incur high communications overhead, due to the need for communicating embeddings between clients and a server.SOLUTION: A method includes: recreating an initial overall embedding based on initial compressed embeddings and initial labels; sending a current version of a server model, to each of client computing devices; obtaining, from the client computing devices updated compressed embeddings, the updated compressed embeddings being compressed by clustering, and updated cluster labels corresponding to the updated compressed embeddings; based on local training by the plurality of client computing devices with the overall embedding and the current server model, recreating an updated overall embedding based on the updated embeddings and the corresponding updated cluster labels; and locally training the server model based on the updated overall embedding.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to the electrical, electronic, and computer fields, and more particularly to artificial intelligence and machine learning. [Background technology]

[0002] Federated learning is a machine learning technique for distributed model training from distributed data. Benefits include maintaining data privacy and reducing communication and storage requirements of a central cloud. Federated learning includes, for example, horizontal federated learning and vertical federated learning. In many practical applications of federated learning, features are vertically partitioned across multiple clients. For example, a bank and an insurance company have different information about the same person. The combination of such information can be used to predict the person's credit score, etc.

[0003] Vertical Federated Learning (VFL) can be used to perform federated learning on such vertically partitioned features. However, standard VFL approaches can incur high communication overhead due to the need to communicate embeddings between the client and server. Summary of the Invention [Problem to be solved by the invention]

[0004] In standard VFL approaches, the need to communicate embeddings between client and server can incur high communication overhead. [Means for solving the problem]

[0005] The present principles provide techniques for vertical federated learning using compressed embeddings. In one aspect, an exemplary method includes obtaining, for a plurality of client computing devices of a federated learning system, an initial compressed embedding, the initial compressed embedding compressed by clustering, the initial compressed embedding including the output of an initial local model for a current mini-batch, and initial cluster labels corresponding to the initial embedding. The method further includes regenerating an initial global embedding based on the initial compressed embedding and the initial cluster labels obtained for the plurality of client computing devices; transmitting, at a server of the federated learning system, a current version of a server model to each of the plurality of client computing devices; and obtaining, at the server of the federated learning system, from the plurality of client computing devices of the federated learning system, an updated compressed embedding, the updated compressed embedding compressed by clustering, and updated cluster labels corresponding to the updated compressed embedding, based on local training by the plurality of client computing devices of the federated learning system using the initial global embedding and the current server model. Still further steps include: at the server of the federated learning system, regenerating an updated global embedding based on the updated compressed embedding and the corresponding updated cluster labels; and at the server of the federated learning system, locally training the server model based on the updated global embedding to obtain an updated version of the server model.

[0006] In another aspect, an exemplary federated learning system includes a memory embodying computer-executable instructions and at least one processor coupled to the memory, the at least one processor operative to execute the computer-executable instructions to: obtain, for a plurality of client computing devices, an initial condensed embedding, the initial condensed embedding compressed by clustering, the initial condensed embedding including the output of an initial local model for a current mini-batch, and initial cluster labels corresponding to the initial condensed embedding. The at least one processor is further operative to execute the computer-executable instructions to: regenerate an initial global embedding based on the initial condensed embedding and the initial cluster labels obtained from the plurality of client computing devices, send a current version of a server model to each of the plurality of client computing devices, and obtain, from the plurality of client computing devices, an updated condensed embedding, the updated condensed embedding compressed by clustering, and updated cluster labels corresponding to the updated condensed embedding, based on local training by the plurality of client computing devices using the global embedding and the current server model. At least one processor is still further operative to execute the computer-executable instructions for regenerating an updated global embedding based on the updated condensed embedding and the corresponding updated cluster labels, and locally training the server model based on the updated global embedding to obtain an updated version of the server model.

[0007] In yet another aspect, an exemplary computer program product includes one or more computer-readable storage media and program instructions collectively stored on the one or more computer-readable storage media, including: program instructions for obtaining, for a plurality of client computing devices, an initial condensed embedding, the initial condensed embedding compressed by clustering, the initial condensed embedding including the output of an initial local model for a current mini-batch, and initial cluster labels corresponding to the initial condensed embedding; program instructions for regenerating an initial global embedding based on the initial condensed embedding and the initial cluster labels obtained from the plurality of client computing devices; program instructions for sending a current version of a server model to each of the plurality of client computing devices; and program instructions for obtaining, from the plurality of client computing devices, an updated condensed embedding, the updated condensed embedding compressed by clustering, and updated cluster labels corresponding to the updated condensed embedding, based on local training by the plurality of client computing devices using the global embedding and the current server model. The program instructions further include program instructions for regenerating an updated global embedding based on the updated compressed embedding and the corresponding updated cluster labels, and program instructions for locally training the server model based on the updated global embedding to obtain an updated version of the server model.

[0008] As used herein, "facilitating" an action includes performing the action, facilitating the action, assisting in the performance of the action, or having the action performed. Thus, by way of example and not limitation, instructions executing on one processor may facilitate an action performed by instructions executing on a remote processor by sending appropriate data or commands to cause or assist in the performance of the action. For the avoidance of doubt, even if an actor facilitates an action by other than performing the action, the action is nevertheless performed by some entity or combination of entities.

[0009] One or more embodiments of the present invention or elements thereof can be implemented in the form of a computer program product including a computer-readable storage medium with computer-usable program code for performing the illustrated method steps. Furthermore, one or more embodiments of the present invention or elements thereof can be implemented in the form of a system (or apparatus) including a memory and at least one processor coupled to the memory and operative to perform the illustrated method steps. Still further, in another aspect, one or more embodiments of the present invention or elements thereof can be implemented in the form of a means for performing one or more of the method steps described herein, which may include (i) a hardware module, (ii) a software module stored on a computer-readable storage medium (or multiple such media) and implemented on a hardware processor, or (iii) a combination of (i) and (ii), any of which implements specific techniques described herein.

[0010] The techniques of the present invention may provide substantial beneficial technical effects. Some embodiments may not have these potential advantages, and these potential advantages are not necessarily required for all embodiments. For example, one or more embodiments may provide one or more of the following:

[0011] To improve the technical process of vertically federated machine learning by increasing the efficiency of communication between clients and servers, thereby reducing data transfer overhead.

[0012] Improving the technical processes that operate information technology (IT) infrastructures, such as cloud-based IT infrastructures, by training predictive federated learning models to predict future behavior and deploying IT resources according to those predictions.

[0013] These and other features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments, which is to be read in connection with the accompanying drawings. [Brief explanation of the drawings]

[0014] [Figure 1] 1 illustrates a vertically federated learning environment in which one or more embodiments of the present invention may be used.

[0015] [Figure 2] A system overview according to one aspect of the present invention is presented.

[0016] [Figure 3] According to one aspect of the present invention, an algorithm for clustering vertical association learning is presented.

[0017] [Figure 4] 1 presents a plot of an exemplary training loss versus the amount of data transmitted upstream, in accordance with an aspect of the present invention.

[0018] [Figure 5] 1 presents a plot of an exemplary training accuracy versus amount of data transmitted upstream, in accordance with an aspect of the present invention.

[0019] [Figure 6] 10 presents a bar graph of data transmitted upstream versus clustering at the end of training for 100 epochs, according to one embodiment of the present invention.

[0020] [Figure 7] 1 illustrates a cloud computing environment according to one embodiment of the present invention.

[0021] [Figure 8] 1 illustrates abstraction model layers according to one embodiment of the present invention.

[0022] [Figure 9] 1 illustrates a computer system that may be useful in implementing one or more aspects of the present invention, and / or elements thereof, and that represents a cloud computing node according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0023] As noted above, standard VFL approaches can incur high communication overhead due to the need to communicate embeddings between clients and servers. One or more embodiments advantageously make VFL communication more efficient. FIG. 1 illustrates a vertically federated learning environment in which one or more embodiments of the present invention may be used. Multiple client / data silos 103-1, 103-2, 103-3, 103-4, 103-5, ... 103K (collectively 103) contain vertically separated data, and the silos / clients are used to collaboratively train a federated model. This model W S A portion of W resides on the trusted server 101, and each client has its own model W1, W2, W3, W4, W5, ... W K The characteristics of the data generated at the first client / data silo are typically different from the characteristics of the data generated at the second client / data silo. K Each of the client models W1, W2, W3, W4, W5, ... W6 is part of the global model, and each can be trained separately on the characteristics of its corresponding client. The global / federated model is composed of the trusted server 101 and the client models W1, W2, W3, W4, W5, ... WK The part that resides in W S Includes:

[0024] Referring now also to FIG. 2, FIG. 2 illustrates at least a portion of a federated model (e.g., two clients 103-1 and 103-2 (hence, two vertical partitions) and a single server 101). Different types of deep neural network models can be used. While a fully connected network is shown as an example, convolutional neural networks (CNNs), long short-term memory (LSTMs), or other types of networks can be used. Features are vertically partitioned across clients 103, and data samples have a shared key (e.g., a numeric value unique to a person, such as that person's Social Security number) and a shared label (e.g., a ground truth prediction result). Client 103-1 is provided with features x1, x2, x3, and x4, while client 103-2 is provided with features x5, x6, x7, and x8. In one or more embodiments, the sampling of mini-batches across clients 103 is synchronized via a shared key. As will be appreciated by those skilled in the art, mini-batch gradient descent is a variation of the gradient descent algorithm that divides a training dataset into small batches that are used to calculate model error and update model coefficients. In one or more embodiments, a mini-batch is chosen to feed forward throughout the network. The chosen mini-batch contains several samples, which are synchronized across clients. For example, all samples relate to the same person. As described, it is assumed that the sampling of the mini-batch can be synchronized using a shared key.

[0025] In the example of Figure 2, the models of clients 103-1 and 103-2 each have three layers, and the third layer of each client model also serves as an input layer to the server model. In each case, an embedding is generated after the second layer. The embedding is fed into the first layer of the server model, which is the third layer of the client model. Embeddings from different clients are of the same dimensionality and can therefore be added. During forward propagation, the embeddings are propagated forward through the server model.

[0026] 1 and 2 is split between the client 103 and the server 101. The client and server models can be trained simultaneously (using detergent information) or alternately. In one or more embodiments, the server model W S The embeddings obtained from the output of the last layer of the client model are sent to the server to train the model. The top neurons in layer 1 of the model of client 103-2 are, in particular, the weights assigned to the connections between the top input features and the top neurons in layer 1.

number

number

number

number

number

number

number

[0027] Considering embeddings, typically in vector form as well. In the example of Figure 2,

number

number

[0028] Client models W1, W2, W3, W4, W5, ... W K To train E, either the current server model can be transmitted and replicated at each client, or sample-specific information acquired at the first layer of the server model can be transmitted to the clients during backpropagation (the latter option is indicated by arrow 111-3). In the former option, for example, client 103-1 is provided with a snapshot of the embeddings from client 103-2 and a snapshot of the server model. Client 103-1 can then perform forward propagation locally throughout the network. Significant information exchange may be required, and one or more embodiments may make this procedure more communication efficient. The thick arrows 111-1, 111-2, 111-3, 111-4, and 111-5 at the bottom of Figure 2 indicate the backpropagation process, where the gradient of the server model E is applied at each stage to the corresponding weights

number

number

number

number

number

[0029] Because the server model can compute the loss, both forward and backward propagation can be performed. In one or more embodiments, clients also need to train their neural networks, but they need a snapshot of the server model to perform backpropagation / enable local iterations. See arrow 111-3. Similarly, clients send snapshots of their embeddings to the server, as described elsewhere herein.

[0030] In existing vertical federated learning techniques, the embedding of each data sample needs to be transmitted between the client 103 and the server 101, which can incur high transmission overhead. To overcome this problem, one or more embodiments use a clustering technique to combine the embeddings of data samples (e.g., those in a mini-batch). The cluster size can be determined based on the similarity of the embeddings and their impact on further training (i.e., the cluster size can be dynamically updated). For example, if features at a particular client have no impact on the final prediction (e.g., all embeddings from this client are the same or very similar), it is sufficient to use a single vector to represent all embeddings, i.e., the cluster size is 1. On the other hand, if features at a client have an impact on the final prediction (i.e., the prediction result is highly dependent on these features), the embeddings of different samples are likely to be very different, and using a large cluster size is appropriate.

[0031] The embedding may also be compressed by transmitting only the differences compared to what was previously transmitted (optionally after quantization). The clustering and compression methods may also be designed to maintain user privacy according to some predefined metric. K-means is a non-limiting example. Clustering is a form of vector quantization. In one or more embodiments, the compressed model is transmitted from the server to the client, i.e., in one or more embodiments, downstream communications are also compressed.

[0032] 3 presents pseudocode for an exemplary algorithm 301 for vertical federated learning with clustering of server models and upstream compression. In lines 1 and 2, the client and server models are initialized (t=0). The kth client at time t=0 is

number

number

number

number

[0033] Assume the batch size is 50. There is one embedding for each of the 50 data points. One or more embodiments apply clustering (k-means clustering is a non-limiting example, as seen in line 7) to the embeddings to compress the upstream data. The k-means (or other) clustering can be modulated to obtain a desired number of centroids. The clustered set of embeddings and cluster labels are then sent to a server. In the 50 data point example, assume that five clusters are found as a result of the k-means (or other) clustering. The centroids and cluster labels of the five clusters are sent to the server (line 8), rather than the entire 50 data points, thereby advantageously reducing the required communication bandwidth. The cluster labels are the output of the k-means (or other) clustering, e.g., a vector indicating that sample 1 belongs to cluster 1, sample 2 belongs to cluster 1, sample 3 belongs to cluster 2, etc. There are 50 entries for the 50 data points.

[0034] On line 10, the server uses the cluster information to

number

number

number

[0035] At this point, the client has all the information it needs to perform local training over some number of iterations Q. In lines 12-22, in parallel, each client performs a global embedding on line 13.

number

number

number

[0036] Increasing Q too much can have a negative impact on convergence. In lines 15-18, the client performs forward and backward propagation. For forward propagation, the client finds an embedding and then feeds the embedding through a copy of the server model obtained in line 11. In line 16, the loss is found and the gradient is calculated in line 17. By applying the gradient, the weights of the local model are updated in line 18. This is done over Q iterations. At the end of Q iterations, the client allows the actual server model to train. Since the weights have been updated, the embedding of each client model is also updated. Each client model forms its updated embedding, and k-means clustering is performed in line 20. Clustered Embeddings

number

[0037] Therefore, in line 20, k-means clustering is performed to compress the upstream data, and in line 21, the clustered embeddings and labels are sent to the server. In line 23, the server uses the cluster information to regenerate updated global model embeddings from those of the individual clients. In the FOR NEXT loop in lines 24-29, the loss and partial derivatives are calculated and local gradients are applied. In line 25, the loss calculation is performed (for y contained in the mini-batch, an appropriate loss function, which can be selected by one skilled in the art, is applied to the server model at time t, iteration q, the global embedding (see the explanation above)). In line 26, the partial derivatives are determined (based on the server model at time t, iteration q) and the local gradients are applied to update the local model for q+1 (line 18). Again, this can be done using standard neural network training techniques such as coordinate descent.

[0038] At the end of T rounds, sufficient training has been performed so that the federated model can be used to achieve accurate classification / inference. The value of T can be chosen heuristically, or the calculated loss can be tracked and the calculation stopped when the loss falls below an acceptable threshold (so that acceptable performance / accuracy is achieved) or does not change significantly between successive rounds / time steps. So, although the pseudocode shows T time steps, T is not fixed in advance but rather can be dynamically determined during the calculation, as will be described.

[0039] experiment

[0040] We performed experiments using two workers 103 and one server 101 for binary classification of in-hospital mortality prediction using a publicly available, anonymous hospital database. The workers 103 used a long short-term memory (LSTM) model, while the server model used two fully connected (FC) layers and a SoftMax output. We used a batch size of 128, and the number of clusters in the batch varied over values ​​of 16 (i.e., sending 16 embeddings), 32, and 64 (i.e., halving the amount of data sent to the server). The "no clustering" case was when there was no clustering, and all 128 data points were transmitted. In the experiments, we plotted the loss for all training data points at the end of each epoch, i.e., one iteration through the entire dataset with a given batch size, because computing this for every batch would be prohibitively expensive.

[0041] In the experiment, the server 101 and the worker 103 were each trained locally for five epochs (Q=5). In Figure 4, the X-axis plots the number of batches seen (the number of kilobytes transmitted upstream), and the Y-axis shows the average training loss of the batches. The embeddings transmitted upstream are compressed (except for the case without clustering, which reduces training loss and therefore converges slowest). Convergence is faster with a cluster size of 64, even faster with a cluster size of 32, and fastest with a cluster size of 16. Stated differently, for a fixed amount of data sent upstream, examples with smaller cluster sizes and therefore more data compression can perform more training rounds and therefore converge faster. The exemplary results shown are averages over three runs with different seeds. In one or more embodiments, the number of transmitted bytes is calculated offline by looking at the number of times the algorithm was run and the number of floats / integers transmitted. For each batch, there are two upstream transmissions. For a single transmission with a batch size of 128, the total number of transmitted bytes is determined as follows:

[0042] Total bytes transmitted = (embedding size x number of clusters x 64 + batch size x 6) / (1024 x 8) bytes.

[0043] In the above, the factor 6 is the base 2 logarithm of 128.

[0044] Still referring to Figure 4, all cases were trained until approximately 95,000 KB of data had been transmitted. By the end of 100 epochs, the configuration with a cluster size of 16 transmitted approximately 95,000 KB upstream. Looking at the convergence of different clustering settings for the same amount of bytes transferred, we can see that for the same number of bytes transferred, the convergence rate is better as the number of clusters decreases. This is because, for the same number of bytes transferred, the number of training rounds (observed by the dots in Figure 4) increases with increasing compression.

[0045] Referring now to Figure 5, when looking at the training accuracy (essentially the opposite of the training loss shown in Figure 4) of different clustering settings for the same number of bytes transferred, we can see that for the same number of bytes transferred, as the number of clusters decreases, the accuracy increases. This is because for the same number of bytes transferred, the number of training rounds (observed by the dots in Figure 5) increases with increasing compression, so higher accuracy is achieved more quickly.

[0046] Figure 6 shows the number of bytes transmitted at the end of training for 100 epochs. We can see that the number of transmitted kilobytes increases as the number of transmitted embeddings increases. In our example, we trained all cases up to 100 epochs (T=100) using 115 batches per epoch. Without clustering, we need to transmit approximately 700+ megabytes upstream. With 16 clusters, we need to transmit only approximately 90 megabytes, with 32 clusters, approximately 180 megabytes, and with 64 clusters, approximately 380 megabytes (roughly half the amount without clustering).

[0047] It will thus be appreciated that one or more embodiments provide a vertical federated learning system that includes multiple clients and servers, where the clients 103 and servers 101 exchange compressed embeddings. In one or more embodiments, a clustering technique is used to combine embeddings of data samples (e.g., in mini-batches). Cluster sizes may be determined, for example, based on the similarity of the embeddings and their impact on further training.

[0048] In some examples, the embedding is compressed by transmitting only the differences compared to what was previously transmitted (optionally after quantization). The compression method can be designed to maintain user privacy according to a predefined metric. The compression model can be transmitted both from the client to the server and from the server to the client.

[0049] Therefore, one or more embodiments use compressed embeddings exchanged between the client and the server to reduce the communication overhead of vertical federated learning. One or more embodiments speed up communication between multiple parties involved in a vertical federated learning scheme. Additionally, in one or more embodiments, the server also trains a portion of the model.

[0050] As such, one or more embodiments provide techniques for using a computing device to improve the efficiency of a federated learning scheme, including receiving, by a central computing device in the federated learning scheme, compressed embeddings trained by worker computers in the federated learning scheme that can be combined with other compressed embeddings trained by other worker computers in the federated learning scheme, in which a federated learning model is jointly trained based on all received compressed embeddings.

[0051] In some examples, the compression embedding is based on data samples accessed by each worker computer.

[0052] There can be a variety of practical applications. For example, a manufacturer and an end-user industrial company may have different information about the same industrial equipment. The combination of such information can be used to predict the necessary maintenance schedule for that industrial equipment. A cloud (or other IT) platform managed by one company may store data from another company, or the IT infrastructure may be managed by one company, but the services may be provided by another company. Both companies store logs: one about the infrastructure (how much bandwidth / storage is being used), and the other about the service itself, such as how the service is performing for different users and the states it has transitioned to in the past. Therefore, there are different sets of logs for the same time period, and it is desired to train a predictive model to predict the future behavior of the IT services and infrastructure.

[0053] Based on the above, it will be understood that an exemplary method according to one aspect of the present invention includes obtaining initial condensed embeddings and initial cluster labels corresponding to the initial condensed embeddings for multiple client computing devices of a federated learning system. The initial condensed embeddings are condensed by clustering and include the output of an initial local model for a current mini-batch. See line 8 of the pseudocode in FIG. 3 . The cluster labels should not be combined with labels in the training data. A further step includes regenerating initial global embeddings based on the initial condensed embeddings and initial cluster labels obtained for the multiple client computing devices. See line 10 of the pseudocode in FIG. 3 . This step can be performed at the server or at each client. A still further step includes, at the server of the federated learning system, sending a current version of the server model to each of the multiple client computing devices. See line 11 of the pseudocode in FIG. 3 .

[0054] A still further step includes, at the server of the federated learning system, obtaining, from the multiple client computing devices of the federated learning system, an updated compressed embedding, where the updated compressed embedding has been compressed by clustering, and updated cluster labels corresponding to the updated compressed embedding, based on local training by the multiple client computing devices of the federated learning system using the initial global embedding and the current server model. See line 21 of the pseudocode in FIG. 3 . A further step includes, at the server of the federated learning system, regenerating an updated global embedding based on the updated compressed embedding and the corresponding updated cluster labels. See line 23 of the pseudocode in FIG. 3 . An additional step includes, at the server of the federated learning system, locally training a server model based on the updated global embedding to obtain an updated version of the server model. See lines 25-27 of the pseudocode in FIG. 3 . Note that in one or more embodiments, the updated labels need only be used to obtain the updated global embedding, which is then used to train and obtain the updated server model.

[0055] In one or more embodiments, the steps of obtaining the initial condensed embedding and the initial cluster labels and regenerating the initial global embedding are performed at a server, and sending the current version of the server model to each of the multiple client computing devices further includes sending the initial global embedding.

[0056] In one or more embodiments, local training by the multiple client computing devices generates an updated local model, and a further step 303 includes performing federated machine learning inference using the updated version of the server model and the updated local model. In a non-limiting example, one practical application of the inference includes deploying information technology resources according to the inference, as at 305 (see, e.g., FIGS. 7 and 8).

[0057] Note that in general, the client and server may be controlled / operated by the same or different entities.

[0058] One or more embodiments further include, for the multiple client computing devices of the federated learning system, determining the output of the initial local models as initial decompressed embeddings for the current mini-batch. Each of the initial local models is based on a vertical slice of the data. See line 6 of the pseudocode in FIG. 3 . Such embodiments also include, for the multiple client computing devices of the federated learning system, compressing the initial decompressed embeddings by clustering to obtain initial compressed embeddings (see line 7 of the pseudocode in FIG. 3 ), and sending the initial compressed embeddings and initial cluster labels to a server of the federated learning system (see line 8 of the pseudocode in FIG. 3 ). Such an embodiment further includes, for multiple client computing devices of the federated learning system, locally training initial local models based on the initial ensemble embedding and the current server model to obtain updated local models, each of the updated local models being based on a different vertical slice of the data (see lines 15-18 of the pseudocode in FIG. 3 ), and, for the multiple client computing devices of the federated learning system, sending updated clustered embeddings and corresponding updated cluster labels based on the updated local models to a server of the federated learning system (see lines 20-21 of the pseudocode in FIG. 3 ).

[0059] Note the significance of the "different" vertical slice of data: in each round, a client uses only data from that client, so in that sense it is the same slice of data. However, due to the sampling of the updated mini-batch, it is not exactly the same data, but rather a different set selected from the same slice associated with that client.

[0060] As noted above, in one or more embodiments, features are vertically partitioned across clients 103, and data samples have a shared key (e.g., a number unique to a person, such as that person's Social Security number) and a shared label (e.g., ground truth prediction result). In one or more embodiments, sampling of mini-batches across clients 103 is synchronized via the shared key.

[0061] One or more embodiments further include repeating the steps of obtaining an initial condensed embedding and initial cluster labels, regenerating the initial global embedding, sending the current version of the server model, obtaining an updated condensed embedding and updated cluster labels, regenerating the updated global embedding, and locally training the server model to obtain an updated version of the server model until convergence. See the FOR END FOR loop in lines 3-29 of the pseudocode in FIG. 3. The value of T may be chosen heuristically, or the calculated loss may be tracked, and the calculation stopped when the loss is below an acceptable threshold (so that acceptable performance / accuracy is achieved) or does not change significantly between successive rounds / time steps. Thus, although the pseudocode shows T time steps, T is not fixed in advance but rather may be dynamically determined during the calculation, as described.

[0062] In one or more embodiments, when the steps are repeated (e.g., t=2), random sampling is performed for the second iteration, again randomly sampling a subset of the entire dataset. In one or more embodiments, a different random sample of data points (mini-batch) is used for each iteration, and the mini-batch is different in each iteration. Thus, in one or more embodiments, mini-batch B t , and the model of the kth client at time t

number

[0063] In one or more embodiments, transmitting the updated clustered embeddings (line 21 of the pseudocode in FIG. 3) includes transmitting differentially updated clustered embeddings. As noted elsewhere above, for example, the embeddings may be compressed by transmitting only the differences compared to the previously transmitted ones (optionally after quantization). The clustering and compression methods may also be designed to maintain user privacy according to some predefined metric.

[0064] In one or more embodiments, when repeating the transmission of the current version of the server model, the current version of the server model is transmitted in compressed form, i.e., in one or more embodiments, a compressed model is transmitted from the server to the client, i.e., in one or more embodiments, downstream communications are also compressed.

[0065] In one or more embodiments, the clustering includes k-means clustering, where k-means is a non-limiting example, and the clustering is a form of vector quantization.

[0066] One or more embodiments further include reducing the number of clusters in the clustering when iterating through the steps of obtaining initial condensed embeddings and initial cluster labels, regenerating the initial global embeddings, sending the current version of the server model, obtaining updated condensed embeddings and updated cluster labels, regenerating the updated global embeddings, and locally training the server model to obtain an updated version of the server model. The cluster size may be determined based on the similarity of the embeddings and their impact on further training (i.e., the cluster size may be dynamically updated). For example, if features at a particular client have no impact on the final prediction (e.g., all embeddings from this client are the same or very similar), it is sufficient to use a single vector to represent all embeddings, i.e., the cluster size is 1. On the other hand, if features at the client have an impact on the final prediction (i.e., the prediction result is highly dependent on these features), the embeddings of different samples are likely to be very different, and using a large cluster size is appropriate.

[0067] Those skilled in the art will understand that "K" is also used for the number of clients and the number of centroids in k-means clustering, and will understand the distinction from the context (of course, the number of clients and the number of centroids are not necessarily the same). Dynamic aspects may include changing how many clusters are used in k-means clustering (i.e., how much compression). This may depend, for example, on bandwidth limitations. Considering that increasing the number of data points in the clustering reduces the number of clusters and compresses the data more, reducing the number of data points in a cluster means that more clusters are needed, resulting in less compression but higher accuracy. For example, there may be fewer clusters as the end of training approaches. Of course, even if that does not occur, the clustering is typically changed with each iteration, so an element of dynamism is already present.

[0068] To choose an initial value for the number of clusters in k-means clustering, heuristics are used, for example, based on the characteristics of the data being analyzed and the size of the mini-batch. For example, given an error bound, each client can determine k. Each client can use a different k if desired. Non-limiting examples are shown in figures, such as Figures 4-6. In the examples of Figures 4 and 5, the batch size was 128 and the number of clusters ranged from 16 to 64. Thus, as non-limiting examples, the upper limit on the number of clusters could be 50% of the batch size (64 / 128), while the lower limit on the number of clusters could be 12.5% ​​of the batch size (16 / 128) or 6.25% of the batch size (8 / 128).

[0069] In one or more embodiments, local training on the client and local training on the server are performed for 5-10 iterations (i.e., Q = 5-10). Note, however, that in some cases, Q can be much larger than 10, depending on the diversity of the data and the learning rate. For example, Q can be 100. Furthermore, the algorithm also works with a single local update, so Q = 1 is an appropriate special case in some cases. Thus, in some cases, Q = 1-100. Q can vary between clients, between the client and the server, and between rounds.

[0070] In another aspect, an exemplary federated learning system includes a memory embodying computer-executable instructions and at least one processor coupled to the memory, the at least one processor operative to execute the computer-executable instructions for: obtaining, for a plurality of client computing devices 103, an initial condensed embedding, where the initial condensed embedding is condensed by clustering, and where the initial condensed embedding includes the output of an initial local model for a current mini-batch, and initial cluster labels corresponding to the initial condensed embedding (see, e.g., FIG. 9 and the accompanying text, and the pseudocode of FIG. 3 above). The at least one processor is further operative to execute the computer-executable instructions for: regenerating an initial global embedding based on the initial condensed embedding and the initial cluster labels obtained from the plurality of client computing devices; sending a current version of the server model to each of the plurality of client computing devices; and obtaining, from the plurality of client computing devices, an updated condensed embedding, where the updated condensed embedding is condensed by clustering, and updated cluster labels corresponding to the updated condensed embedding, based on local training by the plurality of client computing devices using the global embedding and the current server model. The at least one processor is still further operative to execute computer-executable instructions for regenerating an updated global embedding based on the updated compressed embedding and the corresponding updated cluster labels, and locally training a server model based on the updated global embedding to obtain an updated version of the server model.

[0071] In one or more embodiments, the memory and at least one processor are distributed across multiple client computing devices and a server 101 coupled to the multiple client computing devices, where local training by the multiple client computing devices generates updated local models, and the multiple client computing devices are configured to: determine an output of an initial local model as an initial decompressed embedding for the mini-batch; compress the initial decompressed embedding by clustering to obtain an initial compressed embedding; send the initial compressed embedding and the initial cluster labels to a server; locally train the initial local model based on the initial overall embedding and the current server model to obtain an updated local model; and send updated clustered embeddings and corresponding updated cluster labels to a server of the federated learning system based on the updated local model. Further, the server is configured to send a current version of the server model to each of the multiple client computing devices, obtain the updated compressed embeddings and updated cluster labels from the multiple client computing devices, and locally train the server model.

[0072] It will be understood that the clients and servers are so configured by computer-executable instructions that implement appropriate portions of the algorithms disclosed, for example, in pseudocode.

[0073] In one or more embodiments, the server is configured to obtain the initial condensed embedding and initial cluster labels and to regenerate the initial global embedding, and the server is configured to transmit the initial global embedding along with the current version of the server model.

[0074] In one or more embodiments, the server and the plurality of client computing devices are cooperatively configured to perform federated machine learning inference using an updated version of the server model and an updated local model. One non-limiting, exemplary practical application of such inference is deploying information technology resources according to the inference.

[0075] In some examples, the server and the multiple client computing devices are further configured to cooperate to iteratively obtain an initial condensed embedding and initial cluster labels, regenerate an initial global embedding, transmit a current version of the server model, obtain an updated condensed embedding and updated cluster labels, regenerate an updated global embedding, and locally train the server model to obtain an updated version of the server model until convergence.

[0076] As noted above, in some cases, the clustering includes k-means clustering. The clustering includes k-means clustering, and the server and the multiple client computing devices may be further configured to cooperate to reduce the number of clusters in the clustering when iterating through obtaining an initial condensed embedding and initial cluster labels, regenerating an initial global embedding, transmitting a current version of the server model, obtaining updated condensed embeddings and updated cluster labels, regenerating an updated global embedding, and locally training the server model to obtain an updated version of the server model.

[0077] In yet another aspect, a computer program product (see, e.g., FIG. 9 and accompanying text, as well as the pseudocode of FIG. 3 above) includes one or more computer-readable storage media and program instructions collectively stored on the one or more computer-readable storage media, the program instructions including: obtaining, for a plurality of client computing devices, an initial compressed embedding, where the initial compressed embedding has been compressed by clustering, and the initial compressed embedding includes the output of an initial local model for a current mini-batch, and initial cluster labels corresponding to the initial compressed embedding; program instructions for regenerating an initial global embedding based on the initial compressed embedding and the initial cluster labels obtained from the plurality of client computing devices; program instructions for sending a current version of the server model to each of the plurality of client computing devices; and obtaining, from the plurality of client computing devices, an updated compressed embedding, where the updated compressed embedding has been compressed by clustering, and updated cluster labels corresponding to the updated compressed embedding, based on local training by the plurality of client computing devices using the global embedding and the current server model. The program instructions further include program instructions for regenerating an updated global embedding based on the updated compressed embedding and the corresponding updated cluster labels, and program instructions for locally training a server model based on the updated global embedding to obtain an updated version of the server model.

[0078] In some embodiments, the program instructions are executed cooperatively by a plurality of client computing devices 103 and a server 101 coupled to the plurality of client computing devices, and local training by the plurality of client computing devices generates updated local models. The program instructions further include program instructions for configuring the plurality of client computing devices to determine an output of the initial local model as an initial decompressed embedding for the mini-batch, program instructions for configuring the plurality of client computing devices to compress the initial decompressed embedding by clustering to obtain an initial compressed embedding, program instructions for configuring the plurality of client computing devices to send the initial compressed embedding and the initial cluster labels to a server, program instructions for configuring the plurality of client computing devices to locally train the initial local model based on the initial overall embedding and a current server model to obtain an updated local model, and program instructions for configuring the plurality of client computing devices to send updated clustered embeddings and corresponding updated cluster labels based on the updated local model to a server of the federated learning system. The program instructions still further include program instructions for sending a current version of the server model to each of the plurality of client computing devices, obtaining the updated compressed embeddings and updated cluster labels from the plurality of client computing devices, and configuring the server to locally train the server model.

[0079] Although this disclosure includes detailed descriptions of cloud computing, it will be understood that implementation of the teachings cited herein is not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in connection with any other type of computing environment now known or later developed.

[0080] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processes, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. The cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0081] The characteristics are as follows:

[0082] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically as needed, without requiring human interaction with the service provider.

[0083] Broad Network Access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (eg, cell phones, laptops, and PDAs).

[0084] Resource Pool: A provider's computing resources are pooled and serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. Although consumers generally have no control or knowledge of the exact location of the resources provided, there is an implication of location independence in that the location may be specifiable at a higher level of abstraction (e.g., country, state, or data center).

[0085] Rapid Elasticity: Capacity can be rapidly and elastically provisioned, in some cases automatically, for quick scale out, and rapidly released for quick scale in. To the consumer, the capacity available for provisioning often appears unlimited, and can be purchased in any quantity at any time.

[0086] Measured Services: Cloud systems automatically control and optimize resource usage by leveraging measurement capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of utilized services.

[0087] The service model is as follows:

[0088] Software as a Service (SaaS): The consumer is offered the ability to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through a thin-client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0089] Platform as a Service (PaaS): The ability offered to consumers is to deploy applications they create or acquire, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the application hosting environment configuration.

[0090] Infrastructure as a Service (IaaS): The ability offered to consumers is to provision processing, storage, network, and other underlying computing resources on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does have control over the operating systems, storage, deployed applications, and possibly limited control of selected networking components (e.g., host firewalls).

[0091] The deployment model is as follows:

[0092] Private Cloud: Cloud infrastructure is operated solely for an organization. It can be managed by the organization or a third party and can reside on-premise or off-premise.

[0093] Community Cloud: Cloud infrastructure is shared by multiple organizations to support a specific community with shared interests (e.g., mission, security requirements, policy, and compliance considerations). It may be managed by the organization or a third party and may reside on-premises or off-premises.

[0094] Public Cloud: Cloud infrastructure is made available to the general public or large industry groups and is owned by organizations that sell cloud services.

[0095] Hybrid Cloud: A combination of two or more clouds (private, community, or public) that remain distinct entities, but are joined together by standardized or proprietary technologies (e.g., cloud bursting for load balancing between clouds) that enable data and application portability.

[0096] A cloud computing environment is a service oriented environment that is stateless, low-coupling, modular, and emphasizes semantic interoperability. At its core, cloud computing lies an infrastructure that includes a network of interconnected nodes.

[0097] Referring now to FIG. 7, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or cellular phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or any combination thereof, may communicate. The nodes 10 may communicate with each other. The nodes may be physically or virtually grouped (not shown) in one or more networks, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, or any combination thereof, as described above. This enables the cloud computing environment 50 to provide infrastructure as a service, platform as a service, or software as a service, or any combination thereof, without the need for cloud consumers to maintain resources on local computing devices. It should be understood that the types of computing devices 54A-N shown in FIG. 7 are for illustrative purposes only, and that the computing nodes 10 and the cloud computing environment 50 can communicate with any type of computerized device through any type of network or network-addressable connection (e.g., using a web browser), or both.

[0098] Referring now to Figure 8, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 7) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 8 are intended to be illustrative only, and that embodiments of the invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0099] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframe 61, RISC (Minimum Instruction Set Computer) architecture-based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0100] The virtualization layer 70 provides an abstraction layer from which the following example virtual entities can be provided: virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75.

[0101] In one example, the management layer 80 may provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources utilized to perform tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and charging or billing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification of cloud consumers and protection for tasks, data, and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management to ensure required service levels are met. Service level agreement (SLA) planning and execution 85 provides advance arrangements and procurement for cloud computing resources that anticipate future requirements according to SLAs.

[0102] Workload tier 90 provides examples of functionality for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this tier include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instructional delivery 93, data analytics processing 94, transaction processing 95, and at least a portion of (e.g., its server 101) a system for vertically federated learning using compressed embedding 96.

[0103] One or more embodiments of the present invention, or elements thereof, may be implemented in the form of an apparatus including a memory and at least one processor coupled to the memory and operable to perform exemplary method steps. FIG. 9 illustrates a computer system that may be useful in implementing one or more aspects of the present invention, or elements thereof, or both, and that represents a cloud computing node according to one embodiment of the present invention. Referring now to FIG. 9, cloud computing node 10 is merely one example of a suitable cloud computing node and is not intended to suggest any limitation as to the scope of use or functionality of the embodiments of the present invention described herein. Regardless, cloud computing node 10 may implement and / or perform any of the functionality described herein above.

[0104] Cloud computing node 10 includes computer system / server 12, which is operable in many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or combinations thereof, that may be suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0105] The computer system / server 12 may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system / server 12 may be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0106] 9, computer system / server 12 in cloud computing node 10 is shown in the form of a general-purpose computing device. Components of computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 coupling various system components including system memory 28 to processor 16.

[0107] Bus 18 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0108] Computer system / server 12 typically includes a variety of computer system-readable media, which may be any available media that can be accessed by computer system / server 12, including both volatile and nonvolatile media, removable and non-removable media.

[0109] The system memory 28 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 30 or cache memory 32, or a combination thereof. The computer system / server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, typically referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from or writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical medium. In such an example, each may be connected to the bus 18 by one or more data media interfaces. As further shown and described below, the memory 28 may include at least one program product including a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present invention.

[0110] By way of example and not limitation, a program / utility 40 including a set (at least one) of program modules 42 may be stored in memory 28, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may comprise an implementation of a network environment. The program modules 42 generally perform the functions or methods, or combinations thereof, of embodiments of the present invention described herein.

[0111] The computer system / server 12 may communicate with one or more external devices 14, such as a keyboard, pointing device, display 24, etc., one or more devices that allow a user to interact with the computer system / server 12, or any device (e.g., a network card, modem, etc.) that allows the computer system / server 12 to communicate with one or more other computing devices, or a combination thereof. Such communication may occur via an input / output (I / O) interface 22. Furthermore, the computer system / server 12 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, via a network adapter 20. As shown, the network adapter 20 communicates with other components of the computer system / server 12 via a bus 18. While not shown, it should be understood that other hardware or software components, or combinations thereof, may be used with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data archive storage systems, etc.

[0112] Accordingly, one or more embodiments may utilize software running on a general-purpose computer or workstation. Referring to FIG. 9 , such an implementation may employ, for example, a processor 16, memory 28, and an input / output interface 22 to an external device 14, such as a display 24 and a keyboard or pointing device. As used herein, the term “processor” is intended to include any processing device, such as a device including a CPU (Central Processing Unit) or other form of processing circuitry, or both. Furthermore, the term “processor” may refer to two or more individual processors. The term “memory” is intended to include memory associated with a processor or CPU, such as, for example, RAM (Random Access Memory) 30, ROM (Read-Only Memory), fixed memory devices (e.g., hard drives 34), removable memory devices (e.g., diskettes), flash memory, etc. Additionally, as used herein, the phrase “input / output interface” is intended to contemplate an interface to, for example, one or more mechanisms for inputting data into a processing unit (e.g., a mouse) and one or more mechanisms for providing results associated with the processing unit (e.g., a printer). The processor 16, memory 28, and input / output interface 22 may be interconnected, for example, via a bus 18 as part of the data processing unit 12. Suitable interconnections, for example, via the bus 18, may also be provided to a network interface 20, such as a network card, which may be provided to interface with a computer network, and to a media interface, such as a diskette or CD-ROM drive, which may be provided to interface with suitable media.

[0113] Thus, computer software containing instructions or code for carrying out the methods of the present invention, as described herein, can be stored in one or more of the associated memory devices (e.g., ROM, fixed or removable memory) and, when ready for use, can be loaded partially or entirely (e.g., into RAM) and implemented by a CPU. Such software can include, but is not limited to, firmware, resident software, microcode, etc.

[0114] A data processing system suitable for storing and / or executing program code includes at least one processor 16 coupled directly or indirectly to memory elements 28 via a system bus 18. The memory elements may include local memory used during the actual implementation of the program code, bulk storage, and cache memory 32 that provides temporary storage of at least some program code to reduce the number of times the code must be retrieved from bulk storage during implementation.

[0115] Input / output or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system either directly or through intermediary I / O controllers.

[0116] A network adapter 20 may also be coupled to the system to enable the data processing system to couple to other data processing systems or remote printers or storage devices through intermediary private or public networks. Currently available types of network adapters include modems, cable modems, and Ethernet cards.

[0117] As used herein, including in the claims, a "server" includes a physical data processing system (e.g., system 12 as shown in FIG. 9) running a server program. It should be understood that such a physical server may or may not include a display and keyboard.

[0118] One or more embodiments may be implemented at least in part in the context of a cloud or virtual machine environment, by way of example and not limitation. See again Figures 7-8 and accompanying text.

[0119] It should be noted that any of the methods described herein may include the additional step of providing a system including separate software modules embodied in a computer-readable storage medium, where the modules may include any or all of the appropriate elements, e.g., shown in block diagrams and / or described herein, including, by way of example and not limitation, any one, some, or all of the described modules / blocks and / or sub-modules / sub-blocks. The method steps may then be performed using separate software modules and / or sub-modules of such a system running on one or more hardware processors, such as 16. Furthermore, a computer program product may include a computer-readable storage medium having code adapted to be implemented to perform one or more method steps described herein, including providing a system having separate software modules.

[0120] One example of a user interface that may be used in some cases is Hypertext Markup Language (HTML) code served by a server or the like to a browser on a user's computing device, which is parsed by the browser on the user's computing device to create a graphical user interface (GUI).

[0121] Exemplary System and Product Details

[0122] The present invention may be a system, a method, or a computer program product, or a combination thereof, at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0123] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved structures that have instructions recorded on them, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as being ephemeral signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0124] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a respective computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network can comprise copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0125] Computer-readable program instructions for carrying out operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or object-oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can execute computer readable program instructions to individualize the electronic circuitry by utilizing state information in the computer readable program instructions to implement aspects of the present invention.

[0126] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0127] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to make a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium that can direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions for implementing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0128] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to create a computer-implemented process that performs a series of operational steps on the computer, other programmable apparatus, or other device, such that the instructions, which execute on the computer, other programmable apparatus, or other device, implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0129] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0130] While the description of various embodiments of the present invention has been presented for illustrative purposes, it is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, practical applications or technical improvements to technology found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. to a plurality of client computing devices of a federated learning system; an initial condensed embedding, the initial condensed embedding being condensed by clustering, the initial condensed embedding including the output of an initial local model for a mini-batch; initial cluster labels corresponding to the initial condensed embedding; and regenerating an initial global embedding based on the initial condensed embedding and the initial cluster labels obtained for the plurality of client computing devices; transmitting, at a server of the federated learning system, a current version of a server model to each of the plurality of client computing devices; generating an updated local model by local training by the plurality of client computing devices of the federated learning system using the initial ensemble embedding and the current server model; at the server of the federated learning system, from the plurality of client computing devices of the federated learning system based on the local training; an updated compressed embedding, the updated compressed embedding being compressed by clustering; and updated cluster labels corresponding to the updated condensed embedding; and regenerating, at the server of the federated learning system, an updated global embedding based on the updated condensed embedding and the corresponding updated cluster labels; locally training the server model based on the updated ensemble embedding at the server of the federated learning system to obtain an updated version of the server model; A method comprising:

2. 2. The method of claim 1 , wherein the steps of obtaining the initial condensed embedding and the initial cluster labels and regenerating the initial global embedding are performed at the server, and wherein the step of sending the current version of the server model to each of the plurality of client computing devices further includes sending the initial global embedding.

3. The method of claim 2, further comprising performing federated machine learning inference using the updated version of the server model and the updated local model.

4. The method of claim 3 , further comprising enabling information technology resources according to the federated machine learning inference.

5. determining, for the plurality of client computing devices of the federated learning system, the output of initial local models as initial decompressed embeddings for the mini-batches, each of the initial local models being based on a vertical slice of data; compressing the initial decompressed embedding by the clustering for the plurality of client computing devices of the federated learning system to obtain the initial compressed embedding; for the plurality of client computing devices of the federated learning system, transmitting the initial condensed embedding and initial cluster labels to the server of the federated learning system; locally training the initial local models based on the initial ensemble embedding and the current server model for the plurality of client computing devices of the federated learning system to obtain updated local models, each of the updated local models being based on a different vertical slice of data; for the plurality of client computing devices of the federated learning system, sending the updated clustered embeddings and corresponding updated cluster labels based on the updated local models to the server of the federated learning system; The method of claim 3 further comprising:

6. 6. The method of claim 5, further comprising repeating the steps of obtaining the initial condensed embedding and the initial cluster labels, regenerating the initial global embedding, sending the current version of the server model, obtaining the updated condensed embedding and the updated cluster labels, regenerating the updated global embedding, and locally training the server model to obtain the updated version of the server model until convergence.

7. The method of claim 6 , wherein transmitting the updated clustered embeddings comprises transmitting differentially updated clustered embeddings.

8. The method of claim 7 , wherein when repeating the transmission of the current version of the server model, the current version of the server model is transmitted in compressed form.

9. The method of claim 6 , wherein the clustering comprises k-means clustering.

10. 10. The method of claim 9, further comprising: reducing the number of clusters in the clustering when repeating the steps of obtaining the initial compressed embedding and the initial cluster labels, regenerating the initial global embedding, sending the current version of the server model, obtaining the updated compressed embedding and the updated cluster labels, regenerating the updated global embedding, and locally training the server model to obtain the updated version of the server model.

11. The method of any one of claims 5 to 10, wherein the local training on the plurality of client computing devices and the local training on the server are performed for 5 to 10 iterations.

12. a memory embodying computer-executable instructions; and at least one processor coupled to the memory, To a plurality of client computing devices, an initial condensed embedding, the initial condensed embedding being condensed by clustering, the initial condensed embedding including the output of an initial local model for a mini-batch; initial cluster labels corresponding to the initial condensed embedding; and regenerating an initial global embedding based on the initial condensed embedding and the initial cluster labels obtained from the plurality of client computing devices; transmitting a current version of the server model to each of the plurality of client computing devices; from the plurality of client computing devices based on local training by the plurality of client computing devices using the initial ensemble embedding and the current server model; an updated compressed embedding, the updated compressed embedding being compressed by clustering; and updated cluster labels corresponding to the updated condensed embedding; and regenerating an updated global embedding based on the updated condensed embedding and the corresponding updated cluster labels; locally training the server model based on the updated ensemble embedding to obtain an updated version of the server model; and at least one processor operative to execute the computer-executable instructions to perform the the memory and the at least one processor are distributed across the plurality of client computing devices and a server coupled to the plurality of client computing devices; a federated learning system, wherein the local training by the plurality of client computing devices generates an updated local model.

13. The plurality of client computing devices: determining the output of an initial local model as an initial decompressed embedding for the mini-batch; compressing the initial decompressed embedding by the clustering to obtain the initial compressed embedding; sending the initial condensed embedding and initial cluster labels to the server; locally training the initial local model based on the initial global embedding and the current server model to obtain the updated local model; sending the updated clustered embeddings and corresponding updated cluster labels to the server of the federated learning system based on the updated local model; and the server is configured to send the current version of a server model to each of the plurality of client computing devices, obtain the updated compressed embedding and the updated cluster labels from the plurality of client computing devices, and train the server model locally. The federated learning system of claim 12.

14. 14. The federated learning system of claim 13, wherein the server is configured to obtain the initial condensed embedding and the initial cluster labels and to regenerate the initial global embedding, and the server is configured to transmit the initial global embedding along with the current version of the server model.

15. 15. The federated learning system of claim 14, wherein the server and the plurality of client computing devices are cooperatively configured to perform federated machine learning inference using the updated version of the server model and the updated local model.

16. 16. The federated learning system of claim 15, wherein the server and the plurality of client computing devices are further configured to operatively cooperate to use information technology resources in accordance with the federated machine learning inference.

17. 16. The federated learning system of claim 15, wherein the server and the plurality of client computing devices are further cooperatively configured to iterate the obtaining of the initial condensed embedding and the initial cluster labels, the regeneration of the initial global embedding, the transmission of the current version of the server model, the obtaining of the updated condensed embedding and the updated cluster labels, the regeneration of the updated global embedding, and the local training of the server model to obtain the updated version of the server model until convergence.

18. 18. The federated learning system of claim 17, wherein the clustering comprises k-means clustering, and wherein the server and the plurality of client computing devices are further configured to cooperate to reduce a number of clusters in the clustering when iterating through the steps of obtaining the initial condensed embedding and the initial cluster labels, regenerating the initial global embedding, transmitting the current version of the server model, obtaining the updated condensed embedding and the updated cluster labels, regenerating the updated global embedding, and locally training the server model to obtain the updated version of the server model.

19. On the computer, To a plurality of client computing devices, an initial condensed embedding, the initial condensed embedding being condensed by clustering, the initial condensed embedding including the output of an initial local model for a mini-batch; initial cluster labels corresponding to the initial condensed embedding; The steps to obtain regenerating an initial global embedding based on the initial condensed embedding and the initial cluster labels obtained from the plurality of client computing devices; transmitting a current version of the server model to each of the plurality of client computing devices; from the plurality of client computing devices based on local training by the plurality of client computing devices using the initial ensemble embedding and the current server model; an updated compressed embedding, the updated compressed embedding being compressed by clustering; and updated cluster labels corresponding to the updated condensed embedding; The steps to obtain regenerating an updated global embedding based on the updated condensed embedding and the corresponding updated cluster labels; locally training the server model based on the updated global embedding to obtain an updated version of the server model; Execute The computer program product, wherein the local training by the plurality of client computing devices generates updated local models.

20. The computer: configuring the plurality of client computing devices to determine the output of an initial local model as an initial decompressed embedding for the mini-batch; configuring the plurality of client computing devices to compress the initial decompressed embedding by the clustering to obtain the initial compressed embedding; configuring the plurality of client computing devices to transmit the initial condensed embedding and initial cluster labels to a server; configuring the plurality of client computing devices to locally train the initial local model based on the initial global embedding and the current server model to obtain the updated local model; configuring the plurality of client computing devices to send the updated clustered embeddings and corresponding updated cluster labels to the server of a federated learning system based on the updated local model; configuring the server to send the current version of the server model to each of the plurality of client computing devices, obtain the updated condensed embeddings and the updated cluster labels from the plurality of client computing devices, and train the server model locally; 20. The computer program of claim 19, further comprising:

Images