Vertical joint learning with compressed embeddings

Abstract

Embodiments of the present invention relate to methods, systems, and computer program products for vertical joint learning with compressed embeddings. In the method, for multiple client computing devices of a joint learning system, initial compressed embeddings are obtained through clustering compression, including the output of an initial local model for the current microbatch, and initial cluster labels corresponding to the initial embeddings are obtained. Based on the initial embeddings and initial labels, an initial overall embedding is recreated. At the server of the joint learning system, the current version of the server model is sent to each client computing device; from the client computing devices, updated compressed embeddings through clustering compression and updated cluster labels corresponding to the updated embeddings are obtained. Based on local training performed by multiple clients using the overall embeddings and the current server model, at the server, updated overall embeddings are recreated based on the updated embeddings and corresponding updated labels, and the server model is trained locally based on the updated overall embeddings.

Description

Technical Field

[0001] This invention relates to the fields of electrical, electronic and computer science, and more specifically, to artificial intelligence, machine learning and the like. Background Technology

[0002] Federation learning is a machine learning technique used to train distributed models based on distributed data. It offers benefits such as protecting data privacy and reducing the communication and storage requirements of a central cloud. Federation learning includes, for example, horizontal federated learning and vertical federated learning. In many practical applications of federated learning, features are vertically partitioned across multiple clients. For example, banks and insurance companies have different information about the same person. This combination of information can be used to predict that person's credit score, etc.

[0003] To perform joint learning on features of such vertical partitioning, Vertical Joint Learning (VFL) can be used. However, standard VFL methods can incur high communication overhead due to the need to transmit embeddings between the client and server. Summary of the Invention

[0004] The principles of this invention provide a technique for vertical joint learning with compressed embeddings. In one aspect, an exemplary method includes: for a plurality of client computing devices of a joint learning system, obtaining: an initial compressed embedding, compressed by clustering, the initial compressed embedding including the output of an initial local model for a current minibatch; and initial clustering labels corresponding to the initial compressed embedding. The method further includes: recreating an initial overall embedding based on the initial compressed embedding and initial clustering labels obtained for the plurality of client computing devices; sending a current version of a server model to each of the plurality of client computing devices at a server of the joint learning system; and at the server of the joint learning system, obtaining from the plurality of client computing devices of the joint learning system: an updated compressed embedding, compressed by clustering; and updated clustering labels corresponding to the updated compressed embedding, based on local training performed by the plurality of client computing devices of the joint learning system using the initial overall embedding and the current server model. Further steps include: recreating the updated overall embedding based on the updated compressed embedding and the corresponding updated clustering labels at the server of the joint learning system; and training the server model locally based on the updated overall embedding at the server of the joint learning system to obtain an updated version of the server model.

[0005] On the other hand, an exemplary federated learning system includes: a memory containing computer-executable instructions, and at least one processor coupled to the memory and operable to execute the computer-executable instructions to: obtain for a plurality of client computing devices: an initial compressed embedding, the initial compressed embedding being compressed by clustering, the initial compressed embedding including the output of an initial local model for a current microbatch; and initial cluster labels corresponding to the initial compressed embedding. The at least one processor is further operable to execute the computer-executable instructions to: recreate an initial overall embedding based on the initial compressed embedding and initial cluster labels obtained from the plurality of client computing devices; send a current version of a server model to each of the plurality of client computing devices; and obtain from the plurality of client computing devices: an updated compressed embedding, the updated compressed embedding being compressed by clustering; and updated cluster labels corresponding to the updated compressed embedding, based on local training performed by the plurality of client computing devices with the overall embedding and the current server model. The at least one processor is also operable to execute the computer-executable instructions to: recreate an updated overall embedding based on the updated compressed embedding and the corresponding updated cluster labels; and train a server model locally based on the updated overall embedding to obtain an updated version of the server model.

[0006] In another aspect, an exemplary computer program product includes one or more computer-readable storage media, and program instructions co-stored on the one or more computer-readable storage media. The program instructions include: program instructions for obtaining, for a plurality of client computing devices, an initial compressed embedding, compressed by clustering, the initial compressed embedding including the output of an initial local model for a current microbatch; and initial clustering labels corresponding to the initial compressed embedding; program instructions for recreating an initial global embedding based on the initial compressed embedding and initial clustering labels obtained from the plurality of client computing devices; program instructions for sending a current version of a server model to each of the plurality of client computing devices; and program instructions for obtaining, based on local training performed by the plurality of client computing devices using the global embedding and the current server model, an updated compressed embedding, compressed by clustering; and updated clustering labels corresponding to the updated compressed embedding. The program instructions also include: program instructions for recreating an updated global embedding based on the updated compressed embedding and the corresponding updated clustering labels; and program instructions for training a server model locally based on the updated global embedding to obtain an updated version of the server model.

[0007] As used herein, "facilitating" actions include performing an action, making an action easier, assisting in performing an action, or causing an action to be performed. Therefore, by way of example and not limitation, instructions executing on one processor can facilitate an action performed by instructions executing on a remote processor by sending appropriate data or commands to cause or assist in the action to be performed. To avoid confusion, in cases where an actor facilitates an action by performing other actions, that action is still performed by an entity or combination of entities.

[0008] One or more embodiments of the present invention, or elements thereof, may be implemented in the form of a computer program product comprising a computer-readable storage medium having computer-usable program code for performing the illustrated method steps. Furthermore, one or more embodiments of the present invention, or elements thereof, may be implemented in the form of a system (or apparatus) comprising a memory and at least one processor coupled to the memory and operable to perform the exemplary method steps. Additionally, in another aspect, one or more embodiments of the present invention, or elements thereof, may be implemented in the form of a component for performing one or more method steps described herein; such component may include (i) one or more hardware modules, (ii) one or more software modules stored in a computer-readable storage medium (or multiple such media) and implemented on a hardware processor, or (iii) a combination of (i) and (ii); any one of (i)-(iii) implements the specific techniques set forth herein.

[0009] The technology of this invention can provide substantially beneficial technical effects. Some embodiments may not have these potential advantages, and these potential advantages are not necessary for all embodiments. For example, one or more embodiments provide one or more of the following:

[0010] Improve the technical process of vertical joint machine learning by increasing the communication efficiency between the client and the server, thereby reducing data transmission overhead;

[0011] Improve the technical processes of operating IT infrastructure, such as cloud-based IT infrastructure, by training predictive joint learning models to predict future behavior and deploying information technology (IT) resources based on those predictions.

[0012] These and other features and advantages of the invention will become apparent from the following detailed description of illustrative embodiments of the invention, which are read in conjunction with the accompanying drawings. Attached Figure Description

[0013] Figure 1 This illustrates a vertical joint learning environment in which one or more embodiments of the present invention may be employed;

[0014] Figure 2This presents a system overview according to aspects of the invention;

[0015] Figure 3 An algorithm for vertical joint learning by clustering according to an aspect of the invention is shown;

[0016] Figure 4 A graph illustrating an exemplary training loss relative to the amount of data transmitted upstream according to an aspect of the invention is shown.

[0017] Figure 5 A graph illustrating exemplary training accuracy relative to the amount of data transmitted upstream according to an aspect of the invention is shown.

[0018] Figure 6 A histogram showing the relative clustering of data transmitted upstream at the end of training over 100 epochs according to an aspect of the invention;

[0019] Figure 7 Describing a cloud computing environment according to embodiments of the present invention;

[0020] Figure 8 Describing the abstract model layer according to embodiments of the present invention; and

[0021] Figure 9 The description refers to a computer system that can be used to implement one or more aspects and / or elements of the present invention, and also represents a cloud computing node according to an embodiment of the present invention. Detailed Implementation

[0022] As noted, standard VFL methods can result in high communication overhead due to the need to transmit embeddings between the client and server. One or more embodiments advantageously make VFL communication more efficient. Figure 1 A vertical joint learning environment in which one or more embodiments of the present invention can be employed is illustrated. Multiple client / data silos 103-1, 103-2, 103-3, 103-4, 103-5, ..., 103-K (collectively referred to as 103) comprise vertically separated data; the silos / clients are used collaboratively to train a joint model. The model W... S A portion resides on trusted server 101; each client has its own model W1, W2, W3, W4, W5, ..., W K The characteristics of data generated in the first client / data silo are typically different from those of data generated in the second client / data silo. Models W1, W2, W3, W4, W5, ..., W K Each component forms part of the overall model, and each can be trained independently on the features of its corresponding client. The overall / joint model includes the W component residing on the trusted server 101. SAnd client models W1, W2, W3, W4, W5, ..., W K .

[0023] Still referencing Figure 2 This illustrates at least a portion of a joint model (e.g., two clients 103-1 and 103-2 (and thus two vertical partitions), and a single server 101). Different kinds of deep neural network models can be used. A fully connected network is shown as an example, but convolutional neural networks (CNNs), long short-term memory (LSTMs), or other types of networks can also be employed. Features are vertically partitioned across client 103, and data samples have a shared key (e.g., a unique numerical value for an individual, such as that individual's Social Security number) and shared labels (e.g., ground truth predictions). For client 103-1, features X1, X2, X3, and X4 are fed in, while for client 103-2, features X5, X6, X7, and X8 are fed in. In one or more embodiments, micro-batch sampling across client 103 is synchronized via the shared key. As those skilled in the art will understand, micro-batch gradient descent is a variation of the gradient descent algorithm that partitions the training dataset into mini-batches for calculating model errors and updating model coefficients. In one or more embodiments, micro-batches are selected for feedforward through the entire network. The selected micro-batch will contain a number of samples that are synchronized on the client. For example, all samples may involve the same person. As discussed, it is assumed that a shared key can be used to synchronize the micro-batch sampling.

[0024] exist Figure 2 In the example, the client models 103-1 and 103-2 each have three layers; the third layer of each client model also serves as the input layer to the server model. In each case, embeddings are generated after the second layer. The embeddings are fed into the first layer of the server model, which is the third layer of the client model. Embeddings from different clients have the same dimension and can therefore be summed. During forward propagation, the embeddings are forward-propagated through the server model.

[0025] Figure 1 and Figure 2 In the example, the model is split between client 103 and server 101. The client and server models can be trained simultaneously (using latent information) or alternately. To train the server model W... S In one or more embodiments, the embedding obtained from the output of the last layer of the client model is sent to the server. Note that the top neuron in layer 1 of the client 103-2 model specifically receives the weights assigned to the connection between the top input features in layer 1 and the top neuron. Multiply by the feature x8 as input; the top neurons in layer 2 of client model 103-2 specifically receive the weights assigned to the connections between the top neurons in layer 1 and the top neurons in layer 2. Multiply by the output of the top neuron in layer 1 As input, the top neurons in layer 3 specifically receive the weights assigned to the connections between the top neurons in layer 2 and the top neurons in layer 3. Multiply by the output of the top neuron in layer 2 As input, the top neurons in layer 4 specifically receive the weights assigned to the connections between the top neurons in layer 3 and the top neurons in layer 4. Multiply by the output of the top neuron in layer 3 As input.

[0026] Consider embeddings; they are typically in vector form. Figure 2 In the example, Multiply One of the numbers in the embedding vector sent from client 103-2 to the server is formed. As those skilled in the art will understand, embeddings can be used, for example, to reduce the dimensionality of large sparse matrices. Another embedding is sent from client 103-1 to the server. The embeddings from each client are summed and fed forward through the server model. For example, in the case of performing classification, the output is obtained. Then, backpropagation is performed. E() represents the loss computed at the end of the server model. Figure 2 The thick arrows 111-1, 111-2, 111-3, 111-4, and 111-5 at the bottom indicate the gradient (partial derivative) of the loss relative to the associated set of weights during backpropagation. Arrows 115-1, 115-2, 115-3, 115-4, and 115-5 represent the forward propagation of client 103-1 (similar to the arrows 113-1, 113-2, 113-3, 113-4, and 113-5 already discussed for representing the forward propagation of client 103-2).

[0027] In order to train client models W1, W2, W3, W4, W5, ..., W K The current server model can be sent and copied at each client, or sample-specific information obtained at the first layer of the server model can be sent to the client during backpropagation (the latter option is shown by arrows 111-3). In the former option, for example, an embedded snapshot of the server model from client 103-2 is provided to client 103-1. Client 103-1 can then perform forward propagation locally across the network. Significant information exchange may be required, and one or more embodiments make the process more communicatively efficient. Figure 2The thick arrows 111-1, 111-2, 111-3, 111-4, and 111-5 at the bottom illustrate the backpropagation process, where at each step, the server model E is calculated relative to the corresponding weights. The gradient.

[0028] Since the server model can compute the loss, both forward and backward propagation can be performed. In one or more embodiments, clients also need to train their neural networks; however, they require a snapshot of the server model to perform backpropagation / enable local iteration. See arrows 111-3. Similarly, clients send snapshots of their embeddings to the server, as discussed elsewhere in this document.

[0029] In existing vertical joint learning methods, the embedding of each data sample needs to be transferred between client 103 and server 101, which can incur high transmission overhead. To overcome this problem, one or more embodiments use clustering methods to combine the embeddings of data samples (e.g., those in micro-batches). The cluster size can be determined based on the similarity of the embeddings and their impact on further training (i.e., the cluster size can be updated dynamically). For example, when features at a particular client have no impact on the final prediction (e.g., all embeddings from that client are the same or very similar), it is sufficient to use a single vector to represent all embeddings, i.e., the cluster size is 1. On the other hand, when features at the client have an impact on the final prediction (i.e., the prediction strongly depends on these features), the embeddings of different samples are likely to be very different, and using a large cluster size is appropriate.

[0030] Embeddings can also be compressed by sending only the difference compared to what has been previously sent (optionally, after quantization). Clustering and compression methods can also be designed to protect user privacy according to a predefined metric. K-means is a non-limiting example. Clustering is a form of vector quantization. In one or more embodiments, the compressed model is sent from the server to the client; that is, in one or more embodiments, downstream communication is also compressed.

[0031] Figure 3 Pseudocode for an exemplary algorithm 301 for vertical joint learning with a server model and clustering for upstream compression is shown. In lines 1 and 2, the client model and server model are initialized (t = 0). The model for the k-th client at time t = 0 is specified as... The server model at time t=0 is specified as Following the FOR-NEXT loop from lines 3 to 29, the entire technique is repeated for time steps T = 1 to T. In line 4, a micro-batch B is randomly sampled for a given time step. t This micro-batch Bt Synchronized on the client and selected from feature X. From lines 5-9, in parallel, for each client k=1,…,K, the output of the local model is obtained (line 6), the upstream data is compressed by K-means clustering, etc. (line 7), and the embeddings are exchanged by sending the clustered embeddings and labels to the server (line 8). Symbols This represents the micro-batch B based on time t. t And the model of the kth client at time t, using the function h k Micro-batch B was found at client k. t Embedding. Function h k It is an abbreviation for the output of the local neural network model, as will be understood by those skilled in the art.

[0032] Assume the batch size is 50. For each of the 50 data points, there will be one embedding. One or more embodiments apply clustering (k-means clustering is a non-limiting example as shown in line 7) to the embeddings to compress the upstream data. The k-means (or other) clustering can be adjusted to obtain the desired number of centroids. The set of clusters and cluster labels for the embeddings are then sent to the server. In the example of 50 data points, assume that 5 clusters are found as a result of k-means (or other) clustering. The 5 cluster centroids and cluster labels are sent to the server (line 8) instead of all 50 data points, which advantageously reduces the required communication bandwidth. The cluster labels are the output of k-means (or other) clustering; for example, a vector indicating that sample 1 belongs to cluster 1, sample 2 belongs to cluster 1, sample 3 belongs to cluster 2, etc. There will be 50 entries for 50 data points.

[0033] In line 10, the server uses clustering information from the model embeddings used by each client. Recreate the overall model embedding in line 11 and add the updated server model to it. The embedding matrix is ​​sent to all clients. The server reconstructs the embedding matrix based on the centroids of the clusters; each cluster has a centroid, and instead of sending each data point, it only needs to send which cluster each point belongs to and the centroid of that cluster. At the server, during reconstruction, each data point in a cluster is replaced with the centroid of that cluster. This reduces bandwidth because all that needs to be sent are the centroids and data point labels. In one or more embodiments, each client performs multiple iterations of local training but only participates in external communication periodically. However, in one or more embodiments, because the features are distributed, the client needs information from other silos / clients (embeddings from other clients and snapshots of the server model) to allow local iteration. The server obtains the (clustered) information from each client, recreates the overall embedding, and in line 11, sends this overall embedding along with the server model at time t back to each client to facilitate the process.

[0034] At this point, the clients have all the information they need to perform local training for a certain number of iterations Q. From lines 12-22, in parallel, on line 13, each client receives its overall embedding φ. K The copy, then, in the FOR-NEXT loop from lines 14-19, for q=1 to Q, in line 15, based on the microbatch B at time t. t And the model of the k-th client in the q-th iteration at time t. Use function h k Get local embedding (Without communicating with the server). On line 16, loss calculation is performed (applying an appropriate loss function, selectable by a person skilled in the art, to the server model at time t, for each client embedding, for the y (data label / class) contained in the microbatch). A person skilled in the art can heuristically choose a suitable loss function; non-restricted examples include cross-entropy loss, mean squared error, etc. On line 17, (based on the server model at time t) the partial derivatives are determined, and the local gradient is applied to update the local model for q+1 (line 18). This can be done using standard neural network training techniques (e.g., coordinate descent). If Q = 1, the algorithm is reduced to performing forward and backward propagation in each iteration and communicating with the server in each round. Based on heuristics / knowledge of the application and dataset, Q can be increased to values ​​higher than 1. In a non-restricted example, Q is increased to values ​​in the range of 5-10.

[0035] Excessive increases in Q can negatively impact convergence. In lines 15-18, the client performs forward and backward propagation. For forward propagation, the client finds the embeddings and then feeds them to the copy of the server model it obtains in line 11. It finds the loss in line 16 and computes the gradient in line 17. In line 18, the weights of the local model are updated by applying the gradient. This is done for Q iterations. At the end of the Q iterations, the client allows training the actual server model. Since the weights have been updated, the embeddings for each client model are also updated. Each client model forms its updated embeddings, and k-means clustering is performed in line 20. In line 21, the clustered embeddings are... The clustering labels are sent to the server. The server now has the latest embeddings (similar to line 10 but recreated in line 23), and in lines 24-28, training is performed on the latest embeddings. Figure 3 In the unrestricted example, the server is also trained Q times, which allows for faster training while reducing the need for client-server communication. Lines 21 and 23 are similar to lines 8 and 10, where lines 8 and 10 use the initial embeddings, while lines 21 and 23 use the updated embeddings. The Q-value used for the server does not need to be the same as the Q-value used for the client, and different clients do not need to have the same Q-value as each other.

[0036] Therefore, in line 20, K-means clustering is performed to compress the upstream data, and in line 21, the clustered embeddings and labels are sent to the server. In line 23, the server uses the clustering information to recreate an updated overall model embedding from the model embeddings used for each client. In the FOR-NEXT loop from lines 24-29, the loss and partial derivatives are calculated, and the local gradient is applied. In line 25, the loss is calculated (an appropriate loss function, which may be chosen by those skilled in the art (see above discussion), is applied to the server model at time t iterating q, the overall embedding, for y included in the microbatch). In line 26, the partial derivatives are determined (based on the server model at time t iterating q), and the local gradient is applied to update the local model for q+1 (line 18). Again, this can be accomplished using standard neural network training techniques such as coordinate descent.

[0037] At the end of T rounds, sufficient training has been performed to enable the joint model to achieve accurate classification / inference. The value of T can be chosen heuristically, or it can track the calculated loss and stop computation when the loss falls below an acceptable threshold (to achieve acceptable performance / accuracy) or when the loss does not change meaningfully between consecutive rounds / time steps. Thus, the pseudocode indicates T time steps, but T is not a fixed prior value but is dynamically determined during computation, as described.

[0038] experiment

[0039] Using a publicly available anonymized hospital database, experiments were conducted on binary classification for in-hospital mortality prediction using two workers (103) and one server (101). Workers (103) used a Long Short-Term Memory (LSTM) model, while the server model employed two fully connected (FC) layers and a SoftMax output. A batch size of 128 was used, and the number of clusters in a batch varied between values ​​of 16 (i.e., sending 16 embeddings), 32, and 64 (i.e., halving the amount of data sent up to the server). The "no clustering" case is when no clustering occurs and all 128 data points have been transmitted. In our experiments, we plotted the loss at the end of each epoch across all training data points; that is, an iteration over the entire dataset with a given batch size, as computing this for each batch would be very expensive.

[0040] In our experiment, server 101 and worker 103 were each trained locally for five epochs (Q=5). Figure 4 In the diagram, the X-axis plots the number of batches seen (the number of kilobytes transmitted upstream), and the Y-axis shows the average training loss for a batch. The embeddings transmitted upstream are compressed (except in the case of no clustering, where the reduction in training loss and therefore convergence is slowest). Convergence is faster for a cluster size of 64, still faster for a cluster size of 32, and fastest for a cluster size of 16. In other words, for a fixed amount of data transmitted upstream, examples with smaller cluster sizes and therefore more data compression can undergo more rounds of training, resulting in faster convergence. The exemplary results depicted are averages over three runs using different seeds. In one or more embodiments, the number of bytes transmitted is calculated offline by looking at the number of times the algorithm is executed and the number of floating-point / integer numbers transmitted. For each batch, there are two upstream transmissions. For a single transmission with a batch size of 128, the total number of bytes transmitted is determined as follows:

[0041] The total number of bytes transmitted = (embedded size * number of clusters * 64 + batch size * 6) / (1024 * 8) bytes.

[0042] Above, factor 6 is the base-2 logarithm of 128.

[0043] Still referencing Figure 4Training was performed on all cases until approximately 95,000 KB of data had been transferred. Before the end of 100 epochs, approximately 95,000 KB was transferred upstream with a cluster size of 16. When examining the convergence of different clustering settings for the same number of bytes transferred, it can be seen that, for the same number of bytes transferred, the convergence rate improves as the number of clusters decreases. This is because, for the same number of bytes transferred, the number of training epochs (e.g., ...) increases. Figure 4 The value observed at the points in the diagram increases with increasing compression.

[0044] Now go to Figure 5 When examining the training accuracy of different clustering settings for the same number of transmitted bytes (basically compared to...), Figure 4 When the training loss shown is the opposite (as illustrated), it can be seen that, with the same number of bytes transmitted, accuracy increases as the number of clusters decreases. This is because, with the same number of bytes transmitted, the number of training rounds (such as...) increases. Figure 5 The points observed in the data increase with increasing compression, thus enabling faster achievement of higher accuracy.

[0045] Figure 6 The diagram shows the number of bytes transferred at the end of training over one hundred epochs. It can be seen that the number of kilobytes transferred increases with the number of "embeddings" transferred. In this example, training is conducted for up to 100 epochs (T=100) across all cases, with 115 batches per epoch. Without clustering, slightly more than approximately 700 MB needs to be sent upstream. For 16 clusters, only about 90 MB needs to be sent upstream; for 32 clusters, approximately 180 MB; and for 64 clusters, approximately 380 MB (about half the amount without clustering).

[0046] Therefore, it will be understood that one or more embodiments provide a vertical joint learning system comprising multiple clients and servers, wherein client 103 and server 101 exchange compressed embeddings. In one or more embodiments, clustering methods are used to combine the embeddings of data samples (e.g., those in microbatches). The cluster size may be determined, for example, based on the similarity of the embeddings and their impact on further training.

[0047] In some instances, the embedding is compressed by sending only the difference compared to what has been previously sent (optionally, after quantization). The compression method can be designed to protect user privacy based on predefined metrics. The compressed model can be sent from the client to the server, or from the server to the client.

[0048] Therefore, one or more embodiments employ compressed embeddings exchanged between the client and server to reduce the communication overhead of vertical joint learning. One or more embodiments accelerate communication between the multiple parties participating in the vertical joint learning scheme. Additionally, in one or more embodiments, the server also trains a portion of the model.

[0049] Therefore, one or more embodiments provide techniques for improving the efficiency of a joint learning scheme using computing devices, including: receiving compressed embeddings trained by worker computers in the joint learning scheme by a central computing device in the scheme. The compressed embeddings may be combined with other compressed embeddings trained by other worker computers in the joint learning scheme. In the joint learning scheme, a joint learning model is jointly trained based on all received compressed embeddings.

[0050] In some instances, compressed embedding is based on data samples accessed by each worker computer.

[0051] Various practical applications are possible. For example, manufacturers and end-user industrial companies may have different information about the same industrial equipment. This combination of information can be used to predict the maintenance schedule required for that equipment. A cloud (or other IT) platform managed by one company may store data from another company, or the IT infrastructure may be managed by one company, but the services may be provided by another. Both companies store logs: one for the infrastructure (how much bandwidth / storage was used), and the other for the services themselves, such as how the service worked for different users and the states it had transitioned to in the past. Thus, there are different sets of logs but for the same time period, and the goal is to train predictive models to predict the future behavior of the IT services and infrastructure.

[0052] In the discussion up to this point, it will be understood that an exemplary method according to one aspect of the invention includes: obtaining initial compressed embeddings and initial clustering labels corresponding to the initial compressed embeddings for multiple client computing devices of a joint learning system. The initial compressed embeddings are compressed by clustering and include the output of an initial local model for the current microbatch. Reference Figure 3 Line 8 of the pseudocode. The cluster labels should not be confused with the labels in the training data. Another step involves recreating the initial overall embedding based on the initial compressed embeddings and initial cluster labels obtained for multiple client computing devices. Reference Figure 3 Line 10 of the pseudocode; this step can be done at the server or at each client. Another step involves, at the server of the federated learning system, sending the current version of the server model to each of the multiple client computing devices. (See reference) Figure 3 Line 11 of the pseudocode.

[0053] Further steps include: at the server of the federated learning system, obtaining from multiple client computing devices of the federated learning system: an updated compressed embedding, which is compressed through clustering; and updated cluster labels corresponding to the updated compressed embedding—based on local training performed by the multiple client computing devices of the federated learning system using the initial overall embedding and the current server model. (See reference) Figure 3 Line 21 of the pseudocode. Another step involves: at the server of the joint learning system, recreating the updated overall embeddings based on the updated compressed embeddings and the corresponding updated cluster labels. (See reference...) Figure 3 Line 23 of the pseudocode. Additional steps include: training a server model locally at the server of the joint learning system based on the updated overall embeddings to obtain an updated version of the server model. Reference Figure 3 In lines 25-27 of the pseudocode, it is worth noting that in one or more embodiments, the updated labels only need to be used to obtain the updated overall embedding, and then the updated overall embedding is used to train and obtain the updated server model.

[0054] In one or more embodiments, the steps of obtaining the initial compressed embedding and the initial clustering label, as well as the step of recreating the initial overall embedding, are performed at the server, and the step of sending the current version of the server model to each of the plurality of client computing devices further includes sending the initial overall embedding.

[0055] In one or more embodiments, local training performed by multiple client computing devices produces an updated local model, and a further step 303 includes performing joint machine learning inference using an updated version of the server model and the updated local model. In a non-limiting example, a practical application of inference includes deploying IT resources based on inference (e.g., see...). Figure 7 and Figure 8 ), such as in 305.

[0056] It is worth noting that, typically, the client and server can be controlled / operated by the same or different entities.

[0057] One or more embodiments further include: for multiple client computing devices of the joint learning system, determining the output of an initial local model as the initial uncompressed embedding for the current microbatch. Each initial local model is based on a vertical data slice. (Reference) Figure 3 Line 6 of the pseudocode. This embodiment also includes: for multiple client computing devices of the joint learning system, compressing the initial uncompressed embeddings by clustering to obtain the initial compressed embeddings (see reference). Figure 3(Line 7 of the pseudocode); and for multiple client computing devices in the federated learning system, sending the initial compressed embeddings and initial clustering labels to the server of the federated learning system (see reference). Figure 3 (Line 8 of the pseudocode). This embodiment also includes: for multiple client computing devices of the joint learning system, training an initial local model locally based on the initial overall embedding and the current server model to obtain an updated local model, wherein each updated local model is based on another vertical data slice (see reference). Figure 3 (Lines 15-18 of the pseudocode); and for multiple client computing devices in the federated learning system, based on the updated local model, the updated clustering embeddings and corresponding updated clustering labels are sent to the server of the federated learning system (see reference). Figure 3 (Lines 20-21 of the pseudocode).

[0058] It is worth noting the importance of the "other" vertical data slice. In each round, the client only uses the data from that client, so in that sense, it is the same data slice. However, due to the updated micro-batch sampling, it is not exactly the same data, but a different set selected from the same slice associated with that client.

[0059] As described above, in one or more embodiments, features are vertically segmented across client 103, and data samples have a shared key (e.g., a unique numerical value for each individual, such as the individual's Social Security number) and shared tags (e.g., labels for prediction results). In one or more embodiments, micro-batch sampling across client 103 is synchronized via the shared key.

[0060] One or more embodiments further include: repeating the following steps until convergence: obtaining initial compressed embeddings and initial cluster labels; recreating the initial global embeddings; sending the current version of the server model; obtaining updated compressed embeddings and updated cluster labels; recreating the updated global embeddings; and training the server model locally to obtain an updated version of the server model. Reference Figure 3 The FOR-END FOR loop is shown in lines 3-29 of the pseudocode. The value of T can be chosen heuristically, or it can track the calculated loss and stop computation when the loss falls below an acceptable threshold (to achieve acceptable performance / accuracy) or when the loss does not change meaningfully between consecutive rounds / time steps. Therefore, although the pseudocode indicates T time steps, T may not be a fixed prior value, but rather dynamically determined during computation, as described.

[0061] In one or more embodiments, when these steps are repeated (e.g., t=2), random sampling is performed for the second iteration—randomly sampling a subset of the entire dataset again. In one or more embodiments, for each iteration, a different random sample (micro-batch) of data points is used; the micro-batch is different in each iteration. Therefore, in one or more embodiments, in each round, the micro-batch B t The model for the k-th client at time t is different. The client and server models are updated iteratively, and this also changes the embedding in each iteration.

[0062] In one or more embodiments, an updated clustering embedding is sent. Figure 3 Line 21 of the pseudocode includes sending the clustering embedding with difference updates. As described elsewhere, the embedding can be compressed, for example, by sending only the difference compared to what was previously sent (optionally, after quantization). The clustering and compression methods can also be designed to protect user privacy based on a predefined metric.

[0063] In one or more embodiments, when the current version of the server model is repeatedly sent, the current version of the server model is sent in compressed form. That is, in one or more embodiments, a compressed model is sent from the server to the client; i.e., in one or more embodiments, downstream communication is also compressed.

[0064] In one or more embodiments, clustering includes k-means clustering; k-means is a non-limiting example, and clustering is a form of vector quantization.

[0065] One or more embodiments further include reducing the number of clusters in a cluster when repeating the steps of obtaining initial compressed embeddings and initial cluster labels, recreating initial overall embeddings, sending the current version of the server model, obtaining updated compressed embeddings and updated cluster labels, recreating updated overall embeddings, and training the server model locally to obtain an updated version of the server model. The cluster size can be determined based on the similarity of the embeddings and their impact on further training (i.e., the cluster size can be updated dynamically). For example, when features on a particular client have no impact on the final prediction (e.g., all embeddings from that client are identical or very similar), it is sufficient to use a single vector to represent all embeddings, i.e., the cluster size is 1. On the other hand, when features on the client have an impact on the final prediction (i.e., the prediction strongly depends on these features), the embeddings of different samples are likely to be very different, and using a large cluster size is appropriate.

[0066] Those skilled in the art will understand that "K" refers to the number of clients and the number of centroids in the K-means clustering, and will understand the distinction from the context (of course, the number of clients and the number of centroids need not be the same). Dynamic aspects can include changing how many clusters are used in the k-means clustering (i.e., how much compression). This can depend on, for example, bandwidth constraints. Consider increasing the number of data points in a cluster to reduce the number of clusters and compress more data; reducing the number of data points in a cluster means more clusters are needed, and less compression but more accuracy. For example, fewer clusters can be used near the end of training. Of course, even if this doesn't occur, dynamic elements already exist, as the clusters typically change with each iteration.

[0067] To select an initial value for the number of clusters in k-means clustering, heuristics are used, for example, based on the characteristics of the data being analyzed and the size of the microbatch. For example, given an error bound, each client can determine k. Each client can use a different k if needed. Non-limiting examples are shown in the accompanying figures; for example... Figures 4 to 6 .exist Figure 4 and Figure 5 In the example, the batch size is 128, and the number of clusters ranges from 16 to 64. Therefore, as a non-limiting example, the upper limit of the number of clusters could be 50% of the batch size (64 / 128), while the lower limit of the number of clusters could be 12.5% ​​of the batch size (16 / 128) or 6.25% of the batch size (8 / 128).

[0068] In one or more embodiments, local training on the client and local training on the server are performed for 5-10 iterations (i.e., Q = 5-10). However, note that in some cases, Q can be much greater than 10, depending on data diversity and the learning rate. For example, Q can be 100. Furthermore, in some cases, Q = 1 is a relevant special case because the algorithm also works with a single local update. Therefore, in some cases, Q = 1 to 100. Q can vary between clients, between clients and servers, and between rounds.

[0069] On the other hand, joint learning systems include (for example, see...) Figure 9 and related text as described above Figure 3The pseudocode represents a memory containing computer-executable instructions and at least one processor coupled to the memory and operable to execute the computer-executable instructions to: obtain, for a plurality of client computing devices 103, an initial compressed embedding, which is compressed by clustering, the initial compressed embedding including the output of an initial local model for the current microbatch; and initial cluster labels corresponding to the initial compressed embedding. The at least one processor is further operable to execute the computer-executable instructions to: recreate an initial overall embedding based on the initial compressed embedding and initial cluster labels obtained from the plurality of client computing devices; send a current version of the server model to each of the plurality of client computing devices; and obtain, based on local training performed by the plurality of client computing devices with the overall embedding and the current server model, an updated compressed embedding, which is compressed by clustering; and updated cluster labels corresponding to the updated compressed embedding. The at least one processor is also configured to execute the computer-executable instructions to: recreate an updated overall embedding based on the updated compressed embedding and the corresponding updated cluster labels; and train the server model locally based on the updated overall embedding to obtain an updated version of the server model.

[0070] In one or more embodiments, memory and at least one processor are distributed across multiple client computing devices and a server 101 coupled to the multiple client computing devices; local training performed by the multiple client computing devices generates an updated local model, and the multiple client computing devices are configured to: determine the output of the initial local model as an initial uncompressed embedding for the current microbatch; compress the initial uncompressed embedding by clustering to obtain an initial compressed embedding; send the initial compressed embedding and initial cluster labels to the server; train the initial local model locally based on the initial overall embedding and the current server model to obtain an updated local model; and send the updated cluster embedding and corresponding updated cluster labels to the server of the joint learning system based on the updated local model. Furthermore, the server is configured to: send the current version of the server model to each of the multiple client computing devices, obtain updated compressed embeddings and updated cluster labels from the multiple client computing devices, and train the server model locally.

[0071] It will be understood that the client and server are configured in this way, for example, by computer-executable instructions that implement the relevant parts of the algorithm exposed in pseudocode.

[0072] In one or more embodiments, the server is configured to obtain the initial compressed embedding and initial cluster labels, and to recreate the initial overall embedding, and the server is configured to send the initial overall embedding and the current version of the server model.

[0073] In one or more embodiments, a server and multiple client computing devices are collaboratively configured to perform joint machine learning inference using an updated version of the server model and an updated local model. A non-limiting exemplary practical application of this inference is the deployment of information technology resources based on inference.

[0074] In some instances, the server and multiple client computing devices are also collaboratively configured to repeatedly obtain initial compressed embeddings and initial cluster labels, recreate initial global embeddings, send the current version of the server model, obtain updated compressed embeddings and updated cluster labels, recreate updated global embeddings, and train the server model locally to obtain an updated version of the server model, until convergence.

[0075] As mentioned above, in some cases, clustering includes k-means clustering. The server and multiple client computing devices can also be cooperatively configured to reduce the number of clusters in a cluster when repeatedly obtaining initial compressed embeddings and initial cluster labels, recreating initial overall embeddings, sending the current version of the server model, obtaining updated compressed embeddings and updated cluster labels, recreating updated overall embeddings, and training the server model locally to obtain an updated version of the server model.

[0076] On the other hand, computer program products (e.g., see...) Figure 9 and related text as described above Figure 3 The pseudocode includes one or more computer-readable storage media, and program instructions co-stored on one or more computer-readable storage media; the program instructions include: program instructions for obtaining, for a plurality of client computing devices, an initial compressed embedding, which is compressed by clustering, the initial compressed embedding including the output of an initial local model for the current microbatch; and an initial cluster label corresponding to the initial compressed embedding; program instructions for recreating an initial global embedding based on the initial compressed embedding and the initial cluster label obtained from the plurality of client computing devices; program instructions for sending the current version of the server model to each of the plurality of client computing devices; and program instructions for obtaining, based on local training performed by the plurality of client computing devices with the global embedding and the current server model, the following from the plurality of client computing devices: an updated compressed embedding, which is compressed by clustering; and an updated cluster label corresponding to the updated compressed embedding. The program instructions also include: program instructions for recreating an updated global embedding based on the updated compressed embedding and the corresponding updated cluster label; and program instructions for training the server model locally based on the updated global embedding to obtain an updated version of the server model.

[0077] In some embodiments, the program instructions are executed collaboratively by a plurality of client computing devices 103 and a server 101 coupled to the plurality of client computing devices; and local training performed by the plurality of client computing devices produces an updated local model. The program instructions also include: program instructions for configuring the plurality of client computing devices to determine the output of the initial local model as initial uncompressed embeddings for the current microbatch; program instructions for configuring the plurality of client computing devices to compress the initial uncompressed embeddings by clustering to obtain initial compressed embeddings; program instructions for configuring the plurality of client computing devices to send the initial compressed embeddings and initial cluster labels to the server; program instructions for configuring the plurality of client computing devices to train the initial local model locally based on the initial overall embeddings and the current server model to obtain an updated local model; and program instructions for configuring the plurality of client computing devices to send the updated cluster embeddings and corresponding updated cluster labels to the server of the federated learning system based on the updated local model. The program instructions also include program instructions for configuring the server to send the current version of the server model to each of the plurality of client computing devices, to obtain updated compressed embeddings and updated cluster labels from the plurality of client computing devices, and to train the server model locally.

[0078] It should be understood that although this disclosure includes a detailed description of cloud computing, the implementation of the teachings set forth herein is not limited to a cloud computing environment. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0079] Cloud computing is a service delivery model that enables convenient, on-demand network access to a pool of shared, configurable computing resources (e.g., networks, network bandwidth, servers, processing power, storage, applications, VMs, and services) that can be rapidly provisioned and released with minimal management costs or interaction with service providers. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0080] The characteristics are as follows:

[0081] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power (such as server time and network storage) on demand without human interaction with the service provider.

[0082] Wide network access: Capabilities are available on the network and accessed through standard mechanisms that facilitate the use of heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0083] Resource pooling: A provider's computing resources are grouped into resource pools to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. Typically, consumers cannot control or know the exact location of the resources provided, but can specify the location at a higher level of abstraction (e.g., country, state, or data center), thus exhibiting location independence.

[0084] Rapid flexibility: Capabilities can be rapidly and flexibly (in some cases automatically) provided to expand outward quickly and be rapidly released to shrink back down. For consumers, the available capacity often appears unlimited and can be purchased at any time and in any quantity.

[0085] Measurable services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0086] The service model is as follows:

[0087] Software as a Service (SaaS): The capability offered to consumers is the ability to use applications running on a provider's cloud infrastructure. These applications can be accessed from various client devices via thin client interfaces such as web browsers (e.g., web-based email). Aside from limited user-specific application configuration settings, consumers neither manage nor control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities.

[0088] Platform as a Service (PaaS): This provides consumers with the ability to deploy consumer-created or acquired applications on cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers neither manage nor control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the applications they deploy and may also have control over the configuration of the application hosting environment.

[0089] Infrastructure as a Service (IaaS): This provides consumers with the capability to deploy and run any software, including operating systems and applications, on the cloud, providing them with processing, storage, networking, and other basic computing resources. Consumers neither manage nor control the underlying cloud infrastructure, but they have control over the operating system, storage, and deployed applications, and may have limited control over chosen network components (e.g., host firewalls).

[0090] The deployment model is as follows:

[0091] Private cloud: A cloud infrastructure that runs exclusively for a single organization. It can be managed by that organization or a third party, and can exist inside or outside the organization.

[0092] Community cloud: A cloud infrastructure shared by several organizations and supporting a specific community with common interests (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by the organization or a third party and can exist inside or outside the organization.

[0093] Public cloud: Cloud infrastructure available to the general public or large industrial groups and owned by organizations that sell cloud services.

[0094] Hybrid cloud: A cloud infrastructure consisting of two or more clouds (private, community, or public) that remain distinct entities but are bound together by standardized or proprietary technologies that enable data and applications to be ported together (e.g., cloud bursts for load balancing between clouds).

[0095] Cloud computing environments are service-oriented, characterized by statelessness, loose coupling, modularity, and semantic interoperability. The core of cloud computing is its infrastructure, which includes a network of interconnected nodes.

[0096] Now for reference Figure 7 The diagram illustrates an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 to which local computing devices used by cloud consumers can communicate. These local computing devices include personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks (such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above). This allows the cloud computing environment 50 to provide Infrastructure as a Service, Platform as a Service, and / or Software as a Service without requiring cloud consumers to maintain resources on their local computing devices. It should be understood that... Figure 6 The various computing devices 54A-N shown are merely illustrative. The computing node 10 and the cloud computing environment 50 can communicate with any type of computerized device over any type of network and / or network-addressable connection (e.g., using a web browser).

[0097] Now for reference Figure 8 This demonstrates a cloud computing environment of 50 ( Figure 7 This provides a set of functional abstraction layers. First, it should be understood that... Figure 8 The components, layers, and functions shown are for illustrative purposes only, and embodiments of the invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0098] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a mainframe 61; a RISC (Reduced Instruction Set Computer) based server 62; a server 63; a blade server 64; a storage device 65; and a network and network components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0099] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 71; virtual storage 72; virtual network 73, including virtual private network; virtual application and operating system 74; and virtual client 75.

[0100] In one example, management layer 80 may provide the following functionalities: Resource provisioning function 81 provides dynamic acquisition of computing resources and other resources for performing tasks in the cloud computing environment. Metering and pricing provisioning function 82 provides cost tracking for the use of resources within the cloud computing environment and provides bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks and protection for data and other resources. User portal function 83 provides access to the cloud computing environment for consumers and system administrators. Service level management function 84 provides allocation and management of cloud computing resources to meet required service levels. Service level agreement (SLA) planning and enforcement function 85 provides pre-scheduling and procurement of cloud computing resources according to the SLA for its projected future needs.

[0101] Workload layer 90 provides examples of functionalities that can leverage a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; instruction delivery in virtual classrooms 93; data analysis and processing 94; transaction processing 95; and at least a portion 96 of a vertical federated learning system with compressed embedding (e.g., its server 101).

[0102] One or more embodiments of the present invention or elements thereof may be implemented in the form of an apparatus including a memory and at least one processor coupled to the memory and operable to perform exemplary method steps. Figure 9 A computer system is described that can be used to implement one or more aspects and / or elements of the present invention, which also represents a cloud computing node according to an embodiment of the present invention. Reference is now made to... Figure 9 Cloud computing node 10 is merely an example of a suitable cloud computing node and is not intended to impose any limitation on the scope or functionality of the embodiments of the invention described herein. In any case, cloud computing node 10 is capable of implementing and / or performing any of the functions set forth above.

[0103] Within cloud computing node 10, there exists a computer system / server 12 that can operate alongside many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the aforementioned systems or devices.

[0104] Computer system / server 12 can be described in the general context of computer system executable instructions (such as program modules) executed by the computer system. Typically, program modules can include routines, programs, objects, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. Computer system / server 12 can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can reside in local and remote computer system storage media, including memory storage devices.

[0105] like Figure 9 As shown, a computer system / server 12 in a cloud computing node 10 is illustrated in the form of a general-purpose computing device. Components of the computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that couples various system components, including the system memory 28, to the processor 16.

[0106] Bus 18 represents one or more of several types of bus architectures, including memory buses or memory controllers, peripheral buses, accelerated graphics ports, and processor or local buses using any of the various bus architectures. By way of example and not limitation, such architectures include the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA (EISA) bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0107] Computer system / server 12 typically includes various computer system readable media. Such media can be any available media that can be accessed by computer system / server 12, and it includes volatile and non-volatile media, removable and non-removable media.

[0108] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer system / server 12 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be provided for reading from and writing to non-removable non-volatile magnetic media (not shown and generally referred to as "hard disk drives"). Although not shown, disk drives for reading from and writing to removable non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable non-volatile optical disks (such as CD-ROMs, DVD-ROMs, or other optical media) may be provided. In this case, each may be connected to bus 18 via one or more data media interfaces. As will be further described and depicted below, memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present invention.

[0109] A program / utility 40 having at least one set of program modules 42, along with an operating system, one or more applications, other program modules, and program data, may be stored in memory 28, as an example and not a limitation. Each of the operating system, one or more applications, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. Program modules 42 typically perform the functions and / or methods of the embodiments of the invention described herein.

[0110] The computer system / server 12 can also communicate with one or more external devices 14, such as a keyboard, indicating device, display 24, etc.; one or more devices that enable a user to interact with the computer system / server 12; and / or any device that enables the computer system / server 12 to communicate with one or more other computing devices (e.g., a network interface card, modem, etc.). This communication can occur via input / output (I / O) interface 22. Furthermore, the computer system / server 12 can communicate with one or more networks (such as a local area network (LAN), a general area network (WAN), and / or a public network (e.g., the Internet)) via network adapter 20. As shown, network adapter 20 communicates with other components of the computer system / server 12 via bus 18. It should be understood that, although not shown, other hardware and / or software components can be used in conjunction with the computer system / server 12, examples including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0111] Therefore, one or more embodiments can utilize software running on a general-purpose computer or workstation. (Reference) Figure 9 This implementation may employ, for example, a processor 16, memory 28, and an input / output interface 22 to a display 24 and one or more external devices 14 (such as a keyboard, pointing device, etc.). The term "processor" as used herein is intended to include any processing device, such as a processing device including a CPU (Central Processing Unit) and / or other forms of processing circuitry. Further, the term "processor" may refer to more than one individual processor. The term "memory" is intended to include memory associated with a processor or CPU, such as RAM (Random Access Memory) 30, ROM (Read-Only Memory), fixed storage devices (e.g., hard disk drive 34), removable storage devices (e.g., disk), flash memory, etc. Furthermore, the phrase "input / output interface" as used herein is intended to be an interface to, for example, one or more mechanisms (e.g., a mouse) for inputting data to the processing unit and one or more mechanisms (e.g., a printer) for providing results associated with the processing unit. The processor 16, memory 28, and input / output interface 22 may be interconnected, for example, via bus 18, as part of a data processing unit 12. For example, appropriate interconnections via bus 18 can also be provided to network interface 20 (e.g., network card, which can be provided to interface with computer network) and media interface (e.g., disk or CD-ROM drive, which can be provided to interface with appropriate media).

[0112] Therefore, computer software including instructions or code for performing the methods of the present invention as described herein can be stored in one or more associated storage devices (e.g., ROM, fixed or removable memory) and, when ready for use, partially or wholly loaded (e.g., into RAM) and implemented by a CPU. Such software may include, but is not limited to, firmware, resident software, microcode, etc.

[0113] A data processing system suitable for storing and / or executing program code will include at least one processor 16 directly or indirectly coupled to memory element 28 via system bus 18. Memory element may include local memory used during the actual implementation of the program code, mass storage, and cache memory 32 that provides temporary storage for at least some of the program code to reduce the number of times code must be retrieved from mass storage during implementation.

[0114] Input / output or I / O devices (including but not limited to keyboards, displays, indicators, etc.) can be coupled to the system directly or through an intermediate I / O controller.

[0115] Network adapter 20 can also be coupled to the system to enable the data processing system to be coupled to other data processing systems or remote printers or storage devices via an intermediate private or public network. Modems, cable modems, and Ethernet cards are just a few of the currently available types of network adapters.

[0116] As used herein (including the claims), "server" includes a physical data processing system that runs server programs (e.g., such as...). Figure 9 (System 12 shown). It is understood that such a physical server may or may not include a monitor and keyboard.

[0117] One or more embodiments may be implemented, at least in part, in the context of a cloud or virtual machine environment, although this is exemplary and not limiting. Return to References Figures 7-8 And related text.

[0118] It should be noted that any method described herein may include additional steps of providing a system comprising different software modules implemented on a computer-readable storage medium; modules may include any or all of the appropriate elements depicted in the block diagram and / or described herein; any, some, or all of the modules / blocks and / or submodules / subblocks are described by way of example and not limitation. The method steps may then be performed using the different software modules and / or submodules of a system executed on one or more hardware processors (e.g., 16) as described above. Further, the computer program product may include a computer-readable storage medium having code adapted to be implemented to perform one or more method steps described herein, including providing different software modules to the system.

[0119] In some cases, an example of a user interface that can be used is Hypertext Markup Language (HTML) code distributed by a server or similar entity to the browser on the user's computing device. The HTML is then parsed by the browser on the user's computing device to create a graphical user interface (GUI).

[0120] Exemplary system and article details

[0121] This invention can be a system, method, and / or computer program product at any possible level of technical detail integration. The computer program product may include one or more computer-readable storage media having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0122] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures on which instructions are recorded, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0123] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0124] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuits in order to perform aspects of this invention.

[0125] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0126] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0127] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions executed on the computer, other programmable apparatus or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions indicated in the blocks may occur in a non-consecutive order as shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0129] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or improvements to existing technologies on the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for joint learning, comprising: For multiple client computing devices in the federated learning system, the following information is obtained at the server of the federated learning system: The initial compressed embedding, which is compressed through clustering, includes the output of the initial local model for the current microbatch, and The initial clustering label corresponding to the initial compressed embedding; At the server of the joint learning system, the initial overall embedding is recreated based on the initial compressed embedding and the initial clustering labels obtained for the plurality of client computing devices; At the server of the joint learning system, the current version of the server model and the initial overall embedding are sent to each of the plurality of client computing devices; At the server of the joint learning system, based on local training performed by the multiple client computing devices of the joint learning system using the initial overall embedding and the current server model, the following are obtained from the multiple client computing devices of the joint learning system: The updated compressed embedding, which is compressed through clustering, and The updated clustering labels corresponding to the updated compressed embedding; At the server of the joint learning system, an updated overall embedding is recreated based on the updated compressed embedding and the corresponding updated clustering label; as well as At the server of the joint learning system, the server model is trained locally based on the updated overall embedding to obtain an updated version of the server model; The method further includes generating an updated local model through local training performed by the plurality of client computing devices, and performing joint machine learning inference using the updated version of the server model and the updated local model.

2. The method according to claim 1, further comprising: Based on the reasoning, deploy information technology resources.

3. The method according to claim 1, further comprising: For the multiple client computing devices of the joint learning system, the output of the initial local model is determined as the initial uncompressed embedding for the current microbatch, wherein each of the initial local models is based on a vertical data slice; For the multiple client computing devices of the joint learning system, the initial uncompressed embedding is compressed through clustering to obtain the initial compressed embedding; For the multiple client computing devices of the federated learning system, the initial compressed embedding and initial clustering labels are sent to the server of the federated learning system; For the multiple client computing devices of the joint learning system, based on the initial overall embedding and the current server model, the initial local model is trained locally to obtain the updated local model, wherein each of the updated local models is based on another vertical data slice; and For the multiple client computing devices of the federated learning system, based on the updated local model, the updated compressed embedding and the corresponding updated clustering labels are sent to the server of the federated learning system.

4. The method according to claim 3, further comprising: The steps of obtaining the initial compressed embedding and the initial clustering label, recreating the initial global embedding, sending the current version of the server model, obtaining the updated compressed embedding and the updated clustering label, recreating the updated global embedding, and training the server model locally to obtain the updated version of the server model are repeated until convergence.

5. The method according to claim 4, wherein, Sending the updated compressed embedding includes sending the clustering embedding with differential updates.

6. The method according to claim 5, wherein, When the current version of the server model is sent repeatedly, the current version of the server model is sent in compressed form.

7. The method according to claim 4, wherein, The clustering includes k-means clustering.

8. The method according to claim 7, further comprising: When repeating the steps of obtaining the initial compressed embedding and the initial cluster label, recreating the initial overall embedding, sending the current version of the server model, obtaining the updated compressed embedding and the updated cluster label, recreating the updated overall embedding, and training the server model locally to obtain the updated version of the server model, the number of clusters in the cluster is reduced.

9. The method according to claim 3, wherein, The local training on the client and the local training on the server are performed for 5 to 10 iterations.

10. A joint learning system, comprising: Memory containing computer-executable instructions, and At least one processor coupled to the memory, the memory and the at least one processor being distributed across a plurality of client computing devices and a server coupled to the plurality of client computing devices, the processor for the server being operable to execute the computer-executable instructions to: For the aforementioned multiple client computing devices, the following is obtained: The initial compressed embedding, which is compressed through clustering, includes the output of the initial local model for the current microbatch, and The initial clustering label corresponding to the initial compressed embedding; Based on the initial compressed embedding and the initial clustering labels obtained from the plurality of client computing devices, the initial overall embedding is recreated; Send the current version of the server model and the initial overall embedding to each of the plurality of client computing devices; Based on local training performed by the plurality of client computing devices using the overall embedding and the current server model, the following is obtained from the plurality of client computing devices: The updated compressed embedding, which is compressed through clustering, and The updated clustering labels corresponding to the updated compressed embedding; Based on the updated compressed embedding and the corresponding updated clustering labels, the updated overall embedding is recreated; as well as Based on the updated overall embedding, the server model is trained locally to obtain an updated version of the server model; The local training performed by the plurality of client computing devices generates an updated local model, and the server and the plurality of client computing devices are configured collaboratively to perform joint machine learning inference using the updated version of the server model and the updated local model.

11. The joint learning system according to claim 10, wherein, The plurality of client computing devices are configured as follows: The output of the initial local model is determined as the initial uncompressed embedding for the current microbatch; The initial uncompressed embedding is compressed through the clustering to obtain the initial compressed embedding; The initial compressed embedding and initial clustering labels are sent to the server; Based on the initial overall embedding and the current server model, the initial local model is trained locally to obtain the updated local model; and Based on the updated local model, the updated compressed embedding and the corresponding updated clustering labels are sent to the server of the joint learning system.

12. The joint learning system according to claim 11, wherein, The server and the plurality of client computing devices are also configured in a coordinated manner to deploy information technology resources based on the inference.

13. The joint learning system according to claim 11, wherein, The server and the plurality of client computing devices are also configured to repeatedly obtain the initial compressed embedding and the initial clustering label, recreate the initial global embedding, send the current version of the server model, obtain the updated compressed embedding and the updated clustering label, recreate the updated global embedding, and train the server model locally to obtain the updated version of the server model, until convergence.

14. The joint learning system according to claim 13, wherein, The clustering includes k-means clustering, and wherein the server and the plurality of client computing devices are also cooperatively configured to reduce the number of clusters in the clustering when repeatedly obtaining the initial compressed embedding and the initial cluster labels, recreating the initial overall embedding, sending the current version of the server model, obtaining the updated compressed embedding and the updated cluster labels, recreating the updated overall embedding, and training the server model locally to obtain the updated version of the server model.

15. A computer program product comprising program instructions respectively for use on a plurality of client computing devices and a server coupled to the plurality of client computing devices, wherein, The program instructions used on the server include: Program instructions for obtaining the following from multiple client computing devices: The initial compressed embedding, which is compressed through clustering, includes the output of the initial local model for the current microbatch, and The initial clustering label corresponding to the initial compressed embedding; Program instructions for recreating the initial overall embedding based on the initial compressed embedding and the initial clustering labels obtained from the plurality of client computing devices; Program instructions for sending the current version of the server model and the initial overall embedding to each of the plurality of client computing devices; For obtaining the following program instructions from the plurality of client computing devices based on local training performed by the plurality of client computing devices using the overall embedding and the current server model: The updated compressed embedding, which is compressed through clustering, and The updated clustering labels corresponding to the updated compressed embedding; Program instructions for recreating an updated overall embedding based on the updated compressed embedding and the corresponding updated clustering labels; and Program instructions for training the server model locally based on the updated overall embedding to obtain an updated version of the server model; The local training performed by the plurality of client computing devices generates an updated local model, and the server and the plurality of client computing devices are configured collaboratively to perform joint machine learning inference using the updated version of the server model and the updated local model.

16. The computer program product according to claim 15, wherein, The program instructions for use on the plurality of client computing devices include: The program instructions used to determine the output of the initial local model as the initial uncompressed embedding for the current microbatch; Program instructions for compressing the initial uncompressed embedding through the clustering to obtain the initial compressed embedding; Program instructions for sending the initial compressed embedding and initial clustering labels to the server; Program instructions for training the initial local model locally based on the initial overall embedding and the current server model to obtain the updated local model; Program instructions for sending the updated compressed embedding and corresponding updated clustering labels to the server of the federated learning system based on the updated local model.

Images