Privacy-preserving asynchronous federated learning with vertical partitioned data

Abstract

A system and method for asynchronously training a federated learning model. The system includes a coordinator, an active computing device, and a passive computing device in communication with each other. The active computing device has a processor and a storage device storing computer executable code. The computer executable code is configured to train a federated learning model in the active computing device using dimensions of instances in the active computing device and instruct at least one passive computing device to train the federated learning model in the at least one passive computing device using dimensions of instances in the at least one passive computing device. The training instances in the active computing device and the training instances in the at least one passive computing device do not correspond to each other at a same training time.

Description

[0001] Cross-references

[0002] This application claims priority to U.S. Patent Application No. 16 / 910,238, filed June 24, 2020, entitled "Privacy-Preserving Asynchronous Federated Learning for Vertically Partitioned Data," the entirety of which is incorporated herein by reference.

[0003] References are cited and discussed in the description of this disclosure, which may include patents, patent applications, and various publications. The citations and / or discussions of such references are provided solely to clarify the description of this disclosure and do not imply that any such references constitute "prior art" as disclosed herein. All references cited and discussed in the specification are incorporated herein by reference in their entirety, to the same extent as each individual reference is incorporated by reference individually. Technical Field

[0004] This disclosure generally relates to federated learning, and more specifically, to a system and method for privacy-preserving asynchronous federated learning of vertically partitioned data. Background Technology

[0005] The background description provided herein is intended to provide a general overview of the context of this disclosure. Within the scope of this background description, the work of the currently named inventors and descriptions that might not have been considered prior art at the time of filing are not, expressly or impliedly, acknowledged as prior art to this disclosure.

[0006] Federated learning is a machine learning technique that allows algorithms to be trained on multiple distributed edge devices or on servers that store local data samples without exchanging data samples. However, maintaining adequate privacy is a challenge when devices frequently exchange data.

[0007] Therefore, there is a need in this field to address the aforementioned defects and shortcomings. Summary of the Invention

[0008] This disclosure relates to a system for asynchronously training a federated learning model. The system includes an active computing device, at least one passive computing device, and a coordinator that communicate with each other. Each of the active and passive computing devices has a federated learning model and local data. The active computing device includes a processor and a storage device storing computer-executable code. When executed at the processor, the computer-executable code is configured to: train the federated learning model in the active computing device using dimensions of a plurality of training instances in the active computing device; and instruct the at least one passive computing device to train the federated learning model in the at least one passive computing device using dimensions of the training instances in the at least one passive computing device. The training instances in the active and passive computing devices are indexed, such that the index of one training instance selected by the at least one passive computing device is independent of the index of one training instance selected by the active computing device at the same time, such that the training of the federated learning model in the active and passive computing devices is asynchronous. Note that the training instances in the active computing device are different from the training instances in the at least one passive computing device.

[0009] In some embodiments, the parameters of the federated learning model include a constant learning rate.

[0010] In some embodiments, intermediate results are transmitted between the active worker and the at least one passive worker via a tree-structured scheme through the coordinator. For example, the intermediate results may be dot products and adjusted dot products.

[0011] In some embodiments, the computer-executable code is configured to train a federated learning model in the active computing device through the following steps: selecting an i-th training instance in the active computing device, where i is the index of the training instance; calculating an active dot product between the current model weights in the active computing device and the i-th instance in the active computing device; instructing at least one passive computing device to calculate a passive dot product between the current model weights in the at least one passive computing device and the i-th training instance in the at least one passive computing device; summing the active dot product and the passive dot product to obtain a summed dot product; using the summed dot product to calculate the stochastic gradient of the federated learning model; and updating the model weights using the stochastic gradient and learning rate of the federated learning model. In some embodiments, the stochastic gradient is used... Calculated It is the regularization loss corresponding to the loss of the i-th instance. This is the gradient of the regularization loss with respect to the dimensions in the active computing device. Note that the i-th training instance in the active computing device is different from the i-th training instance in at least one passive computing device. The i-th training instance in the active computing device may include some dimensions of the instance, while the i-th training instance in at least one passive computing device may include some other dimensions of the instance.

[0012] In some embodiments, the computer-executable code is configured to train a federated learning model in the active computing device through the following steps: for each iteration in a first number of iterations: compute the fully local gradients of instances in the active worker; execute in parallel and independently in each of the active worker and the at least one passive worker; select one of the instances; compute the current dot product between the current model weights and the selected instance, and the reference dot product between the reference model weights and the selected instance; instruct the at least one passive computing device to compute the current gradient of the at least one passive computing device using the instance corresponding to the selected instance. The process involves: calculating the current dot product from the active computing device and the reference dot product from the at least one passive computing device; summing the current dot product from the active computing device and the current dot product from the at least one passive computing device to obtain a summed current dot product; summing the reference dot product from the active computing device and the reference dot product from the at least one passive computing device to obtain a summed reference dot product; calculating the stochastic gradient of the federated learning model using the summed current dot product and the summed reference dot product; updating the model weights using the stochastic gradient and learning rate of the federated learning model to obtain updated model weights; and defining the updated model weights as new reference model weights. In some embodiments, the stochastic gradient is used... Calculated It is the regularization loss corresponding to the loss of the i-th instance, f i (w s f(w) is the regularization loss corresponding to the loss of the i-th instance with respect to the reference model weights. s ) is the empirical risk of regularization. They are f(w s The gradient of the dimension in the active computing device. In some embodiments, summing the current dot product is performed using a first tree structure scheme, and summing the reference dot product is performed using a second tree structure scheme, which are significantly different from the first tree structure scheme.

[0013] In some embodiments, the computer-executable code is configured to train a federated learning model in the active computing device by: computing the local gradients of instances in the active workers; and executing in parallel and independently in each active worker and at least one passive worker: selecting one instance; computing the current dot product between the current model weights and the selected instance; instructing the at least one passive computing device to compute the current dot product of the at least one passive computing device using the instance corresponding to the selected instance; summing the current dot products from the active computing device and the current dot products from the at least one passive computing device to obtain a summed dot product; using the summed dot product to compute the stochastic gradient of the federated learning model; and updating the model weights using the stochastic gradient and learning rate of the federated learning model to obtain updated model weights, and updating the local gradients. In some embodiments, the stochastic gradient uses... Calculated It is the regularization loss corresponding to the loss of the i-th instance. yes The gradient with respect to the dimension of the i-th instance in the active computing device, It is the local gradient in the active computing device.

[0014] In some embodiments, the computer-executable code is configured to predict the value of an input instance after training the federated learning model by: calculating the dot product between the model weights and the input instance for each of the active worker and the at least one passive worker; and summing the dot products from the active worker and the at least one passive worker using tree-structured communication to obtain a summed dot product, wherein the summed dot product is the value predicted for the input instance.

[0015] In some embodiments, the computer-executable code is configured to predict the value of an input instance after training the federated learning model by the following steps: for each of the active worker and the passive worker: generating a random number; calculating the dot product between the model weights and the input instance; adding the random number to the dot product to obtain an adjusted dot product; summing the adjusted dot products from the active worker and the at least one passive worker using a first tree structure communication to obtain a summed adjusted dot product; summing the random number from the active worker and the at least one passive worker using a second tree structure communication to obtain a summed random number; and subtracting the summed random number from the summed adjusted dot product to obtain a summed dot product, wherein the summed dot product is a predicted value for the input instance.

[0016] In some aspects, this disclosure relates to a method. In some embodiments, the method includes: training a federated learning model in an active computing device using dimensions of a plurality of training instances in the active computing device; and instructing at least one passive computing device to train the federated learning model in the at least one passive computing device using dimensions of the plurality of training instances in the at least one passive computing device, wherein the active computing device and the at least one passive computing device communicate with each other to index the plurality of training instances in the active computing device and the plurality of training instances in the at least one passive computing device, wherein the index of one training instance among the plurality of training instances selected by the at least one passive computing device is independent of the index of one training instance among the plurality of training instances selected by the active computing device at the same time, such that the training of the federated learning model in the active computing device and the training of the federated learning model in the at least one passive computing device are asynchronous.

[0017] In some embodiments, the parameters of the federated learning model include a constant learning rate, and intermediate results are transferred between the active worker and the at least one passive worker using a tree-structured scheme via a coordinator that communicates with the active worker and the at least one passive worker.

[0018] In some embodiments, training the federated learning model in the active computing device includes: selecting an i-th training instance in the active computing device, where i is the index of the training instance; calculating an active dot product between the current model weights in the active computing device and the i-th training instance in the active computing device; instructing at least one passive computing device to calculate a passive dot product between the current model weights in the at least one passive computing device and the i-th instance in the at least one passive computing device; summing the active dot product and the passive dot product to obtain a summed dot product; using the summed dot product to calculate the stochastic gradient of the federated learning model; and updating the model weights using the stochastic gradient and the learning rate of the federated learning model. In some embodiments, the stochastic gradient is used... Calculated It is the regularization loss corresponding to the loss of the i-th instance. It is the gradient of the regularization loss of the i-th instance with respect to the dimension in the active computing device.

[0019] In some embodiments, training the federated learning model in the active computing device includes: for each iteration in a first number of iterations: computing the fully local gradients of instances in the active worker; performing in parallel and independently in each of the active worker and the at least one passive worker: selecting one instance from the instances; computing the current dot product between the current model weights and the selected instance, and the reference dot product between the reference model weights and the selected instance; instructing the at least one passive computing device to use the instance corresponding to the selected instance to compute the current dot product of the at least one passive computing device and the reference dot product of the at least one passive computing device; summing the current dot products from the active computing device and the current dot products from the at least one passive computing device to obtain a summed current dot product, summing the reference dot products from the active computing device and the reference dot products from the at least one passive computing device to obtain a summed reference dot product; computing the stochastic gradient of the federated learning model using the summed current dot product and the summed reference dot product; updating the model weights using the stochastic gradient and learning rate of the federated learning model to obtain updated model weights; and defining the updated model weights as new reference model weights. In some embodiments, the stochastic gradient is used Calculated It is the regularization loss corresponding to the loss of the i-th instance, f i (w s f(w) is the regularization loss corresponding to the loss of the i-th instance with respect to the reference model weights. s ) is the regularization empirical risk. They are Regarding the gradient of dimensions in the active computing device. In some embodiments, summing the current dot product is performed using a first tree structure scheme, and summing the reference dot product is performed using a second tree structure scheme, which are significantly different from the first tree structure scheme.

[0020] In some embodiments, training the federated learning model in the active computing device includes: computing the local gradients of instances in the active computing device;

[0021] Each of the active worker and the at least one passive worker executes in parallel and independently: selecting an instance from the instances; calculating the current dot product between the current model weights and the selected instance; instructing the at least one passive computing device to use the instance corresponding to the selected instance to calculate the current dot product of the at least one passive computing device; summing the current dot products from the active computing device and the current dot products from the at least one passive computing device to obtain a summed dot product; using the summed dot product to calculate the stochastic gradient of the federated learning model; and updating the model weights using the stochastic gradient and learning rate of the federated learning model to obtain updated model weights, and updating the local gradient. In some embodiments, the stochastic gradient is used... Calculated It is the regularization loss corresponding to the loss of the i-th instance. yes The gradient with respect to the dimension of the i-th instance in the active computing device, It is the local gradient in the active computing device.

[0022] In some aspects, this disclosure relates to a non-transitory computer-readable medium storing computer-executable code. The computer-executable code, when executed at a processor of a computing device, is configured to perform the methods described above.

[0023] These and other aspects of this disclosure will become apparent from the following description of preferred embodiments taken in conjunction with the accompanying drawings and description, although variations and modifications therein may affect the novel conception and scope of this disclosure without departing from it. Attached Figure Description

[0024] The accompanying drawings illustrate one or more embodiments of this disclosure and, together with the written description, serve to explain the principles of this disclosure. Where possible, the same reference numerals are used throughout the drawings to refer to the same or similar elements of the embodiments.

[0025] Figure 1A Synchronous computation is illustrated schematically.

[0026] Figure 1B Asynchronous computation according to certain embodiments of this disclosure is schematically depicted.

[0027] Figure 2 An asynchronous federated learning system for vertically partitioned data according to certain embodiments of this disclosure is schematically depicted.

[0028] Figure 3A A tree structure according to certain embodiments of the present disclosure is schematically depicted.

[0029] Figure 3B Another tree structure according to certain embodiments of the present disclosure is schematically depicted.

[0030] Figure 4A A system for asynchronous federated learning according to certain embodiments of the present disclosure is schematically depicted.

[0031] Figure 4B A worker for asynchronous federated learning according to certain embodiments of this disclosure is schematically depicted.

[0032] Figure 5A An AFSGD application for asynchronous federated learning according to certain embodiments of the present disclosure is illustrated schematically.

[0033] Figure 5B An AFSVRG application for asynchronous federated learning according to certain embodiments of the present disclosure is illustrated schematically.

[0034] Figure 5C An AFSAGA application for asynchronous federated learning according to certain embodiments of the present disclosure is illustrated schematically.

[0035] Figure 6A An AFSGD training process for asynchronous federated learning according to certain embodiments of the present disclosure is schematically depicted.

[0036] Figure 6B An AFSVRG training process for asynchronous federated learning according to certain embodiments of this disclosure is schematically depicted.

[0037] Figure 6C An AFSAGA training process for asynchronous federated learning according to certain embodiments of the present disclosure is schematically depicted.

[0038] Figure 7A A flow for calculating predicted values ​​for a sample according to certain embodiments of the present disclosure is schematically depicted.

[0039] Figure 7B A flow for calculating predicted values ​​for a sample according to certain embodiments of the present disclosure is schematically depicted.

[0040] Figure 8 The dataset used in experiments according to certain embodiments of this disclosure is shown.

[0041] Figure 9A and 9B The convergence of different algorithms for classification tasks according to certain embodiments of this disclosure is schematically depicted.

[0042] Figure 9C and 9DThe convergence of different algorithms for regression tasks according to certain embodiments of this disclosure is schematically depicted.

[0043] Figures 10A-10D The convergence of different algorithms for binary classification tasks on larger datasets according to certain embodiments of this disclosure is schematically depicted.

[0044] Figure 11 The speedup results of asynchronous algorithms compared to synchronous algorithms according to certain embodiments of the present disclosure are summarized.

[0045] Figure 12 The asynchronous efficiency according to certain embodiments of the present disclosure is illustrated schematically, wherein eight workers operate on a URL dataset for a binary classification task.

[0046] Figures 13A-13C The scalability of certain embodiments of this disclosure is illustrated, wherein the scalability is evaluated on a URL dataset used for a classification task.

[0047] Overview of this disclosure

[0048] In some embodiments, the model uses symbols and equations defined as follows:

[0049] SGD is an abbreviation for Stochastic Gradient Descent. SVRG is an abbreviation for Stochastic Variance Reduced Gradient. SAGA is a fast incremental gradient method proposed by Aaron Defazio and Francis Bach in 2014. VP is an abbreviation for Vertically Partitioned or Vertically Partitioned Data, a common data distribution in federated learning. AFSGD-VP is an abbreviation for Asynchronous Federated SGD algorithm for vertically partitioned data, AFSVRG-VP is an abbreviation for Asynchronous Federated SVRG algorithm for vertically partitioned data, and AFSAGA-VP is an abbreviation for Asynchronous Federated SAGA algorithm for vertically partitioned data.

[0050] x represents a data instance, which can be used in all workers (computing devices) or servers in federated learning. i It is one of the instances indexed by i, where i is selected from 1 to n, and n is a positive integer. n represents the total number of instances. It is an instance in the local worker l. Each instance xi There are d dimensions, each representing a feature of the instance. Instance x i These can be distributed across different workers, each of which can contain certain features of instances corresponding to certain dimensions. y represents the label of the instance. i It is instance x i The label is used to represent the values ​​of y. For binary problems, each y value can be +1 or -1; for regression problems, y can be a real number. S is a set of instance data x. i and its corresponding label y i The sample set, the sample set can be represented as D l It is the local data of the l-th worker, with the same size as the sample set S.

[0051] For real numbers, Let d represent the dimension, where each dimension has a real number value.

[0052] w is the model weight or model coefficient of the model in this disclosure. The dimension of the model weight w corresponds to the dimension of the instance x. In other words, when the instance has d dimensions, the model weight w also has d dimensions. The model can be, for example, a linear model or any other type of model depending on the problem to be solved. These are the model weights of the l-th worker. The model weights are read inconsistently from all workers, from any two different workers. They may be in different local iteration stages. This relates to asynchronous training in this disclosure. This represents a snapshot of the model weights after a certain number of iterations. In some embodiments, the d dimensions of instance x and the d dimensions of model weights w are in vector form.

[0053] w T x is an exemplary linear model, and is the dot product between the model weights w and the instance x. T x i The model weights and the instance x with index i are... i The dot product between them.

[0054] L(w T x i y i ) is a linear model w T The loss function of x, where the loss function represents the dot product w. T x i With instance x i The tag y i The differences between them.

[0055] f(w) can be obtained from Defined regularization empirical risk, where g(w) is the regularization term and f i f(w) is the regularization loss function. In some embodiments, f(w) is also referred to as the objective of the training task.

[0056] Indicates about instance x i stochastic gradient, Indicates about instance x i Historical stochastic gradient, This represents the historical gradient. It is the stochastic gradient corresponding to the feature of the l-th worker.

[0057] v represents the unbiased stochastic gradient. In Algorithm 3, In Algorithm 4, In Algorithm 5.

[0058] α i yes The latest historical gradient. Let q represent the number of workers. Let n be a positive integer representing the number of instances. γ is the learning rate of the model. b is a random number in the range [0, 2π], which can be generated by a random number generator. l These are random numbers generated on the l-th worker. T0, T1, and T2 are tree-structured communication schemes. Leaf(·) represents all the leaves of a tree.

[0059] Privacy-preserving federated learning for vertically partitioned data has shown promising results as an emerging solution for multi-party federated modeling applications, where data holders (such as government agencies, privacy finance companies, and e-commerce companies) collaborate throughout the learning process, rather than relying on a trusted third party to hold the data. However, existing federated learning algorithms for vertically partitioned data are limited to synchronous computation.

[0060] In federated learning systems, imbalances in computational / communication resources among parties are common, and synchronous computation often encounters problems. To improve computational efficiency, this disclosure provides, in some aspects, an asynchronous training algorithm for vertically partitioned data while maintaining data privacy. In some embodiments, the provided algorithms, methods, and systems include asynchronous federated SGD algorithms on vertically partitioned data (AFSGD-VP) and their SVRG and SAGA variants. Furthermore, this disclosure provides convergence analysis of AFSGD-VP and its SVRG and SAGA variants under strong convexity conditions. This disclosure also describes their model privacy, data privacy, computational complexity, and communication costs. AFSGD-VP and its SVRG and SAGA variants are the first asynchronous federated learning algorithms for vertically partitioned data.

[0061] Figure 1A and Figure 1B The advantages of using asynchronous computation according to certain embodiments of this disclosure are illustrated schematically. For example... Figure 1A As shown, during synchronous computation, workers 1-4 need to be coordinated at predetermined time intervals. Within each time interval of workers 1-4, the computation may include computation time and idle time. Conversely, as... Figure 1B As shown, no coordination is required between workers 1 to 4, and the computations within workers 1 to 4 are continuous. When one worker completes a task, it will begin the next task without any idle time. Therefore, because asynchronous computation keeps all computing resources constantly busy, Figure 1B The asynchronous computation ratio shown Figure 1A The synchronous computation shown is much more efficient. In some respects, this disclosure uses a method similar to SGD to asynchronously train data on vertical partitions, a fact not disclosed by anyone else in the art.

[0062] Contributions to this disclosure include, for example: (1) This disclosure provides an asynchronous federated stochastic gradient algorithm (AFSGD-VP) for vertically partitioned data and proves the convergence speed of AFSGD-VP under strong convexity conditions; (2) This disclosure provides an asynchronous federated SVRG algorithm (AFSVRG-VP) for vertically partitioned data and proves the convergence speed of AFSVRG-VP under strong convexity conditions; (3) This disclosure provides an asynchronous federated SAGA algorithm (AFSAGA-VP) for vertically partitioned data and proves the convergence speed of AFSAGA-VP under strong convexity conditions.

[0063] Problem Statement. The exemplary problems described below are for illustrative purposes only and are not intended to limit the scope and form of this disclosure. In some embodiments, this disclosure uses w TThe linear form of x is considered in the model. However, other model forms are also possible depending on the problem to be solved. Given a training set... there will be and y i ∈{+1, -1} is used for binary classification or Used for regression. Regarding the sample (x) i y i The loss function for the model weights w can be expressed as L(w) T x i y i Therefore, this disclosure considers optimizing the following regularization empirical risk minimization problem:

[0064]

[0065] In this disclosure, g(w) is a regularization term, and each f i : All are considered to be smooth and possibly nonconvex functions. Clearly, the empirical risk minimization problem is a special case of problem (1). Besides the empirical risk minimization problem, problem (1) also summarizes a large number of important regularization learning problems, such as l2 regularized logistic regression, ridge regression, and least-squares support-vector machine (LS-SVM). As mentioned earlier, in many real-world machine learning applications, the input of the training sample (x, y) is vertically partitioned into q parts, i.e., partitions with d features as disclosed in this disclosure. Therefore, this disclosure has in Stored on the l-th worker. Based on whether the labels are included in the worker, this disclosure categorizes workers into two types: active workers and passive workers. Active workers are data providers holding sample labels, while passive workers only have sample input. Active workers will act as the master server in federated learning, while passive workers will play the role of the client. This disclosure enables D... l This represents the data stored on the l-th worker. Note the label y. i Distributed across active workers. The objective of this disclosure can be expressed as: enabling active workers to cooperate with passive workers, thereby resolving vertically partitioned data in parallel and asynchronously with SGD and its SVRG and SAGA variants while maintaining the privacy of the vertically partitioned data. The problem of minimizing regularized empirical risk on (1).

[0066] SGD, SVRG, and SAGA. SGD-like algorithms have been popular for solving large-scale machine learning problems. The updated framework for SGD-like algorithms includes various variants of variance reduction methods. Specifically, given an unbiased stochastic gradient v (i.e., ... The update rule of an SGD-like algorithm can be expressed as:

[0067] w←w-γv (2),

[0068] Where w is the model weight, γ is the learning rate, and v is the stochastic gradient. The specific forms of the unbiased stochastic gradient v for SGD, SVRG, and SAGA are described below.

[0069] SGD: In each iteration, SGD (Bottou, 2010) independently processes the sample (x) i y i Sampling is performed using the sample (x) i y i stochastic gradient Update the solution as follows:

[0070]

[0071] in It is f i The stochastic gradient of (w).

[0072] SVRG: For SVRG (Gu et al., 2018; Xiao et al., 2014), stochastic gradients were not used directly. Instead, the solution is updated using an unbiased stochastic gradient v, as shown below:

[0073]

[0074] in This represents a snapshot of w after a certain number of iterations.

[0075] SAGA: For SAGA (Defazio et al., 2014), the formula for the unbiased stochastic gradient v is as follows:

[0076]

[0077] Where α i yes The latest historical gradient can be updated online. In some embodiments, α i It is dynamic and will be updated during the model's training process. In contrast, history... It is a static reference gradient.

[0078] The system architecture disclosed herein. As described above, AFSGD-VP, AFSVRG-VP, and AFSAGA-VP are privacy-preserving asynchronous federated learning algorithms for vertically partitioned data. Figure 2 A privacy-preserving asynchronous federated learning system according to certain embodiments of this disclosure is schematically depicted. For example... Figure 2 As shown, the system uses a tree-structured communication method, which achieves model privacy and data privacy.

[0079] Tree-structured communication. To obtain w T x i This disclosure requires accumulating local results from different workers. Zhang et al., 2018, proposed an efficient tree-structured communication scheme to obtain the global sum, which is faster than the simple strategy of directly sending the results of all workers to the coordinator for summation. Figure 3A and Figure 3B Two different tree-structured communications according to certain embodiments of this disclosure are schematically depicted. Taking four workers as an example, such as... Figure 3A As shown, this disclosure pairs workers so that when worker 1 adds the result of worker 2, worker 3 can simultaneously add the result of worker 4. Finally, the results of the two pairs of workers are sent to the coordinator, and this disclosure obtains the global sum. In some embodiments, this disclosure uses a tree-structured communication scheme to obtain w T x i It should be noted that tree-structured communication schemes are used in conjunction with asynchronous modes to obtain w T x i Therefore, this disclosure does not include data from different processors. Align the number of iterations to compute w T x i This is very different from the synchronization mode used in Zhang; all of Zhang's... They all have the same number of iterations.

[0080] Based on the tree-structured communication scheme, the following algorithm 1 is calculated on the l-th active worker. The basic algorithm. In some embodiments, after the model is trained, Algorithm 1 can be used on the l-th active worker for a new instance x. i Making predictions. In some embodiments, as shown in Algorithm 1, each worker has certain features or dimensions of an instance, and correspondingly certain dimensions of model weights corresponding to the features or dimensions of the instance stored in that worker. In some embodiments, the tree-structured communication in Algorithm 1 can use any form of tree structure.

[0081]

[0082] Data privacy and model privacy. To maintain the privacy of vertically partitioned data and models, this disclosure will protect data privacy. and model weights Each data point is stored privately in the first worker. This disclosure does not include local data. and local model weights Directly transmitted to other workers. To obtain w T x i This disclosure is calculated locally. And only Transferred to other workers for calculating w T x, as shown in Algorithm 1. According to Values ​​inferred from local model coefficients and local instance This is no easy task. Therefore, this disclosure achieves data privacy and model privacy.

[0083] Although according to The value accurately infers the local model coefficients and local instance This is not an easy task, and it carries the risk of approximate inference attacks. The risk is higher when the dimensionality of the instance and model weights is small. To address this issue, this disclosure employs a more secure algorithm to compute the values ​​in Algorithm 2. Specifically, this disclosure will use random number b l′ Add to dot product The adjusted dot product is obtained in the first step, and then the sum of the adjusted dot products is calculated using a tree-structured communication scheme on tree structure T1. This can improve delivery The data and model security of the value operations. Finally, this disclosure needs to address the issue of data and model security. China Resumption The value of b. To prevent leakage of subtree T1's b... l′ Any sum, this disclosure calculates using a significantly different tree structure T2 on all workers. Using two significantly different tree structures to convey the adjusted dot product and enhanced randomness enhances data security. In some embodiments, the two tree structures T1 and T2 may or may not be, provided they are significantly different from each other. Figure 3A and Figure 3B The tree structure shown.

[0084] In some embodiments, a significant difference is defined as tree structures T1 and T2 not having a common subtree. Specifically, for two tree structures T1 and T2 on all workers {1, ..., q}, if no subtree of T1 exists... subtree of T2 These two subtrees have dimensions greater than 1 and less than T1 and T2, such that These two tree structures are significantly different.

[0085]

[0086]

[0087] In some embodiments, during training, Algorithm 1 or Algorithm 2 can be invoked via Algorithms 3-5. After the model is trained, both Algorithm 1 and Algorithm 2 can be used on the l-th active worker for new instance x. i Make predictions. It's important to note that Algorithm 1 typically offers sufficient data privacy, especially when the instance dimensionality is high. When the instance dimensionality is low, such as less than 5 dimensions, or only 1 or 2 dimensions, Algorithm 2 is preferred for predictions to maintain a high level of model security.

[0088] Algorithms and methods for applying the algorithms. In some embodiments, this disclosure provides three asynchronous federated stochastic gradient algorithms, AFSGD-VP, AFSVRG-VP, and AFSAGA-VP, and uses them to process federated data. In some embodiments, AFSVRG-VP is the preferred algorithm.

[0089] AFSGD-VP. AFSGD-VP repeats the following four steps simultaneously for each worker without encryption. (1) Selecting an index: AFSGD-VP randomly selects index i from {1, ..., n} and selects it from local data D. l Get local instance (2) Calculation AF SGD-VP uses a tree-structured communication scheme with asynchronous modes (i.e., Algorithm 1 or 2) to obtain... in This indicates that w is read inconsistently from different workers, two from different workers. It may be in different local iteration stages. It should be noted that this disclosure always has the current... (3) Calculate the stochastic local gradient: based on This disclosure enables the computation of unbiased stochastic local gradients. (4) Update: AF SGD-VP passed Update local model weights Where γ is the learning rate. The application of the AFSGD-VP algorithm is summarized in Algorithm 3 below.

[0090]

[0091]

[0092] AFSVRG-VP. Due to the random sampling similar to the SGD algorithm, the stochastic gradient in AFSGD-VP has a large variance. In order to handle the large variance, the AFSVRG-VP of this disclosure utilizes the SVRG technique to reduce the variance of the stochastic gradient and proposes a fast AFSGD-VP algorithm (i.e., AFSVRG-VP). Algorithm 4 summarizes the AFSVRG-VP algorithm according to some embodiments of this disclosure. Compared with AFSGD-VP, AFSVRG-VP has the following three differences: (1) The first is that AFSVRG-VP calculates the full local gradient in the outer loop. Use it as a snapshot of the full gradient, where the superscript s denotes the s-th output loop. (2) The second is that this disclosure not only calculates Also calculate (w) for each iteration s ) f x i (3) The third is that AFSVRG-VP calculates the unbiased stochastic local gradient as Instead

[0093]

[0094] AFSAGA-VP. As mentioned above, the stochastic gradient in SGD has a large variance due to random sampling. To handle this large variance, the AFSAGA-VP of this disclosure uses the SAGA technique to reduce the variance of the stochastic gradient. This disclosure summarizes the AFSAGA-VP algorithm in Algorithm 5. Specifically, this disclosure maintains a latest historical local gradient. The table, which is accessed via... The update rules for each iteration are implemented based on the latest historical local gradient table. The unbiased stochastic local gradient in AFSAGA-VP is calculated as follows:

[0095]

[0096] Theoretical Analysis. This disclosure provides convergence, security, and complexity analyses for AFSGD-VP, AFSSVRG-VP, and AFSAGA-VP.

[0097] Convergence Analysis. This disclosure first presents the assumptions of strong convexity, (block coordinate) Lipschitz smoothness, and bounded gradient in block coordinates, which are the standards for convex analysis. Then, for AFSGD-VP, this disclosure proves that the convergence rate is... To achieve accuracy ∈. For AFSVRG-VP and AFSAGA-VP, this disclosure proves that the convergence rate is .

[0098] Security Analysis. This disclosure demonstrates that the AFSGD-VP, AFSVRG-VP, and AFSAGA-VP algorithms can prevent exact inference attacks (see Definition 1) and approximate inference attacks (see Definition 2) under the semi-honesty assumption.

[0099] Definition 1 (Exact Inference Attack). An exact inference attack on the l-th worker is an attack that accurately infers certain feature groups of a sample x or model w belonging to another worker without direct access.

[0100] Definition 2 (∈-approximation inference attack). An ∈-approximation inference attack on the l-th worker is an attack that, without direct access, identifies certain feature groups of a sample x (model w) belonging to another worker. Inferred as With accuracy ∈ (i.e. or ).

[0101] The assumption of semi-honesty (semi-honesty safety). All workers will follow the algorithm to perform correct calculations. However, it may retain records of intermediate calculation results that may be used later to infer data and models for other workers.

[0102] Complexity Analysis. This disclosure presents the computational complexity and communication cost of AFSGD-VP, AFSSVRG-VP, and AFSAGA-VP, as described below.

[0103] The computational complexity of one iteration of AFSGD-VP is O(d+q). Therefore, the total computational complexity of AFSGD-VP is O((d+q)t), where t represents the number of iterations. Furthermore, the communication cost of one iteration of AFSGD-VP is O(q), and the total communication cost is O(qt).

[0104] For AFSVRG-VP, the computational complexity and communication cost of line 3 in Algorithm 4 are O((d+q)l) and O(ql), respectively. Assuming the number of inner loops in AFSVRG-VP is t, the total computational complexity of AFSVRG-VP is O((d+q)(l+t)S), and the communication cost is O(q(l+t)S).

[0105] For AFSAGA-VP, the computational complexity and communication cost of line 2 in Algorithm 5 are O((d+q)l) and O(ql), respectively. Assuming the number of loops in AFSAGA-VP is t, the total computational complexity of AFSAGA-VP is O((d+q)(l+t)), and the communication cost is O(q(l+t)). Detailed Implementation

[0106] The present disclosure is described in more detail in the following examples, which are intended to be illustrative only, as many modifications and variations therein will be apparent to those skilled in the art. Various embodiments of the present disclosure are now described in detail. Referring to the accompanying drawings, throughout the views, the same numerals indicate the same parts. Unless the context clearly specifies otherwise, the terms “a,” “an,” and “the” as used herein and throughout the claims have the meaning of the plural. Furthermore, as used in the description and claims of this disclosure, unless the context clearly specifies otherwise, “in” has the meaning of “in” and “on”. Additionally, headings or subheadings may be used in the specification for the reader's convenience, but these do not affect the scope of the present disclosure. Furthermore, some terms used in this specification are given more specific definitions below.

[0107] The terms used in this specification generally have their ordinary meaning in the art, in the context of this disclosure, and in the specific context in which each term is used. Certain terms used to describe this disclosure are discussed below or elsewhere in the specification to provide practitioners with additional guidance regarding the description of this disclosure. It will be understood that the same thing can be expressed in more than one way. Therefore, alternative language and synonyms may be used for any one or more terms discussed herein, and have no particular significance in whether a term is elaborated or discussed herein. This disclosure provides synonyms for certain terms. The use of one or more synonyms does not preclude the use of other synonyms. The use of examples anywhere in this specification, including examples of any terms discussed herein, is merely illustrative and in no way limits the scope and meaning of this disclosure or any exemplary terms. Likewise, this disclosure is not limited to the various embodiments given in this specification.

[0108] As described herein, the term "module" can refer to or include Application Specific Integrated Circuit (ASIC); electronic circuitry; combinational logic circuitry; Field Programmable Gate Array (FPGA); processor (shared, dedicated, or grouped) that executes code; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-a-chip. The term "module" can include memory (shared, dedicated, or grouped) that stores code executed by the processor.

[0109] As described herein, the term "code" can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, and / or objects. The term "shared" as used above means that some or all of the code from multiple modules can be executed using a single (shared) processor. Furthermore, some or all of the code from multiple modules can be stored in a single (shared) memory. The term "group" as used above means that some or all of the code from a single module can be executed using a group of processors. Furthermore, a group of memories can be used to store some or all of the code from a single module.

[0110] As described herein, the term "interface" generally refers to a communication tool or device used at the interaction point between components to perform data communication between components. Generally, interfaces can be applied at both the hardware and software levels, and can be unidirectional or bidirectional. Examples of physical hardware interfaces can include electrical connectors, buses, ports, cables, terminals, and other I / O devices or components. Components communicating with the interface can be, for example, multiple components of a computer system or peripheral devices.

[0111] This disclosure relates to computer systems. As illustrated in the accompanying drawings, computer components may include physical hardware components as shown in solid boxes and virtual software components as shown in dashed boxes. Those skilled in the art will understand that, unless otherwise stated, these computer components may be implemented as software, firmware, or hardware components or combinations thereof, but are not limited to these forms. The apparatuses, systems, and methods described herein may be implemented by one or more computer programs executed by one or more processors. Computer programs include processor-executable instructions stored on a non-transitory tangible computer-readable medium. Computer programs may also include stored data. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory, magnetic storage, and optical storage.

[0112] This disclosure will now be described more fully below with reference to the accompanying drawings, in which embodiments of the disclosure are illustrated. However, this disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the scope of the disclosure to those skilled in the art.

[0113] Figure 4A A system for federated learning according to certain embodiments of this disclosure is schematically depicted. Figure 4A As shown, system 400 includes a coordinator 410, a network 430, and multiple workers 450. The coordinator 410 and the multiple workers 450 communicate with each other via the network 430. In some embodiments, each coordinator 410 and worker 450 shown in FIG. 4 can be a server computer, cluster, cloud computer, general-purpose computer, headless computer, or special-purpose computer providing federated learning capabilities. In some embodiments, each coordinator 410 and worker 450 is a server computing device. In some embodiments, the coordinator 410 is configured to collect certain data from both active and passive workers 450 upon receiving a request from an active worker 450. The data collected by the coordinator 410 may include, for example, the model weights w of each worker and the dot product between instances. In some embodiments, the coordinator 410 is configured to use one of a variety of tree-structured communications to collect data from the workers 450. In some implementations, some or all of the workers 450 include a model for federated learning, and each worker 450 may have its own privacy data. Worker 450 can be an active worker or a passive worker. When worker 450 initiates federated learning, it acts as an active worker. Active worker 450 can request coordinator 410 to collect certain data from workers or notify other workers to perform certain functions. Other workers are referred to as passive workers. Active and passive workers 450 can work together to train the federated learning model, but privacy data is protected within the respective workers and is not shared with other workers; each worker can have its own model weights. In some embodiments, the federated learning model is an asynchronous federated (AF) learning model, and the privacy data is vertically partitioned data. Network 430 can be a wired or wireless network, and can take various forms such as public networks and private networks. Examples of network 430 can include, but are not limited to, a local area network (LAN) or a wide area network (WAN) including the Internet. In some embodiments, two or more different networks 430 can be used to connect coordinator 410 and worker 450.

[0114] Figure 4B A worker 450 according to certain embodiments of the present disclosure is schematically depicted. In some embodiments, worker 450 is a server computing device and functions as an active worker. However, worker 450 can be any of workers 450-1 to 450-q, because each worker can initialize AF training and operate as an active worker, and each worker can operate as a passive worker and receive instructions from the active worker. Figure 4B As shown, worker 450 may include, but is not limited to, processor 452, memory 454, and storage device 456. In some embodiments, worker 450 may include other hardware and software components (not shown) to perform its respective tasks. Examples of such hardware and software components may include, but are not limited to, other required memory, interfaces, buses, input / output (I / O) modules or devices, network interfaces, and peripherals.

[0115] Processor 452 may be a central processing unit (CPU) configured to control the operation of worker 450. Processor 452 may execute the operating system (OS) or other applications of worker 450. In some embodiments, worker 450 may have multiple CPUs as processors, such as two CPUs, four CPUs, eight CPUs, or any suitable number of CPUs.

[0116] Memory 454 may be volatile memory, such as random-access memory (RAM), used to store data and information during operation of worker 450. In some embodiments, memory 454 may be an array of volatile memory. In some embodiments, worker 450 may operate on multiple memories 454. In some embodiments, worker 450 may also include a graphics card to assist processor 452 and memory 454 in image processing and display.

[0117] Storage device 456 is a non-volatile data storage medium used to store the operating system (not shown) and other applications of worker 450. Examples of storage device 456 may include non-volatile memory such as flash memory, memory card, USB drive, hard disk drive, floppy disk, optical drive, solid-state drive, or any other type of data storage device. In some embodiments, worker 450 may have multiple storage devices 456, which may be the same type of storage device or different types of storage devices. Applications of worker 450 may be stored in one or more storage devices 456 within worker 450.

[0118] In this embodiment, processor 452, memory 454, and storage device 456 are components of worker 450 (e.g., a server computing device). In other embodiments, worker 450 may be a distributed computing device, where processor 452, memory 454, and storage device 456 are shared resources from multiple computing devices in a predefined region.

[0119] In addition, storage device 456 includes an asynchronous federated (AF) application 458 and privacy data 472. The AF application 458, when executed, is configured to train an AF model or use the AF model for prediction. In some embodiments, the AF model is a linear model. In some embodiments, the AF application 458 may be... Figure 5A The AFSGD application shown is 558A. Figure 5B The AFSVRG application shown is 558B and Figure 5C Any of the AFSAGA applications 558C shown. In some embodiments, each of the applications 558A, 558B, and 558C is called a model, which includes model weights that can be trained using training data, and the model can make predictions using the trained model weights. Privacy data 472 is specific to each worker, but different workers can have the same index for an instance. For example, an instance x i It can have 100 dimensions of features, and this instance x i The index is i. This instance x i Instances can be stored across three different workers (450), each with the same index. The first worker stores the first 20 dimensions of the instance, the second worker stores the remaining 70 dimensions, and the third worker stores the final 10 dimensions. These 100 dimensions can then be linked together using the same index (i). i Each dimension of an instance can correspond to a feature of the instance, and each feature can be represented by a vector in the corresponding dimension.

[0120] Figure 5A An AFSGD application 558A according to certain embodiments of this disclosure is schematically depicted. For example... Figure 5AAs shown, the AFSGD application 558A includes a model weight initialization module 560A, a sampling module 562A, a local dot product module 564A, a dot product summation module 566A, a stochastic gradient module 568A, and an update module 570A. In some embodiments, the AFSGD application 558A may include other applications or modules necessary for the operation of the AFSGD application 558A, such as a listener configured to receive notifications or instructions from the coordinator 410 or other workers 450. It should be noted that modules 560A to 570A are all implemented by computer-executable code or instructions, or data tables or databases, or a combination of hardware and software, which together constitute an application. In some embodiments, each module may also include sub-modules. Alternatively, some modules may be combined into a stack. In other embodiments, some modules may be implemented as circuits rather than executable code. In some embodiments, the modules may also be collectively referred to as a model, which can be trained using training data and, after training, can be used for prediction. In some embodiments, the functionality of the AFSGD application 558A corresponds to the algorithm 3 described above.

[0121] The model weight initialization module 560A is configured to initialize the model weights for the AFSGD application 558A (or the AFSGD model, which can be a linear model). The model weights are then provided to the local dot product module 564A for initialization. In some embodiments, the dimension of the model weights w is the same as the dimension of the instance x. In some embodiments, each dimension of the model weights w is a real number. During initialization, the model weight initialization module 560A can provide a random value, a value of 0, or any other arbitrary value for each dimension of the model weights. During training using training samples (instances and their corresponding labels), the model weights can be updated and stored in the AFSGD application 558A. During prediction, the latest model weights can be used to make predictions based on instances. In some embodiments, during or after initialization, the AFSGD model of the active worker is also configured to instruct the passive workers to initialize their respective model weights and run their respective training. In some embodiments, the model weight initialization module 560A is also configured to instruct the sampling module 562A to begin sampling instances.

[0122] The sampling module 562A is configured to, upon receiving an instruction from the model weight initialization module 560A, randomly select index i from {1, ..., n} and sample data from local data D. l Get local instance The local instance is then sent to the local dot product module 564A. Here, n is the number of instances in the training data used to train the AFSGD application 558A, and i is the index of the instance. Each sample includes an instance. and its corresponding label yi The tag y i It can be used in other modules, such as the stochastic gradient module 568A for obtaining stochastic gradients.

[0123] The local area module 564A is configured to, upon receiving the initialization model weights from the model weight initialization module 560A... and local instances from sampling module 562A At that time, calculate the local dot product between the initialized model weights and the local instance. The local dot product is then sent to the dot product summing module 566A. In some embodiments, the AFSGD application 558 will have updated model weights after one round of training. Subsequent rounds of training, such as training with a second randomly selected local instance or subsequent instances, will use the updated model weights instead of the initial weights.

[0124] The dot product summation module 566A is configured to, upon receiving a local dot product, request the coordinator 410 to calculate the product corresponding to instance x. i The sum of the dot products In response to the request, the coordinator 410 is configured to use tree-structured communication to sum the dot products from all workers to obtain the result corresponding to instance x. i The dot product of the summations is calculated and provided to the stochastic gradient module 568A. The tree structure can be T1, T2, or any other type. In some embodiments, the dot product from each worker is calculated using the latest model weights of that worker.

[0125] The stochastic gradient module 568A is configured to, upon receiving the summation of the dot product... At that time, the stochastic gradient is calculated based on the summation of the dot product. The stochastic gradient is then sent to the update module 570A. In some embodiments, the stochastic gradient can be calculated based on equations (1) and (3).

[0126] The update module 570A is configured to, upon receiving a stochastic gradient from the stochastic module 568A, The model weights are updated using stochastic gradients and a learning rate γ. In some embodiments, the learning rate is constant. In some embodiments, the update module 570 may also instruct the sampling module 562A to select another sample and use that sample to refine the updated model weights. After iterating the training process using all training instances, the update module 570A is configured to present the current model weights as the model's output. The trained model with the updated model weights can then be used for prediction.

[0127] In some embodiments, the AFSGD application 558A may also include a user interface and / or a scheduler. The user interface can be used to configure parameters of the AFSGD application 558A, to train the AFSGD application 558A, and to perform predictions using the AFSGD application 558A. The scheduler can be used to schedule operations and data flows between modules of the AFSGD application 558A, between active workers and the coordinator, and between active workers and other workers.

[0128] Figure 5B An AFSVRG application 558B according to certain embodiments of this disclosure is schematically depicted. For example... Figure 5B As shown, the AFSVRG application 558B includes a model weight initialization module 560B, a local gradient module 561B, a sampling module 562B, a local dot product module 564B, a dot product summation module 566B, a stochastic gradient module 568B, and an update module 570B. In some embodiments, the AFSVRG application 558B may include other applications or modules necessary for the operation of the AFSVRG application 558B, such as a listener configured to receive notifications or instructions from the coordinator 410 or other workers 450. It should be noted that modules 560B to 570B are all implemented by computer-executable code or instructions or data tables or databases, and they together form an application. In some embodiments, each module may also include sub-modules. Alternatively, some modules may be combined into a stack. In other embodiments, some modules may be implemented as circuits rather than executable code. In some embodiments, the modules may also be collectively referred to as a model or AFSVRG model, which can be trained using training data and, after training, can be used for prediction. In some embodiments, the functionality of AFSVRG application 558B corresponds to algorithm 4 described above. In some embodiments, AFSAGA application 558C defines a learning rate γ and provides privacy data 472.

[0129] The model weight initialization module 560B is configured to initialize the weights of the AFSVRG application 558B and initialize the model weights. The model weights are provided to the local gradient module 561B. In some embodiments, the dimension of the model weights is the same as the dimension of the local instances. In some embodiments, each dimension of the model weights is a real number. During initialization, the model weight initialization module 560B can provide a random value, a value of 0, or any other arbitrary value for each dimension of the model weights. The superscript of the model weights represents an outer loop that iterates from 0 to S-1, where S is a predefined positive integer. The subscript of the model weights represents an inner loop with n iterations corresponding to n training instances. During training using training samples (instances and their corresponding labels), the model weights can be updated and stored in the AFSVRG application 558B. During prediction, the latest model weights are available for instance-based prediction. In some embodiments, during or after initialization, the AFSVRG model of the active worker is also configured to instruct the passive workers to initialize their respective model weights and run their respective training. In some embodiments, the model weight initialization module 560B is also configured to instruct the sampling module 562B to begin sampling instances.

[0130] The local gradient module 561B is configured to compute the full local gradient for each iteration s from 0 to S-1 and send this full local gradient to the stochastic variance reduction gradient (SVRG) module 568B. In some embodiments, S is a predefined positive integer, and the value of S is configured based on the form of the model, the problem to be solved, and the size of the training data. In some embodiments, S is configured to make the training of the model converge. In some embodiments, S can be set to 100 or 1000 empirically. In some embodiments, the local gradient module 561B is configured to instruct the coordinator 410 to use the equation To compute the fully local gradient, coordinator 410 is configured to use a tree-structured communication scheme for summation. In some embodiments, the local gradient module 561B is also used to add reference model weights (or the latest historical model weights). Defined as the current model weights of the model Right now And the current model weights Provided to the local dot product module 564B. In some embodiments, the loop in which the local gradient module 561B selects s ​​from integers 0, 1, 2, ..., S-1 is called the outer loop and is executed locally in the l-th active worker. Note that the model is configured to store a series of model weights as a reference. The current model weights at the end of each outer loop are defined as a reference model weight, which is used for the next outer loop.

[0131] The sampling module 562B is configured to, when the current model weights Defined as reference model weights (or the latest historical model weights). The latter can be defined as the initialization of model weights. Then, randomly select index i from {1, ..., n}, and start from local data D. l Get local instance The local instance is then sent to the local dot product module 564B. Here, n is the number of training samples used to train the AFSVRG application 558B, and i is the index of the training sample. Each training sample contains an instance. and its corresponding label y i The tag y i It can be used in other modules, such as the SVRG module 568B for acquiring stochastic gradients. In some embodiments, each worker or computing device has a label y. i .

[0132] The local area module 564B is configured to, upon receiving a local instance from the sampling module 562B, and the current model weights from the local gradient module 561B or stored in the model. and reference model weights At that time, calculate the current local dot product between the current model weights and the local instance. Calculate the reference local dot product (or the latest historical dot product) between the reference model weights and the instances. The current local dot product and the reference dot product are then sent to the dot product summing module 566B. (Reference model weights) It is a reference point; each outer loop has a defined one. The current model weights are updated after each inner loop, so each inner loop has a specific current model weight. For example, in the first inner loop of the first outer loop, the current model weights and reference model weights All of these will be used to initialize model weights. In the second inner loop of the first outer loop, the current model weights will be the model weights obtained after the first inner loop of the first outer loop, referencing the model weights. Still initializing model weights In the first inner loop of the second outer loop, the current model weights... Both the reference model weights and the model weights will be updated in the last inner loop of the first outer loop; while in the second inner loop of the second outer loop, the current model weights will be updated. The model weights will be obtained after the first inner loop of the second outer loop, while the reference model weights will still be the model weights obtained in the last inner loop of the first outer loop.

[0133] The dot product summation module 566B is configured to, upon receiving the current dot product from the local dot product module 564B, sum the dot product... and reference dot product At that time, the coordinator 410 is requested to calculate the sum of the current dot products from the relevant workers. and the sum of the reference dot products from the relevant workers (w s ) T x i These dot products correspond to instance x i In some embodiments, the dot product from each worker is calculated using the current model weights and reference model weights of that worker. In some embodiments, the coordinator 410 is configured to use a tree structure to obtain the sum of the dot products, which can be T1, T2, or any other type of form. The coordinator 410 obtains the currently summed dot products. dot product of summation and reference (w) s ) T x i Subsequently, the dot product summation module 566B is also configured to send the summed dot product to the random SVRG module 568B.

[0134] The SVRG module 568B is configured to, upon receiving the summation of the dot product... and (w s ) T x i At that time, the gradient of the random variance from the summation of the dot product is reduced. The calculated stochastic gradient is then sent to the update module 570A. In some embodiments, the SVRG module 568B is configured to use the equation Calculate the gradient of decreasing random variance. In some embodiments, the stochastic gradient can be calculated based on the above equations (1) and (4).

[0135] The update module 570B is configured to, upon receiving the random variance reduction gradient from the random module 568B, At this time, the model weights are updated using stochastic variance-decreasing gradients and a learning rate γ. In some embodiments, the learning rate is constant. In some embodiments, the learning rate is... The update module 570B is used to update the model weights. After updating the model weights, the update module 570B also notifies the sampling module 562B to sample another instance for another round of inner loop training. When the inner loop training is completed by exhausting the training instances, the update module 570B also provides the current model weights as the (s+1)th reference model weights and notifies the local gradient module 561B to start another round of outer loop training. The outer loop ends when iterating from 0 to S-1, or until the model weights converge, and the latest model weights are the output of the AFSVRG application 558B. The trained model with the updated model weights can then be used for prediction.

[0136] In some embodiments, the AFSVRG application 558B may also include a user interface and / or a scheduler. The user interface can be used to configure parameters of the AFSVRG application 558B for training and prediction. The scheduler can be used to schedule operations and data flows between modules of the AFSVRG application 558B, between active workers and the coordinator, and between active workers and other workers.

[0137] Figure 5C An AFSAGA application 558C according to certain embodiments of this disclosure is schematically depicted. For example... Figure 5C As shown, the AFSAGA application 558C includes a model weight initialization module 560C, a local gradient module 561C, a sampling module 562C, a local dot product module 564C, a dot product summation module 566C, an SAGA gradient module 568C, and an update module 570C. In some embodiments, the AFSAGA application 558C may include other applications or modules necessary for the operation of the AFSAGA application 558C, such as a listener configured to receive notifications or instructions from the coordinator 410 or other workers 450. It should be noted that modules 560C to 570C are all implemented by computer-executable code or instructions, or data tables or databases, and they together form an application. In some embodiments, each module may also include sub-modules. Alternatively, some modules may be combined into a stack. In other embodiments, some modules may be implemented as circuits rather than executable code. In some embodiments, these modules may also be collectively referred to as the AFSAGA model. The model can be trained using training data and, once trained, can be used for prediction. In some embodiments, the functionality of AFSAGA application 558C corresponds to that of algorithm 5 described above. In some embodiments, AFSAGA application 558C defines a learning rate γ and provides privacy data 472.

[0138] The model weight initialization module 560C is configured to initialize the model weights for AFSAGA application 558C, and will initialize the model weights. The model weights are provided to the local gradient module 561C. In some embodiments, the dimensions of the model weights are the same as the dimensions of the local instances. In some embodiments, each dimension of the model weights is a real number. During initialization, the model weight initialization module 560C can provide a random value, a value of 0, or any other arbitrary value for each dimension of the model weights. During training using training samples (instances and their corresponding labels), the model weights are updated and stored in the AFSAGA application 558C. In some embodiments, during or after initialization, the AFSAGA model of the active worker is also configured to instruct the passive workers to initialize their respective model weights and run their respective training. In some embodiments, the model weight initialization module 560C is also configured to instruct the sampling module 562C to begin sampling instances.

[0139] The local gradient module 561C is configured to compute a local gradient for each training instance from 1 to n, and send the computed local gradient of the instance to the SAGA module 568C. In some embodiments, n is a positive integer corresponding to the total number of training instances. In some embodiments, the local gradient module 561C is configured to use an equation... Calculate local gradient Referring to equations (1) and (5) above, calculate f. i (w) It is necessary to compute the dot product from different workers. In some embodiments, the local gradient module 561C instructs the coordinator 410 to use a tree-structured communication scheme to perform certain parts of the computation.

[0140] The sampling module 562C is configured to initialize model weights when training begins. When, or when using the first instance to update the model weights after training. At that time, randomly select index i from {1, ..., n}, and select from local data D l Get local instance The local instance is then sent to the local dot product module 564C. Here, n is the number of instances in the training data used to train the AFSAGA application 558C, and i is the index of the instance. Each training sample contains an instance. and its corresponding label y i The tag y i It can be used in other modules, such as the SAGA module 568C for obtaining stochastic gradients. In some embodiments, each worker or computing device has a label y. i .

[0141] The local area unit module 564C is configured to, upon receiving data from the model weight initialization module 560C or from the current model weights stored in the model, [follow this configuration]. And a local instance from the sampling module 562C. At that time, calculate the local dot product between the current model weights and the selected instances. The local dot product is then sent to the dot product summing module 566C. In some embodiments, the AFSAGA application 558C will use the initialized model weights in the first round of training and the updated model weights in subsequent training.

[0142] The dot product summation module 566C is configured to, upon receiving the current dot product from the local dot product module 564C, sum the dot product... At that time, the coordinator 410 is requested to calculate the sum of the dot products from all relevant workers. These dot products correspond to instance x i In some embodiments, the dot product of each worker is computed using the current model weights of each worker. In some embodiments, the coordinator 410 is configured to use a tree structure to obtain the sum of the dot products, the tree structure being T1, T2, or any other type of form. The sum is obtained in coordinator 410. Subsequently, the dot product summation module 566C is also configured to send the sum to the SAGA module 568C. It is important to note that the relevant workers are configured to perform their own training independently, rather than simultaneously using instances corresponding to the same index i; the worker performance is asynchronous.

[0143] The SAGA module 568C is configured to, upon receiving a summation of dot products, perform... At that time, the stochastic gradient is calculated based on the summation of the dot product. The calculated stochastic gradient is then sent to the update module 570C. In some embodiments, the SAGA module 568C is configured to use an equation... Calculate the stochastic gradient. In some embodiments, the stochastic gradient can be calculated based on the above equations (1) and (5).

[0144] The update module 570C is configured to update upon receiving a stochastic gradient from the SAGA gradient module 568C. At that time, the model weights are updated using stochastic gradients and a learning rate γ. And update the local gradient In some embodiments, the learning rate is a constant learning rate. In some embodiments, different learning rates are used. and An update is performed. After the update, the update module 570C is also configured to notify the sampling module 562C to sample another instance for another round of training. With the updated model weights, the model can be used to make predictions on the input instance.

[0145] In some embodiments, the AFSAGA application 558C may also include a user interface and / or a scheduler. The user interface can be used to configure parameters of the AFSAGA application 558C for training and prediction. The scheduler can be used to schedule operations and data flows between modules of the AFSAGA application 558C, between active workers and coordinators, and between active workers and other workers.

[0146] Figure 6A An asynchronous federated stochastic gradient descent training process for vertically partitioned data (AFSGD-VP) according to certain embodiments of this disclosure is schematically depicted. In some embodiments, the training process corresponds to Algorithm 3 described above. In some embodiments, the training process consists of, for example, Figure 4B The server computing device, such as the worker 450 shown, performs the execution, specifically by... Figure 5A The AFSGD shown is performed using the 558A. It should be specifically noted that, unless otherwise stated in this disclosure, the steps of the AFSGD-VP training procedure or method may be arranged in a different order, and therefore are not limited to... Figure 6A The order is shown. In some embodiments, process 600A is executed on the first active worker of worker 450. The first worker includes specific local data D that is not shared with other workers. l Furthermore, the l-th worker has its model parameters, such as the learning rate γ. In some embodiments, the learning rate γ is a constant value across different iterations of the model.

[0147] like Figure 6A As shown, in step 602A, the l-th worker operates as the active worker, and the AFSGD application 558A initializes the model weights of the l-th worker using the model weight initialization module 560A. The initialized model weights are then sent to the local dot product module 564A. (Model weights) There is d l There are 3 dimensions, each with a real number value. For initialization purposes, the dimension values ​​can be randomly set. Then, the l-th worker iteratively executes steps 604-614 on all training data. In some embodiments, the model weight initialization module also instructs the sampling module 562A to begin sampling. Furthermore, the l-th worker instructs the passive worker to initialize its respective model weights and locally execute steps 604-614 to train its local AFSGD model. In some embodiments, the active and passive workers execute the steps in parallel. However, the workers do not need to wait for input from other workers; the process is asynchronous. In other words, the active and passive workers do not need to use instances corresponding to the same index i to train their respective models in the same round of training.

[0148] In step 604A, upon receiving an instruction from the model weight initialization module 560A, the sampling module 562A randomly selects an index i from {1, ..., n} and retrieves the weights from the local data D. l Get local instance The local instance is then sent to the local dot product module 564A. In some embodiments, since the parallel operations of the workers are asynchronous, the workers do not need to select corresponding instances with the same index. In other words, the active worker does not need to send the random seed i to other workers, and each worker is likely to use different instances for training simultaneously.

[0149] In step 606A, after receiving the initialized model weights from the model weight initialization module 560A and the local instance from the sampling module 562A... At that time, the local dot product module 564A calculates the dot product between the model weights and the local instance. The dot product is then sent to the dot product summing module 566A. In some embodiments, after training with the first instance and obtaining updated model weights, the updated model weights are used instead of the initialized model weights to select another random instance in the next round and compute the local dot product. In other words, this disclosure only requires initializing the model weights once at the start of training.

[0150] In step 608A, upon receiving the dot product from the local dot product module 564A, the dot product summation module 566A requests the coordinator 410 to calculate the dot product with instance x. i The sum of the corresponding dot products In response to the request, coordinator 410 calculates the sum of the dot products from all relevant workers corresponding to index i and sends the summed dot products to stochastic gradient module 568A. In some embodiments, relevant workers may be predefined, each including data corresponding to index i. In some embodiments, coordinator 410 uses a tree structure scheme for summation. In some embodiments, the following equation is used for summation: In some embodiments, using Figure 7A or Figure 7B The method shown is used for summation.

[0151] In step 610A, the summed dot product is received from the dot product summing module 566A. At that time, the stochastic gradient module 568A calculates the stochastic gradient of the model using the summed dot product and sends the stochastic gradient to the model update module 570A. In some embodiments, the stochastic gradient is calculated using an equation... Computational. In some embodiments, gradients can be calculated based on the problem to be solved. For example, for binary classification, the gradient can be calculated using equation (1) and the summation of the dot product.

[0152] In step 612A, upon receiving the stochastic gradient At this time, the update module 570A updates the model weights using stochastic gradients and a learning rate γ, and instructs the sampling module 562A to select another sample for further training. In some embodiments, the update is performed in the following manner: Updated model weights It is saved as the output of process 600A.

[0153] In step 614A, the AFSGD application 558A continues the training process 604A–612A by selecting additional training samples. In some embodiments, the training process ends when all training instances have been used for training. After the model has been trained using all training instances, the model weights are optimized. At this point, the user can input an input instance into the model, which can then provide prediction results using the updated model weights.

[0154] Figure 6B An asynchronous federated random variance reduction gradient training process for vertically partitioned data (AFSVRG-VP) according to certain embodiments of this disclosure is schematically depicted. In some embodiments, the training process corresponds to Algorithm 4 described above. In some embodiments, the training process consists of, for example, Figure 4B The server computing device, such as the worker 450 shown, performs the task, and specifically... Figure 5B The AFSVRG application shown is executed using the 558B. It should be specifically noted that, unless otherwise stated in this disclosure, the steps of the AFSVRG-VP training procedure or method may be arranged in a different order, and therefore are not limited to... Figure 6B The order is shown. In some embodiments, process 600B is executed on the first active worker of worker 450. The first worker includes specific local data D that is not shared with other workers. l Furthermore, the l-th worker has its model parameters, such as the learning rate γ. In some embodiments, the learning rate γ is a constant value across different iterations of the model.

[0155] like Figure 6B As shown, in step 602B, AFSVRG applies the model weight initialization module 560B of 558B to initialize the model weights of the l-th worker. And the model weights Send to local gradient module 561B. Model weights. There are dl dimensions, and the value of each dimension is a real number. Here we use... replace This indicates the model weights, as there are two loop steps in the following steps. The superscript corresponds to the outer loop, and the subscript corresponds to the inner loop. In this step, the model weight initialization module 560B also instructs the sampling module 562B to start sampling instances, and also instructs the passive workers to initialize their respective model weights and run their respective training.

[0156] In step 603B, upon receiving the model weights At that time, the local gradient module 561B instructs the coordinator 410 to use a tree-structured communication scheme to calculate the full local gradient. Use the reference model weights (or the latest historical model weights). Defined as model weights and model weights Provided to the local dot product module 564B. When s is 0, there are no historical model weights, and the model weights will be initialized. Defined as the current model weights The loop selected from integers 0, 1, 2, ..., S-1 is called the outer loop, which is executed locally by the l-th active worker. In some embodiments, the value of S can be empirically set to 100 or 1000, depending on the problem to be solved and the number of inner loops. In some embodiments, the value of S is defined such that the fully local gradient is close to 0 in S-1 rounds.

[0157] In step 604B, upon receiving the instruction from the model weight initialization module 560B, the sampling module 562B randomly selects an index i from {1, ..., n} and retrieves the data from the local data D. l Get local instance And send the local instance to the local dot product module 564B. Local instance This corresponds to a random index i. Step 604B is executed in parallel across multiple workers, each of which can independently and asynchronously select and train the model. Workers do not need to select corresponding instances with the same index i within approximately the same timeframe, and the active worker does not need to send index i to other workers. When different workers are simultaneously performing their respective training, the i selected by different workers may differ from each other within the same timeframe. In some embodiments, the process of randomly selecting instances is referred to as the inner loop.

[0158] In step 606B, upon receiving a local instance from sampling module 562B and the current model weights from the local gradient module 561B or stored in the model. and reference weight At that time, the local dot product module 564B calculates the current dot product between the current model weights and the instances. and the reference dot product between the reference model weights and instances. The current dot product and the reference dot product are then sent to the dot product summation module 566B.

[0159] In step 608B, upon receiving the current dot product from the local dot product module 564B... and reference dot product At that time, the dot product summation module 566B instructs the coordinator 410 to sum the q current local dot products from the q workers to obtain the summed current dot product. Summing the reference dot products of the q workers yields the summed reference dot product. The summed dot product is then sent to the dot product summing module 566B. The dot product summing module 566B then processes the summed dot product... and (ws) T x i Send to SVRG module 568B. In some embodiments, the following equation is used for summation: and In some embodiments, a tree structure scheme is used for summation. In some embodiments, a tree structure is used... Figure 7A or Figure 7B The method shown performs each summation.

[0160] In step 610B, upon receiving the summed dot product from dot product summing module 566B... and (w s ) T x i At that time, the SVRG module 568B calculates the stochastic variance reduction gradient of the model based on the summed dot product and sends the stochastic variance reduction gradient to the model update module 570B. In some embodiments, an equation is used. Calculate the stochastic variance-reducing gradient. In some embodiments, if the model is a binary model, the stochastic gradient can be calculated based on the above equations (1) and (4).

[0161] In step 612B, upon receiving the gradient of decreasing random variance Then, update module 570B updates the model weights. In some embodiments, the update is performed in the following manner:

[0162] In step 614B, steps 604B to 612B are iteratively executed in each worker using the training instance. The iterations of steps 604B to 612B are called the inner loop. In some embodiments, different workers execute this inner loop in parallel, but they work asynchronously because each worker independently selects its own i-th training instance. In other words, different workers can train on different instances simultaneously.

[0163] In step 616B, after training the model using all training instances in the inner loop, update the definition of module 570B. It is important to note that These are multiple historical records of the model weights used as references; each iteration of the outer loop corresponds to one reference model weight. and These are the current model weights that are updated during each inner loop of training.

[0164] In step 618B, steps 603B to 616B are repeated for another iteration until a predetermined number of S iterations are completed. In some embodiments, the repetition of steps 603B to 616B is referred to as the outer loop. In some embodiments, the completion of an iteration can also be determined when the model weights converge. After completing S iterations or until the current model weights converge, the model weights are saved for the model, and the trained and optimized model weights can be used for prediction.

[0165] Figure 6C An asynchronous federated SAGA training process for vertical partitioned data (AFSAGA-VP) according to certain embodiments of this disclosure is schematically depicted. In some embodiments, the training process corresponds to Algorithm 5 described above. In some embodiments, the training process consists of, for example, Figure 4B The server computing device, such as the worker 450 shown, performs the task, and specifically... Figure 5C The AFSAGA application shown is based on the 558C. It should be specifically noted that, unless otherwise stated in this disclosure, the steps of the AFSAGA-VP training procedure or method may be arranged in a different order, and therefore are not limited to... Figure 6C The order is shown. In some embodiments, process 600C is executed on the first active worker of worker 450. The first worker includes specific local data D that is not shared with other workers. l Furthermore, the l-th worker has its model parameters, such as the learning rate γ. In some embodiments, the learning rate γ is a constant value across different iterations of the model.

[0166] like Figure 6C As shown, in step 602C, AFSAGA applies the model weight initialization module 560C of 558C to initialize the model weights of the l-th worker. Model weights From Select and set model weights Send to the local gradient module 561C. In some embodiments, Represents model weights Having d lEach dimension has a real value. In this step, the model weight initialization module 560C also instructs the sampling module 562C to begin sampling instances, and also instructs the passive workers to initialize their respective model weights and run their respective training.

[0167] In step 603C, upon receiving the model weights Subsequently, the local gradient module 561C uses a tree-structured communication scheme to calculate the local gradient. The computed local gradients are stored locally and then provided to the SAGA gradient model 568C. This step initializes the local gradients for each instance, which will be updated in the following steps.

[0168] In some embodiments, the following steps are performed in parallel by multiple workers. However, each worker can independently and asynchronously select and train the model. Workers do not need to select corresponding instances with the same index at approximately the same time, and the active worker does not need to send index i to other workers. When different workers are training simultaneously, the i selected by different workers can be different from each other within the same time frame.

[0169] In step 604C, upon receiving the instruction from the model weight initialization module 560C, the sampling module 562B randomly selects index i from 1 to n and extracts it from the local data D. l Get local instance And send the local instance to the local dot product module 564C. Local instance Corresponding to random index i.

[0170] In step 606C, upon receiving a local instance from sampling module 562C When using local dot product modules 561C and the current model weights stored in the model, the local dot product module 564C calculates the dot product between the model weights and the instances. And send the dot product to the dot product summation module 566C.

[0171] In step 608C, upon receiving the dot product from the local dot product module 564C... When summing the corresponding local dot products from other relevant workers, the dot product summing module 566C sums the q local dot products from the q workers to obtain a summed dot product, and sends the summed dot product to the SAGA gradient module 568C. In some embodiments, the summation is performed using the following equation: In some embodiments, prior to the summation operation, the first active worker or coordinator 410 instructs the passive workers to calculate their respective dot products. In some embodiments, using... Figure 7A or Figure 7BThe method shown is used for summation.

[0172] In step 610C, upon receiving the summed dot product from the dot product summing module 566C, the SAGA gradient module 568C calculates the stochastic gradient of the model and sends the gradient to the model update module 570C. In some embodiments, an equation is used. Calculate the gradient.

[0173] In step 612C, upon receiving the gradient At that time, the update module 570C was accessed via... Update the model weights, and through Update the local gradient.

[0174] In step 614C, steps 604C to 612C are iteratively executed using all training instances one by one. In some embodiments, all relevant workers execute steps 604C to 612C in parallel, but the relevant workers do not simultaneously use instances corresponding to the same index i to train the model; therefore, the training of the workers is asynchronous. After training using all training instances, the update module 570C updates the model weights. This is the output of the model. At this point, the user can input either an input or an instance into the model, which can then use the updated model weights to make predictions.

[0175] Figure 7A A dot product summation method according to certain embodiments of the present disclosure is schematically depicted. In some embodiments, the method corresponds to Algorithm 1 described above. In some embodiments, the method is, for example... Figure 4B The processor 450 and other server computing devices shown in the diagram perform the operations, specifically by local dot product modules 564A, 564B, or 564C and dot product summation modules 566A, 566B, or 566C. (For ease of explanation) Figure 7A and Figure 7B In the steps described, modules 564A, 564B, or 564C are named local dot product module 564, and dot product summation modules 566A, 566B, or 566C are named dot product summation module 566. In some embodiments, worker 450 cooperates with coordinator 410 for summation. It should be noted that, unless otherwise stated in this disclosure, the steps of the method can be arranged in different orders, and are therefore not limited to... Figure 7A The order shown.

[0176] In step 702A, the local dot product module 564 of the active worker calculates its dot product. Meanwhile, asynchronous federated application 458 instructs the other workers among the 1 to q workers to compute their respective dot products. In other words, the workers can compute their respective dot products in parallel. Here, l′=1,...,q denotes one of the workers.

[0177] In step 704A, the dot product summation module 566 of the active workers instructs the coordinator 410 to calculate the summed dot products. Specifically, the coordinator 410 uses tree-structured communication to sum the dot products from all relevant workers. The summed dot products are calculated by... definition.

[0178] The dot product is obtained by parallel computation of the dot product by the q workers 450 and summation of the dot product by the coordinator 410 using a tree structure, which can be used in other modules.

[0179] In some embodiments, when the user inputs instance x i When the summation of the dot product ξ is used to predict instance x, i In some embodiments, for a binary classification problem, x i The prediction is sgn(x) i ), where if the dot product of the summations is greater than 0, sgn(x) i If x is 1, then x is -1; otherwise, x is -1. For regression problems, x... i The prediction is directly about x i The summation of the dot product.

[0180] Figure 7B A method for summing dot products according to certain embodiments of the present disclosure is schematically depicted. In some embodiments, the method corresponds to Algorithm 2 described above. In some embodiments, the method is, for example... Figure 4B The process is executed by a server computing device such as the worker 450 shown, and specifically by the local dot product module 564 and the dot product summation module 566. It should be noted that, unless otherwise stated in this disclosure, the steps of the method can be arranged in different orders, and therefore are not limited to... Figure 7B The order shown.

[0181] In step 702B, the local dot product module 564 of the active worker generates a random number b. l′ In some embodiments, the generated random numbers have values ​​in the range [0, 2π].

[0182] In step 704B, the local dot product module 564 of the active worker calculates its adjusted dot product. Simultaneously, the local dot product module 564 of the active worker also instructs another worker among the 1 to q workers to generate its own random number and compute its respective adjusted dot product. In other words, the workers can generate their own random numbers and compute their respective adjusted dot products in parallel. Here, l′=1,...,q refers to one of the workers.

[0183] In step 706B, after the q workers have generated their respective random numbers and calculated their respective adjusted dot products, the dot product summing module 566 of the active workers instructs the coordinator 410 to calculate the summed adjusted dot products. Specifically, the coordinator 410 uses a first tree structure communication to sum the adjusted dot products from all relevant workers. The summed adjusted dot products are generated by... Definition. In some embodiments, the first tree structure can be any form of tree structure, such as... Figure 3A The tree structure T1 is shown.

[0184] In step 708B, the dot product summation module 566 of the active workers instructs the coordinator 410 to calculate the sum of random numbers. Specifically, the coordinator 410 uses a second tree structure to sum the random numbers from all relevant workers. The sum of random numbers is generated by... Define a tree structure where {1, ..., q} are workers. In some embodiments, the second tree structure is significantly different from the first tree structure. In some embodiments, the second tree structure can be any form of tree structure, such as... Figure 3B The tree structure T2 shown.

[0185] In step 710B, the dot product summing module 566 of the active worker subtracts the random number of the sum from the adjusted dot product to obtain the final dot product. In some embodiments, using To perform this step.

[0186] By adding random number elements to the adjusted dot product and using a first tree structure and a second tree structure that are significantly different from each other, data privacy in the communication is effectively protected.

[0187] In some embodiments, when an instance has a large number of dimensions, preferably, Figure 7A The process shown is more Figure 7B The process shown is simpler and faster. In some embodiments, when the instance has only a few dimensions, preferably, Figure 7B Increased security. In some embodiments, Figure 7A and Figure 7B The steps in the process can be used for prediction.

[0188] In some aspects, this disclosure relates to a non-transitory computer-readable medium for storing computer-executable code. In some embodiments, the computer-executable code may be software stored in storage device 456 as described above. When executed, the computer-executable code may perform one of the methods described above.

[0189] In some aspects, this disclosure relates to a method for predicting the outcome of an instance using a trained model. In some embodiments, this disclosure uses Figures 6A-6C The steps described in any of them are used to train the model and use Figure 7A and Figure 7B The steps described in any of these embodiments make a prediction for a given sample. In some embodiments, this disclosure predicts whether a loan should be issued to a customer based on online financial information from customers of digital financial companies, online shopping patterns from e-commerce companies, and banking information from traditional banks. This prediction can be initiated by the server of any one of the three entities, but the three entities do not need to share their real customer data, and the three entities can be trained asynchronously.

[0190] As described above in conjunction with the methods and modules, in some embodiments, asynchronous training instructs the active worker and the passive worker to train their respective models by selecting their own random samples. Therefore, at any given time, the active worker can train its model using the instance corresponding to the first index, while one of the passive workers can train its model using the instance corresponding to the second index. Once the active worker has completed training using the first instance, it can immediately continue training by selecting, for example, a third instance corresponding to the third index. Similarly, once the passive worker has completed training using the second instance, it can immediately continue training by selecting, for example, a fourth instance corresponding to the fourth index. One of the active or passive workers can complete training using an instance without waiting for the other worker.

[0191] In contrast, for synchronous training, despite instances having different data characteristics, the active and passive workers simultaneously train their models using instances corresponding to the same index. When one of the active or passive workers finishes training with the first instance corresponding to the first index, it waits for the other workers to finish the same round of training with their respective first instances corresponding to the same first index. When all workers have completed a round of training, the active and passive workers then begin training with their respective instances corresponding to, for example, the same second index. Therefore, the concept of synchronous training here does not mean that the workers train their models using exactly the same data, nor does it mean that they start operating at exactly the same point in time, because the active worker may need to start first, and due to the different complexity of the instances they use and the different computational capabilities the workers possess, the time when the workers finish the same round of training may differ slightly.

[0192] Example. Exemplary experiments have been conducted using models according to certain embodiments of this disclosure.

[0193] For classification and regression tasks, asynchronous vertical SGD, SVRG, and SAGA algorithms (i.e., AFSGD-VP, AFSVRG-VP, and AFSAGA-VP) are compared with synchronous versions of vertical SGD, SVRG, and SAGA (denoted as FSGD-VP, FSRRG-VP, and FSAGA-VP, respectively), where FSRRG-VP is almost identical to FD-SVRG described by Zhang et al., 2018. For classification tasks, this disclosure considers an l2-norm regularized logistic regression model, as follows:

[0194]

[0195] For regression tasks, this disclosure uses a ridge linear regression method with l2 norm regularization, as follows:

[0196]

[0197] To fully demonstrate the scalability of the asynchronous vertical federated learning algorithm of this disclosure, this disclosure... Figure 8 The eight datasets summarized in Table 1 for binary classification and regression tasks were used for experiments. Two real-world, relatively small financial datasets, UCICreditCard and GiveMeSomeCredit, were obtained from the Kaggle website (https: / / www.kaggle.com / datasets). The other six datasets were obtained from the LIBSVM website (https: / / www.csie.ntu.edu.tw / cjlin / libsvmtools / datasets / ). This disclosure randomly splits the news20, url, and webspam datasets into training and testing data in a 4:1 ratio. Because the testing data contains more instances, this disclosure also uses the rcv1 testing data for training and the training data for testing.

[0198] Experimental Design. In our experiments, we not only validated the theoretical results of AFSGD-VP, AFSVRG-VP, and AFSAGA-VP, but also demonstrated that our algorithm is more efficient than the corresponding synchronous algorithms (i.e., FSGD-VP, FSVRG-VP, and FSAGA-VP). Our asynchronous algorithm is implemented within a decentralized framework, where workers possess their own partial data and model parameters. There is no master node for aggregating data / features / gradients, which could potentially lead to unintended leakage of user information. Instead, we use methods such as... Figure 2The coordinator collects products computed based on local data and parameters from other workers. Each worker node can independently invoke the coordinator to enable asynchronous model updates. Local products are aggregated in a demand-based manner, meaning that a worker node only requests the coordinator to retrieve local products from other workers when it needs to update its local parameters. Unlike horizontal federated learning, it is more difficult for an attacker to recover information from the user data in a worker node using local products than gradients.

[0199] Experimental Setup. We ran all experiments on a 32-node cluster of 20-core Intel Xeon E5-2660 2.60GHz (Haswell) processors. The nodes were connected to a 56Gb FDR. We used OpenMPI v3.1.1 with multi-threading support to enable communication between the worker processes and Armadillo v9.700.3 for efficient matrix computation. Each worker was placed on a different machine node. For the l2 regularization term, we set the coefficient λ = 1e for all experiments. -4 We also selected the optimal learning rate ∈ (5e) for each algorithm for different learning tasks. -1 ,1e -1 5e -2 ,1e -2 (...). There exists a synthetic discrete node that may be 40% to 300% slower than the fastest working node to simulate real-world application scenarios. In reality, it is quite normal for different parties in a federated learning system to possess varying computational and communication capabilities and resources.

[0200] Implementation Details. In our asynchronous algorithm, each worker node performs computations fairly independently. The main thread of the worker process performs the main workload of gradient calculation and model update operations. Another listener thread continues to listen for requests and send the local product back to the request source. The computation graph of the worker can be summarized as follows: (1) Randomly select the index of the data. (2) Call the coordinator to broadcast the index to the listeners of other workers. (3) Reduce the sum of the local products returned from the listeners. (4) Perform gradient calculation and model parameter update. It should be noted that the local product is calculated based on the current parameters of the worker. However, in general, some workers may update their parameters more often than others. Unlike the asynchronous level algorithm, although the worker process runs asynchronously, all parameters used by the worker to calculate the gradient are up-to-date. Broadcast and reduction operations are implemented in the tree structure scheme to reduce communication costs.

[0201] Classify tasks.

[0202] Evaluation was conducted using real-world financial datasets. We first compared our asynchronous federated learning algorithm with a synchronous version on a financial dataset to demonstrate its ability to solve real-world applications. In the asynchronous algorithm, each worker saves its local parameters for testing at fixed time intervals. In the synchronous setting, since all workers run at the same speed, each worker saves its parameters after a fixed number of iterations. We followed this scheme in our other experiments.

[0203] The original feature counts for the UCICreditCard and GiveMeSomeCredit datasets were 23 and 10, respectively. We applied one-hot encoding to the categorical features and standardized the other features column-by-column. After simple data preprocessing, the feature counts became 90 and 92, respectively.

[0204] Four working nodes were used in this part of the experiment. Figure 9A and Figure 9B The convergence of different algorithms for classification tasks according to certain embodiments of this disclosure is schematically depicted. For example... Figure 9A and Figure 9B As shown, our asynchronous vertical algorithm (solid line) consistently outperforms its synchronous algorithm (dashed line). The suboptimal y-axis function represents the error of the objective function towards the global optimum. The shape of the convergence curve is primarily determined by the optimization method we choose: SGD, SVRG, and SAGA. SGD typically has higher error accuracy than SVRG, while SAGA's error accuracy is similar to SVRG. Therefore, the convergence speed is mainly affected by computational and communication complexity. In the asynchronous setup, there is no inefficient idle time waiting for other workers, resulting in a much higher update frequency, which leads to faster convergence of the asynchronous algorithm relative to clock time.

[0205] Evaluation was conducted using large-scale datasets. Previous experiments demonstrated that our asynchronous federated learning algorithm can solve real-world financial problems more effectively. In this section, we will further validate our algorithm using large-scale benchmark datasets, characterized by a large number of data instances and high-dimensional features. Experiments were conducted on the new20 and rcv1 datasets with 8 worker nodes; and on the url and webspam datasets with 16 worker nodes. Figures 10A-10D The results are shown in the figure. The speedup of asynchronous algorithms becomes more pronounced as the total computational budget increases. Therefore, they are much more efficient in large-scale practical applications. Our asynchronous SGD, SVRG, and SAGA outperformed their synchronous counterparts in experiments on all four datasets.

[0206] Regression task. To further illustrate the advantage of asynchronous algorithms being scalable to a variety of tasks, we also consider, for example... Figure 9C and Figure 9DExperiments were conducted on the regression problem shown. Both E20060-tfidf with fewer data instances and more features, and YearPredictionMSD with more data instances and fewer features, were tested. This experiment used four working nodes, and similar conclusions were drawn as before.

[0207] Asynchronous efficiency. Figure 11 Table 2 summarizes the speedup results of asynchronous algorithms compared to synchronous algorithms. The speedup is based on the algorithm achieving a certain optimal accuracy (1e for SVRG and SAGA). -4 ; SGD based on different datasets is 1e -2.5 or 1e -1.5 The time is calculated.

[0208] To further analyze the efficiency of our asynchronous algorithm, we quantified the time consumption composition of asynchronous and synchronous algorithms, such as... Figure 12 As shown, in the synchronous algorithm, the execution time and update frequency are scaled by the discretizer. The computation time of the discretizer is much longer than that of the non-discreteizer, which leads to a significant amount of synchronization time required for the non-discreteizer in the synchronous algorithm. In our asynchronous algorithm, however, the non-discreteizer picks up the latest product information from the discretizer without waiting for the discretizer to complete its current iteration. Therefore, the synchronization time is eliminated. Although this increases communication costs because each worker needs to independently aggregate products from other workers, we can gain a significant advantage in terms of update frequency.

[0209] Scalability. Figures 13A-13C The diagram shows the scalability in terms of the number of workers. Synchronous algorithms fail to solve the discrete problem and perform poorly. Using synchronization barriers causes non-discrete workers to wait ineffectively for discrete workers. Our asynchronous algorithm performs ideally at the beginning because it solves the discrete problem well; however, as the number of workers continues to grow, it deviates from its ideal value because communication overhead limits speedup.

[0210] In summary, this disclosure provides an asynchronous federated SGD (AFSGD-VP) algorithm for vertically partitioned data, along with its SVRG and SAGA variants. To the best of our knowledge, AFSGD-VP and its SVRG and SAGA variants are the first asynchronous federated learning algorithms for vertically partitioned data. Importantly, this disclosure provides the convergence speed of AFSGD-VP and its SVRG and SAGA variants under the condition of strong convexity of the objective function. This disclosure also proves model privacy and data privacy. Extensive experimental results on various vertically partitioned datasets not only validate the theoretical results of AFSGD-VP and its SVRG and SAGA variants but also demonstrate that the algorithm is more efficient than the corresponding synchronous algorithms.

[0211] The foregoing description of exemplary embodiments of this disclosure is presented for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit this disclosure to its precise form. In view of the foregoing teachings, many modifications and variations are possible.

[0212] The embodiments were chosen and described to explain the principles of this disclosure and its practical application, enabling others skilled in the art to utilize this disclosure and various embodiments, and to have various modifications suitable for a particular intended use. Alternative embodiments will become apparent to those skilled in the art to which this disclosure pertains without departing from the spirit and scope of this disclosure. Therefore, the scope of this disclosure is defined by the appended claims, and not by the foregoing description and the exemplary embodiments described therein.

[0213] List of references (included in the entire text through citations):

[0214] 1.Nadeem Badshah, Facebook to contact 87million users affected by databreach, The Guardian, April 8, 2018.

[0215] 2.Amir Beck and Luba Tetruashvili, On the Convergence of blockcoordinate descent type methods, SIAM Journal on Optimization, 2013, 23(4), 2037-2060.

[0216] 3. Léon Bottou, Large-scale machine learning with stochastic gradient descent, Proceedings of COMPSTAT'2010, Springer, 2010, 177-186.

[0217] 4.Chih-Chung Chang and Chih-Jen Lin, LIBSVM: a library for supportvector machines, ACM Transactions on Intelligent Systems and Technology, 2011, 2:1-27.

[0218] 5.Kewei Cheng,Tao Fan,Yilun Jin,Yang Liu,Tianjian Chen,and QiangYang,Secureboost:A lossless federated learning framework,2019,arXiv:1901.08755.

[0219] 6.Bryan Conroy and Paul Sajda,Fast,exact model selection andpermutation testing for 12-regularized logistic regression,InternationalConference on Artificial Intelligence and Statistics,2012,246-254.

[0220] 7.Aaron Defazio,Francis Bach,and Simon Lacoste-Julien,SAGA:a fastincremental gradient method with support for non-strongly convex compositeobjectives,NIPS’14:Proceedings of the 27th International Conference on NeuralInformation Processing Systems,2014,1646-1654.

[0221] 8.Wenliang Du and Mikhail J.Atallah,Privacy-preserving cooperativestatistical analysis,Proceedings of the 17 th Annual Computer SecurityApplications Conference,2001,102-110.

[0222] 9.Regulation(EU)2016 / 679 of the European Parliament and of theCouncilof 27 April 2016,https: / / eur-lex.europa.eu / legal-content / EN / TXT / ?uri=CELEX:02016R0679-20160504.

[0223] 10.Cong Fang,Chris Junchi Li,Zhouchen Lin,and Tong Zhang,SPIDER:near-optimal non-convex optimization via stochastic path-integrated differentialestimator,Advances in Neural Information Processing Systems 31,2018,689-699.

[0224] 11.Adria Gascon,Phillipp Schoppmann,Borja Balle,Mariana Raykova,JackDoerner,Samee Zahur,and David Evans,Secure linear regression on verticallypartitioned datasets,IACR Cryptology ePrint Archive,2016,892.

[0225] 12.Adria Gascon,Phillipp Schoppmann,Borja Balle,Mariana Raykova,JackDoerner,Samee Zahur,and David Evans,Privacy-preserving distributed linearregression on high-dimensional data,Proceedings on Privacy EnhancingTechnologies,2017,(4):345-364.

[0226] 13.Richard L.Graham,Timothy S.Woodall,and Jeffrey M.Squyres,Open MPI:A flexible high performance MPI,International Conference on ParallelProcessing and Applied Mathematics,2005,228-239.

[0227] 14.Bin Gu,Zhouyuan Huo,Cheng Deng,and Heng Huang,Faster derivative-free stochastic algorithm for shared memory machines,Proceedings of the 35thInternational Conference on Machine Learning,2018,1807-1816.

[0228] 15.Bin Gu,Zhouyuan Huo,and Heng Huang,Asynchronous stochastic blockcoordinate descent with variance reduction,2016,arXiv:1610.09447.

[0229] 16.Stephen Hardy,Wilko Henecka,Hamish Ivey-Law,Richard Nock,GiorgioPatrini,Guillaume Smith,and Brian Thorne,Private federated learning onvertically partitioned data via entity resolution and additively homomorphicencryption,2017,arXiv:1711.10677.

[0230] 17.Thomas Hofmann,Aurelien Lucchi,Simon Lacoste-Julien,and BrianMcWilliams,Variance reduced stochastic gradient descent with neighbors,Advances in Neural Information Processing Systems 28,2015,2305-2313.

[0231] 18.Zhouyuan Huo and Heng Huang,Asynchronous mini-batch gradientdescent with variance reduction for non-convex optimization,Thirty-First AAAIConference on Artificial Intelligence,2017,2043-2049.

[0232] 19.Alan F.Karr,Xiaodong Lin,Ashish P.Sanil and Jerome P.Reiter,Privacy-preserving analysis of vertically partitioned data using securematrix products,Journal of Official Statistics,2009,25(1):125-138.

[0233] 20.Vyacheslav Kungurtsev,Malcolm Egan,Bapi Chatterjee,and DanAlistarh,Asynchronous stochastic subgradient methods for general nonsmoothnonconvex optimization,2019,arXiv:1905.11845.

[0234] 21.Rémi Leblond,Fabian Pedregosa,and Simon Lacoste-Julien,ASAGA:Asynchronous parallel SAGA,Proceedings of the 20th International Conferenceon Artificial Intelligence and Statistics,2017,54:46-54.

[0235] 22.Xingguo Li,Tuo Zhao,Raman Arora,Han Liu,and Mingyi Hong,Animproved convergence analysis of cyclic block coordinate descent-type methodsfor strongly convex minimization,Proceedings of the 19th InternationalConference on Artificial Intelligence and Statistics,2016,51:491-499.

[0236] 23.Xingguo Li,Tuo Zhao,Raman Arora,Han Liu,and Mingyi Hong,On fasterconvergence of cyclic block coordinate descent-type methods for stronglyconvex minimization,J.Mach.Learn.Res.18,2018,1-24.

[0237] 24.Yang Liu,Yingting Liu,Zhijie Liu,Junbo Zhang,Chuishi Meng,and YuZheng,2019,arXiv:1905.10053.

[0238] 25.Yang Liu,Zhuo Ma,Ximeng Liu,Siqi Ma,Surya Nepal,and Robert H.Deng,Boosting privately:privacy-preserving federated extreme boosting for mobilecrowdsensing,Cryptography and Security,2019,arXiv:1907.10218.

[0239] 26.Horia Mania,Xinghao Pan,Dimitris Papailiopoulos,Benjamin Recht,Kannan Ramchandran,and Michael I Jordan,Perturbed iterate analysis forasynchronous stochastic optimization,2015,arXiv:1507.06970.

[0240] 27.Qi Meng,Wei Chen,Jingcheng Yu,Taifeng Wang,Zhi-Ming Ma,and Tie-YanLiu,Asynchronous stochastic proximal optimization algorithms with variancereduction,2016,arXiv:1609.08435.

[0241] 28.Yu Nesterov,Efficiency of coordinate descent methods on huge-scaleoptimization problems,SIAM Journal on Optimization,2012,22(2):341-362.

[0242] 29.Lam M Nguyen,Jie Liu,Katya Scheinberg,and Martin Takac,SARAH:Anovel method for machine learning problems using stochastic recursivegradient,Proceedings of the 34th International Conference on MachineLearning,2017,70:2613-2621.

[0243] 30.Richard Nock,Stephen Hardy,et al.,Entity resolution and federatedlearning get a federated resolution,2018,arXiv preprint arXiv:1803.04035.

[0244] 31.Nhan H Pham,Lam M Nguyen,Dzung T Phan,and Quoc Tran-Dinh,ProxSARAH:An efficient algorithmic framework for stochastic compositenonconvex optimization,2019,arXiv:1902.05679.

[0245] 32.Conrad Sanderson and Ryan Curtin,Armadillo:a template-based C++library for linear algebra,Journal of Open Source Software,2016,1(2):26.

[0246] 33.Ashish P.Sanil,Alan F.Karr,et al.,Privacy preserving regressionmodelling via distributed computation,Proceedings of the Tenth ACM SIGKDDInternational Conference on Knowledge Discovery and Data Mining,2004,677-682.

[0247] 34.Mark Schmidt,Nicolas Le Roux,and Francis Bach,Minimizing finitesums with the stochastic average gradient,Mathematical Programming B,Springer,2017,162(1-2):83-112.

[0248] 35.Xia Shen,Moudud Alam,Freddy Fikse,and Lars Ronnegard,A novelgeneralized ridge regression method for quantitative genetics,Genetics,2013,193(4):1255-1268.

[0249] 36.Jinhyun So,Basak Guler,Amir Salman Avestimehr,and Payman Mohassel,CodedPrivateML:A Fast and Privacy-Preserving Framework for DistributedMachine Learning,2019,arXiv:1902.00641.

[0250] 37.Johan AK Suykens and Joos Vandewalle,Least squares support vectormachine classifiers,Neural Processing Letters,1999,9(3):293-300.

[0251] 38.Jaideep Vaidya and Chris Clifton,Privacy preserving associationrule mining in vertically partitioned data,Proceedings of the Eighth ACMSIGKDD International Conference on Knowledge Discovery and Data Mining,2002,639-644.

[0252] 39.Li Wan,Wee Keong Ng,Shuoguo Han,and Vincent C.S.Lee,Privacypreservation for gradient descent methods,Proceedings of the 13th ACM SIGKDDInternational Conference on Knowledge Discovery and Data Mining,2007,775-783.

[0253] 40.ZheWang,Kaiyi Ji,Yi Zhou,Yingbin Liang,and Vahid Tarokh,SpiderBoost:A class of faster variance-reduced algorithms for nonconvexoptimization,2018,arXiv:1810.10690.

[0254] 41.Lin Xiao and Tong Zhang,A proximal stochastic gradient method withprogressive variance reduction,SIAM Journal on Optimization,2014,24(4):2057-2075.

[0255] 42.Qiang Yang,Yang Liu,Tianjian Chen,and Yongxin Tong,Federatedmachine learning:Concept and applications,ACM Transactions on IntelligentSystems and Technology(TIST),2019,10(2):article 12.

[0256] 43.Hwanjo Yu,Jaideep Vaidya,and Xiaoqian Jiang,Privacy-preserving SVMclassification on vertically partitioned data,Pacific-Asia Conference onKnowledge Discovery and Data Mining,Springer,2006,647-656.

[0257] 44.Gong-Duo Zhang,Shen-Yi Zhao,Hao Gao,and Wu-Jun Li,Feature-distributed SVRG for high-dimensional linear classification,2018,arXiv:1802.03604.

[0258] 45.Shen-Yi Zhao and Wu-Jun Li,Fast asynchronous parallel stochasticgradient descent:a lock-free approach with convergence guarantee,ThirtiethAAAI Conference on Artificial Intelligence,2016,2379-2385.

[0259] 46.Ian Goodfellow,Yoshua Bengio,Aaron Courville,Deep learning,MITPress,2016,http: / / www.deeplearningbook.org.

Claims

1. A system for asynchronously training a federated learning model, comprising a proactive computing device and at least one passive computing device in communication with each other, wherein each of the proactive computing device and the passive computing device comprises a federated learning model and local data, the proactive computing device comprising a processor and a storage device storing computer executable code that, when executed at the processor, is configured to: train the federated learning model in the proactive computing device using dimensions of a plurality of training instances in the proactive computing device; and instruct the at least one passive computing device to train the federated learning model in the at least one passive computing device using dimensions of a plurality of training instances in the at least one passive computing device, wherein index the plurality of training instances in the proactive computing device and the plurality of training instances in the at least one passive computing device, an index of a training instance in the plurality of training instances selected by the at least one passive computing device being independent of an index of a training instance in the plurality of training instances selected by the proactive computing device at the same time, such that the training of the federated learning model in the proactive computing device and the training of the federated learning model in the at least one passive computing device are asynchronous.

2. The system of claim 1, wherein, parameters of the federated learning model comprise a constant learning rate.

3. The system of claim 1, further comprising a coordinator in communication with the active computing device and the at least one passive computing device, wherein, intermediate results are communicated between the proactive worker and the at least one passive worker via the coordinator using a tree structure scheme.

4. The system of claim 1, wherein, the computer executable code is configured to train the federated learning model in the proactive computing device by: selecting an i-th instance in the proactive worker, where i is an index of a training instance; computing an active dot product between current model weights in the proactive computing device and dimensions of the i-th instance in the proactive computing device; instructing the at least one passive computing device to compute a passive dot product between current model weights in the at least one passive computing device and dimensions of the i-th instance in the at least one passive computing device; summing the active dot product and the passive dot product to obtain a summed dot product; computing a stochastic gradient of the federated learning model using the summed dot product; and updating the model weights using the stochastic gradient of the federated learning model and a learning rate. the computer executable code is configured to train the federated learning model in the proactive computing device by:

5. The system of claim 4, wherein, The random gradient is computed using is the regularization loss corresponding to the loss of the ith instance, is the regularization loss corresponding to the loss of the ith instance, is the gradient of the regularization loss with respect to the dimension in the active computing device.

6. The system of claim 1, wherein, for each iteration in a first number of iterations: computing a full local gradient of instances in the proactive computing device; in parallel and independently in each of the proactive worker and the at least one passive worker: selecting one of the instances; computing a current dot product between current model weights and the selected instance and a reference dot product between reference model weights and the selected instance; instructing the at least one passive computing device to compute a current dot product of the at least one passive computing device and a reference dot product of the at least one passive computing device using instances corresponding to the selected instance; ​ summing the current dot product from the active computing device and the current dot product from the at least one passive computing device to obtain a summed current dot product, summing the reference dot product from the active computing device and the reference dot product from the at least one passive computing device to obtain a summed reference dot product; computing a stochastic gradient of the federated learning model using the summed current dot product and the summed reference dot product; and updating model weights using the stochastic gradient of the federated learning model and a learning rate to obtain updated model weights; and defining the updated model weights as new reference model weights.

7. The system of claim 6, wherein, The stochastic gradient is computed using , is a regularization loss corresponding to the loss of the i-th instance, is a regularization loss corresponding to the loss of the i-th instance with respect to the reference model weights, is a regularized empirical risk, , and are , and are gradients with respect to the dimensions in the active computing device.

8. The system of claim 6, wherein, The summing of current dot products is performed using a first tree structure scheme, and the summing of reference dot products is performed using a second tree structure scheme, the first tree structure scheme being substantially different from the second tree structure scheme.

9. The system of claim 1, wherein, The computer executable code is configured to train a federated learning model in the active computing device by: computing a local gradient of an instance in the active computing device; performing, in parallel and independently in each of the active worker and the at least one passive worker: selecting one of the instances; computing a current dot product between the current model weights and the selected instance; instructing the at least one passive computing device to compute a current dot product of the at least one passive computing device using an instance corresponding to the selected instance; summing the current dot product from the active computing device and the current dot product from the at least one passive computing device to obtain a summed dot product; computing a stochastic gradient of the federated learning model using the summed dot product; and updating the model weights using the stochastic gradient of the federated learning model and a learning rate to obtain updated model weights, and updating the local gradient.

10. The system of claim 6, wherein, The stochastic gradient is computed using is computed, is a regularization loss corresponding to the loss of the i-th instance, is is a gradient with respect to a dimension of the i-th instance in the active computing device, is a local gradient in the active computing device.

11. The system of claim 1, wherein, The computer executable code is configured to, after training the federated learning model, predict a value of an input instance by: computing a dot product between model weights and the input instance for each of the active worker and the at least one passive worker; and summing the dot product from the active worker and the dot product from the at least one passive worker using tree structure communication to obtain a summed dot product, wherein the summed dot product is the predicted value for the input instance.

12. The system of claim 1, wherein, The computer executable code is configured to, after training the federated learning model, predict a value of an input instance by: for each of the active worker and the passive worker: generating a random number, computing a dot product between model weights and the input instance, adding the random number to the dot product to obtain an adjusted dot product; summing the adjusted dot product from the active worker and the adjusted dot product from the at least one passive worker using first tree structure communication to obtain a summed adjusted dot product; summing the random number from the active worker and the random number from the at least one passive worker using second tree structure communication to obtain a summed random number; and subtracting the summed random numbers from the summed adjusted dot products to obtain summed dot products, wherein the summed dot products are prediction values for the input instances.

13. A method for asynchronously training a federated learning model, comprising: training, by an active computing device, a federated learning model in the active computing device using dimensions of a plurality of training instances in the active computing device; and indicating, by at least one passive computing device, the at least one passive computing device to train a federated learning model in the at least one passive computing device using dimensions of a plurality of training instances in the at least one passive computing device, wherein the active computing device and the at least one passive computing device are in communication with each other, index the plurality of training instances in the active computing device and the plurality of training instances in the at least one passive computing device, an index of a training instance in the plurality of training instances selected by the at least one passive computing device is independent of an index of a training instance in the plurality of training instances selected by the active computing device at the same time, such that the training of the federated learning model in the active computing device and the training of the federated learning model in the at least one passive computing device are asynchronous.

14. The method of claim 13, wherein, parameters of the federated learning model include a constant learning rate, intermediate results are communicated between the active worker and the at least one passive worker using a tree structure scheme via a coordinator in communication with the active worker and the at least one passive worker.

15. The method of claim 13, wherein, the training the federated learning model in the active computing device includes: selecting an i-th training instance in the active computing device, where i is an index of an instance; computing an active dot product between current model weights in the active computing device and dimensions of the i-th training instance in the active computing device; indicating the at least one passive computing device to compute a passive dot product between current model weights in the at least one passive computing device and dimensions of the i-th instance in the at least one passive computing device; summing the active dot product and the passive dot product to obtain a summed dot product; using the summed dot product to compute a stochastic gradient of the federated learning model; and updating the model weights using the stochastic gradient of the federated learning model and a learning rate, wherein the stochastic gradient is computed using is computed, is a regularization loss corresponding to the loss of the ith instance, is a gradient of the regularization loss with respect to a dimension in the active computing device.

16. The method of claim 13, wherein, the training the federated learning model in the active computing device includes: for each iteration in a first number of iterations: computing a full local gradient of an instance in the active computing device; performing, in parallel and independently in each worker of the active worker and the at least one passive computing device: selecting one of the instances; computing a current dot product between current model weights and the selected instance and a reference dot product between reference model weights and the selected instance; indicating the at least one passive computing device to compute the current dot product of the at least one passive computing device and the reference dot product of the at least one passive computing device using an instance corresponding to the selected instance; summing the current dot product from the active computing device and the current dot product from the at least one passive computing device to obtain a summed current dot product, summing the reference dot product from the active computing device and the reference dot product from the at least one passive computing device to obtain a summed reference dot product; computing a stochastic gradient of the federated learning model using the summed current dot product and the summed reference dot product; and updating the model weights using the stochastic gradient of the federated learning model and a learning rate to obtain updated model weights; and defining the updated model weights as new reference model weights, wherein the stochastic gradient is using , is a regularization loss corresponding to the loss of the i-th instance, is a regularization loss corresponding to the loss of the i-th instance with respect to the reference model weights, is a regularized empirical risk, 、 and are gradients with respect to the dimensions in the active computing device. 、 and are gradients with respect to the dimensions in the active computing device.

17. The method of claim 16, wherein, the summing of the current dot product is performed using a first tree structure scheme, the summing of the reference dot product is performed using a second tree structure scheme, the first tree structure scheme is substantially different from the second tree structure scheme.

18. The method of claim 13, wherein, the training the federated learning model in the active computing device includes: computing a local gradient of an instance in the active computing device; performing, in parallel and independently in each of the active worker and the at least one passive worker: selecting one of the instances; computing a current dot product between the current model weights and the selected instance; instructing the at least one passive computing device to compute a current dot product of the at least one passive computing device using an instance corresponding to the selected instance; summing the current dot product from the active computing device and the current dot product from the at least one passive computing device to obtain a summed dot product; computing a stochastic gradient of the federated learning model using the summed dot product; and updating the model weights using the stochastic gradient of the federated learning model and a learning rate to obtain updated model weights, and updating the local gradient, wherein the stochastic gradient is computed using , is a regularization loss corresponding to the loss of the i-th instance, is is a gradient with respect to a dimension of the i-th instance in the active computing device, is a local gradient in the active computing device.

19. A non-transitory computer-readable medium storing computer-executable code, wherein, the computer executable code, when executed at a processor of an active computing device, is configured to: train a federated learning model in the active computing device using dimensions of a plurality of training instances in the active computing device; and instruct at least one passive computing device to train a federated learning model in the at least one passive computing device using dimensions of a plurality of training instances in the at least one passive computing device, wherein the active computing device and the at least one passive computing device are in communication with each other; and wherein a plurality of training instances in the active computing device and a plurality of training instances in the at least one passive computing device are indexed, an index of a training instance in the plurality of training instances selected by the at least one passive computing device is independent of an index of a training instance in the plurality of training instances selected by the active computing device at a same time, such that the training of the federated learning model in the active computing device and the training of the federated learning model in the at least one passive computing device are asynchronous. communicate intermediate results between the active worker and the at least one passive worker via a coordinator using a tree structure scheme, the coordinator being in communication with the active computing device and the at least one passive computing device.

20. The non-transitory computer-readable medium of claim 19, wherein, ​

Images