Asynchronous multi-faceted weighted federated learning systems and methods

Abstract

Systems and methods for asynchronously training a federated learning model are presented. The system includes a master device and K institutional clients. The master device is configured to execute an aggregation thread and K client management threads. The kth executed client management thread includes: during initialization, instructing the kth client to initiate a first training iteration of its local federated learning model to obtain a training event E k ; or upon receiving an aggregation event E aggr from the aggregation thread, instructing the kth client to conduct a tth training iteration of its local federated learning model to obtain a training event E k ; and sending E k to the aggregation thread. The executed aggregation thread includes: upon receiving E k , updating a global federated learning model in the master device to obtain an aggregation event E aggr ; and sending the aggregation event E aggr to the kth client management thread.

Description

[0001] Cross-referencing

[0002] References are cited and discussed in the description of this disclosure, which may include patents, patent applications, and various publications. The citation and / or discussion of such references are provided solely to clarify the description of this disclosure and do not imply that any such reference is “prior art” as disclosed herein. All references cited or discussed in this specification are incorporated herein by reference in their entirety, to the same extent as each individual reference is incorporated by reference individually. Technical Field

[0003] This disclosure generally relates to federated learning, and more specifically to an asynchronous multifaceted weighted federated learning system and method for accelerating collaborative modeling among multiple institutions. Background Technology

[0004] The background description provided herein is intended to provide a general overview of the context of this disclosure. Within the scope of this background description, the inventors' work, and descriptions that may not conform to the prior art at the time of application, are neither explicitly nor implicitly acknowledged as prior art to this disclosure.

[0005] In recent years, data privacy has become a significant concern. The General Data Protection Regulation (GDPR) came into effect in the European Union (EU) in 2018 to protect the privacy and security of EU citizens, while the California Consumer Privacy Act (CCPA) of 2018 provides consumers with control over the personal information of businesses. Despite these increasingly stringent privacy laws, different organizations still need to collaborate on building some common models using their own institutional data.

[0006] Federated learning is a well-known solution to this challenging problem. Through homomorphic encryption, multi-party computation, and other techniques, federated learning enables multiple participants to collaboratively train a general machine learning / deep learning model without sharing data, while ensuring data privacy and security. Federated learning consists of two phases: 1) Each client (training participant) uses, but does not share, its own dataset to train a local model, and then sends the parameters of the local model to the master device; 2) After receiving all / part of the local models, the master device (training coordinator) aggregates the global model using various aggregation algorithms. When aggregation is performed synchronously, there is idle time, and waiting for synchronization wastes computational resources. When aggregation is performed asynchronously, convergence is difficult.

[0007] Therefore, there is a need in this field to address the aforementioned defects and shortcomings. Summary of the Invention

[0008] In some aspects, the present disclosure relates to an asynchronous multi-aspect weighted federated learning algorithm that can compress idle time, solve the problem of long synchronization latency and the problem of convergence difficulty in multi-institution collaborative modeling. In some embodiments, the asynchronous algorithm solves the problem of long synchronization latency and implements fully asynchronous aggregation in the institution federated learning setting, which is different from the partial asynchrony in device asynchronous aggregation. In some embodiments, the present disclosure designs two types of events (aggregation event, local model training management event) on the server side, defines a thread pool, and each thread in the thread pool manages the training of an institution, which is fully asynchronous and different from the partial asynchrony using a periodic trigger method. Therefore, the algorithm can compress all idle time of each institution, realize fully asynchronous aggregation, and accelerate the training speed of the global model. In addition, the multi-aspect weighted aggregation method of the present disclosure can improve the convergence difficulty problem in the multi-institution collaborative modeling setting by considering the influence of all factors, including the skewed dataset size of each client, the delay of the local model, and the accuracy fluctuation of the local model. The multi-aspect characteristics of the present disclosure guarantee the convergence of the global model and further accelerate the training speed of the global model.

[0009] In some aspects, the present disclosure relates to a federated learning system. In some embodiments, the system includes a host computing device comprising a processor and a storage device having stored therein computer executable code. The computer executable code, when executed at the processor, is configured to:

[0010] performing a kth client management thread of the K client management threads by: during initialization, instructing a kth client computing device to initiate a first training iteration of a kth client federated learning model to obtain a training event E k ; upon receiving an aggregation event E aggr from the aggregation thread, instructing the kth client computing device to perform a tth training iteration of the kth client federated learning model using the aggregation event E aggr to obtain the training event E k ; and sending the training event E k to the aggregation thread; and

[0011] performing the aggregation thread by: upon receiving the training event E k from the kth client management thread, updating a global federated learning model in the host computing device to obtain the aggregation event E aggr ; and sending the aggregation event E aggr to the kth client management thread.

[0012] In some embodiments, the training event E kincludes a client number k, a global round number t, a client reporting timestamp ts, a dataset size n for training the kth client federated learning model k , an accuracy p of the tth training iteration of the kth client federated learning model t，k , and a local model weight W of the kth client federated learning model after the tth training iteration t，k，ts . In certain embodiments, k ∈ K, K is a positive integer, representing a total number of client computing devices, t ∈ T, T is a positive integer, representing a total number of training iteration rounds for each client federated learning model stored in a respective one of the K clients.

[0013] In certain embodiments, the step of updating the global federated learning model to obtain the aggregation event E aggr is performed using the following equation:

[0014]

[0015] where W t is an updated weight of the global federated learning model after merging the received training event E k , W t-1 is a weight of the global federated learning model before merging the received training event E k , n is a total dataset size of all K clients, a t is a delay discount rate of the tth training iteration of the kth client federated learning model, p t，k is an accuracy of the tth training iteration of the kth client federated learning model.

[0016] In certain embodiments, the delay discount rate a t is a function of t, ts and a, t represents an earliest time when one of the K federated learning models completes the tth training iteration, a is a hyperparameter representing a delay weight value. In certain embodiments, the delay discount rate a t is calculated using at least one of a linear function, a polynomial function and an exponential function.

[0017] In certain embodiments, each of the K client federated learning models is initialized by knowledge distillation. In certain embodiments, each of the K client federated learning models is initialized randomly.

[0018] In certain embodiments, each of the K client federated learning models is trained using stochastic gradient descent (SGD).

[0019] In some embodiments, the federated learning is institutional federated learning. In some embodiments, K is a positive number in the range of 2 to 100. In some implementations, K is in the range of 2 to 10.

[0020] In some embodiments, the global federated learning model includes T global federated learning model versions, and the k-th client federated learning model is configured to update a corresponding global federated learning model version among the T global federated learning model versions after each training iteration.

[0021] In some aspects, this disclosure relates to a federated learning method. In some embodiments, the method includes:

[0022] K client management threads are executed through the main computing device, wherein the kth client management thread is executed by the following operation: during initialization, the kth client computing device is instructed to start the first training iteration of the kth client federated learning model to obtain training event E. k Upon receiving an aggregation event E from the aggregation thread aggr At that time, the k-th client computing device is instructed to use the aggregate event E. aggr The k-th client federated learning model is trained for the t-th iteration to obtain the training event E. k ; and the training event E k Send to the aggregation thread; and

[0023] The aggregation thread is executed via the main computing device, wherein the aggregation thread is executed by the following operation: upon receiving the training event E from the k-th client management thread. k At that time, the global federated learning model in the main computing device is updated to obtain the aggregation event E. aggr ; and the aggregation event E aggr Send to the k-th client management thread.

[0024] In some embodiments, the training event E k This includes the client ID k, the global round number t, the client report timestamp ts, and the dataset size n for training the k-th client federated learning model. k The accuracy p of the k-th client-side federated learning model in the t-th training iteration. t，k And the local model weights W of the k-th client-side federated learning model after the t-th training iteration. t，k，tsIn certain embodiments, k e K, K is a positive integer, representing a total number of client computing devices, t e T, T is a positive integer, representing a total number of training iteration rounds for each client federated learning model stored in a respective one of the K clients.

[0025] In certain embodiments, updating the global federated learning model to obtain the aggregation event E aggr is performed using the following equation:

[0026]

[0027] where W t is the updated weight of the global federated learning model after merging the received training event E k , W t-1 is the weight of the global federated learning model before merging the received training event E k , n is a total dataset size of all K clients, a t is a delay discount rate of the t-th training iteration of the k-th client federated learning model, p t，k is an accuracy of the t-th training iteration of the k-th client federated learning model.

[0028] In certain embodiments, the delay discount rate a t is a function of t, ts and a, t represents the earliest time at which one of the K federated learning models completes the t-th training iteration, a is a hyperparameter representing a delay weight value; wherein the delay discount rate a t is calculated using at least one of a linear function, a polynomial function and an exponential function.

[0029] In certain embodiments, each of the K client federated learning models is initialized by knowledge distillation.

[0030] In certain embodiments, the federated learning is institutional federated learning.

[0031] In certain aspects, the present disclosure relates to a non-transitory computer-readable medium storing computer executable code. The computer executable code, when executed at a processor of a computing device, is configured to perform the above-described method.

[0032] In certain embodiments, the present disclosure is used in a case where an e-commerce company and a transportation carrier collaboratively build an NLP-based high-performance model to represent / extract logistics information, while the two institutions cannot violate data protection laws and share their customer data with each other.

[0033] These and other aspects of the present disclosure will become apparent from the following description of the preferred embodiment taken in conjunction with the following drawings, although changes can be made and equivalents employed, without departing from the true spirit and scope of the novel concept of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings illustrate one or more embodiments of the present disclosure and together with the written description, serve to explain the principles of the present disclosure. Wherever possible, the same reference numbers are used throughout the drawings to refer to the same or like elements.

[0035] Figure 1A Synchronous computation according to certain embodiments of the present disclosure is schematically depicted.

[0036] Figure 1A Asynchronous computation according to certain embodiments of the present disclosure is schematically depicted.

[0037] Figure 2 Asynchronous federated learning system according to certain embodiments of the present disclosure is schematically depicted.

[0038] Figure 3 Framework for multi-institutional collaborative modeling using asynchronous aggregation algorithm according to certain embodiments of the present disclosure is schematically depicted.

[0039] Figure 4 Algorithm operating in a master device of a federated learning system according to certain embodiments of the present disclosure is schematically depicted.

[0040] Figure 5 Algorithm operating in a client of a federated learning system according to certain embodiments of the present disclosure is schematically depicted.

[0041] Figure 6 Update order of models according to certain embodiments of the present disclosure is schematically depicted.

[0042] Figure 7 Training method of an asynchronous federated learning model according to certain embodiments of the present disclosure is schematically depicted.

[0043] Figure 8 Training method of a local federated learning model according to certain embodiments of the present disclosure is schematically depicted. DETAILED DESCRIPTION

[0044] The present disclosure is described in more detail in the following examples, which are intended to be illustrative only as numerous modifications and variations will be apparent to persons skilled in the art. Various embodiments of the present disclosure are now described in detail. Referring to the drawings, like numbers indicate like parts throughout the views. As used in the description of the disclosure and the claims the meaning of "a", "an", and "the" includes plural references unless the context clearly dictates otherwise. Also, as used in the description of the disclosure and the claims the meaning of "in" includes "in" and "on" unless the context clearly dictates otherwise. Furthermore, titles or subtitles can be used in the specification for the convenience of a reader, which are neither intended nor should be interpreted to limit the scope of the disclosure. Moreover, some terminology used in the specification and the claims have been discussed above and the above discussion is incorporated by reference herein in its entirety.

[0045] The terminology used in the present specification is generally intended to correspond to the terminology in the art, in the context of the present disclosure, and in the particular context in which each term is used. Certain terminology used in the description of the present disclosure is discussed below or elsewhere in the specification, to provide the practitioner with additional guidance regarding the description of the present disclosure. It will be appreciated that the same thing can be said in more than one way. Consequently, alternative language and synonyms can be used for any one or more of the terms discussed herein. No special significance is to be placed upon whether or not a term is elaborately recited herein or discussed in detail. The present disclosure provides certain synonyms for certain terms, which should be taken to read disjunctively. The use of any of these terms in various places in the specification is not intended to exclude the use of other terms in other places. The use of a term in the specification or claims has no special significance beyond its ordinary meaning.

[0046] As described herein, the term "module" can refer to, be part of, or include an Application Specific Integrated Circuit (ASIC); an electronic circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor (shared, dedicated, or group) that executes code; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip. The term "module" can include memory (shared, dedicated, or group) that stores code executed by the processor.

[0047] The term "code," as used herein, can include software, firmware, and / or microcode, and can refer to program(s), routine(s), function(s), class(es), and / or object(s). Shared meaning of the term as used above means that a single (shared) processor can execute portions of or all of code from multiple modules. Also, portions or all of code from multiple modules can be stored in a single (shared) memory. Group meaning of the term as used above means that a group of processors can execute portions of or all of code from a single module. Also, a group of memories can be used to store some or all code from a single module.

[0048] As described herein, the term "interface" generally refers to a communication tool or device used at the interaction point between components to perform data communication between components. Generally, interfaces can be applied at both the hardware and software levels, and can be unidirectional or bidirectional. Examples of physical hardware interfaces can include electrical connectors, buses, ports, cables, terminals, and other I / O devices or components. Components communicating with the interface can be, for example, multiple components of a computer system or peripheral devices.

[0049] This disclosure relates to computer systems. As shown in the accompanying drawings, computer components may include physical hardware components, shown as solid line blocks, and virtual software components, shown as dashed line blocks. Those skilled in the art will understand that, unless otherwise stated, these computer components may be implemented as software, firmware, or hardware components or combinations thereof, but are not limited to these forms.

[0050] The apparatus, systems, and methods described herein can be implemented by one or more computer programs executed by one or more processors. The computer program includes processor-executable instructions stored on a non-transitory tangible computer-readable medium. The computer program may also include stored data. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory, magnetic storage, and optical storage.

[0051] This disclosure will now be described more fully below with reference to the accompanying drawings, in which embodiments of the disclosure are illustrated. However, this disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the scope of the disclosure to those skilled in the art.

[0052] Some symbols and terms used in certain embodiments of this disclosure are as follows: K is the number of clients; B is the number of local mini-batches (hyperparameter); E is the number of local epochs (hyperparameter); T is the number of global epochs (hyperparameter); α is the delay weight (hyperparameter); W is the model parameter; η is the learning rate (hyperparameter); L is the loss function; f(t, ts) is the staleness function.

[0053] In some embodiments, the model aggregation algorithm in a federated learning setting is based on a synchronization mechanism (e.g., FedSGD, FedAvg, FedMA, etc.), which means that the collaborating parties must wait for most of the participants to finish training in each round of aggregation. Figure 1A Synchronous computation according to certain embodiments of this disclosure is illustrated schematically. Figure 1AAs shown, when performing synchronous computation, there is a need to coordinate the clients 1-3 at certain time intervals. At each time interval for the clients 1-3, the computation can include computation time and idle time, so that the clients 1-3 can start the next round of training at the same time. In a device federated learning setting, 1) the number of training participants (devices) is large, and if some participants are discarded / ignored due to unstable environment, the synchronous aggregation algorithm can still proceed; 2) the dataset for each device is small, so the end time interval for training the local model in a global round among devices is small, and the smaller the interval, the less the idle time. However, the interval for multi-agency collaborative modeling is large, and synchronization is difficult because: 1) the number of training participants is small, and the data of each client is important, and the local model of each client cannot be ignored in the aggregation round, otherwise the collaborative model building is meaningless; 2) although the training environment in the multi-agency setting is relatively stable, it is still difficult to synchronize due to the uneven size of the data sets and the heterogeneous environment of different agencies. In addition, large models (such as BERT) are used among agencies, and large models will result in a large end time interval for training the local model.

[0054] In the above embodiment, the synchronization mechanism results in longer idle time / synchronization waiting, wasting computing resources. In contrast, asynchronous aggregation can reduce idle time and can complete more rounds (5 / 8 / 5 rounds) than synchronous aggregation (all 3 rounds) in the same time period, speeding up the training process. Figure 1B Asynchronous computation according to certain embodiments of the present disclosure is schematically depicted. As shown, Figure 1B As shown, there is no need for coordination among the clients 1-3, and the computation in the workers 1-3 is continuous. After one of the workers 1-3 completes a round of training, the next round of training begins, with no idle time. Therefore, asynchronous computation is more efficient because it keeps all computing resources busy all the time. Although asynchronous aggregation is more efficient, it is more difficult to converge in the same training rounds than synchronous aggregation. It should be noted Figure 1A and Figure 1B As shown, the computation / IO / network cycle of each client in each round changes because the running environment of each client can change (e.g., network transmission is unstable under public network), the size of the data set can change, and so on.

[0055] To further improve Figure 1A and Figure 1B The present disclosure provides an asynchronous multi-aspect weighted federated learning system and method, which realizes true asynchrony. Figure 2 A federated learning system according to certain embodiments of the present disclosure is schematically depicted. As shown, Figure 2As shown, the system 200 includes a master device 210, a network 230, and K client devices 250. The master device 210 and the K client devices 250 communicate with each other through the network 230. In certain embodiments, Figure 2 Each of the master device 210 and the client devices 250 as shown can be a server computer, a cluster, a cloud computer, a general-purpose computer, a headless computer, or a special-purpose computer that provides federated learning capability. In certain embodiments, each of the master device 210 and the client devices 250 can be a server computing device. In certain embodiments, the master device 210 is also referred to as a coordinator, a server, or a master computing device 210; each client device 250 is also referred to as a worker or a client computing device. In certain embodiments, the master device 210 includes a global federated learning model, and each client device 250 includes a local federated learning model and respective private data. The master device 210 and the client devices 250 can work together to train the federated learning model, but the private data is protected in the respective client and is not shared with other client devices 250 or the master device 210, and each client device 250 has respective model weights. In certain embodiments, the federated learning model is an asynchronous federated learning model, and the private data is horizontally partitioned data. The network 230 can be a wired or wireless network and can be in various forms, such as public networks and private networks. Examples of the network 230 can include, but are not limited to, a local area network (LAN) or a wide area network (WAN), including the Internet. In certain embodiments, two or more different networks 230 can be used to connect the master device 210 and the client devices 250.

[0056] As Figure 2As shown, the master device 210 can include, but is not limited to, a processor 212, a memory 214, and a storage 216. In certain embodiments, the master device 210 can include other hardware components and software components (not shown) to perform their corresponding tasks. Examples of these hardware and software components can include, but are not limited to, other required memories, interfaces, buses, input / output (I / O) modules or devices, network interfaces, and peripheral devices. The processor 212 can be a central processing unit (CPU) configured to control the operation of the master device 210. The processor 212 can execute an operating system (OS) or other applications of the master device 210. In certain embodiments, the master device 210 can have multiple CPUs as processors, such as two CPUs, four CPUs, eight CPUs, or any suitable number of CPUs. The memory 214 can be a volatile memory, such as a random access memory (RAM), for storing data and information during the operation of the master device 210. In certain embodiments, the memory 214 can be a volatile memory array. In certain embodiments, the master device 210 can run on multiple memories 214. In certain embodiments, the master device 210 can also include a graphics card to assist the processor 212 and the memory 214 in image processing and display. The storage 216 is a non-volatile data storage medium for storing the operating system (not shown) and other applications of the master device 210. Examples of the storage 216 can include a non-volatile memory, such as a flash memory, a memory card, a USB drive, a solid state drive, a hard disk drive, a floppy disk, an optical drive, a solid state hard drive, or any other type of data storage device. In certain embodiments, the master device 210 can have multiple storages 216, which can be the same storage or different types of storages, and the applications of the master device 210 can be stored in one or more storages 216 of the master device 210. In this embodiment, the processor 212, the memory 214, and the storage 216 are components of the master device 210 (e.g., a server computing device). In other embodiments, the master device 210 can be a distributed computing device, and the processor 212, the memory 214, and the storage 216 are shared resources from multiple computing devices in a predetermined area.

[0057] The storage 216 also includes an asynchronous federated learning application 218. The asynchronous federated learning application 218, when executed, is configured to coordinate the training of the global federated learning model in the master device 210 and the training of the local federated learning model in each of the K clients 250. The asynchronous federated learning application 218 includes an aggregation thread module 220, K client management thread modules 222-226, and a global federated learning model 228.

[0058] In some embodiments, the asynchronous federated learning application 218 may include other applications or modules required for the operation of the asynchronous federated learning application 218, such as listeners for receiving notifications or instructions from threads and clients 250, or preparation modules for defining events and threads. The defined threads include aggregate threads defined by the aggregate thread module 220 and K client management threads defined by the K client management thread modules 222-226. The K client management threads include a first client management thread or client 1 management thread, a second client management thread or client 2 management thread, ..., the kth client management thread or client k management thread, ..., and the Kth client management thread or client K management thread. The aggregate thread is configured to update the global federated learning model 228, and each of the K client management threads is configured to instruct one of the corresponding clients 250 to update its local federated learning model. The defined events include training events from any client 250 and aggregate events from the master device 210. A training event E is generated for one round of training of the local federated learning model in the corresponding client k. k Training event E k This can include the client ID k, the global round number t (iteration number or round number), the client report timestamp ts, and the data size n. k And the parameters W of the local federated learning model after this round of training. t，k，ts k∈K, where K is a positive integer representing the total number of clients (250). t∈T, where T is a positive integer representing the total number of global epochs for training the federated learning model on each client. Clients can report timestamps (ts) to report the generation of training event E. k The time ts is the end time of the current training round of the local federated learning model in client k, where time ts is relative to the time of round t or the start time of training in round t. Data size n k It is the size of the dataset k. Aggregate event E aggr Corresponding to receiving training event E from one of the client 250 k Subsequently, an update to the global federated learning model in master device 210 occurs, event E. aggr This can include the client ID k, the global round number t, and the global federated learning model parameters W. t Using event E k Update parameter W t Then, it will include the update parameter W. t Event E aggr Send back to client k. Each of the K clients has T E. k Therefore, during the training of the global and local models, there will be T×K E... k Therefore, the global model weights W t It will be updated T×K times.

[0059] It should be noted that modules 220-228 are each implemented through computer-executable code or instructions, or data tables or databases, or a combination of hardware and software, which together form an application. In some embodiments, each module further includes sub-modules, or some modules may be combined into a stack. In other embodiments, some modules may be implemented as circuits rather than executable code. In some embodiments, the modules may also be collectively referred to as a model, which can be trained by aggregating weights from a local federated learning model from client 250, and this model can send the updated weights of the global federated learning model 228 to client 250. In some embodiments, the functionality of asynchronous federated learning application 218 corresponds to... Figure 4 Algorithm 1 is shown in the figure.

[0060] like Figure 2 As shown, aggregation thread module 220 is configured to provide an aggregation thread when executed. K client management thread modules 222–226 are configured to provide K client management threads when executed. Each of the K client management threads corresponds to one of the K clients 250. The following description will use client k as an example, which can be any one of the K clients from client 1 to client K.

[0061] The aggregation thread is configured to receive training events E from the client k management thread. k Using training event E k Update the global federated learning model 228 to obtain the aggregated event E containing the updated global weights. aggr and aggregate event E aggr This is sent back to the client-side management thread (e.g., k), so that it can invoke client-side k to perform the next training iteration using the updated global model weights. It should be noted that the training event E... k This can correspond to any training iteration of the local federated learning model in client k, and the corresponding aggregation event E. aggr The data will be sent to client k via the management thread, allowing client k to begin the next training iteration of the local federated learning model. Once client k completes the predefined training iteration, the aggregation thread will update the weights of the global federated learning model 228 as the final model weights, and send these final model weights to all clients 250 as the final weights for all local models.

[0062] The aggregation thread is configured to update the global federated learning model 228 by calculating delayed weight values ​​and using these delayed values ​​to calculate the weights of the global federated learning model. The delayed weight values ​​are calculated using the following equation:

[0063] α t= f(t, ts, a) (1)

[0064] In certain embodiments, the disclosure defines the monotonically decreasing function f(t, ts, a) using various mathematical functions, such as linear function, polynomial function, exponential function, etc. For example:

[0065] Linear function:

[0066] Polynomial function: f = (1 + ts - t) -α (3)

[0067] Exponential function: f = e -α(ts-t) (4)

[0068] where t in equations (1)-(4) is not the global round number, but the time when the client that finishes the tthglobal round earliest finishes the tthglobal round, ts is the timestamp when each local client finishes its tthglobal round, a is the delay weight value (hyperparameter), a e (0, 1), e is the natural logarithm. In certain embodiments, the exponential function (4) is used, which is superior to the linear function (2) and the polynomial function (3). In certain embodiments, ts - t in equations (2)-(4) is defined as follows: (1) for the tthtraining round, the K clients finish at different times, the client that finishes the tthtraining round earliest finishes at timestamp A, (2) all other clients finish their tthtraining round at some time after timestamp A, client k finishes its tthtraining round at timestamp ts, (3) ts - t can be the time period from timestamp A to timestamp ts, which can be a value in seconds or milliseconds, (4) in certain embodiments, ts - t can not be a value of time period, but a normalized value in the range of 0-1, where the larger the time difference between timestamp A and timestamp ts, the closer the value to 1.

[0069] In the institutional federated learning environment, due to environment heterogeneity, skewed dataset size, and large model, the delay of the local model is more serious. The aggregation method according to certain embodiments of the disclosure uses a delay discount rate a t As an important aspect, the delay impact is quantified and benefits the global model convergence.

[0070] After obtaining the delay weight value a t Subsequently, the aggregation thread is further configured to update the weights of the global federated learning model 228 by:

[0071]

[0072] where W t is the parameter or weight of the global federated learning model 228 after merging the local model parameters W t，k，ts ​t-1 is the merged W t，k，ts global federated learning model's weights, n is the total dataset size from all K clients 250, a t is the delayed weight value of the t-th training iteration of the federated learning model of client k, p t，k is the accuracy of the t-th training iteration of the federated learning model of client k. It should be noted that the weight W t-1 is not an absolute value, as the weight is updated each time one of the K client student models is updated. W t-1 is actually the weight of the student model round-(t-1) in the master device, and the weight is updated K times during the training of the model. Similarly, the weight W t is also updated K times.

[0073] Considering various aspects that affect the training convergence of the global federated learning model, including the skewed dataset size of each client, the delay of the local model, and the accuracy fluctuation of the local model, as shown in equation (5) above, the present disclosure combines the three aspects, i.e., the dataset size, the delay discount rate, and the accuracy, to improve the convergence of the global federated learning model 228 during training. From these three aspects, the present disclosure can alleviate the error caused by the skewed dataset size of each client, the delay of the local model, and the accuracy fluctuation of the local model when aggregating the global model.

[0074] The first aspect of the weighted aggregation method is the data size ratio. In the institutional federated learning environment, the number of institutions is less than the number of devices in the device federated learning environment. Therefore, the skewed dataset size between institutions has a greater impact on the convergence of the global model. In certain embodiments, the client with a larger data set size ratio contributes more to the global model parameters. Therefore, as shown in equation (5) above, during the aggregation process, the present disclosure is configured to use the data set size ratio n k / n as an important aspect, quantifies the contribution of each institution based on the data set size and facilitates the convergence of the global model. Here n k is the data size of client k, and n is the total data size of all K clients.

[0075] The second aspect of the weighted aggregation method is the delay discount rate. As described above, the present disclosure is configured to calculate the delay discount rate using equation (1): a t =f(t, ts, a), which defines a monotonically decreasing function using the difference ts-t and the hyperparameter a.

[0076] A third aspect of the weighted aggregation method is the accuracy of the local models. In certain embodiments, each institution such as client k uses a train / test split and cross-validation to train the local model in each round to avoid overfitting, and outputs the local model and its accuracy p t，k represent the performance of the current local model. Integrating a poorly performing local model into the global model can increase the error and degrade the performance of the global model. Therefore, the present disclosure is configured to use the accuracy p t，k as another aspect to quantify the contribution of the local model. Adding the accuracy aspect helps the convergence of the global model. Therefore, a client with a better-performing local model contributes more to improving the performance of the global model. t，k

[0077] By incorporating these three aspects in equation (5), certain embodiments of the present disclosure effectively improve the convergence of the global federated learning model.

[0078] In addition to the one aggregation thread described above, the executed asynchronous federated learning application 218, specifically the executed client management thread model 222-226, also provides K client management threads. Each of the K client management threads is configured to manage the operations of a corresponding one of the K clients. The role of each of the K client management threads is described with the client k management thread as an example. In certain embodiments, the client k management thread is configured to instruct the client k to run the first training iteration when initialized. The client k management thread can invoke the training of the client k using the ClientUpdate function, which has parameters including the client number k, the round t, and the current weights of the global model. Because it is the first training iteration of the client k, the round value is 1 and the current weights are null. For the second training iteration, the round value is 2 and the current global weights are W2. For the tth training iteration, the round value is t and the current global weights are W t . The ClientUpdate(k, t, W t ) will run at the client k and send the running results to the client k management thread so that the client k management thread generates the current training event E k based on the running results. After the generation is completed, the client k management thread is also configured to send the current training event E k to the aggregation thread.

[0079] ​The global federated learning model 228 can be any type of federated learning model used to aggregate the client federated learning model parameters to update the global federated learning model weights and send the updated global weights to the corresponding client models. In certain embodiments, the global federated learning model 228 includes K global models, each corresponding to a training round.

[0080] The clients 250 include K clients from client 1 to client K. Each client 250 can be a computing device having a processor, a memory, and a storage. The description of the processor, the memory, and the storage of the client 250 is substantially the same as the description of the processor 212, the memory 214, and the storage 216 of the master device 210. The storage of one client 250 includes a private dataset, a teacher model, and a student model. The detailed information about the dataset, the teacher model, and the student model of the client k is described below. As Figure 2 As shown, the client k includes a dataset k, a teacher model k, and a student model k.

[0081] When starting the training of the local model in the client k in the first iteration or the first round, the local model parameters need to be initialized. An overlarge initialization can cause gradient explosion, while an undersized initialization can cause gradient vanishing. Therefore, a good initialization can prevent the gradient of the neural network activation from vanishing or exploding. In certain embodiments, the Glorot initialization (also known as Xavier initialization) is used for the initialization, in which the present disclosure initializes each weight according to a Gaussian distribution. In certain embodiments, for simplicity, the client k does not include the teacher-student model structure, and the initialization of the deep neural network is random.

[0082] In certain embodiments, when the model is very large, such as the BERT model, the present disclosure uses the knowledge distillation method extracted from the pre-trained teacher model to initialize the local student model parameters. The master device (server) will use the student model for iterative aggregation. Figure 3 A framework is shown for using an asynchronous aggregation algorithm for multi-institution collaborative modeling and using knowledge distillation to initialize the local student model according to certain embodiments of the present disclosure. As Figure 3As shown, the master device 310 has a global student model database, which includes T global student model versions. Each client 350, e.g., client k, has a dataset k, a teacher model k, and a student model k. The trained teacher model k can be transferred to the student model k. The transfer from the teacher model k to the student model k is performed during initialization, after which the student model k and the dataset k can be used for subsequent coordination with the global student model database. In certain embodiments, the knowledge distillation is performed according to Hinton et al., “Distilling the knowledge in a neural network,” Geoffrey Hinton, Oriol Vinyals, and Jeff Dean, arXiv: 1503.02531, 2015, which is incorporated by reference in its entirety.

[0083] After the training of the local model in the client k is initiated, and the local model (student model k) is invoked again in the second or later round, the local model is configured to perform a new training iteration with the received (current) global weights. In certain embodiments, the local model in the client k will receive the global model weights W t defined as the current local model weights W t，k，ts and update the local model weights W t，k，ts using the dataset k by:

[0084] For each local round i ∈ E:

[0085] For each batch b ∈ B:

[0086]

[0087] where i is the local round number, E is a hyperparameter denoting the total number of local rounds per training iteration or per round of training of the local model, E is a positive integer. b is the batch number of the training data, B is a hyperparameter denoting the total number of batches of training data from the dataset k, B is a positive integer. η is the learning rate of the local model (hyperparameter), is the loss function using the current local weights and the training data of the current batch, W t+1，k，ts is the local weights after the t-th round of training. In certain embodiments, the training of the local model is performed using SGD. After the local model training is completed, the local model of the client k is also used to return the training results to the client management thread. In certain embodiments, the training results include t+1, ts, n k , P t+1，k , and W t+1，k，ts .

[0088] Other clients in the clients 250 can have the same teacher model-student model structure as the client k, but different datasets from each other.

[0089] In certain embodiments, the asynchronous federated learning application 218 can also include a user interface and / or a scheduler. The user interface can be used to configure parameters of the application, as well as to train global and local models. The scheduler can be used to schedule operations and data flow between modules of the application, as well as between the master device and the clients. In certain embodiments, the asynchronous federated learning application 218 can also be stored in one of the clients, which acts as both a master device to coordinate training of the global model and as a client to train the local model.

[0090] Figure 4 An algorithm operating in a master device according to certain embodiments of the present disclosure is schematically depicted. Figure 5 An algorithm operating in a client according to certain embodiments of the present disclosure is schematically depicted. In certain embodiments, the master device is the master device 210 shown in Figure 2 and the client is any one of the clients 250 shown in Figure 2 . This algorithm shows the details of the fully asynchronous aggregation. As shown in Figure 4 , the algorithm defines two types of events for the asynchronous aggregation, namely, the aggregation event E aggr and the local model training management event E k . Once the aggregation thread in the server receives the event E k , which means that the local model from the client k is ready, the server can immediately update the global model. Once the training management thread in the server receives the event E aggr , which means that the global model is updated using the local model from the client k, the server can immediately inform the client k to use the new weights of the global model for the next training iteration. Through these two types of events, the algorithm can reduce the idle time of each agency in the related synchronous-based aggregation algorithm due to the synchronous waiting.

[0091] In contrast, for the asynchronous algorithm in the device federated learning environment, the training of each agency is triggered periodically by the management thread on the server, and there is a lot of idle time waiting for the periodic scheduling. In the agency federated learning environment, the number of agencies is small, and the present disclosure is designed such that the server can maintain a pool of client management threads, one thread in the pool manages the training of one agency, which is more efficient than using one thread to manage multiple devices in the device federated learning environment, further compressing the idle time in the scheduling process. Therefore, Figure 4 and Figure 5 the asynchronous algorithm shown in and is fully asynchronous, which can compress all idle time of each agency, and there is no idle time for any agency.

[0092] Figure 6 The update order of the model according to certain embodiments of the present disclosure is schematically depicted. Figure 6 The components shown in Figure 3 , Figure 6 The data flow shown in Figure 4 and Figure 5 As shown in Figure 6 , the global student model database includes T student model versions corresponding to T global rounds. The present disclosure trains each student model stored in a corresponding client T times, and the T times of local model training results of each client are respectively used to update the T student models in the master device. In one aspect, the client student model training is performed in parallel and independently, and is controlled by K client management threads. In another aspect, each student model in the master device is updated once upon receiving the training result from one client student model, so the weights of the student models in the master device are influenced by each client student model. For example, student model k is trained for T rounds. After the t-th training round of student model k, the training result will be used to update student model round-t in the master device. Therefore, each of the student models 1 to K will be trained t times, and each of the student models student model round-1 to student model round-T in the master device will be updated K times.

[0093] Figure 7 The training method of the asynchronous federated learning model according to certain embodiments of the present disclosure is schematically depicted. In certain embodiments, the training process corresponds to Figure 4 and Figure 5 Algorithm 1 and Algorithm 2 shown in. In certain embodiments, the method can be implemented on the system shown in Figure 2 It should be noted in particular that the steps of the method can be performed in different orders unless otherwise specified in the present disclosure, and are not limited to the order shown in Figure 7 .

[0094] As shown in Figure 7 , when the asynchronous federated learning application 218 is initialized, it defines the aggregation event E aggr and the training event E k , and runs the aggregation thread 702 and K client management threads. For brevity, only the client k management thread 704 is shown. Client k will run its local model training when invoked by the client k management thread 704. In certain embodiments, the asynchronous federated learning application 218 also defines the training iteration or training round for the client, for example, the global round number T.

[0095] At step 710, the client k management thread 704 is executed, which defines the function ClientUpdate(k, t, W t) begins the first training iteration of client k, where k is the corresponding client number, t is the global round, and W t is the global model parameter or global model weight of client k at round t. Because it is the client k management thread 704, the client number is k; because it is the first training iteration of client k, t is 1; because it is the beginning of model training, there is no global weight, and the global model weight is denoted as “None”. Thus, the client k management thread 704 defines the function as ClientUpdate(k, 1, None). It should be noted that the initialization of the local model training can begin at substantially the same time. In other words, the functions ClientUpdate(1, 1, None), ClientUpdate(2, 1, None),..., ClientUpdate(k, 1, None), and ClientUpdate(K, 1, None) are defined or prepared at substantially the same time.

[0096] After the function is defined at step 712, the client k management thread 704 sends the function ClientUpdate(k, 1, None) to client k, or in other words, calls client k to execute the function ClientUpdate(k, 1, None). Because the initialization of the local model training is at substantially the same time, the K client management threads actually send the functions ClientUpdate(1, 1, None) to ClientUpdate(K, 1, None) to the respective K clients at substantially the same time.

[0097] At step 714, upon receiving the function ClientUpdate(k, 1, None), because t is 1 and / or W1 is “None”, client k determines that this is the first training iteration. Client k initializes the parameters W t，k，ts , where t is the iteration number, k is the client number, and ts is the client reported timestamp. Because it is the first iteration and the timestamp is 0, the parameters W t，k，ts are initialized as W 1，k，0 . In certain embodiments, the initialization of W 1，k，0 is random. In certain embodiments, the initialization of W 1，k，0 is with knowledge distillation.

[0098] At step 716, after initializing the parameters W 1，k，0 , client k uses the parameters W 1，k，0Training the local model. In some embodiments, the local model is trained for E rounds or E local rounds. In some embodiments, in each local round, client k trains the model using each batch of dataset k in B batches. That is, the local model will be updated B times in each local round. After each of the E local training rounds, the local parameters W are... 1，k，0 Updated to W 2，k，ts , where ts is the timestamp after the first training iteration of the local model.

[0099] In step 718, client k will (t+1, ts, n) k p t+1 ,k,W t+1，k，ts Returning to the ClientUpdate function in the client management thread 704, where the values ​​of the above parameters are (2, ts, n) k p 2，k W 2，k，ts It should be noted that the global iteration count t now increases from 1 to 2.

[0100] In step 720, when parameters are received from client k, client k management thread 704 prepares training event E including all received parameters. k .

[0101] In step 722, before training event E is ready... k Afterwards, client-side management thread 704 will E k Send to aggregation thread 702.

[0102] In step 724, upon receiving training event E k At that time, aggregation thread 702 uses equation (1) to calculate the delay weight value: α t = f(t, ts, α). Return to reference step 718, where t is 2, ts is the timestamp at the end of the first training iteration of the local model in client k, and α is the delay weight value, which is a predefined hyperparameter.

[0103] In step 726, after calculating the delay weight α2, aggregation thread 702 updates the global model using equation (5): Where t is 2; W t-1 This is the current global model weight. If this is the first time updating the global model, W... t-1 It can be an initial value or a random value; n k p is the size of dataset k; n is the total size of data from all K clients; p t，k For p 2，k W represents the accuracy of the first training iteration of the local model in client k; t，k，ts (This is W)2，k，ts ) is the local model weight of the local model in client k after the first training iteration. Thus, the global model weight W t (here is W2) is obtained. It should be noted that the global model weight W t (here is W2) represents the current model weight, if later another client finishes its first training iteration and provides the corresponding training event, the aggregation thread will take the current model weight as W t-1 and update the global model again using equation (5). It should be noted that the global model updated in this step is a version of the global model, i.e. Figure 6 the student model round -1 as shown.

[0104] In step 728, the aggregation thread 702 prepares the aggregation event E aggr based on the global model weight W2, and sends the aggregation event E aggr to the client k management thread 704.

[0105] In step 730, upon receiving the aggregation event E aggr , the client k management thread 704 prepares the ClientUpdate function, here ClientUpdate(k, t, W t ) is actually ClientUpdate(k, 2, W2).

[0106] In step 732, the client k management thread 704 sends the current ClientUpdate(k, t, W t ) function to the client k or calls the client k to execute the ClientUpdate(k, t, W t ) function.

[0107] In step 734, upon receiving the ClientUpdate(k, t, W t ) function from the client k management thread 704, the client k takes the weight W t as the local model weight, and performs the second training iteration of the local model using the local model weight. In other words, the local model weight is not the random weight or the knowledge distillation weight in the first training iteration, but the current global model weight.

[0108] Steps 736-752 are basically the same as steps 716-732, which complete the second training iteration of the local model of the client k, and update the global model using the updated local model parameters. The global model updated in this step is a version of the global model, i.e. Figure 6 the student model round -2 as shown.

[0109] The above steps 734-752 are repeated until the local model of client k completes T training iterations. Specifically, steps 754-766 are substantially the same as steps 734-746. The T training iterations of the local model are completed at steps 754 and 756, and the local model training result is used to update the global model at step 766. The global model updated in this step is a version of the global model, i.e. Figure 6 the student model round-T is shown.

[0110] It should be noted that the global weights in step 766 can not be the final global model weights. In fact, the global weights are not obtained until each of the K clients completes its step 766. By then, the global model has been trained. The weights of the trained global model can then be sent to all K local models so that the K clients can use the final determined weights to complete their local models and use the completed local models for prediction or other applications.

[0111] Figure 8 A method of training a local federated learning model is schematically depicted in accordance with certain embodiments of the present disclosure. In certain embodiments, the training process corresponds to Algorithm 2 shown in Figure 5 In certain embodiments, the method can be implemented by any one of the K clients shown in Figure 2 It should be noted that, unless otherwise specified in the present disclosure, the steps of the method can be in different orders and are not limited to the order shown in Figure 8

[0112] At step 802, upon receiving a call to ClientUpdate indicating that it is the first training iteration of the local model, model k initializes the local model parameters. Specifically, the parameter of the ClientUpdate function is now k, 1, None, and the initialization is either by knowledge distillation or randomly. The local model weights after initialization are W 1，k，0 where “1” indicates that it is the first training iteration of the local model, “k” indicates the client number, and “0” indicates that it is the beginning of the first training iteration of the local model.

[0113] At step 804, client k trains the local model for the first local round and trains the local model using B batches of data. For each batch b in the B batches, client k updates the local model weights using equation (6). After B training periods, client k completes the first local round of training.

[0114] At step 806, from the second local round to the E-th local round, client k repeatedly trains the local model to complete the first training iteration of the local model. The local model weights are incremented by 1, so that W​1，k，0 Update to W 2，k，ts where "2" indicates the first training iteration of the local model is complete, "k" indicates the client number, and "ts" indicates the timestamp when the local model in client k ends the first training iteration.

[0115] At step 808, client k sends the current local model weights W 2，k，ts (the iteration number "2" indicates the first iteration is complete), the timestamp ts, the data size n k in client k, and the accuracy p t+1 of the local model (here, p2, which indicates the model accuracy of the first training iteration) to the client k management thread to prepare the training event E k The client k management thread then sends E k to the aggregation thread of the global model to update the weights of the global model, resulting in updated global model weights.

[0116] At step 810, after aggregation, the aggregation thread sends the updated global weights to the client k management thread, which invokes client k for a second training iteration of the local model.

[0117] At step 812, upon receiving the invocation of ClientUpdate indicating that it is the second training iteration of the local model, model k takes the updated global model weights as the local model weights and repeats steps 804-806 to train the local model and repeats step 808 to further update the global model weights with the updated local model weights. The training of the local model is repeated for T training iterations.

[0118] At step 814, after T training iterations of the local model in client k, the local model weights are sent to the global model to update the global model weights. When all K clients have completed their respective T training iterations of the local model, the updated global model weights are the final model weights. That is, the training of the K local models is asynchronous, and thus the time required for the T training iterations of the local model is different. The model training is complete when each of the K clients has completed its T training iterations.

[0119] Referring back to Figure 7 When all T training iterations of each of the K clients are complete, the global model weights at step 766 will be the final model weights. At this point, the aggregation thread in the host device 210 sends the final weights to all K clients as the weights of the K local models.

[0120] In certain aspects, the present disclosure relates to a method of using a trained local model to predict an outcome of an instance.

[0121] In certain aspects, the present disclosure relates to a non-transitory computer- readable medium storing computer-executable code. In certain embodiments, the computer- executable code can be software stored in the storage 216 as described above. The computer-executable code, when executed, can perform one of the above-described methods.

[0122] Certain embodiments of the present disclosure also have the following advantages: (1) After each local model training iteration, the training results are directly used to update the global model, and the updated global model weights are sent back to the local model for the next training iteration. Therefore, there is no interval between the local model training iterations of each client. This process is completely asynchronous, ensuring the training efficiency of the local model. Specifically, the entire training process is significantly accelerated.(2) In order to accurately evaluate the contribution of the local model weight to the global model weight, especially to improve the training convergence of the global model, the present disclosure considers the data volume of the local model, the delay discount rate of the local model, and the accuracy of the local model training. By updating the global weight by synthesizing these three aspects, the accuracy and convergence of the global weight are guaranteed.(3) The local model is initialized by knowledge distillation, which further reduces the required computing resources and improves the performance of the related computing device.

[0123] The method of the present disclosure is suitable for multi-institutional collaborative modeling, in which the number of training participants is small, and each client cannot be ignored, otherwise the collaborative modeling is meaningless. Although the training environment is stable in the multi-institutional setting, the model can be very large, and synchronization is difficult. Certain synchronization methods can reduce the impact of device client disconnection and instability, but cannot be directly applied to the multi-institutional setting. The large interval in training the local model makes the training process more difficult to synchronize in the multi-institutional setting, resulting in longer synchronization waiting time when aggregating the model in the multi-institutional setting, thereby slowing down the training process. Through a completely asynchronous design and considering three different aspects, certain embodiments of the present disclosure can be well targeted at the multi-institutional federated learning scene.

[0124] In summary, the present disclosure provides an asynchronous multi-aspect weighted federated learning algorithm for effectively and efficiently accelerating multi-institutional collaborative modeling. More specifically, the present disclosure has: (1) a completely asynchronous algorithm to reduce synchronization waiting delay and compress all idle time; (2) a multi-aspect weighted aggregation method to reduce the error of aggregating the global model and accelerate the convergence speed of the model.

[0125] The foregoing description of exemplary embodiments of the present disclosure has been presented for the purposes of illustration and description and is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings.

[0126] The embodiments are chosen and described in order to explain the principles of the disclosure and its practical application to thereby enable others skilled in the art to utilize the disclosure and various embodiments and with various modifications as are suited to the particular use contemplated. Alternative embodiments will become apparent to those skilled in the art to which the present disclosure pertains, without departing from the scope of the present disclosure. Accordingly, the scope of the present disclosure is defined only by the claims appended hereto and equivalents thereof.

Claims

1. A system comprising a master computing device for federated learning, wherein, The master computing device comprises a processor and a storage device having stored thereon computer executable code that, when executed at the processor, is configured to: performing a kth client management thread of K client management threads by: During initialization, instruct the kth client computing device to initiate a first training iteration of the kth client federated learning model to obtain a training event E k ; Upon receiving an aggregation event E aggr from an aggregation thread, instructing the kth client computing device to use the aggregation event E aggr perform a tth training iteration on the kth client federated learning model to obtain a training event E k ; and sending the training event E k to the aggregation thread; and performing the aggregation thread by: Upon receiving the training event E from the k-th client management thread k At that time, the global federated learning model in the main computing device is updated to obtain the aggregation event E. aggr ;as well as The aggregation event E aggr is sent to the kth client management thread.

2. The system of claim 1, wherein The training event E k includes a client number k, a global round number t, a client report timestamp ts, a dataset size n for training the kth client federated learning model k , an accuracy p of the tth training iteration of the kth client federated learning model t，k , and a local model weight W of the kth client federated learning model after the tth training iteration t，k，ts ; and wherein k e K, K is a positive integer, representing a total number of client computing devices, t e T, T is a positive integer, representing a total number of iteration rounds of training each client federated learning model stored in a respective one of the K clients.

3. The system of claim 2, wherein, updating the global federated learning model to obtain the aggregation event E aggr is performed using the following equation: where W t is the updated weight of the global federated learning model after merging the received training events E k parameters, W t-1 is the weight of the global federated learning model before merging the received training events E k parameters, n is the total dataset size of all K clients, a t is the delayed discount rate of the t-th training iteration of the k-th client federated learning model, p t，k is the accuracy of the t-th training iteration of the k-th client federated learning model.

4. The system of claim 3, wherein, the delay discount rate a t is a function of t, ts, and a, t denotes the earliest time at which one of the K federated learning models completes the t-th training iteration, a is a hyperparameter denoting a delay weight value.

5. The system of claim 4, wherein, calculating the delayed discount rate a using at least one of a linear function, a polynomial function, and an exponential function t .

6. The system of claim 1, wherein, Each of the K client federated learning models is initialized by knowledge distillation.

7. The system of claim 1, wherein, Each of the K client federated learning models is initialized randomly.

8. The system of claim 1, wherein, Each of the K client federated learning models is trained using stochastic gradient descent (SGD).

9. The system of claim 1, wherein, The federated learning is institutional federated learning.

10. The system of claim 1, wherein, The global federated learning model comprises T global federated learning model versions, the kth client federated learning model is configured to update a corresponding one of the T global federated learning model versions after each training iteration.

11. A federated learning method, comprising: performing K client management threads, wherein a kth client management thread of the K client management threads is performed by: During initialization, the kth client computing device is instructed to initiate a first training iteration of the kth client federated learning model to obtain a training event E k ; Upon receiving an aggregation event E aggr from an aggregation thread, instructing the kth client computing device to use the aggregation event E aggr perform a tth training iteration on the kth client federated learning model to obtain a training event E k ; and sending the training event E k to the aggregation thread; and performing the aggregation thread by a master computing device, wherein the aggregation thread is performed by: upon receiving the training event E k from the kth client management thread, updating a global federated learning model in the host computing device to obtain an aggregated event E aggr ; and The aggregation event E aggr is sent to the kth client management thread.

12. The method of claim 11, wherein The training event E k includes a client number k, a global round number t, a client report timestamp ts, a dataset size n for training the kth client federated learning model k , an accuracy p of the tth training iteration of the kth client federated learning model t，k , and a local model weight W of the kth client federated learning model after the tth training iteration t，k，ts ; and wherein k e K, K is a positive integer, representing a total number of client computing devices, t e T, T is a positive integer, representing a total number of iteration rounds of training each client federated learning model stored in a respective one of the K clients.

13. The method of claim 12, wherein, updating the global federated learning model to obtain the aggregation event E aggr is performed using the following equation: where W t is the updated weight of the global federated learning model after merging the received training events E k parameters of the global federated learning model, W t-1 is the weight of the global federated learning model before merging the received training events E k parameters, n is the total dataset size of all K clients, a t is the delayed discount rate of the t-th training iteration of the k-th client federated learning model, p t，k is the accuracy of the t-th training iteration of the k-th client federated learning model.

14. The method of claim 13, wherein, the delay discount rate a t is a function of t, ts, and a, t represents the earliest time at which one of the K federated learning models completes the tth training iteration, a is a hyperparameter representing a delay weight value; wherein the delay discount rate a t is calculated using at least one of a linear function, a polynomial function, and an exponential function.

15. The method of claim 11, wherein, Each of the K client federated learning models is initialized by knowledge distillation.

16. The method of claim 11, wherein, The federated learning is institutional federated learning.

17. A non-transitory computer-readable medium storing computer- executable code, wherein, The computer executable code, when executed at a processor of a computing device, is configured to: performing a kth client management thread of K client management threads by: During initialization, the kth client computing device is instructed to initiate a first training iteration of the kth client federated learning model to obtain a training event E k ; Upon receiving an aggregation event E aggr from an aggregation thread, instructing the kth client computing device to use the aggregation event E aggr to perform a tth training iteration on the kth client federated learning model to obtain a training event E k ; and sending the training event E k to the aggregation thread; and performing the aggregation thread by: Upon receiving the training event E from the k-th client management thread k At that time, the global federated learning model in the main computing device is updated to obtain the aggregation event E. aggr ;as well as The aggregation event E aggr is sent to the kth client management thread.

18. The non-transitory computer readable medium of claim 17, wherein The training event E k includes a client number k, a global round number t, a client report timestamp t s , a dataset size n for training the kth client federated learning model k , an accuracy p of the tth training iteration of the kth client federated learning model t，k , and a local model weight W of the kth client federated learning model after the tth training iteration t，k，ts ; and wherein k e K, K is a positive integer, representing a total number of client computing devices, t e T, T is a positive integer, representing a total number of iteration rounds of training each client federated learning model stored in a respective one of the K clients.

19. The non-transitory computer-readable medium of claim 18, wherein, updating the global federated learning model to obtain the aggregation event E aggr The step of updating the global federated learning model to obtain the aggregation event E is performed using the following equation: where W t is the updated weight of the global federated learning model after merging the received training events E k parameters, W t-1 is the weight of the global federated learning model before merging the received training events E k parameters, n is the total dataset size of all K clients, a t is the delayed discount rate of the t-th training iteration of the k-th client federated learning model, p t，k is the accuracy of the t-th training iteration of the k-th client federated learning model.

20. The non-transitory computer-readable medium of claim 19, wherein, the delay discount rate a t is a function of t, ts, and a, t denotes the earliest time at which one of the K federated learning models completes the t-th training iteration, a is a hyperparameter denoting a delay weight value.

Images