Model training system, method and device based on adaptive semi-asynchronous federated learning

Abstract

The present invention discloses a model training system, method and device based on adaptive semi-asynchronous federated learning. The system includes a client and a server. The client and the server implement iterative updates of the model through multiple rounds of interaction to complete global model training. During the whole process, the client data is always stored locally on the client, and the client can join or exit the model training of this system at any time. Due to the performance differences of the clients, different clients take different times to complete a local training. In order to improve the efficiency of global model training, the server starts to update the local model parameters immediately after receiving a certain number of local model parameters, and after waiting for a certain period of time, according to the arrival of the local models of the remaining clients, it adaptively decides whether to perform a global synchronous model update or a global asynchronous model update. The present invention can be flexibly applied to the application scenarios of distributed machine learning and distributed deep learning, and has high practicality.

Description

Technical Field

[0001] The present invention belongs to the technical field of federated learning, and relates to a model training system, method and device, and specifically to a model training system, method and device based on adaptive semi-asynchronous federated learning. Background Art

[0002] With the popularization of the Internet of Things (IoT) and the continuous development of big data, people are no longer concerned about the amount of data, but are beginning to care about data privacy and security. Many countries and regions have begun to enact laws to protect user data privacy. Federated learning is a machine learning framework proposed by Google in 2016. It can effectively help multiple institutions use data and conduct machine learning modeling while meeting the requirements of user privacy protection, data security and government regulations. As a privacy protection paradigm in distributed machine learning, it can effectively solve the problem of data islands, in which edge clients collaborate to train global shared models without leaking their local data to the server.

[0003] There are currently two basic federated learning computing communication frameworks. One is synchronous federated learning. In synchronous federated learning, the server needs to wait until all participating devices have completed uploading their model parameters before performing model aggregation in each round. Due to differences in device resources and differences in training data on devices, faster devices must wait for slower devices in each round of training, which greatly reduces the training efficiency of federated learning. The other is asynchronous federated learning. In asynchronous federated learning, the server only waits for some participating devices to upload their model parameters in each round before participating in the global aggregation task, which speeds up the model aggregation. Although asynchronous federated learning smoothes the impact of device performance differences on federated learning efficiency to a certain extent, for unselected terminal devices, this asynchronous federated learning suppresses the potential of these devices to accelerate model convergence, and there is also a problem of wasted device training progress. In addition, this asynchronous federated learning allows the server to perform global aggregation immediately after collecting a small number of local models, but if the training process wants to cover all data samples, more training rounds are required, which may lead to overfitting, causing the model to develop in the direction of local optimal solutions, reducing the model's predictive ability on global data samples. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a model training system, method and device based on adaptive semi-asynchronous federated learning. The present invention fully considers the differences in terminal device performance and gives full play to the potential contribution of all terminal devices to model convergence, so that the server can accelerate the training of the global model.

[0005] The technical solution adopted by the method of the present invention is: a model training method based on adaptive semi-asynchronous federated learning, using a model training system based on adaptive semi-asynchronous federated learning; the system includes m clients with different performances and a server;

[0006] The method adaptively divides the clients into two parts according to the actual completion status of each round of clients: fast clients and slow clients; the k clients that first submit local model parameters to the server are recorded as the fast clients of this round, and the remaining mk clients are recorded as the slow clients of this round;

[0007] The server first aggregates the local model parameters of k fast clients to obtain a local model, and then waits for a preset time τ before judging:

[0008] If all local model parameters of mk slow clients are sent to the server within the preset time τ, the local model parameters of all clients are aggregated into a global model, and the global model parameters are sent to all clients for the next round of iterative training;

[0009] If the local model parameters of mk slow clients are not all sent to the server within the preset time τ, the server first sends the local model parameters obtained by aggregation to k fast clients; then after receiving the local model parameters of mk slow clients, it aggregates the local models of mk slow clients again, and sends the local model parameters to mk slow clients for the next round of iterative training;

[0010] When the global model converges to a predetermined accuracy or the number of model training rounds reaches a preset threshold, the entire model training process ends.

[0011] Preferably, if the global model parameter update frequency is greater than a preset value A, the time threshold τ is reduced; if the global model parameter update frequency is less than a preset value B, the time threshold τ is increased.

[0012] As a preferred method, the clients are adaptively divided into fast clients and slow clients according to the actual completion status of each round of clients; in each round of iteration, the server adaptively performs global asynchronous model update and global synchronous model update;

[0013] The adaptability is reflected in the fact that fast clients and slow clients are not specified by the system, nor are they fixed, but are dynamically determined based on the specific execution efficiency of the client and its network environment. The speed at which different clients calculate and submit local model training parameters to the server depends on the amount of computing resources the client has and the quality of the network environment.

[0014] Adaptation is also reflected in the fact that whether the server performs a global synchronous model update or a global asynchronous model update in each round is not specified by the system, but is dynamically determined based on the actual operating conditions of the client; if the actual upload time of all client local model parameters in a certain round of model iteration training is within the threshold range, the system will perform a global synchronous model update, otherwise it will perform a global asynchronous model update.

[0015] The technical solution adopted by the device of the present invention is: a model training device based on adaptive semi-asynchronous federated learning, comprising:

[0016] one or more processors;

[0017] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the model training method based on adaptive semi-asynchronous federated learning.

[0018] The present invention also provides a storage medium storing a program; when the program is executed by a processor, the model training method based on adaptive semi-asynchronous federated learning is implemented.

[0019] The beneficial effects of the present invention are as follows: all terminal devices willing to participate in federated learning can join the training process at any time as clients. The server aggregates when receiving training results of clients that meet a predetermined number, and determines whether to perform global synchronous aggregation or global asynchronous aggregation of client training results through a preset time threshold. During the entire training process, fast clients participating in model training will not be affected by slow clients, and at the same time, the model training covers the data sets of all clients, giving full play to the potential contribution of slow clients to global model convergence. In practical applications, terminal devices acting as clients generally have speed differences, so the present invention has more practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 : System model diagram of an embodiment of the present invention;

[0021] Figure 2 : Flow chart of the method of an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0023] Please see Figure 1, an adaptive semi-asynchronous federated learning system provided by the present invention specifically includes multiple clients (assuming there are m clients) and a server. Each client has its own dataset. The client and the server jointly obtain a global model covering all client datasets through multiple rounds of model training iterations. The basic process of model training in the federated learning manner is as follows: The client trains a local model based on its own dataset locally and uploads the local model parameters to the server. The server aggregates them to obtain the global model parameters for this round of training iteration and sends them to all clients for model update, so that the client can perform the next round of local model training based on the updated global model parameters. This process continues until the global model converges to a predetermined accuracy rate or the number of model training rounds reaches a preset threshold. The time consumption of the client for model training often varies due to different hardware conditions of its own. Therefore, in the federated learning method, during model training, some clients may have completed the model training for this round, but some other clients are still in the process of model training. The server must wait for these slow clients to submit the local model parameters for this round before performing the aggregation operation.

[0024] To solve this problem, in the system of the present invention, the clients are divided into two categories according to their processing capabilities: fast clients (assuming there are k fast clients, k < m) and slow clients (assuming there are m - k slow clients). In a certain round of model update iteration, if all k fast clients have sent their local model parameters for this round to the server, the server waits for a time threshold τ:

[0025] If during the waiting period, not all m - k slow clients have uploaded their local model parameters for this round to the server, then the server aggregates the local model parameters sent by the k fast clients to obtain a local model parameter and sends it to these k fast clients for model update. These k fast clients will perform the next round of model training iteration based on this; in addition, when the server receives the local model parameters for this round uploaded by all m - k slow clients, the server aggregates the local model parameters sent by the m - k slow clients to obtain a local model parameter and sends it to these m - k slow clients for model update. These m - k slow clients will perform the next round of model training iteration based on this.

[0026] If within the waiting time, the server receives the local model parameters for this round uploaded by all m - k slow clients, then the server aggregates the local model parameters sent by the m devices to obtain a global model parameter and sends it to all m clients for model update. The m clients will perform the next round of model training iteration based on this.

[0027] The local model parameters refer to the model parameters obtained by a client based on its own dataset; the so-called local model parameters refer to the model parameters obtained by the server by aggregating the local model parameters provided by some clients; the so-called global model parameters refer to the model parameters obtained by the server by aggregating the local model parameters provided by all clients. The local model, local model and global model refer to the same training model. The difference lies in the different datasets covered. The local model is trained based on the dataset of one client, the local model is trained based on the dataset of some clients, and the global model is trained based on the dataset of all clients.

[0028] In this embodiment, the adaptation includes two aspects: the client is adaptively divided into two parts - fast client and slow client in each round of iteration; the server adaptively performs global asynchronous model update and global synchronous model update in each round of iteration.

[0029] The first aspect of the adaptability is that the fast client and the slow client are not specified by the system, nor are they fixed, but are dynamically determined according to the specific execution efficiency of the client and its network environment. The speed at which different clients calculate and submit local model training parameters to the server depends on the amount of computing resources the client has and the quality of the network environment. In terms of the amount of computing resources and the quality of the network environment, some clients participating in federated learning may have obvious differences, while some clients are very similar. At the same time, in the federated learning mode, the client needs to perform multiple rounds of model iteration training. The speed at which the same client calculates and submits local model parameters to the server in each round actually depends on its own currently available computing resources and network environment. Therefore, dynamically determining fast clients and slow clients based on the actual performance of the client in each round is one of the core manifestations of the adaptability of the present invention.

[0030] The second aspect of the adaptability is that whether the server performs a global synchronous model update or a global asynchronous model update in each round is not specified by the system, but is dynamically determined based on the actual operation of the client. If the actual upload time of all client local model parameters is very close in a certain round of model iteration training, the system will perform a global synchronous model update, otherwise a global asynchronous model update. Therefore, dynamically deciding whether to perform a global synchronous model update or a global asynchronous model update based on the overall actual performance of all clients in each round is another core embodiment of the adaptability of the present invention.

[0031] Please see Figure 2, the model used in this embodiment is a binary SVM model. The goal of binary SVM model training is to find the segmentation hyperplane w·x+b=0 with the largest geometric spacing based on a given training data set, where w is the normal vector of the hyperplane and b is the intercept of the hyperplane. Therefore, the model parameters of SVM are (w, b). This embodiment describes the process of applying the semi-asynchronous federated learning method and system proposed in the present invention to binary SVM model training, and the specific process includes the following steps:

[0032] See Figure 2 In step 1, the server sets the threshold k and τ and initializes the SVM model parameters And send it to all clients participating in model training. In addition, the server sets the maximum number of iterations R for model update and the target accuracy that the model aims to achieve.

[0033] Assume that the size of the data set of the i-th (i=1,2,…,m) client is n i , the sample data is (x i,j ,y i,j ), where j = 1, 2, ..., n i , x i,j is the feature vector of the jth sample of the ith client, y i,j is the label of the jth sample of the i-th client, y i,j ∈{-1,+1}. Represents the local SVM model parameters obtained by the i-th client in the r-th (1≤r≤R)-th model update iteration.

[0034] See Figure 2 In step 2, assuming that this iteration is the rth model update iteration, then the i-th client will update its own data set. and the global model parameters currently received from the server for the r-1th iteration (possibly or or ) Perform local SVM model training to obtain the local SVM model parameters for this iteration And W r i is sent to the server.

[0035] Local SVM model parameters It refers to the SVM model parameters trained based on the data set owned by the i-th client.

[0036] See Figure 2Step 3: The server generates global SVM parameters based on the received local SVM parameters. For the sake of ease of description, in this embodiment, we assume that in each iteration, the 1st, 2nd, ..., kth clients are fast clients, and the k+1th, k+2th, ..., mth clients are slow clients.

[0037] Step 3 specifically includes the following sub-steps:

[0038] Step 3.1: Once the server receives the local model parameters uploaded by the first, second, ..., kth clients, it updates the local model parameters for these k clients: Model parameters are the SVM model parameters covering these k client data sets.

[0039] Local SVM model parameters It refers to the SVM model parameters trained based on the data sets owned by the first, second, ..., kth clients.

[0040] Step 3.2: The server waits for a waiting time determined by the time threshold τ, and decides whether to perform a global synchronization model update based on the upload status of the local model parameters of the remaining m–k clients during the waiting period;

[0041] If the remaining m–k clients all upload their local model parameters during the waiting period (i.e., the waiting time is less than or equal to the time threshold τ), then the server will perform a global synchronization model update for all m clients. The specific steps are as follows: The server updates the local model parameters of these m–k clients: Model parameters is the SVM model parameter covering these m–k client data sets, and then calculate and The average value of the SVM model parameters covering all m client data sets is , and Sent to all m clients, and m clients enter the r+1th round of iterative updates.

[0042] Local SVM model parameters It refers to the SVM model parameters trained based on the data sets owned by the k+1th, k+2th, ..., mth clients.

[0043] Global SVM model parameters It refers to the SVM model parameters obtained by training based on the data sets owned by all m clients.

[0044] If during the waiting period (i.e., the waiting time is equal to the time threshold τ), only some of the remaining m–k clients have uploaded their local model parameters, and some clients have not completed local model training and uploaded their local model parameters, then the server will perform local model updates on k clients and m–k clients in turn (i.e., perform global asynchronous model updates), specifically as follows: the server will The k clients then enter the r+1th round of model update iteration process. At the same time, after receiving the local model parameters of all the remaining m–k clients, the server calculates the local model parameters related to these m–k clients. , and It is sent to these m–k clients, and these m–k clients then enter the r+1th round of model update iteration process;

[0045] See Figure 2 Step 4: Repeat steps 2 and 3. When the global model converges to a predetermined accuracy or the number of model training rounds reaches a preset threshold R, the entire model training process ends.

[0046] Assume that the model training process ends because the number of iterations reaches R. If the server performs a global synchronization model update in the Rth iteration, the server will update the model parameters obtained in this global synchronization model update iteration. Output as the final training result; if the server performs a global asynchronous model update in the Rth iteration, the server will not update the model after the waiting time threshold τ. Instead of sending the corresponding k clients, continue to wait until the local model parameters of the remaining m–k clients are collected and calculated. Then calculate Will Output as the final training result.

[0047] If the global synchronization model update occurs frequently, it means that the time threshold τ is set too large and the time threshold τ should be appropriately reduced; if the global synchronization model update is not triggered for a long time, it means that the time threshold τ is set too small and the time threshold τ should be appropriately increased.

[0048] The present invention also provides a model training device based on adaptive semi-asynchronous federated learning, comprising:

[0049] one or more processors;

[0050] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the model training method based on adaptive semi-asynchronous federated learning.

[0051] The present invention also provides a storage medium storing a program; when the program is executed by a processor, the model training method based on adaptive semi-asynchronous federated learning is implemented.

[0052] The present invention effectively solves the contradiction of how to resolve the speed mismatch between fast clients and slow clients in the model training of federated learning. The present invention is a solution to how to improve the training efficiency and effect of the global model when there are differences in the training efficiency of the local model on the client. The method can be used in the efficient distributed model training of multiple models in multiple application fields. While solving the data island problem, it also achieves good training efficiency and effect, and has wide practicality.

[0053] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A model training method based on adaptive semi-asynchronous federated learning, using a model training system based on adaptive semi-asynchronous federated learning; the system includes m clients with different performances and a server; Features: The method adaptively divides the clients into two parts according to the actual completion status of each round of clients: fast clients and slow clients; the k clients that first submit local model parameters to the server are recorded as the fast clients of this round, and the remaining mk clients are recorded as the slow clients of this round; The server first aggregates the local model parameters of k fast clients to obtain a local model, and then waits for a preset time τ before judging: If all local model parameters of mk slow clients are sent to the server within the preset time τ, the local model parameters of all clients are aggregated into a global model, and the global model parameters are sent to all clients for the next round of iterative training; If the local model parameters of mk slow clients are not all sent to the server within the preset time τ, the server first sends the local model parameters obtained by aggregation to k fast clients; then after receiving the local model parameters of mk slow clients, it aggregates the local models of mk slow clients again, and sends the local model parameters to mk slow clients for the next round of iterative training; When the global model converges to a predetermined accuracy or the number of model training rounds reaches a preset threshold, the entire model training process ends; If the global model parameter update frequency is greater than the preset value A, the time threshold τ is reduced; if the global model parameter update frequency is less than the preset value B, the time threshold τ is increased; According to the actual completion status of each round of clients, the clients are adaptively divided into fast clients and slow clients; in each round of iteration, the server adaptively performs global asynchronous model updates and global synchronous model updates; The adaptability is reflected in the fact that fast clients and slow clients are not specified by the system, nor are they fixed, but are dynamically determined based on the specific execution efficiency of the client and its network environment. The speed at which different clients calculate and submit local model training parameters to the server depends on the amount of computing resources the client has and the quality of the network environment. Adaptation is also reflected in the fact that whether the server performs a global synchronous model update or a global asynchronous model update in each round is not specified by the system, but is dynamically determined based on the actual operating conditions of the client; if the actual upload time of all client local model parameters in a certain round of model iteration training is within the threshold range, the system will perform a global synchronous model update, otherwise it will perform a global asynchronous model update.

2. A model training device based on adaptive semi-asynchronous federated learning, It is characterized in that include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the model training method based on adaptive semi-asynchronous federated learning as described in claim 1.

3. A storage medium storing a program; It is characterized in that When the program is executed by the processor, the model training method based on adaptive semi-asynchronous federated learning described in claim 1 is implemented.

Images