A federated learning training method, system and device for synchronous falling behind problem

Abstract

A federated learning training method, system and device for the synchronization dropout problem, the method of which dynamically adjusts the local iteration number of the next round of training of the client according to the local training time and local model accuracy of the client in each round through a synchronization aggregation algorithm; through a client selection algorithm, in each round of training, a client with short training time and high local model accuracy is selected with a higher probability to participate in the current round of training, realizing the simultaneous improvement of the training speed and accuracy of federated learning; the system, device and medium thereof can realize the simultaneous improvement of the training speed and accuracy of federated learning for the federated learning training method for the synchronization dropout problem, solving the problems of low accuracy and slow convergence speed in the synchronization aggregation scheme in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of privacy protection technology, specifically relating to a federated learning training method, system, and device for addressing the problem of synchronization falling behind. Background Technology

[0002] Compared to traditional machine learning, federated learning ensures that user data does not leave their local machine, thus protecting user data privacy. The federated averaging algorithm is a classic example of federated learning. It consists of a server and multiple clients, with each client holding its own local data. The server first initializes a global model and sends its parameters to the clients. Each client trains the global model using its local data, obtains its local model, and uploads its parameters back to the server. The server receives all the local model parameters from the clients, weights and aggregates them to obtain a new global model. After multiple rounds of communication, the global model converges.

[0003] However, due to differences in data volume and computing power among clients, synchronous aggregation methods suffer from client lag, requiring the server to wait for all clients to complete their local training before aggregation can proceed. This lag effect severely impacts the learning efficiency and convergence speed of federated learning. Existing solutions to the lag problem are generally categorized into asynchronous aggregation, semi-asynchronous aggregation, and synchronous aggregation methods. While asynchronous and semi-asynchronous aggregation can effectively alleviate the lag problem, they can lead to clients with long training times not receiving updates for extended periods, resulting in outdated models. Synchronous aggregation solutions reduce the differences in local training time among clients by adjusting the amount of local training data or the number of local iterations. However, changes in client data volume and the number of local iterations significantly impact model accuracy, leading to decreased accuracy and slower convergence.

[0004] Patent application CN114219097A, entitled "A Federated Learning Training and Prediction Method and System Based on Heterogeneous Resources," discloses a federated learning training and prediction method and system based on heterogeneous resources. This invention first determines the learning task and prediction function of the convolutional neural network model by the server, and models the computing and communication resources of heterogeneous devices in the federated learning scenario, distinguishing the heterogeneous characteristics of resources of different devices. It then proposes two strategies to adjust the way clients are selected in each round of federated learning, and to set a maximum number of local iterations for clients during training, in order to fully utilize client computing resources, reduce communication overhead, and address the impact of lagging behind. However, in dynamically adjusting the maximum number of local iterations for clients, this invention only considers the length of the client's local training time, without considering the impact of the client's local model accuracy. This can easily lead to low global model accuracy and slow convergence speed in federated learning. Summary of the Invention

[0005] To overcome the problems existing in the prior art, this invention discloses a federated learning training method, system, and device for the synchronous fall-off problem. First, a synchronous aggregation algorithm dynamically adjusts the number of local iterations for the next training round of the client based on two parameters: the client's local training time and local model accuracy in each round. Second, a client selection algorithm is proposed, which supports selecting clients with shorter training times and higher local model accuracy to participate in the current training round with a higher probability, thereby achieving a simultaneous improvement in the training speed and accuracy of federated learning. This solves the problems of low accuracy and slow convergence speed in the synchronous aggregation schemes of the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] The specific steps of the federated learning training method for addressing the synchronous falling behind problem are as follows:

[0008] Step 1. Initialize the federated learning system, setting the client universal set S and client subset S r ;

[0009] Step 2. After initialization, each client C in the complete client set S of the federated learning system... i Obtain the training sample set;

[0010] Step 3. The central server sends data to the client subset S. r Each client C i Send global model parameters θ r and the maximum number of local iterations T i k ;

[0011] Step 4. Client subset Sr Each client C i The global model is trained locally to obtain the local model parameters;

[0012] Step 5. The central server aggregates local model parameters;

[0013] Step 6. The central server updates the client subset S. r Each client C i The maximum number of local iterations T for the next training iteration i k+1 And remove the client subset S from this round of training. r Client C has local model accuracy lower than average accuracy and local training time higher than average training time. i ;

[0014] Step 7. The central server selects a subset S of clients to participate in the next round of training. r .

[0015] The specific method for step 1 is as follows:

[0016] The federated learning system consists of one central server and N clients. To initialize the federated learning system, the universal set of clients is set to S = {C1, C2, ..., C...}. i ,…C N}, where N≥2; set the maximum number of communication rounds in federated learning to R, R≥2; set the subset of clients participating in training in the r-th round of communication to be Where 1≤r≤R; the i-th client C i The maximum number of iterations for the k-th local training iteration is T. i k Where 1≤i≤N, 1≤k≤r; initialize the global model parameters of the convolutional neural network as θ. 0 .

[0017] The specific method for step 2 is as follows:

[0018] Each client C i X locally held Z data items i and its corresponding label Y i Composition of training sample set D i ={X i ,Y i}, where Z≥50, X i ={x i1 ,x i2 ,…,x iz ,…,x iZ}, x iz Indicate C i The z-th data item held, xiz The corresponding label is y iz .

[0019] The specific method for step 3 is as follows:

[0020] The central server communicates with the subset S of clients participating in the training in the r-th round. r Each client C in i Send the global model parameters θ for the r-th round r and the maximum number of local iterations T for each client i k .

[0021] The specific method for step 4 is as follows:

[0022] Client subset S r Each client C in i Receive global model parameters θ r and the maximum number of local iterations T i k Using its own training sample set D i Perform T on the global model i k The local model parameters are obtained through the r-th iteration of training. And calculate the accuracy F of the local model. i r and local training time The result F i r and Uploaded to the server.

[0023] The specific method for step 5 is as follows:

[0024] The central server calculates the parameters of each received local model. With global model parameters θ r cosine similarity And based on the local model accuracy F i r and By aggregating the local model parameters, we obtain the global model parameters θ for the (r+1)th round. r+1 The server determines whether r = R is true; if so, the training ends, and the parameter θ is obtained. r+1 If the global model is not available, proceed to step 6.

[0025] The specific method for step 6 is as follows: The central server updates the maximum local iteration count T for the next training session for each client in the client subset. i k+1And remove clients from the current training client subset whose local model accuracy is lower than the average accuracy and whose local training time is higher than the average training time:

[0026] The central server is based on the accuracy F of the local model. i r and local training time Determine client C in the client subset i The next time, i.e., the (k+1)th time, the maximum number of local iterations T during training. i k+1 The central server removes m clients from the current training client subset whose local model accuracy is lower than the average accuracy and whose local training time is higher than the average training time. r And calculate this m r The deviation between the local training time and the average training time of each excluded client. Deviation between local model accuracy and average accuracy

[0027] The specific method for step 7 is as follows:

[0028] If any client in the complete client set S has not participated in training, then the server will select clients from the client set SS. r Randomly select m r A client that has not participated in training before participates in the (r+1)th round of training; if all clients in the client set S have participated in training, then the server selects clients from the client set S. r Based on the local training time deviation of each client Deviation from local model accuracy Select m r One client participates in the (r+1)th round of training; let r = r+1, and execute step 3.

[0029] A federated learning training system for addressing the problem of synchronization falling behind, specifically comprising:

[0030] System initialization module: Used to initialize the federated learning system in steps 1 to 3, setting the client global set S and client subset S. r And obtain each client C in the complete client set S. i The training sample set is given to the client subset S. r Each client C i Send global model parameters θ r and the maximum number of local iterations T i k ;

[0031] Federated Learning Training Module: Used in steps 4 and 5, for the client subset S r Each client Ci Training is performed locally using a custom dataset. The number of local training iterations is determined by the maximum local iteration count calculation module, and the server calculates the number of iterations based on the client subset S. r Each client C i The local model is aggregated based on its accuracy and similarity.

[0032] Maximum local iteration count calculation module: used in step 6, where the central server calculates the number of local iterations based on the subset S of clients participating in the training. r Each client C i The local model accuracy and local training time determine the client subset S r The maximum number of local iterations for each client when it participates in training next time;

[0033] Client selection module: Used in step 7 to select each client C from the complete client set S. i The local training time and local model accuracy are obtained from the client subset S participating in the next round of training. r .

[0034] A federated learning training device for the problem of synchronization falling behind includes:

[0035] Memory, used to store computer programs;

[0036] The processor is configured to implement the federated learning training method for the synchronous falling behind problem as described in any one of steps 1 to 7 when executing the computer program, thereby achieving a synchronous improvement in the speed and accuracy of federated learning training.

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] (1) In this invention, the server dynamically adjusts the maximum number of local iterations for the next round of training of the client based on the local training time and local model accuracy of each round of the client. For clients with short local training time but low local model accuracy, the number of iterations is increased, and for clients with long local training time but high local model accuracy, the number of iterations is reduced. This avoids the defect in the prior art where adjusting the number of local iterations affects the model accuracy, effectively improves the convergence speed of federated learning, and avoids the problem of synchronization falling behind.

[0039] (2) The client selection algorithm proposed in this invention can discard clients with long training time and low local model accuracy in each round, and select clients with short training time and high local model accuracy to participate in training with a higher probability, which effectively improves the model accuracy and fairness of federated learning.

[0040] In summary, this invention uses a synchronous aggregation algorithm to dynamically adjust the number of local iterations for the next training round of the client based on two parameters: the client's local training time and local model accuracy in each round. Through a client selection algorithm, clients with shorter training times and higher local model accuracy are selected to participate in each training round with a higher probability, thus simultaneously improving the training speed and accuracy of federated learning. This addresses the problems of low accuracy and slow convergence speed in existing synchronous aggregation schemes. Attached Figure Description

[0041] Figure 1 This is a schematic diagram illustrating the implementation process of the federated learning training method for the synchronous falling behind problem provided in an embodiment of the present invention.

[0042] Figure 2 This is a schematic diagram of the structure of a federated learning training system for the synchronous falling behind problem provided in an embodiment of the present invention.

[0043] Figure 3 This is a flowchart illustrating the client selection method in the federated learning training method for the synchronous falling behind problem provided in an embodiment of the present invention. Detailed Implementation

[0044] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0045] Reference Figure 1 A federated learning training method for the synchronous falling behind problem includes the following steps:

[0046] (1) Initialize the federated learning system and set up the client set and client subset:

[0047] A federated learning system consists of one central server and N clients. During system initialization, the total set of clients is S = {C1, C2, ..., C...}. i ,…C N}, where N≥2; set the maximum number of communication rounds in federated learning to R, R≥2; let the subset of clients participating in training in the r-th round of communication be}. Where 1≤r≤R; the i-th client C i The maximum number of iterations for the k-th local training iteration is T. i k Where 1≤i≤N, 1≤k≤r. Initialize the global model parameters of the convolutional neural network as θ. 0 ;

[0048] In this embodiment, the number of federated learning clients N = 100, and the initial maximum number of local training iterations T is set. i 1=10, maximum number of communication rounds R=500, the number of client subsets participating in training in each round is 20, the convolutional neural network model initialized by the central server is a classification model, the convolutional neural network consists of two composite layers and two fully connected layers stacked in sequence, the composite layer consists of a convolutional layer, a max pooling layer and an activation function layer, wherein the convolutional kernel of the convolutional layer is 5×5, the number of output channels of the first convolutional layer is 10, the number of output channels of the second convolutional layer is 20, the activation function layer uses the ReLU function, and the size of the pooling kernel of the max pooling layer is 2×2.

[0049] (2) After initialization, each client in the client set of the federated learning system obtains the training sample set:

[0050] Each client C i X locally held Z data items i and its corresponding label Y i Composition of training sample set D i ={X i ,Y i}, where Z≥50, X i ={x i1 ,x i2 ,…,x iz ,…,x iZ}, x iz Indicate C i The z-th data item held, x iz The corresponding label is y iz ;

[0051] In this invention, each client C i The Z data items held can be any of image data, audio data, and text data. In this embodiment, Z = 600, and the MNIST dataset is used as the image dataset to train a classification model. The MNIST dataset is a grayscale handwritten digit image dataset widely used in the field of federated learning. It contains 60,000 training images, each of which is a 28×28 grayscale handwritten digit image, and its label is the real digit corresponding to the handwritten digit, with a value of 0 to 9. In this embodiment, each client C i Each possesses 600 different cards with real labels Y i grayscale handwritten digit image X i .

[0052] (3) The central server sends the global model parameter θ to each client in the client subset. r and the maximum number of local iterations T i k :

[0053] The server communicates with the subset S of clients participating in the training during the r-th round of communication.r Each client C in i Send the global model parameters θ for the r-th round r and the maximum number of local iterations T for each client i k ;

[0054] (4) Each client in the client subset performs local training on the global model to obtain local model parameters:

[0055] Client subset S r Each client C in i Receive global model parameters θ r and the maximum number of local iterations T i k Using its own training sample set D i Perform T on the global model i k The local model parameters are obtained through the r-th iteration of training. And calculate the accuracy F of the local model. i r and local training time The result F i r and The specific steps for uploading to the server are as follows:

[0056] (4a) Client subset S r Each client C in i The training sample set D i The data is used as input to the neural network model for forward propagation to obtain each data xi. z The corresponding prediction result y′ in the t-th iteration iz,t The loss value L for this iteration is calculated using the cross-entropy loss function. i,t The calculation formula is:

[0057]

[0058] Where ∑· represents the summation operation;

[0059] (4b) Each client passes through L i,t partial derivatives with respect to local model parameters And according to The local model parameters are updated to obtain the local model for the t-th iteration, where the update formula is:

[0060]

[0061] Where η represents the learning rate;

[0062] (4c) Each client determines t=T i k If true, obtain the local model parameters. Then execute step (4d); otherwise, let t = t + 1 and execute step (4a).

[0063] (4d) Each client calculates the accuracy Fi of the local model in the r-th round. r In this embodiment, F i r Accuracy can be expressed as the ratio of the number of correctly predicted samples to the total number of predicted samples, and the local training time can be recorded. Local model parameters Local model accuracy F i r Training time Package and upload to the server.

[0064] (5) The central server aggregates local model parameters:

[0065] The server calculates each received local model parameter. With global model parameters θ r cosine similarity And based on the local model accuracy F i r and By aggregating the local model parameters, we obtain the global model parameters θ for the (r+1)th round. r+1 The server determines whether r = R is true; if so, it terminates the training and obtains the parameter θ. r+1 The global model is used; otherwise, step (6) is executed. The calculation formulas for cosine similarity and model aggregation are as follows:

[0066]

[0067]

[0068] Where ||·|| is the L2 norm operation, <·> represents the inner product operation, α represents the proportion of local model accuracy, and 0≤α≤1;

[0069] (6) The central server updates the maximum number of local iterations T for each client in the client subset to participate in the next training session. i k+1 And remove clients from the current training client subset whose local model accuracy is lower than the average accuracy and whose local training time is higher than the average training time:

[0070] The server is based on the accuracy F of the local model. i rand local training time Determine client C i The next time, i.e., the (k+1)th time, the maximum number of local iterations T during training. i k+1 The server removes m clients whose local model accuracy is lower than the average accuracy and whose local training time is higher than the average training time during this training round. r And calculate this m r Deviation between local training time and average training time for each client Deviation between local model accuracy and average accuracy

[0071] Average accuracy and average training time The calculation formulas are as follows:

[0072]

[0073]

[0074] Among them, card(S) r ) represents set S r The number of elements in the text.

[0075] The server is based on the accuracy F of the local model. i r and local training time Determine client C i The maximum number of local iterations T during the (k+1)th training iteration i k+1 The specific implementation steps are as follows:

[0076] Server computing client subset S r Average accuracy of the local model in the client and average training time For clients whose local model accuracy is lower than average accuracy and whose local training time is lower than average training time, the maximum number of local iterations for the next training iteration is increased. The calculation formula is as follows:

[0077]

[0078] For clients whose local model accuracy is higher than average accuracy and whose local training time is higher than average training time, reduce the maximum number of local iterations for the next training iteration. The calculation formula is as follows:

[0079]

[0080] For clients whose local model accuracy is lower than average accuracy and whose local training time is higher than average training time, and for clients whose local model accuracy is higher than average accuracy and whose local training time is lower than average training time, the maximum number of iterations remains unchanged, and the calculation formula is as follows:

[0081] T i k+1 =T i k

[0082] Server computing client subset S r Client C has local model accuracy lower than average accuracy and local training time higher than average training time. n Local training time deviation Deviation from local model accuracy The specific implementation steps are as follows:

[0083] Training time bias is represented by client C. i The difference between the current local training time and the average training time of this round is calculated using the following formula:

[0084]

[0085] Local model accuracy deviation is represented by client C i The difference between the accuracy of the local model in this round and the average training accuracy in this round is calculated using the following formula:

[0086]

[0087] (7) The central server selects a subset of clients to participate in the next round of training. See [link / reference] Figure 3 :

[0088] If the client set SS is the sum of the client subsets and the total client set, then... r If there are clients that have not participated in the training, the server will randomly select m from these clients. r One client that has not participated in training participates in the (r+1)th round of training; if the client set SS is the sum of the total client set minus the client subset. r If all clients have participated in training, then the server will select the client set SS. r Based on the local training time deviation of each client Deviation from local model accuracy Select m r One client participates in the (r+1)th round of training. Let r = r+1, and execute step (3), such as... Figure 3 As shown, the specific implementation steps are as follows:

[0089] (7a) Obtain the client set SS rEach client C i Training time deviation of the most recent training session and training accuracy deviation

[0090] (7b) For the client set SS respectively r Training time bias for each client and training accuracy deviation Perform min-max normalization to obtain the training time bias coefficient t. i and training accuracy deviation coefficient q i The normalization formula is:

[0091]

[0092]

[0093] in, and These represent the client set SS. r The minimum and maximum values ​​of the training time bias. and These represent the client set SS. r The minimum and maximum values ​​of the training accuracy deviation.

[0094] (7c) Calculate the client set SS r The probability of each client being selected is calculated using the following formula:

[0095]

[0096] Where β represents the proportion of training time bias, and 0≤β≤1.

[0097] (7d) The server determines the client set SS r The probability P of each client i Select the client to participate in training in round r+1.

[0098] like Figure 2 The federated learning training system architecture shown for the synchronous fall-off problem includes:

[0099] System initialization module 201: Used to initialize the federated learning system.

[0100] Federated Learning Training Module 202: Selected clients train locally using their own datasets. The number of local training iterations is determined by the maximum number of local iterations calculation module. The server aggregates local models based on the accuracy and similarity of each client's local model.

[0101] Maximum local iterations calculation module 203: The server determines the maximum number of local iterations for each client to participate in training next, based on the local model accuracy and local training time of each client participating in training.

[0102] Client selection module 204: Obtains a subset of clients to participate in the next round of training based on each client's local training time and local model accuracy.

Claims

1. A federated learning training method for the synchronous falling behind problem, characterized in that, The specific steps are as follows: Step 1. Initialize the federated learning system and set up the client set. With client subset The federated learning system includes one central server and There are 2 clients, N≥2; the maximum number of communication rounds in federated learning is set to... , ;Set the first The subset of clients participating in the training via round-robin communication is ,in, ;No. Client No. The maximum number of iterations for each local training session is: ,in, , Initialize the global model parameters of the convolutional neural network as follows: ; Step 2. Complete client set of the federated learning system after initialization Each client Obtain the training sample set; for each client Locally held data and their corresponding tags Composition of training sample set ,in, ; Step 3. The central server sends data to a subset of clients. Each client Send global model parameters and maximum number of local iterations ; Step 4. Client Subset Each client Train the global model locally to obtain local model parameters; client subset Each client in Receive global model parameters and maximum number of local iterations Using their own training sample set Perform global model The local iteration training yields the... Local model parameters of the wheel And calculate the accuracy of the local model. and local training time ; will be obtained , and Uploaded to the server; Step 5. The central server aggregates local model parameters; The central server calculates the parameters of each received local model. With global model parameters cosine similarity And based on the accuracy of the local model and Aggregate local model parameters to obtain the first Round global model parameters Server judgment If the condition is met, training ends, and the parameters are obtained. The global model in, ; Server judgment If the condition is met, training ends, and the parameters are obtained. If the global model is not available, proceed to step 6. Step 6. The central server updates the client subset. Each client Maximum number of local iterations for the next training session And remove the subset of clients from this round of training. Clients with local model accuracy below average and local training time exceeding average training time ; Specifically, this refers to: a subset of computing clients. The average accuracy of the local model and average training time ; According to each client of and The maximum number of local iterations for the next iteration is determined according to the following rules. : right < and < For clients, increase their maximum local iteration count: ; right > and > For clients, reduce their maximum local iteration count: ; right < and > The client, and > and < For clients, keep their maximum number of local iterations unchanged: ; At the same time, remove the client subset from this round. China satisfies < and > The number of clients is ; Calculate the deviation between the local training time and the average training time of these excluded clients. And the deviation between the accuracy of the local model and the average accuracy. ; Step 7. The central server selects a subset of clients to participate in the next round of training. Specifically: If there are clients in the complete client set S that have not participated in training, the server subtracts the set of clients from the complete client set. Random selection The first client that had not participated in the training participated in the... Rotational training; If all clients in the complete client set S have participated in the training, then the server... The training time deviation is based on the most recent training time of each client. Deviation from local model accuracy ,choose The client participated in the first Rotational training; The selection method includes: right Each client in and The training time bias coefficient is obtained by performing min-max normalization separately. and training accuracy deviation coefficient : in, and Representing client sets respectively The minimum and maximum values ​​of the training time bias. and Representing client sets respectively The minimum and maximum values ​​of the training accuracy deviation; calculate The probability of each client being selected : Where β represents the proportion of training time bias, 0≤β≤1; The server determines the probability. Select the A subset of clients participating in the training round ; make Then return to step 3.

2. The federated learning training method for the synchronous falling behind problem according to claim 1, characterized in that, Step 1 also includes initializing the federated learning system and setting the client's complete set to... .

3. The federated learning training method for the synchronous falling behind problem according to claim 1, characterized in that, The data and their corresponding tags Composition of training sample set ,in, , express The first data, The corresponding tags are .

4. The federated learning training method for the synchronous falling behind problem according to claim 1, characterized in that, The specific method for step 3 is as follows: The central server sends to the first A subset of clients participating in training via round-robin communication Each client in Send the Global model parameters of the wheel and the maximum number of local iterations for each client .

5. A system based on the federated learning training method for the synchronous falling behind problem according to any one of claims 1 to 4, characterized in that, Specifically, it includes: System initialization module: Used to initialize the federated learning system in steps 1 to 3, setting up the full client set and client subsets, obtaining the training sample set of each client in the full client set, and sending global model parameters to each client in the client subset. and maximum number of local iterations ; Federated learning training module: used in steps 4 and 5, each client in the client subset trains locally using its own dataset. The number of local training iterations is determined by the maximum number of local iterations calculation module. The server aggregates the local models based on the accuracy and similarity of the local models of each client in the client subset. Maximum local iterations calculation module: In step 6, the central server determines the maximum number of local iterations for each client in the client subset when it participates in training next, based on the local model accuracy and local training time of each client in the client subset participating in training. Client selection module: Used in step 7, based on the client's complete set. The local training time and local model accuracy of each client are used to obtain a subset of clients that will participate in the next round of training.

6. A federated learning training device for the problem of synchronization falling behind, characterized in that, include: Memory, used to store computer programs; The processor is configured to implement the federated learning training method for the synchronous falling behind problem described in any one of steps 1 to 4 when executing the computer program, thereby achieving a simultaneous improvement in the speed and accuracy of federated learning training.

Images