Federal learning algorithm based on adaptive aggregation and dynamic training

Abstract

The invention discloses a federal learning algorithm based on adaptive aggregation and dynamic training, and belongs to the field of deep learning, and the learning algorithm specifically comprises the following steps: S1, each client receives an initial global model parameter, and carries out model training by using local data; s2, locally updating the weight of the model by adopting stochastic gradient descent; s3, after local training is completed, the client side calculates a model loss value of the current round, and the loss is reserved to serve as a reference for subsequent adaptive aggregation; s4, on the client side, each client predicts the time demand of the next round of training by adopting a dynamic training strategy according to the historical training time of the client; according to the method, the accuracy and generalization ability of the model are improved, resource idling is avoided, it is ensured that computing resources of all clients are fully utilized, the convergence speed of the global model is increased, the computing overhead is reduced, resource utilization is optimized, and the cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to a federated learning algorithm based on adaptive aggregation and dynamic training. Background Art

[0002] With the rapid development of digitalization and information technology, the amount of data in all walks of life continues to grow. Especially in industries such as healthcare, finance, and smart manufacturing, the diversity and sensitivity of data make it a key issue to effectively use data while protecting data privacy. Traditional centralized machine learning methods rely on centralizing data from various sources to a central server, which not only brings potential data leakage and privacy risks, but also brings high costs and delays due to data transmission and storage. In addition, such methods place a heavy burden on data center resources and do not meet industry compliance requirements when processing high-privacy data.

[0003] In this context, distributed machine learning technology has become the focus of exploration in various fields. Federated learning, as a distributed learning method that does not require centralized data, has gradually attracted widespread attention. Federated learning allows each participant (i.e., client) to complete model training locally and send the trained model parameters to a central server for aggregation, which not only preserves the privacy of the data but also achieves the effect of collaborative modeling. In particular, federated learning can continuously update and improve model performance based on local data, making full use of decentralized data resources while protecting data privacy. However, the widespread application of federated learning still faces major challenges in practice.

[0004] Among them, the non-independent and identically distributed (Non-IID) property of data is a key problem. In practical applications, the local data of each client often has different distribution characteristics, which makes the global model susceptible to data distribution differences when aggregating model parameters of different clients, and it is difficult to achieve good generalization capabilities on all clients. In addition, there are significant differences in computing resources of different clients. Differences from hardware performance to network bandwidth may cause some clients to have slow training speeds or even disconnection problems during training. Traditional federated learning algorithms generally randomly select participating clients in each round of aggregation. Although this method theoretically guarantees the convergence of the model, in actual operation, inefficient client selection will prolong the overall training time and may affect the overall accuracy and reliability of the model. Therefore, in view of the unique advantages of federated learning in data privacy protection and distributed collaborative training, how to optimize data heterogeneity and the utilization of client computing resources is still the focus and difficulty of current federated learning research. Summary of the invention

[0005] The purpose of the present invention is to solve the defects in the prior art and propose a federated learning algorithm based on adaptive aggregation and dynamic training.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A federated learning algorithm based on adaptive aggregation and dynamic training. The specific steps of the learning algorithm are as follows:

[0008] S1. Each client receives the initial global model parameters and uses its local data to train the model.

[0009] S2, locally update the model weights using stochastic gradient descent;

[0010] S3. After local training is completed, the client calculates the model loss value of the current round and retains the loss as a reference for subsequent adaptive aggregation;

[0011] S4. On the client side, each client uses a dynamic training strategy based on its historical training time to predict the time required for the next round of training;

[0012] S5. Dynamically adjust the number of iterations of local training based on the prediction results to ensure maximum utilization of computing resources;

[0013] S6. After completing local training, the client uploads the updated model parameters to the server.

[0014] S7, after receiving the model updates from each client, the server adopts an adaptive aggregation strategy to update the global model;

[0015] S8. The server sends the aggregated global model update to each client. After receiving the global model, the client continues the next round of local training. This process continues until the global model reaches the preset training accuracy or meets other training stop conditions.

[0016] As a further solution of the present invention, the specific steps of the dynamic training strategy in S4 are as follows:

[0017] Q1.1: When the global model is being trained, each client records the forward propagation time t of the global model during the training process in real time. f and the back propagation time t b , and based on the recorded forward propagation time t f and the back propagation time t b , calculate the training time T of the small batch in this round of training b ;

[0018] Q1.2: Based on the training time T of the small batch b, the amount of small batch data A, the batch size B and the corresponding client training round E, calculate the local training time of the global model in each client, and after each round of training, each client records the corresponding local training time T;

[0019] Q1.3: Before a new round of training begins, the client retrieves the local training duration of the past N rounds and estimates the next round of training time based on the training duration of the past N rounds. After predicting the next round of training time, it adds it to the set communication window T c Combined with training to obtain the local round LE size for the next round.

[0020] As a further solution of the present invention, the specific calculation formula for the small batch data training time in Q1.1 is as follows:

[0021]

[0022] Where, T b represents the training time of a small batch; x i represents the i-th training data; B represents the batch size; t f represents the forward propagation time; t b represents the back-propagation time;

[0023] The specific calculation formula for the local training duration described in Q1.2 is as follows:

[0024]

[0025] In the formula, T represents the local training time; E represents the training round; M represents the number of small batches in one training; A represents the amount of data in a small batch; B represents the batch size;

[0026] The specific calculation formula for the next round of training time described in Q1.3 is as follows:

[0027]

[0028] Where, T r represents the duration of the next round of training; N represents the total number of global model training rounds recorded by the client; T i Represents the training duration of the i-th round;

[0029]

[0030] Where LE represents the local round size of the next round; T c Represents the communication window between the client and the server and is used to coordinate the communication frequency between the server and the client.

[0031] As a further solution of the present invention, the communication window T in Q1.3c The size of determines the local computing time of each round of training on the client, and also affects the synchronization frequency of the model parameters.

[0032] As a further solution of the present invention, the specific steps of updating the global model using the adaptive aggregation strategy in S7 are as follows:

[0033] Q2.1: While the global model is being trained locally on each client, the client calculates in real time and records the loss value of the local model during the t-1th round of communication and the change in local model training loss between round t and round t-1 of communication on the client

[0034] Q2.2: According to as well as Calculate the improvement p of the client's global model in this round of training relative to the previous round k , and then use the exponential function to normalize it, and then set the two hyperparameters σ1 and σ2 according to the actual experimental situation, and determine the training data volume S of the corresponding client k , and calculate the aggregate weight w of the client k ;

[0035] Q2.3: When the aggregate weight of all clients w k After the calculation is completed, based on the calculated aggregate weight w of each client k , the server performs weighted averaging of the global model parameters of all clients to aggregate the central model on the server side, and then sends the aggregated global model update to each client. After receiving the global model, the client continues with the next round of local training.

[0036] As a further solution of the present invention, the improvement amount p in Q2.2 k The specific calculation formula is as follows:

[0037]

[0038] In the formula, Represents the loss value of the local model during the t-1th round of communication on the client; Represents the change in local model training loss between round t and round t-1 of communication on the client;

[0039] The specific calculation formula for the server-side central model aggregation described in Q2.3 is as follows:

[0040]

[0041] In the formula, w k Represents client clk Aggregation weight; σ1 and σ2 represent hyperparameters, which are used to adjust the relative importance of loss value and data volume in aggregation; S k Represents client cl k The amount of training data.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] This federated learning algorithm based on adaptive aggregation and dynamic training adopts a dynamic training strategy when the global model is trained on the client side. Before the start of a new round of training, each client estimates the training time of the next round based on its historical training time. After predicting the training time of the next round, it is combined with the set communication window for training to obtain the size of the next local round. According to the prediction results, the number of iterations of local training is dynamically adjusted to ensure the maximum utilization of computing resources. After completing local training, the client uploads the updated model parameters to the server. While the global model is being trained locally in each client, the client calculates in real time and records the loss value of the local model during the t-1th round of communication and the local model training loss between the tth round and the t-1th round of communication on the client. , calculate the improvement of the client's global model in this round of training relative to the previous round, and then use the exponential function to standardize it. At the same time, determine the amount of training data for the corresponding client, and calculate the aggregation weight of the client. When the aggregation weight calculation of all clients is completed, based on the calculated aggregation weights of each client, the server performs weighted averaging on the global model parameters of all clients to perform server-side central model aggregation. After that, the server updates the aggregated global model and sends it to each client. After receiving the global model, the client continues to perform the next round of local training, which improves the accuracy and generalization ability of the model, avoids idle resources, ensures that the computing resources of each client are fully utilized, accelerates the convergence speed of the global model, reduces computing overhead, optimizes resource utilization, and reduces costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0045] Figure 1 A flowchart of a federated learning algorithm based on adaptive aggregation and dynamic training proposed by the present invention;

[0046] Figure 2 This is a comparison chart of the accuracy of the FedAD model of the federated learning algorithm based on adaptive aggregation and dynamic training proposed by the present invention;

[0047] Figure 3This is a system block diagram of a federated learning algorithm based on adaptive aggregation and dynamic training proposed in the present invention. DETAILED DESCRIPTION

[0048] Reference Figure 1-3 , a federated learning algorithm based on adaptive aggregation and dynamic training. The specific steps of this learning algorithm are as follows: Each client receives the initial global model parameters and uses its local data to train the model.

[0050] Stochastic gradient descent is used to update the model weights locally.

[0051] After local training is completed, the client calculates the model loss value of the current round and retains the loss as a reference for subsequent adaptive aggregation.

[0052] On the client side, each client adopts a dynamic training strategy based on its historical training time to predict the time required for the next round of training.

[0053] Specifically, refer to Figure 3 It can be seen that when the global model is trained, each client records the forward propagation time t of the global model in the training process in real time. f and the back propagation time t b , and based on the recorded forward propagation time t f and the back propagation time t b , calculate the training time T of the small batch in this round of training b , according to the training time T of the small batch b , the amount of data in the small batch A, the batch size B and the corresponding client training round E, calculate the local training time of the global model in each client, after each round of training, each client records the corresponding local training time T, before the start of a new round of training, the client automatically retrieves the local training time of the past N rounds, and estimates the next round of training time based on the training time of the past N rounds. After predicting the next round of training time, it is combined with the set communication window T c Combined with training to obtain the local round LE size for the next round.

[0054] It should be further explained that the specific calculation formula for small batch data training time is as follows:

[0055]

[0056] Where, T b represents the training time of a small batch; x i represents the i-th training data; B represents the batch size; t f represents the forward propagation time; t brepresents the back-propagation time;

[0057] The specific calculation formula for local training duration is as follows:

[0058]

[0059] In the formula, T represents the local training time; E represents the training round; M represents the number of small batches in one training; A represents the amount of data in a small batch; B represents the batch size;

[0060] The specific calculation formula for the next round of training time is as follows:

[0061]

[0062] Where, T r represents the duration of the next round of training; N represents the total number of global model training rounds recorded by the client; T i Represents the training duration of the i-th round;

[0063]

[0064] Where LE represents the local round size of the next round; T c Represents the communication window between the client and the server and is used to coordinate the communication frequency between the server and the client.

[0065] In addition, it should be noted that the communication window T c The size of determines the local computing time of each round of training on the client, and also affects the synchronization frequency of model parameters; the training time of the current round is predicted based on the average training time of the previous N rounds, and combined with the communication window size T between the client and the server c , dynamically set the number of local rounds LE of the client in this round of training to optimize the allocation of computing resources and improve training efficiency.

[0066] Based on the prediction results, the number of local training iterations is dynamically adjusted to ensure maximum utilization of computing resources.

[0067] After completing local training, the client uploads the updated model parameters to the server.

[0068] After receiving model updates from each client, the server uses an adaptive aggregation strategy to update the global model.

[0069] Specifically, refer to Figure 3 It can be seen that during the period when the global model is trained locally in each client, the client calculates in real time and records the loss value of the local model during the t-1th round of communication and the change in local model training loss between round t and round t-1 of communication on the client according to as well as Calculate the improvement p of the client's global model in this round of training relative to the previous round k , and then use the exponential function to normalize it, and then set the two hyperparameters σ1 and σ2 according to the actual experimental situation, and determine the training data volume S of the corresponding client k , and calculate the aggregate weight w of the client k , when the aggregate weight of all clients w k After the calculation is completed, based on the calculated aggregate weight w of each client k , the server performs weighted averaging of the global model parameters of all clients to aggregate the central model on the server side, and then sends the aggregated global model update to each client. After receiving the global model, the client continues with the next round of local training.

[0070] It should be further explained that the improvement amount p k The specific calculation formula is as follows:

[0071]

[0072] In the formula, Represents the loss value of the local model during the t-1th round of communication on the client; Represents the change in local model training loss between round t and round t-1 of communication on the client;

[0073] The specific calculation formula for server-side central model aggregation is as follows:

[0074]

[0075] In the formula, w k Represents client cl k Aggregation weight; σ1 and σ2 represent hyperparameters, which are used to adjust the relative importance of loss value and data volume in aggregation; S k Represents client cl k The amount of training data.

[0076] The server sends the aggregated global model update to each client. After receiving the global model, the client continues the next round of local training. This process continues until the global model reaches the preset training accuracy or meets other training stop conditions.

Claims

1. A federated learning algorithm based on adaptive aggregation and dynamic training, characterized in that: The specific steps of the learning algorithm are as follows: S1. Each client receives the initial global model parameters and uses its local data to train the model. S2, locally update the model weights using stochastic gradient descent; S3. After local training is completed, the client calculates the model loss value of the current round and retains the loss as a reference for subsequent adaptive aggregation; S4. On the client side, each client uses a dynamic training strategy based on its historical training time to predict the time required for the next round of training; S5. Dynamically adjust the number of iterations of local training based on the prediction results to ensure maximum utilization of computing resources; S6. After completing local training, the client uploads the updated model parameters to the server; S7, after receiving the model updates from each client, the server adopts an adaptive aggregation strategy to update the global model; S8. The server sends the aggregated global model update to each client. After receiving the global model, the client continues the next round of local training. This process continues until the global model reaches the preset training accuracy or meets other training stop conditions.

2. The federated learning algorithm based on adaptive aggregation and dynamic training according to claim 1, characterized in that: The specific steps of the dynamic training strategy described in S4 are as follows: Q1.1: When the global model is being trained, each client records the forward propagation time t of the global model during the training process in real time. f and the back propagation time t b , and based on the recorded forward propagation time t f and the back propagation time t b , calculate the training time T of the small batch in this round of training b ; Q1.2: Based on the training time T of the small batch b , the amount of small batch data A, the batch size B and the corresponding client training round E, calculate the local training time of the global model in each client, and after each round of training, each client records the corresponding local training time T; Q1.3: Before a new round of training begins, the client retrieves the local training duration of the past N rounds and estimates the next round of training time based on the training duration of the past N rounds. After predicting the next round of training time, it adds it to the set communication window T c Combined with training to obtain the local round LE size for the next round.

3. The federated learning algorithm based on adaptive aggregation and dynamic training according to claim 2, characterized in that: The specific calculation formula for the small batch data training time mentioned in Q1.1 is as follows: Where, T b represents the training time of a small batch; x i represents the i-th training data; B represents the batch size; t f represents the forward propagation time; t b represents the back-propagation time; The specific calculation formula for the local training duration described in Q1.2 is as follows: In the formula, T represents the local training time; E represents the training round; M represents the number of mini-batches in one training; A represents the amount of data in a mini-batch; B represents the batch size; The specific calculation formula for the next round of training time described in Q1.3 is as follows: Where, T r Represents the duration of the next round of training; N represents the total number of global model training rounds recorded by the client; T i Represents the training duration of the i-th round; Where LE represents the local round size of the next round; T c Represents the communication window between the client and the server and is used to coordinate the communication frequency between the server and the client.

4. The federated learning algorithm based on adaptive aggregation and dynamic training according to claim 2, characterized in that: The communication window T described in Q1.3 c The size of determines the local computing time of each round of training on the client, and also affects the synchronization frequency of the model parameters.

5. The federated learning algorithm based on adaptive aggregation and dynamic training according to claim 2, characterized in that: The specific steps of updating the global model using the adaptive aggregation strategy described in S7 are as follows: Q2.1: While the global model is being trained locally on each client, the client calculates in real time and records the loss value of the local model during the t-1th round of communication and the change in local model training loss between round t and round t-1 of communication on the client Q2.2: According to as well as Calculate the improvement p of the client's global model in this round of training relative to the previous round k , and then use the exponential function to normalize it, and then set the two hyperparameters σ1 and σ2 according to the actual experimental situation, and determine the training data volume S of the corresponding client k , and calculate the aggregate weight w of the client k ; Q2.3: When the aggregate weight of all clients w k After the calculation is completed, based on the calculated aggregate weight w of each client k , the server performs weighted averaging of the global model parameters of all clients to aggregate the central model on the server side, and then sends the aggregated global model update to each client. After receiving the global model, the client continues with the next round of local training.

6. The federated learning algorithm based on adaptive aggregation and dynamic training according to claim 5, characterized in that: The improvement amount p described in Q2.2 k The specific calculation formula is as follows: In the formula, Represents the loss value of the local model during the t-1th round of communication on the client; Represents the change in local model training loss between round t and round t-1 of communication on the client; The specific calculation formula for the server-side central model aggregation described in Q2.3 is as follows: In the formula, w k Represents client cl k Aggregation weight; σ1 and σ2 represent hyperparameters, which are used to adjust the relative importance of loss value and data volume in aggregation; S k Represents client cl k The amount of training data.

Images