Method for jointly optimizing client selection and bandwidth allocation based on DRL in federated learning

Abstract

The application discloses a kind of method for jointly optimizing client selection and bandwidth allocation based on DRL in federal learning, called CSBWA, the application formulates an optimization problem, aims at minimizing the total cost of system, defined as the weighted sum of training time and energy consumption. By jointly optimizing client selection and bandwidth allocation, less time cost and energy consumption can be used to ensure the long-term performance of FL. The method relies on the REINFORCE algorithm based on DRL, inputs the observed client state and historical bandwidth information into the policy network, and automatically executes effective client scheduling and bandwidth allocation strategies according to feedback rewards. When applied to the training of datasets MNIST, Fashion-MNIST and CIFAR10, CSBWA effectively reduces time cost and energy consumption while ensuring the performance of wireless federal learning model compared with four other advanced methods.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, in particular to the intersection of federated learning model training algorithms and deep reinforcement learning model training, and is a method for jointly optimizing client selection and bandwidth allocation based on DRL in federated learning. Background Art

[0002] It involves two major background technologies: federated learning and deep reinforcement learning.

[0003] 1) Federated Learning

[0004] Today, billions of edge devices are connected to the internet, generating massive amounts of data that can be used to train more powerful and accurate models. To fully tap the value of this massive amount of edge data, federated learning is considered a promising solution. Its greatest advantage lies in its ability to combine data from multiple parties while protecting client data privacy. However, in wireless federated learning networks (WFLNs), a classic deployment scenario for federated learning, FL requires substantial computing power, communication bandwidth, and storage space, which undoubtedly limits its deployment in resource-constrained wireless network environments.

[0005] Given the limited communication resources in WFLNs, a straightforward approach to improving model training efficiency is to design an efficient client selection strategy during FL training. For example, some related work has considered fairness in client selection and designed algorithms to ensure fairness; however, these works rarely consider the quality of client data samples. Some work has considered selecting clients based on the bias of their data samples, but these efforts do not fully account for client heterogeneity.

[0006] During iterative distributed FL training, high-volume model weights or updates are frequently transmitted between the server and clients. Therefore, designing an efficient communication management mechanism is crucial for improving FL training efficiency. Some work has described a bisection-based iterative strategy to obtain a resource allocation scheme that optimizes training time while minimizing energy consumption. However, these works specify that all clients participate in each round of FL training, without considering bandwidth constraints.

[0007] 2) Scheduling Optimization Based on Deep Reinforcement Learning

[0008] Deep reinforcement learning (DRL) is a type of machine learning that incorporates neural networks from deep learning into reinforcement learning models. In this model, an agent learns to take optimal actions or strategies and adapt to the environment by performing actions and observing rewards. Markov decision processes (MDPs) model the environment in which the agent operates in reinforcement learning. Almost all problems in reinforcement learning can be formally represented as a Markov decision process. Several studies have demonstrated the widespread application of DRL in scheduling tasks such as job scheduling and resource allocation, promoting automated system management. Some work has demonstrated the ability to perform client selection or computing resource allocation based on DRL within a limited budget by considering multiple features that influence FL learning performance. However, these studies have not considered jointly optimizing client selection and bandwidth allocation to further improve efficiency. Summary of the Invention

[0009] The purpose of this invention is to provide a method for jointly optimizing client selection and bandwidth allocation based on DRL in federated learning, aiming to reduce the training and communication overhead of federated learning as much as possible while ensuring the performance of FL.

[0010] The specific technical solution for achieving the purpose of the present invention is:

[0011] A method for jointly optimizing client selection and bandwidth allocation based on DRL in federated learning, characterized in that the method comprises the following steps:

[0012] S1, the first round of federated learning (FL), the server initializes a global model w 0 ; In the next round, the server processes the local model uploaded by the client based on the formula Perform weighted averaging to obtain the updated global model w j , where D i represents the size of the local dataset of client i, represents the local model of client i in round j-1, K represents the number of selected clients;

[0013] S2, the server will global model w j (j=0, 1, ...) is sent to all clients, the total number of clients is N;

[0014] S3, the agent obtains the status information of all clients The agent makes decisions based on the REINFORCE algorithm, selects K clients and allocates corresponding upload bandwidth to the selected clients; specifically, it obtains the status information of all clients. in

[0015] D i : The size of the local dataset of client i, which can be directly obtained before FL model training;

[0016] The data quality of client i in the jth round of training; all clients first train the global model downloaded from the server with a subset of the local dataset; then, each local model is trained with a small portion of the test data on the server, and the corresponding loss value is obtained; a more skewed data distribution is reflected as a larger loss value, and the loss value is used. i To express the quality of data distribution

[0017] The CPU frequency held by client i in the jth round of training; the computation time of each client training the local model is closely related to the CPU frequency used for calculation. Indicates the computing power of the client;

[0018] The upload bandwidth allocated to client i in the j-1th round of training; the future network bandwidth is related to the historical bandwidth information; at the same time, bandwidth information is the main factor affecting the system cost, so consider the bandwidth information of the previous time window Add to the state set; when j = 1, under the maximum bandwidth (maximum bandwidth is expressed as B) limit, allocate B / N bandwidth to each client; for subsequent rounds, the agent makes a decision in the j-1 round to allocate bandwidth to the selected client i, then

[0019] S4, input the state set S into the policy network, which is a neural network composed of three fully connected layers. The agent makes a decision based on the REINFORCE algorithm. j =(i, b i ), i∈K, where b i =p i B,, where p i It represents the probability value of the policy network outputting client i, that is, obtaining the selected client set K and the bandwidth b allocated to each selected client;

[0020] S5. Each client i, i∈K, uses its own local dataset to train the global model downloaded from the server to obtain an updated local model Then pass Upload its local model to the server. After receiving the local models of all selected clients, the server obtains a new global model w through weighted aggregation. j And model accuracy acc j ;

[0021] S6. Agent calculates the reward value r j+1 ;

[0022] S7, repeat steps S1-S6 until acc j When the target accuracy Γ is reached, the episode ends and the policy network updates its network parameters θ, including:

[0023]

[0024] Where γ is the discount factor, γ∈[0,1], J is the maximum number of training rounds, and the reward value of each round of reinforcement learning is expressed as r:

[0025]

[0026] Where T j represents the total training time of the jth round, E j represents the total energy consumption of the jth round, Γ represents the target accuracy value set for the specified task; α, β represent weight factors, which are used to adjust the preference of the objective function, α, β∈[0,1], and α+β=1; where T j , E j The calculation is as follows:

[0027]

[0028]

[0029]

[0030]

[0031]

[0032] Among them, λ i represents the effective capacitance coefficient of the computing chipset of client i, c i Indicates the CPU cycles required for the client to train each data sample, represents the transmission power of client i in round j, M i is the channel state of the client, N0 represents the complex Gaussian channel noise variance, κ represents the number of local iterative training times for each local model, and Mo i Indicates the size of the model to be transferred; V i jIndicates the transmission rate of the model uploaded by client i; represents the local training time cost of client i in round j, represents the communication time of client i in round j; represents the local training energy consumption of client i in round j, represents the communication energy consumption of client i in round j; in synchronous FL, the total time cost of each round depends on the client that spends the longest time in the round, and the total energy cost is the accumulation of the energy costs of all selected clients; the loss calculation of the parameters of the policy network is:

[0033]

[0034]

[0035] in B is the maximum communication bandwidth; R(τ) represents the total reward value of the τth episode of training, Represents the average expectation of the total reward value of all episodes; sets the maximum number of rounds max_episode to 200.

[0036] In step S3, the agent makes a decision based on the REINFORCE algorithm, selects K clients, and allocates corresponding upload bandwidth to the selected clients. The state set of the input policy network in this step comprehensively considers the heterogeneity of different clients, such as computing heterogeneity and data heterogeneity, and further allocates corresponding bandwidth to the selected clients based on the algorithm, which is conducive to more full and efficient utilization of bandwidth resources. Therefore, this method can ensure the long-term performance of FL with less time cost and energy consumption.

[0037] The present invention considers using the baseline REINFORCE algorithm to update the policy network because if the reward obtained is always positive, then when updating, the probability will increase, but the degree of increase will vary. In actual sampling, only a few actions can be sampled at a certain time, and some actions cannot be sampled. Some sampled actions will continue to increase their probability, while the probability of unsampled actions will decrease. This is obviously unreasonable. Therefore, using the baseline-based REINFORCE algorithm, by subtracting the baseline from the total reward, will not affect the update expectation, but will affect the update variance, which can help the algorithm converge faster. In formula (8), if the total reward G obtained by sampling an action sequence is better than the baseline, then when the gradient is updated, its probability will increase, that is, the action sequence is enhanced.

[0038] The beneficial effects of the present invention include:

[0039] 1) Considering the limited communication bandwidth and client heterogeneity in WFLNs, this paper formalizes the joint client selection and bandwidth allocation problem, which can ensure the long-term performance of FL while using lower time cost and energy consumption.

[0040] 2) The present invention designs a FL method, CSBWA, and implements it to optimize the FL training process. By modeling the entire joint optimization process as an MDP process, the present invention only uses client information and the currently available wireless channel information, and uses the DRL-based REINFORCE algorithm to formulate an effective joint optimization strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of the present invention;

[0042] Figure 2 It is a schematic diagram of the present invention;

[0043] Figure 3 Updated flowchart for policy network;

[0044] Figure 4 Schematic diagram of the DRL agent training process;

[0045] Figure 5 This is an example diagram of the experimental results of the time overhead required for the present invention to achieve the target accuracy in three task training;

[0046] Figure 6 This is an example diagram of the experimental results of the energy consumption required to achieve the target accuracy in three task trainings of the present invention. DETAILED DESCRIPTION

[0047] The present invention is further described in detail with reference to the following specific examples and accompanying drawings. The processes, conditions, experimental methods, and the like for implementing the present invention, except for those specifically mentioned below, are common knowledge and common general knowledge in the art. Those skilled in the art can derive other drawings and other implementation methods based on these drawings without inventive effort.

[0048] The present invention provides a method for joint optimization of client selection and bandwidth allocation based on DRL in federated learning, which includes the following steps:

[0049] S1. The server initializes the global model w 0 / The server takes the weighted average of the local models received by the mobile terminal to obtain a new global model w j ;

[0050] S2, the server will global model w 0 / w j Send to all valid clients;

[0051] S3, the agent obtains the status information of all clients The agent makes decisions based on the REINFORCE algorithm, selects a specified number of clients, and allocates corresponding upload bandwidth to them;

[0052] S4: The selected client uses the local dataset to train the global model downloaded in S2 to obtain an updated local model. Each selected client uploads a local model based on the bandwidth allocated by S3;

[0053] S5. The server aggregates all local models uploaded by the selected clients and updates the policy network parameters:

[0054] The step S1 specifically includes: in the first round of FL, the server initializes a global model w 0 ; In the next round, the server takes a weighted average of the local models uploaded by the client Get an updated global model;

[0055] S2, the server will global model w 0 / w j Send to all valid clients;

[0056] S3, Agent makes decisions based on the Policy network, including:

[0057] S3-1. Get status information of all clients in Specifically include:

[0058] D i : The data sample size of client i can be directly obtained before FL model training;

[0059] The data quality of each client. Unbalanced data distribution can significantly reduce the learning performance of the model, so selecting clients with high-quality (as evenly distributed as possible) data samples to participate in the training round is conducive to balancing the impact of poor data sets. In order to reflect the skewness of client data distribution, all clients first train the global model downloaded from the server with a subset of the local data set. Then, each local model is trained with a small portion of the test data on the server to obtain the corresponding loss value. A more skewed data distribution is reflected as a larger loss value. Therefore, the loss value loss is used. i To indicate the quality of data distribution.

[0060] The CPU frequency held by the client, the computation time of each client training local model is closely related to the CPU frequency used for calculation. Can be used to represent the computing power of the client.

[0061] Because the future network bandwidth is related to the historical bandwidth information. At the same time, bandwidth information is the main factor affecting the system cost. The present invention uses the bandwidth information of the previous time window to calculate the bandwidth information of the previous time window. Add to the state set. When j = 1, a certain amount of bandwidth is randomly allocated to each client under the maximum bandwidth limit. For subsequent rounds, since the agent makes a decision to allocate the corresponding bandwidth to the selected client, the bandwidth information of the previous timestamp can be stored.

[0062] S3-2, input the state set S into the policy network, and the agent makes a decision based on the REINFORCE algorithm. j =(i, b i ), i∈K, where b i =p i B,, That is, a selected client set K is obtained and a corresponding bandwidth b is allocated to each selected client.

[0063] S4, each client i (i∈K) uses its own local data set to train the global model downloaded from the server, obtains an updated local model, and then uses b i Upload its local model to the server. After receiving the local models of all selected clients, the server obtains a new global model w through weighted aggregation. j And model accuracy acc j , then the agent calculates the reward value r j+1 .

[0064] Repeat S1-S4 until the global model reaches the target accuracy Γ, and then update the policy network parameters θ. The policy network update specifically includes:

[0065]

[0066] Where γ is the discount factor, γ∈[0,1], J is the maximum number of training rounds, and the reward value of each round of reinforcement learning is expressed as r:

[0067]

[0068] Where T j Indicates the total training time of the jth round, E j represents the total energy consumption of the jth round, and Γ represents the target accuracy value set for the specified task.j , E j The calculation is as follows:

[0069]

[0070]

[0071]

[0072]

[0073]

[0074] Among them, λ i represents the effective capacitance coefficient of the computing chipset of client i, c i Indicates the CPU cycles required for the client to train each data sample, represents the transmission power of client i in round j, M i is the channel state of the client, N0 represents the complex Gaussian channel noise variance, κ represents the number of local iterative training times for each local model, and Mo i Indicates the size of the transmission model. V i j Indicates the transmission rate of the model uploaded by client i; represents the local training time cost of client i in round j, represents the communication time of client i in round j; represents the local training energy consumption of client i in round j, represents the communication energy consumption of client i in round j; in synchronous FL, the total time cost of each round depends on the client that spends the longest time in the round, and the total energy cost is the accumulation of the energy costs of all selected clients. The loss calculation of the parameters of the policy network is:

[0075]

[0076]

[0077] in B is the maximum communication bandwidth.

[0078] Our CSBWA approach relies on a DRL-based reinforcement algorithm to automatically optimize client selection and bandwidth allocation strategies. The decision-making process considers each client's data sample size, data quality, computing power, and available historical bandwidth information, enabling low time and energy costs while maintaining long-term FL learning performance.

[0079] Example

[0080] See attached Figure 1 , which is a flow chart of the present invention.

[0081] Consider a typical federated learning method, with a total of 100 clients. In each communication round, 10 clients are randomly selected to participate in the training. Each client uses the SGD optimizer on the local dataset for 5 iterations per round. Figure 2 As shown, this embodiment provides a joint optimization method for client selection and bandwidth allocation based on DRL in federated learning, including:

[0082] S1. The server initializes the global model w 0 / The server takes the weighted average of the local models received by the mobile terminal to obtain a new global model w j ; Specifically include: In the first round of FL, the server initializes a global model w 0 ; In the next round, the server takes a weighted average of the local models uploaded by the client Get an updated global model;

[0083] S2, the server will global model w 0 / w j Send to all valid clients;

[0084] S3. Get status information of all clients The agent makes decisions based on the REINFORCE algorithm, selects a specified number of clients, and allocates corresponding upload bandwidth to them. Specifically:

[0085] S3-1. Get status information of all clients in Specifically include:

[0086] D i : The data sample size of client i can be directly obtained before FL model training;

[0087] The data quality of each client. Unbalanced data distribution can significantly reduce the learning performance of the model, so selecting clients with high-quality (as evenly distributed as possible) data samples to participate in the training round is conducive to balancing the impact of poor data sets. In order to reflect the skewness of client data distribution, all clients first train the global model downloaded from the server with a subset of the local data set. Then, each local model is trained with a small portion of the test data on the server to obtain the corresponding loss value. A more skewed data distribution is reflected as a larger loss value. Therefore, the loss value loss is used. i To indicate the quality of data distribution.

[0088] The CPU frequency held by the client, the computation time of each client training local model is closely related to the CPU frequency used for calculation. Can be used to represent the computing power of the client.

[0089] Because the future network bandwidth is related to the historical bandwidth information. At the same time, bandwidth information is the main factor affecting the system cost. The present invention uses the bandwidth information of the previous time window to calculate the bandwidth information of the previous time window. Add to the state set. When j = 1, a certain amount of bandwidth is randomly allocated to each client under the maximum bandwidth limit. For subsequent rounds, since the agent makes a decision to allocate the corresponding bandwidth to the selected client, the bandwidth information of the previous timestamp can be stored.

[0090] S3-2, input the state set S into the policy network, and the agent makes a decision based on the REINFORCE algorithm. j =(i, b i ), i∈K, where b i =p i B,, That is, a selected client set K is obtained and a corresponding bandwidth b is allocated to each selected client.

[0091] S4: The selected client uses the local dataset to train the global model downloaded in S2 to obtain an updated local model. According to the bandwidth allocated by S3, each selected client uploads a local model; the server aggregates the local models uploaded by all selected clients; specifically: each client i (i∈K) uses its own local data set to train the global model downloaded from the server, obtains an updated local model, and then uses b i Upload its local model to the server. After receiving the local models of all selected clients, the server obtains a new global model w through weighted aggregation. j And model accuracy acc j , then the agent calculates the reward value r j+1 .

[0092] Repeat S1-S4 until the global model reaches the target accuracy Γ, and then update the policy network parameters θ. The update process is as shown in the attached figure. Figure 3 As shown in the figure, the policy network update specifically includes:

[0093]

[0094] Where γ is the discount factor, γ∈[0,1], J is the maximum number of training rounds, and the reward value of each round of reinforcement learning is expressed as r:

[0095]

[0096] Where T j represents the total training time of the jth round, E j represents the total energy consumption of the jth round, and Γ represents the target accuracy value set for the specified task. j , E j The calculation is as follows:

[0097]

[0098]

[0099]

[0100]

[0101]

[0102] Among them, λ i represents the effective capacitance coefficient of the computing chipset of client i, c i Indicates the CPU cycles required for the client to train each data sample, represents the transmission power of client i in round j, M i is the channel state of the client, N0 represents the complex Gaussian channel noise variance, κ represents the number of local iterative training times for each local model, and Mo i Indicates the size of the transmission model. V i j Indicates the transmission rate of the model uploaded by client i; represents the local training time cost of client i in round j, represents the communication time of client i in round j; represents the local training energy consumption of client i in round j, represents the communication energy consumption of client i in round j; in synchronous FL, the total time cost of each round depends on the client that spends the longest time in the round, and the total energy cost is the accumulation of the energy costs of all selected clients. The loss calculation of the parameters of the policy network is:

[0103]

[0104]

[0105] in B is the maximum communication bandwidth.

[0106] Experimental verification:

[0107] To verify the versatility of this invention, two different typical networks were selected for the experiment: a CNN (two 5×5 convolutional layers) and a LeNet-5 (composed of two convolutional layers and three fully connected layers). This example used a CNN for the MNIST classification task; and a LeNet-5 for the Fashion-MNIST and CIFAR-10 classification tasks. The number of active clients in the experiment was N = 100, and the number of clients was selected as K = 10.

[0108] Client-side local training uses the SGD optimizer with a momentum of 0.9 and a weight decay of 5e-4. The learning rate starts at 0.1 and decays by a factor of 10 at [0.3, 0.6, 0.8] × total epochs. Regarding data distribution, the experiment considers both uniform data distribution (IID) and non-IID data distribution. The non-IID data distribution is configured as follows: 1) When each client contains different percentages of all class samples, for each client, σ = 0.5 means that 50% of the data samples belong to a single label, and the remaining 50% of the data samples contain nine labels; σ = 0.7 means that 70% of the data samples belong to a single label, and the remaining 30% of the data samples contain nine labels. 3) H = 2 indicates that each client contains data samples from two classes.

[0109] Attachment Figure 4This paper reports the total reward trends for three different tasks using two typical models in this embodiment (abbreviated as CSBWA) under both uniform and uneven data distribution scenarios. Figure a shows the total reward obtained by the agent when training converged on the MNIST dataset, Figure b shows the total reward obtained when training converged on the Fashion-MNIST dataset, and Figure c shows the total reward obtained when training converged on the CIFAR-10 dataset. For each learning task, it can be observed that after several hundred rounds of training, the total training reward quickly converges to a stable high value, demonstrating that the agent in this invention can learn how to intelligently jointly optimize client selection and bandwidth allocation strategies. In order to demonstrate the beneficial effects of the present invention, a comparison is also made with other existing methods, including (1) FedAvg: randomly selects clients and evenly distributes the total bandwidth; (2) FedCS: selects as many clients as possible within the maximum time limit in each round and evenly distributes the total bandwidth; (3) ACSBA: uses the DDQN algorithm to adaptively select clients to participate in the global model update; (4) Favor: describes the data distribution of the client according to the model weight uploaded by the client, and selects the client based on the DDQN algorithm to accelerate the convergence speed of FL training. This algorithm evenly distributes bandwidth. (5) CSBWA-ab: The only difference from the embodiment is that the bandwidth is evenly distributed for each selected client. For each learning task, it can be observed that the total training reward can converge to a stable high value after hundreds of training rounds. For example, MNIST requires about 500 rounds to reach the maximum reward value, Fashion-MNIST requires about 450 rounds, and CIFAR-10 requires about 450 rounds. This shows that through CSBWA, the agent can intelligently learn how to jointly optimize client selection and bandwidth allocation strategies. Experiments show that setting the target accuracy Γ to 99% for MNIST training, 85% for Fashion-MNIST training, and 54% for CIFAR-10 training is reasonable.

[0110] In addition, Appendix 1 shows the number of training rounds required to achieve the target accuracy Γ on three datasets under IID and different non-IID settings, where / indicates that the task converged to an accuracy lower than the target accuracy. As can be seen, CSBWA achieves the best performance across different tasks. For the MNIST dataset, CSBWA reduces the number of training rounds by 12%, 14.1%, 13.6%, and 20.3%, respectively, compared to the suboptimal baseline methods. For Fashion-MNIST, CSBWA reduces the number of training rounds by 15.2%, 12.5%, 22.3%, and 2.9%, respectively, compared to the suboptimal methods. For the CIFAR-10 dataset, CSBWA reduces the number of training rounds by 27.2%, 8.8%, 6.0%, and 10.5%, respectively. ACSBA, Favor, and CSBWA, which make intelligent decisions based on DRL, generally require fewer training rounds to achieve the target accuracy, validating the effectiveness of DRL-based algorithms. Furthermore, CSBWA takes into account the client's data sample size, data quality, computing power heterogeneity, and maximum bandwidth constraints, enabling faster training convergence with less energy consumption. Furthermore, the number of training rounds for the CSBWA algorithm on the CIFAR-10 dataset is smaller than that of CSBWA-ab, demonstrating that CSBWA is more suitable for learning tasks involving large-scale data samples.

[0111] Attachment Figure 5 (a) reports the total time cost of achieving the target accuracy for the MNIST dataset classification task during the test process under four different data distribution settings, (b) is the total time cost of achieving the target accuracy for the Fashion-MNIST dataset classification task during the test process, and (c) is the total time cost of achieving the target accuracy for the CIFAR-10 dataset classification task during the test process. Figure 6 (a) reports the total energy cost of achieving the target accuracy for the MNIST dataset classification task during the test process under four different data distribution settings, (b) reports the total energy cost of achieving the target accuracy for the Fashion-MNIST dataset classification task during the test process, and (c) reports the total energy cost of achieving the target accuracy for the CIFAR-10 dataset classification task during the test process. The experimental results of each trial are the average of three experimental results. Figure 5 It can be seen from the figure that CSBWA and CSBWA-ab can converge to the target accuracy faster. Figure 6 It shows that compared with the other four methods, CSBWA and CSBWA-ab consume the least energy; since the CSBWA of the present invention takes into account the computing power and historical bandwidth information of different clients and incorporates these two features into the policy network, it can achieve model convergence faster and consume less energy when the bandwidth is limited.

[0112] Table 1. The number of communication rounds required for training three tasks to achieve the target accuracy.

[0113]

[0114] The above description is only a detailed description of the preferred embodiments and principles of the present invention. For ordinary technicians in this field, based on the ideas provided by the present invention, there may be changes in the specific implementation methods, and these changes should also be considered as the scope of protection of the present invention.

Claims

1. A method for jointly optimizing client selection and bandwidth allocation based on DRL in federated learning, characterized in that: The method comprises the following steps: S1, federated learning is the first round of FL, the server initializes a global model w 0 ; In the next round, the server processes the local model uploaded by the client based on the formula Perform weighted averaging to obtain the updated global model w j , where D i represents the size of the local dataset of client i, represents the local model of client i in round j-1, K represents the number of selected clients; S2, the server will global model w j (j=0, 1, ...) is sent to all clients, the total number of clients is N; S3, the agent obtains the status information of all clients The agent makes decisions based on the REINFORCE algorithm, selects K clients and allocates corresponding upload bandwidth to the selected clients; specifically, it obtains the status information of all clients. in D i : The size of the local dataset of client i, which can be directly obtained before FL model training; The quality of the data distribution of client i in the jth round of training; all clients first train the global model downloaded from the server with a subset of the local dataset; then, each local model is trained with a small portion of the test data on the server, and the corresponding loss value is obtained; a more skewed data distribution is reflected as a larger loss value, and the loss value is used. i To express the quality of data distribution The CPU frequency held by client i in the jth round of training; the computation time of each client training the local model is closely related to the CPU frequency used for calculation. Indicates the computing power of the client; The upload bandwidth allocated to client i in the j-1th round of training; the future network bandwidth is related to the historical bandwidth information; at the same time, bandwidth information is the main factor affecting the system cost, so consider the bandwidth information of the previous time window Add to the state set; when j = 1, under the maximum bandwidth B limit, allocate B / N bandwidth to each client; for subsequent rounds, the agent makes a decision in the j-1 round to allocate bandwidth to the selected client i, then S4, input the state set S into the policy network, which is a neural network composed of three fully connected layers. The agent makes a decision based on the REINFORCE algorithm. j =(i, b i ), i∈K, where b i =p i B, where p i It represents the probability value of the policy network outputting client i, that is, obtaining the selected client set K and the bandwidth b allocated to each selected client; S5. Each client i, i∈K, uses its own local dataset to train the global model downloaded from the server to obtain an updated local model Then pass Upload its local model to the server. After receiving the local models of all selected clients, the server obtains a new global model w through weighted aggregation. j And model accuracy acc j ; S6. Agent calculates the reward value r j+1 ; S7, repeat steps S1-S6 until acc j When the target accuracy Γ is reached, the episode ends and the policy network updates its network parameters θ, including: Where γ is the discount factor, γ∈[0,1], J is the maximum number of training rounds, and the reward value of each round of reinforcement learning is expressed as r: Where T j represents the total training time of the jth round, E j represents the total energy consumption of the jth round, Γ represents the target accuracy value set for the specified task; α, β represent weight factors, which are used to adjust the preference of the objective function, α, β∈[0,1], and α+β=1; where T j , E j The calculation is as follows: Among them, λ i represents the effective capacitance coefficient of the computing chipset of client i, c i Indicates the CPU cycles required for the client to train each data sample, represents the transmission power of client i in round j, M i is the channel state of the client, N0 represents the complex Gaussian channel noise variance, κ represents the number of local iterative training times for each local model, and Mo i Indicates the size of the model to be transferred; V i j Indicates the transmission rate of the model uploaded by client i; represents the local training time cost of client i in round j, represents the communication time of client i in round j; represents the local training energy consumption of client i in round j, represents the communication energy consumption of client i in round j; in synchronous FL, the total time cost of each round depends on the client that spends the longest time in the round, and the total energy cost is the accumulation of the energy costs of all selected clients; the loss calculation of the parameters of the policy network is: in B is the maximum communication bandwidth; R(τ) represents the total reward value of the τth episode of training, Represents the average expectation of the total reward value of all episodes; sets the maximum number of rounds max_episode to 200.

Images