Heterogeneous federated learning optimization method and system based on resource constraint

Abstract

The invention discloses a heterogeneous federated learning optimization method and system based on resource constraint, and the method comprises the following steps: training a machine learning model through federated learning, initializing or aggregating a global model through a server, distributing the training intensity of clients, and distributing the training intensity to the selected clients participating in training; the server distributes the total training intensity of all the clients in training through training a deep reinforcement learning model, and distributes the training intensity of each client participating in training through a deterministic distribution algorithm; and after the client finishes local training, the trained model parameters and the deep reinforcement learning state are transmitted to the server, and the deep reinforcement learning model is updated according to historical training information. Under the constraint that the server has calculation resource budget compensation, the calculation time delay, the communication time delay, the calculation resources and the model convergence speed of the heterogeneous equipment in federated learning are balanced, the training speed is improved, and the consumption of calculation resource budget and the training time delay are reduced.

Description

Technical Field

[0001] The present invention belongs to the field of heterogeneous federated learning optimization, and specifically relates to a heterogeneous federated learning optimization method and system based on resource constraints. Background Art

[0002] Since Google proposed Federated Learning (FL), this distributed learning framework has become a hot research direction. The introduction of Federated Learning enables different devices to collaborate on knowledge while protecting their own privacy data. In today's Internet environment that focuses on privacy protection, Federated Learning has alleviated the data island phenomenon to a certain extent and enhanced the generalization performance of the model.

[0003] Instead of collecting data on a centralized server, FL supports distributed model training. The classic FL framework performs multiple rounds of communication and training. In each round of communication, the client obtains the global model from the server and uses the local data set for local training. After completing local training, the client returns the model parameter gradients to the central server, which combines these gradients to improve the global model. This process is repeated until the target accuracy is reached. The original data does not leave the client during the training process. FL can effectively protect data privacy and reduce the burden of wireless communication.

[0004] In the entire FL system, with the increase and development of mobile devices, the performance differences of mobile devices are becoming larger and larger. It is hoped that the global model of federated learning will be sufficiently improved in accuracy and training time. So far, many studies and efforts have been used to improve the performance of FL, including improving model accuracy, reducing communication costs, or minimizing training delays. Most FL schemes run under synchronous conditions. The synchronous aggregation method means that the central server needs to wait for all or a specified number of clients to complete local training and upload parameters before aggregating the global model. In this method, the time for each round of training depends on the slowest client, which will greatly extend the duration of each round of communication. Considering the hardware and network conditions of the client, the difference in training time will become larger. Some clients with limited training capabilities or poor network conditions will slow down FL training and cause lagging problems.

[0005] To alleviate the above problems, asynchronous FL was proposed, which allows the global model to be aggregated when receiving local parameter updates from any client. Although the asynchronous method can reduce waiting time, it will lead to unstable model aggregation and gradient lag problems, reducing the training accuracy of the model. Other studies have tried to alleviate network bottlenecks from decentralized communication, but this approach will affect the stability and reliability of the system.

[0006] Another strategy to solve the problem of falling behind is to allocate customized and diversified training intensity to clients based on their training efficiency. Specifically, for clients with limited training efficiency, the server will allocate less local training intensity (i.e., the number of local training iterations), and for clients with higher efficiency, more local training intensity can be allocated. The purpose of this approach is to make the training time of all clients closer and reduce the waiting delay of the server. However, the difference in training intensity will cause unfairness to clients and clients will lack motivation to participate. In order to incentivize client participation, the initiator of the FL task will provide participants with monetary rewards to encourage client participation. Most works assume that the budget for each round of training is limited. But in most cases, the budget is usually constrained throughout the training process, rather than within a round. The focus needs to be on the global budget allocation for training.

[0007] An existing study AMBLE proposed an adaptive method (AMBLE: Adjusting mini-batch and local epoch for federated learning with heterogeneous devices Juwon Park, Daegun Yoon, Sangho Yeo, Sangyoon 0h), which dynamically adjusts the mini-batch size and local iteration number of training, and adopts a linear learning rate scaling strategy to ensure the balance of gradient updates, thereby improving the model convergence speed and accuracy. However, the limitation of this technology is that it only considers the performance of the device, without considering the budget constraints and energy consumption of the client, which lacks incentives for participants. Summary of the invention

[0008] The main purpose of the present invention is to overcome the above-mentioned defects in the prior art and propose a resource-constrained heterogeneous federated learning optimization method, which is used to optimize the federated learning model of heterogeneous devices within the scope of computing resource budget constraints and based on client heterogeneity, considering the allocation of global budget and single-round budget. The method takes into account the hardware and network differences in computing and communication capabilities of the client, can minimize the global model loss and training delay in the FL task, and allocate the global budget to the appropriate training round.

[0009] The present invention is achieved by at least one of the following technical solutions.

[0010] A heterogeneous federated learning optimization method based on resource constraints includes the following steps:

[0011] S1. Determine the parameter server task and initialize the global model parameters of federated learning and the deep reinforcement learning model;

[0012] S2. Use the deep reinforcement learning model to learn and optimize task information, and use the proximal strategy optimization algorithm to train the deep reinforcement learning model;

[0013] S3. The parameter server outputs the total training intensity of the participating training clients through the deep reinforcement learning model, and allocates the training intensity of each participating training client according to the deterministic algorithm;

[0014] S4. The client trains with local data, updates the trained local model parameters to the parameter server, and transmits the information status of the deep reinforcement learning model to the parameter server;

[0015] S5. The parameter server updates the deep reinforcement learning model based on historical information, aggregates the parameters uploaded by each client, and performs global model updates.

[0016] Furthermore, in step S1, the client hardware setting of federated learning specifically includes:

[0017] Define a set of clients C = {c1, c2, ..., c i , ..., c N}, assuming that the i-th client c i The computation time t for each local training iteration i,cmp for:

[0018]

[0019] in Indicates the i-th client c i The CPU cycle frequency in the kth training round, μi, represents the number of CPU cycles that the CPU executes 1 bit of training data, b i The number of bits required to represent a local training iteration; c N Indicates the Nth client;

[0020] The client communication time includes the time to download the global model and the time to upload the local model parameter update. Considering only the upload time as the communication time, the client c of the kth training round is i Communication time for:

[0021]

[0022] Where ξ represents the size of the client local model parameters, Indicates the communication bandwidth;

[0023] Client c i The training time in the kth training round for:

[0024]

[0025] in, Represents client c i The number of training iterations in the kth training round, then the training time T for the kth round k for:

[0026]

[0027] Among them C k Represents the set of clients selected by the server to participate in the k-th training round.

[0028] Furthermore, in step S1, the server training task of federated learning specifically includes:

[0029] The server receives and updates the client's model parameters and performs aggregation. i The local model loss function value F i (w) is:

[0030]

[0031] in To represent the client c i In the local dataset D i The loss function of the jth example above, w represents the parameters of the global model of federated learning, and the loss function of the global model is defined as:

[0032]

[0033] Define the training task objectives as:

[0034]

[0035] Where K represents the total number of rounds of federated learning training, w K represents the model parameters of the global model in round K, F(w K ) represents the training loss function value of the global model in the Kth round, B k represents the budget used to motivate clients to participate in training in the kth training round, T k represents the training time of the kth round;

[0036] Constraints and Restrictions:

[0037]

[0038] in For client c i Compensation for the computational resource budget consumed by a local training iteration in the kth training round, represents the global budget of the federated learning task, represents a positive integer, Represents client c i The number of training iterations in the kth training round, Client c i At the kth training round of training time, C k Represents the set of clients selected by the server to participate in the k-th training round.

[0039] Furthermore, in step S2, the proximal strategy optimization algorithm collects strategy π k′ =π(θ k′ ), and then use the advantage estimation algorithm to learn the advantage, where θ k′ represents the network parameters of the policy network of the deep reinforcement learning model at the k′th step of training, π k′ and π(θ k′ ) represents the strategy of the strategy network at step k′;

[0040] The advantage estimation algorithm is defined as the difference between the expected cumulative reward of action a and the total expected cumulative reward:

[0041]

[0042] in, represents the attractiveness of action a; s represents the current state of the deep reinforcement learning model, π θ represents the policy of the current policy network, Indicates that the policy network takes action a in state s, following the policy π θ is the expected cumulative reward from the current moment, Indicates that the policy network follows the policy π in state s θ The expected cumulative reward when represents mathematical expectation;

[0043] After getting the estimated advantage, calculate the strategy and update it:

[0044]

[0045] where θ k′+1 represents the policy network parameters of the k′+1th step, θ is θ k′+1 , represents the loss function of the proximal policy optimization algorithm; where Defined as:

[0046]

[0047] in Indicates that the model changes from the current policy π k′In the generated trajectory distribution, the expected value of a function, T represents the time from time 0 to the present, r t (θ) represents the importance sampling ratio, represents the estimated value of the advantage estimation function, ∈ is a hyperparameter used to control the magnitude of the change between the new and old strategies.

[0048] Furthermore, the state, action, and reward information of the deep reinforcement learning model are as follows:

[0049] Status: The status of the deep reinforcement learning model needs to record the loss function value and training time of the k-1th round of training. The remaining global budget of the server needs to be considered. The status records the ratio of the used budget to the original global budget. k′ represents the k′th training round in federated learning, B k′ represents the budget consumption of the k′th training round; represents the global budget of the federated learning task;

[0050] The states are defined as:

[0051] s k-1 ={F k-1 , Q k-1 , T k-1 , k-1};

[0052] where s k-1 represents the k-1th round state, F k-1 represents the loss function value of the global model in the previous round; T k-1 is the training time of the k-1th round; Q k-1 Indicates the ratio of the budget used in the k-1th round to the original global budget;

[0053] Action: Given the current state, the action of the deep reinforcement learning model agent is determined by the policy network. The total training intensity of all clients of the deep reinforcement learning model agent in the current round is a k for:

[0054] a k =τ k ;

[0055] where τ k represents the total training intensity in the kth training round;

[0056] Reward: The formula of the reward function is:

[0057] r k =α(F k-1 -F k )-β(Q k -Q k-1 )-T k ;

[0058] Where α and β are constants, α>0 and β>0, T k is the training time of the kth round, r k That is, reward feedback, which is used to guide the learning process and goal-oriented behavior of the intelligent agent.

[0059] Furthermore, after the deep reinforcement learning model outputs the total training intensity of a round, it is distributed to each client through a deterministic algorithm. The optimization goal of the deterministic algorithm is:

[0060]

[0061] in, Represents client c i The number of training iterations in the kth training round, B k represents the budget used to motivate clients to participate in training in the kth training round, T k is the training time of the kth round, α′ represents the hyperparameter used to balance the importance of budget consumption and training delay;

[0062] The constraints are:

[0063]

[0064] Among them C k represents the set of clients selected by the server to participate in training in the kth training round, B k′ represents the budget consumed in the k′th training round, represents a positive integer, represents the global budget of the federated learning task, τ k represents the total training intensity in the kth training round;

[0065] in For client c i Compensation for the computational resource budget consumed by a local training iteration in the kth training round, represents the global budget of the federated learning task, τ k represents the total training intensity in the kth training round; the role of α′ is to balance the importance of the two objectives.

[0066] Further, the deterministic algorithm is as follows:

[0067] The set of clients C selected by the server to participate in training in the kth training round k Each client c i As the slowest client, set the number of local training times. The training time of the slowest client satisfies:

[0068]

[0069] where c i′ Represents any client participating in training, Represents client c i′ training time, Indicates the training time of the slowest client;

[0070] Since each client participating in training should be assigned at least one local training time, the slowest client c i The number of local training times should satisfy:

[0071]

[0072] in, Represents client c i The number of training iterations in the kth training round, τ k represents the total training intensity in the kth training round;

[0073] Enumerate the number of local training iterations of the slowest client value, The range is [1, τ k -|C k |+1], in the selected After that, other clients participating in the training meet the following conditions:

[0074]

[0075] in Represents client c i′ The number of local training rounds in the kth training round, Indicates the slowest client c i The computation time of a local training is Indicates the slowest client c i The communication time, Represents client c i′ The communication time, Represents client c i′ The computation time of a local training; for The upper bound of .

[0076] 8. The heterogeneous federated learning optimization method based on resource constraints according to claim 7 is characterized in that the client c i′ Number of local training runs in the kth training round The upper bound of The following inequality is satisfied:

[0077]

[0078] where τk represents the total training intensity in the kth training round;

[0079] like The above two inequalities are not satisfied, indicating that C k \{c i}, there is a client that is assigned a local training number less than 1, or the local training number assigned to the client cannot be τ k After allocation, there is no feasible solution, and the next step is required. An attempt; where {c i} means only client c is included i The collection of C k \{c i} means that except for the slowest client c i All clients participating in the training except

[0080] if If the above two inequalities are satisfied, it means that there is a feasible solution, and the remaining training intensity needs to be distributed to the remaining clients C participating in the training k \{c i}; Through the greedy strategy, C k \{c i}Clients are sorted from small to large according to the computing resources consumed per training iteration. First, each client is guaranteed to be allocated at least one training iteration. The largest number of training iterations is allocated to the front client. At the same time, Repeatedly enumerate in the range [1, τ k -|C k |+1] within And all the clients participating in the training are the slowest client c i , get the minimum B k +α′T k This is the optimal solution for the target.

[0081] A system for implementing the heterogeneous federated learning optimization method based on resource constraints includes:

[0082] Heterogeneous device clients: including smartphones, IoT devices, and edge devices. Each device performs federated learning local model training locally and shares model updates while ensuring data privacy.

[0083] Training intensity allocation setting: At the beginning of each round of training, the server allocates the number of local training iterations required for this round of training to each client participating in the training based on the deep reinforcement learning model and deterministic algorithm;

[0084] Deep reinforcement learning module: including storing the client's historical training information, updating the deep reinforcement learning model, and outputting actions at the beginning of a training round, and converting them into the total training intensity of all participating clients in a round;

[0085] Deterministic algorithm module: After the deep reinforcement learning model outputs the total training intensity, it allocates the training intensity of each client according to the client's training time and unit consumption budget, and calculates the budget consumption;

[0086] Global update and aggregation module: After training the local model, the client will upload the local model update and deep reinforcement learning status information obtained from the training to the server. The server will aggregate the global model and regularly update the policy network and value network of the deep reinforcement learning model.

[0087] A computer device comprising a memory and a processor;

[0088] The memory is used to store computer programs;

[0089] The processor is used to execute the computer program and implement a heterogeneous federated learning optimization method based on resource constraints as described in any one of claims 1 to 8 when executing the computer program.

[0090] Compared with the prior art, the present invention has the following beneficial effects:

[0091] (1) The present invention proposes a heterogeneous federated learning optimization method based on resource constraints. The present invention takes into account the device heterogeneity of the federated learning system, the global budget compensation constraints, and the impact of training delay on the system, and focuses on optimizing the model accuracy while reducing training delay and computing resource consumption.

[0092] (2) In order to solve the problems caused by the dynamics of the actual FL system, the present invention proposes a deep reinforcement learning model, which uses a deep reinforcement learning (DRL) agent to adaptively determine the total training intensity of each round of training clients to ensure the convergence speed and convergence effect of the model.

[0093] (3) The present invention proposes a deterministic algorithm, the purpose of which is to assign a specific local training intensity to each client participating in the training and manage the budget of the current training round. After theoretical analysis and practice, the proposed deterministic algorithm effectively minimizes the training delay and computing resource budget compensation in each round of training. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 This is a flow chart of a heterogeneous federated learning optimization method based on resource constraints according to an embodiment of the present invention;

[0095] Figure 2 is a flow chart of a deterministic algorithm according to an embodiment of the present invention;

[0096] Figure 3 The comparison results of the model accuracy of the embodiment of the present invention and other algorithms over time;

[0097] Figure 4 This is a comparison result of the model accuracy of the embodiment of the present invention and other algorithms with respect to budget consumption. DETAILED DESCRIPTION

[0098] In order to better understand the specific implementation of the present invention, the present invention is described in detail below with reference to the accompanying drawings and embodiments. The present invention aims to propose a method and system for optimizing heterogeneous federated learning. The present invention allocates the number of local training iterations that each client should train through a deep reinforcement learning model and a deterministic algorithm design.

[0099] A system of a heterogeneous federated learning optimization method based on resource constraints in this embodiment includes:

[0100] Heterogeneous device clients: The computing and storage capabilities of client devices vary. They can be smartphones, IoT devices, edge devices, etc. Each device performs model training locally and shares model updates while ensuring data privacy.

[0101] Training intensity allocation setting: At the beginning of each round of training, the server allocates the number of local training iterations required for this round of training to each client participating in the training based on the deep reinforcement learning model (DRL model) and deterministic algorithm.

[0102] Deep reinforcement learning module: stores the client's historical training information, including model loss value, consumed budget ratio, and current round budget time. Updates the DRL network and outputs actions at the beginning of a training round, which are converted into the total training intensity of all participating clients in a round.

[0103] Deterministic algorithm module: After the DRL model outputs the total training intensity, it allocates the training intensity to each client according to the client's training time and unit consumption budget, and calculates the budget.

[0104] Global update and aggregation: After training the local model, the client uploads the model update and DRL status information obtained from the training to the server. The server will aggregate the global model and regularly update the DRL policy network and value network.

[0105] like Figure 1 As shown, a heterogeneous federated learning optimization method based on resource constraints in this embodiment includes the following steps:

[0106] S1. The parameter server determines the task and initializes the global model parameters and DRL model of federated learning:

[0107] Assume that in the federated learning system (classic FL algorithm FedAvg), the parameter of the global model of federated learning is w, and define a set of clients C = {c1, c2, ..., c i , ..., c N}, assuming that the i-th client c i With local dataset D i , the dataset size is |D i |, client c i The loss function is defined as:

[0108]

[0109] Accordingly, the loss function of the global model can be defined by the N clients participating in the training as:

[0110]

[0111] where F i (w) indicates client c i The local model loss function value, F(w) represents the loss function of the global model;

[0112] A classic FL optimization goal is to find the optimal solution parameter w* that minimizes the loss function value:

[0113]

[0114] The federated learning system consists of a central server and a group of clients, where the client set is C = {c1, c2, ..., c N}, for any client c i ∈C, assuming that in the kth training round, client c i The CPU cycle frequency is The number of CPUs executing one bit of training data is μ i , the number of bits to perform one training iteration is b i , and the communication bandwidth is Then the client c i The computation time consumed by a unit training iteration is:

[0115]

[0116] in Indicates the i-th client c i The CPU cycle frequency at the kth training round, μ i represents the number of CPU cycles that the CPU executes for 1 bit of training data, bi The number of bits required to represent a local training iteration; c N Indicates the meaning of the Nth client;

[0117] Since the download time is negligible compared to the upload time, only the upload time is considered as the communication time. In the kth training round, the client c i The communication time can be expressed as:

[0118]

[0119] Where ξ represents the size of the client local model parameters, Indicates the communication bandwidth;

[0120] In order to motivate clients to participate, the task publisher needs to determine the total budget to use the budget to motivate clients to participate in training. represents the global budget of the FL task and is represented by B k Define the budget used to incentivize client participation in the kth training round, To define the client c i The number of training iterations (i.e., training intensity) of client c i The training time in the kth training round can be obtained by the following formula:

[0121]

[0122] in, Represents client c i The number of training iterations in the kth training round, therefore, the training time for the kth training round is:

[0123]

[0124] Among them C k represents the set of clients selected by the server to participate in training in the kth training round;

[0125] Client c i The resource consumption in the kth round of training is computing energy consumption and communication energy consumption. Communication energy consumption is independent of the client's computing power and training intensity, and is easy to supplement or ignore. The client's computing resource consumption that needs to be calculated is mainly considered. Assume that the client's computing consumption per unit training iteration is In a round of training, client c i The computational energy consumption is at least

[0126] S2, uses the deep reinforcement learning (DRL) model to learn the optimization task information, uses the proximal policy optimization (PPO) algorithm to train the random policy in an on-policy manner, and explores the sampling actions according to the latest version of its random policy;

[0127] The deep reinforcement learning model algorithm used in the present invention is the PPO algorithm. The PPO algorithm trains random policies in an on-policy manner and explores sampled actions based on the latest version of its random policy. It is mainly used to solve continuous control tasks. The core idea of ​​PPO is to prevent the policy from being updated too quickly or too much by limiting the gradient of each policy update, thereby improving the stability of training. It introduces a new optimization objective function that uses clipping to limit the changes between the new and old policies.

[0128] The main process of PPO is to collect a set of trajectories on the strategy, such as π k′ =π(θ k′ ) is the strategy at step k′, where θ k′ represents the network model parameters of the deep reinforcement learning model policy network at the k′th step of training, π k′ and π(θ k′ ) represent the strategy of the policy network at the k′th step. PPO learns the advantage through the advantage estimation algorithm. The advantage is defined as That is, the difference between the expected cumulative reward of a specific action a and the total expected cumulative reward, which describes the attractiveness of the action, s represents the current state of the deep reinforcement learning model, and π θ represents the policy of the current policy network, Indicates that the policy network takes action a in state s, following the policy π θ is the expected cumulative reward from the current moment, and Indicates that the policy network follows the policy π in state s θ The expected cumulative reward when Represents mathematical expectation. After obtaining the estimated advantage, the calculation strategy is updated as follows:

[0129]

[0130] where θ k′+1 represents the policy network parameters of the k′+1th step, θ is θ k′+1 , Represents the loss function of the PPO algorithm; Defined as:

[0131]

[0132] in Indicates that the model changes from the current policy πk′ In the generated trajectory distribution, the expected value of a function, T represents the time from time 0 to the present, r t (θ) represents the importance sampling ratio, represents the estimated value of the advantage estimation function, ∈ is a hyperparameter used to control the magnitude of the change between the new and old strategies;

[0133] The strategy will take K′ steps of mini-batch stochastic gradient descent (minibatch SGD) updates.

[0134] S3. The parameter server outputs the total training intensity of the participating training clients through the deep reinforcement learning model, and allocates the training intensity of each participating training client according to the deterministic algorithm;

[0135] Since it is very difficult to mathematically express and calculate the accuracy of the loss function before training, it is very difficult to design a deterministic algorithm to optimize the task objective. This method proposes to use a deep reinforcement learning model to adaptively learn the training performance of the client and determine the total training intensity of all clients in each round of communication based on historical data.

[0136] The calculation of the DRL model is performed on the central server, and the DRL model of the central server consists of a policy network and a value network. At the beginning of the kth round of training, the policy network receives the current state s k-1 , and evaluate the impact of future rewards, then the central server executes action a k After the kth round of training, the client uploads the model parameters and DRL status information, and the DRL agent receives the status s from the client. k and reward r k , and used for the next round of training. The goal of the DRL model is to maximize the expected cumulative discounted reward R = ∑ k γ k-1 r k , where γ is a future discount factor between 0 and 1.

[0137] In this method, given a state s k , the DRL agent decides the action through the policy network, action a k It is defined as the total training intensity τ of the clients participating in the training in one round k , that is, the number of local training iterations of the client training in training round k, the formula is expressed as:

[0138] a k =τ k ;

[0139] where τ kRepresents the total training intensity in the kth training round; the purpose of this part is to balance the accuracy of the global model and the consumption of training resources.

[0140] The deterministic algorithm distributes the training intensity of each participating client:

[0141] In the kth round of training, the server randomly selects a client set C from all clients k Participate in training. The DRL model outputs the total training intensity τ k After that, you need to give the client set C k Assign specific training intensity to each client in . Given τ k , when all client data are evenly distributed, the training loss and accuracy of the current training round can be guaranteed to a certain extent. In addition to shortening the training time and minimizing the budget used by the current training round, the following local training intensity allocation problem is defined:

[0142]

[0143] Where α′ is a hyperparameter used to balance the importance of budget consumption and training latency;

[0144] The constraints are:

[0145]

[0146] Among them C k represents the set of clients selected by the server to participate in training in the kth training round, B k′ represents the budget consumed in the k′th training round, Represents a positive integer.

[0147] The goal of the above formula is to minimize the weighted sum of the training time and the budget used in the current training round, where α′ is a parameter used to balance the importance of these two objectives. In the constraints, the first two constraints indicate that the budget used should not exceed the remaining global budget, and the last two constraints limit the total training intensity of the client to at least be greater than 0, and the cumulative sum is equal to the total training intensity, that is, τ k .

[0148] The specific steps of the deterministic algorithm are as follows:

[0149] In the kth training round, the client set C k The training time of the current training round is determined by C k The main idea of ​​the algorithm is to enumerate the possible slowest clients. Specifically, C k Each client c i As the slowest client, enumerate the possible training intensities. iAs the slowest client, the following formula must hold:

[0150]

[0151] The training intensity is assigned to each client according to the greedy rule, where c i′ Represents any client participating in training, Represents client c i′ training time, Indicates the training time of the slowest client.

[0152] Due to C k Each client in should be assigned at least one local training iteration, so the number of iterations assigned to c i The number of local training iterations should satisfy:

[0153]

[0154] In the range [1, τ k -|C k |+1] to enumerate possible Value. Then, assuming c i is the slowest client, for k Other clients in c i′ , we can deduce that:

[0155]

[0156] in Represents client c i′ The number of local training rounds in the kth training round, Indicates the slowest client c i The computation time of a local training, Indicates the slowest client c i The communication time, Represents client c i′ The communication time, Represents client c i′ The computation time of a local training;

[0157] use To express The upper bound of .

[0158] In C k In, it is assumed is the maximum training time, then The value of a feasible solution must satisfy the following inequality:

[0159]

[0160]

[0161] where τ k is the total training intensity of the kth training round. Make conditional judgments on the above two inequalities.

[0162] If the above conditions cannot be met, it means that k \{c i}, there is at least one client whose assigned number of local training iterations is less than 1, or the number of local training iterations that can be assigned to the client is not greater than the given total training intensity τ k In this case, client c i There is no feasible solution for the allocation of , so the next An attempt. i} means only client c is included i The set of \ indicates exclusion, C k \{c i} means that except for the slowest client c i All clients participating in the training except

[0163] if The above conditions are met, indicating that client c i Allocation It can be guaranteed that there is at least one feasible solution such that τ k Can be distributed to all clients participating in the training, and ensure that c i It is C k The slowest client in the group. Allocate the remaining training intensity to C k \{c i}, and use Indicates the remaining number of local training iterations allocated to the client.

[0164] Due to C k \{c i Each client in} must perform at least one training, Initialization Allocation Then we have the following formula:

[0165]

[0166] Indicates that the remaining training intensity is allocated using the greedy rule; according to the client's computing energy consumption, the client C k \{c i} Sort from small to large, that is in Indicates that except client c i Other clients ci′ The computing energy consumed by a local training.

[0167] Use the greedy rule to distribute the remaining training intensity Consider C one by one k \{c i}, for example, for the lth client If the remaining training intensity is still greater than Assign the maximum training intensity to it, that is Otherwise, assign the remaining training intensity value The distribution formula is:

[0168]

[0169] Then update in Indicates the number of clients The assigned training intensity.

[0170] The above allocation process is repeated until all remaining training intensities are allocated or the budget is exhausted.

[0171] In the range [1, τ k -|C k |+1] Try possible value, find B k +α′T k The optimal solution for the minimum value of .

[0172] Using the above method, try C k Each client in is regarded as the slowest client, and the case with the minimum objective function value is recorded. The algorithm flow is as follows Figure 2 shown.

[0173] S4. The client trains with local data and transmits training model parameter updates and deep reinforcement learning information to the server:

[0174] S5. After the server assigns the exact training intensity, the client performs training, uploads the model parameter updates to the server for aggregation, and transmits the state information of deep reinforcement learning to the server.

[0175] The objective of DRL is to maximize the expected cumulative discounted reward R = ∑ k γ k-1 r k , where γ is the discount factor for future rewards, which is between 0 and 1. The DRL states that need to be recorded in the heterogeneous federated learning system mentioned in the present invention are the loss function value and training time of the k-1th round. In addition, the remaining global budget also needs to be considered, and the ratio of the used budget to the original global budget can be used as the state, that is:

[0176]

[0177] Where k′ represents the k′th training round in federated learning, B k ′ represents the budget consumption of the k′th training round;

[0178] Therefore, the state after the k-1th round of training is defined as:

[0179] s k-1 ={F k-1 , Q k-1 , T k-1 , k-1};

[0180] where F k-1 Represents the global model loss function value of the previous round.

[0181] After the FL system executes the algorithm, three aspects need to be considered, namely the model training loss value, training time, and the budget used in the current training round. The reward function can be expressed as:

[0182] r k =α(F k-1 -F k )-β(Q k -Q k-1 )-T k ;

[0183] Where α>0 and β>0, both are positive constants, which are used to balance the loss function of the current training round, the proportion of the used budget, and the training time.

[0184] The server updates the deep reinforcement learning model based on historical information and aggregates the global model:

[0185] The deep reinforcement learning model includes a policy network and an evaluation network. Every time the server collects a certain number of policy trajectories, it will update and adjust the DRL model. At the same time, the server's global model of federated learning aggregates the parameter updates uploaded by the client through the following formula:

[0186]

[0187] Next, the server will determine whether the global budget is exhausted or the model performance has reached the requirement. If both conditions are not met, the process of the next training round will continue. Otherwise, the training of this reinforcement learning epoch will be terminated and the training of the next reinforcement learning epoch will be carried out to allow the reinforcement learning agent to collect enough strategy trajectories for adjustment and update.

[0188] For the deep reinforcement learning model, the external environment is a federated learning model. The deep reinforcement learning model needs to record the state information in the federated learning environment and update the DRL model after each training round. Feedback and adjustment are performed in the interaction with the federated learning environment. In the entire system, the client's hardware and network conditions are heterogeneous, and the training delay and consumption of computing resources are different. The system takes into account the global budget and delay minimization during the training process, and improves the convergence efficiency of the global model.

[0189] As a preferred embodiment, simulation experiments were conducted on three public real datasets, including MNIST, Fashion-MNIST, and CIFAR-10 datasets. The MNIST dataset and the Fashion-MNIST (F-MNIST for short) dataset are two similar datasets, both of which consist of a training dataset of 60,000 samples and a test set of 10,000 samples. The test set is divided into 10 categories, and the sample images consist of 28×28 grayscale pixels. The difference between the two datasets is that MNIST is handwritten digits and F-MNIST is clothing. The CIFAR-10 dataset consists of a training set of 50,000 samples and a test set of 10,000 samples. It is also divided into 10 different classes, and the images consist of 32×32 RGB pixels.

[0190] For the MNIST and F-MNIST datasets, the CNN model was trained with two 5×5 convolutional layers, with 10 output channels in the first layer and 20 output channels in the second layer, each followed by a 2×2 max pooling layer, followed by two fully connected layers and an output layer with 10 units. The model for the F-MNIST dataset had an additional dropout layer between the two fully connected layers, and the learning rate of the model was 0.005.

[0191] For the CIFAR dataset, the CNN model was trained with two 5×5 convolutional layers, the first layer had 6 output channels, the second layer had 16 output channels, each layer was followed by a 2×2 max pooling layer, followed by 3 fully connected layers and an output layer with 10 units, and the learning rate of the model was set to 0.05.

[0192] As an example, this example sets the total number of clients N = 100, and each client has the same number of data samples. Similar to the traditional algorithm FedAvg, the central server randomly selects 10 clients to participate in the training in each training round. The total training intensity range in the kth training round is set to [1×|C k |, 5×|C k |]. For each client c i, the CPU cycle frequency range is set to [1.0,5.0] GHz, and the communication time of each client is randomly set in [0.5,20] seconds. The number of CPU cycles executed by one bit μ i It is set to 40 times / second and the effective capacitance coefficient of the computing chipset is set to 0.15.

[0193] For the hyperparameters related to DRL, the learning rate of the actor network and the critic network is set to 0.00005, and the reward discount factor is set to γ ​​= 0.95.

[0194] As an embodiment, this embodiment is compared with two baseline algorithms, FedAvg and AMBLE, and the optimization method provided by the present invention is called BFL-HTI with DRL. FedAvg is a classic algorithm proposed by Google, which assigns the same training intensity to each client in each round of communication; the AMBLE algorithm adaptively adjusts the local mini-batch training size and the number of local training iterations for heterogeneous clients. Figure 3 The comparison results of the present invention in terms of model accuracy changing with training time are shown. Figure 3 (a) shows that the accuracy of the model of the present invention increases faster than that of other models (FedAvg, AMBLE) under the MNIST data set. Figure 3 (b) shows that under the F-MNIST dataset, the accuracy of the model of the present invention increases faster than that of other models. Figure 3 (c) shows that under the CIFAR-10 dataset, the model accuracy convergence speed of the present invention has obvious advantages. Figure 4 The comparison results of the model accuracy of the present invention with the budget consumption are shown. Figure 4 (a) shows that under the MNIST dataset, the model budget consumption of the present invention is the least. Figure 4 (b) shows that under the F-MNIST dataset, the model budget consumption of the present invention is the least. Figure 4 (c) shows that under the CIFAR-10 dataset, the model budget consumption of the present invention is significantly less than that of other models. The results show that in the early stages of training, the test accuracy of different algorithms is most different, and the optimization method proposed in the present invention can accelerate model convergence and reduce training time, and achieve the target accuracy with the minimum global budget, which verifies the effectiveness and superiority of the present invention and provides a feasible optimization solution for actual heterogeneous federated learning systems.

[0195] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A heterogeneous federated learning optimization method based on resource constraints, characterized in that: The steps include: S1. Determine the parameter server task and initialize the global model parameters of federated learning and the deep reinforcement learning model; S2. Use the deep reinforcement learning model to learn and optimize task information, and use the proximal strategy optimization algorithm to train the deep reinforcement learning model; S3. The parameter server outputs the total training intensity of the participating training clients through the deep reinforcement learning model, and allocates the training intensity of each participating training client according to the deterministic algorithm; S4. The client trains with local data, updates the trained local model parameters to the parameter server, and transmits the information status of the deep reinforcement learning model to the parameter server; S5. The parameter server updates the deep reinforcement learning model based on historical information, aggregates the parameters uploaded by each client, and performs global model updates.

2. According to the resource constraint-based heterogeneous federated learning optimization method of claim 1, it is characterized in that: In step S1, the client hardware settings for federated learning include: Define a set of clients C = {c1,c2,…,c i ,…,c N }, assuming that the i-th client c i The computation time t for each local training iteration i,cmp for: where f i k Indicates the i-th client c i The CPU cycle frequency at the kth training round, μ i represents the number of CPU cycles that the CPU executes for 1 bit of training data, b i The number of bits required to represent a local training iteration; c N Indicates the Nth client; The client communication time includes the time to download the global model and the time to upload the local model parameter update. If only the upload time is considered as the communication time, then the client c of the kth training round i Communication time for: Where ξ represents the size of the client local model parameters, Indicates the communication bandwidth; Client c i The training time in the kth training round for: in, Represents client c i The number of training iterations in the kth training round, then the training time T for the kth round k for: Among them C k Represents the set of clients selected by the server to participate in the k-th training round.

3. According to the resource constraint-based heterogeneous federated learning optimization method of claim 1, it is characterized in that: In step S1, the server training task of federated learning specifically includes: The server receives and updates the client's model parameters and performs aggregation. i The local model loss function value F i (w) is: in To represent the client c i In the local dataset D i The loss function of the jth example above, w represents the parameters of the global model of federated learning, and the loss function of the global model is defined as: Define the training task objectives as: Where K represents the total number of rounds of federated learning training, w K represents the model parameters of the global model in round K, F(w K ) represents the training loss function value of the global model in the Kth round, B k represents the budget used to motivate clients to participate in training in the kth training round, T k represents the training time of the kth round; Constraints and Restrictions: in For client c i Compensation for the computational resource budget consumed by a local training iteration in the kth training round, represents the global budget of the federated learning task, represents a positive integer, Represents client c i The number of training iterations in the kth training round, Client c i At the kth training round of training time, C k Represents the set of clients selected by the server to participate in the k-th training round.

4. According to the resource constraint-based heterogeneous federated learning optimization method of claim 1, it is characterized in that: In step S2, the proximal policy optimization algorithm collects the policy π k′ =π(θ k′ ), and then use the advantage estimation algorithm to learn the advantage, where θ k′ represents the network parameters of the policy network of the deep reinforcement learning model at the k′th step of training, π k′ and π(θ k′ ) represents the strategy of the strategy network at step k′; The advantage estimation algorithm is defined as the difference between the expected cumulative reward of action a and the total expected cumulative reward: in, represents the attractiveness of action a; s represents the current state of the deep reinforcement learning model, π θ represents the policy of the current policy network, Indicates that the policy network takes action a in state s, following the policy π θ is the expected cumulative reward from the current moment, Indicates that the policy network follows the policy π in state s θ The expected cumulative reward when represents mathematical expectation; After getting the estimated advantage, calculate the strategy and update it: where θ k′+1 represents the policy network parameters of the k′+1th step, θ is θ k′+1 , represents the loss function of the proximal policy optimization algorithm; where Defined as: in Indicates that the model changes from the current policy π k′ In the generated trajectory distribution, the expected value of a function, T represents the time from time 0 to the present, r t (θ) represents the importance sampling ratio, represents the estimated value of the advantage estimation function, ∈ is a hyperparameter used to control the magnitude of the change between the new and old strategies.

5. The heterogeneous federated learning optimization method based on resource constraints according to claim 1 is characterized in that: The state, action, and reward information of the deep reinforcement learning model are as follows: Status: The status of the deep reinforcement learning model needs to record the loss function value and training time of the k-1th round of training. The remaining global budget of the server needs to be considered. The status records the ratio of the used budget to the original global budget. k′ represents the k′th training round in federated learning, B k′ represents the budget consumption of the k′th training round; represents the global budget of the federated learning task; The states are defined as: s k-1 ={F k-1 ,Q k-1 ,T k-1 ,k-1}; where s k-1 represents the k-1th round state, F k-1 represents the loss function value of the global model in the previous round; T k-1 is the training time of the k-1th round; Q k-1 Indicates the ratio of the budget used in the k-1th round to the original global budget; Action: Given the current state, the action of the deep reinforcement learning model agent is determined by the policy network. The total training intensity of all clients of the deep reinforcement learning model agent in the current round is a k for: a k =t k ; where τ k represents the total training intensity in the kth training round; Reward: The formula of the reward function is: r k =α(F k-1 -F k )-β(Q k- Q k-1 )-T k ; Where α and β are constants, α>0 and β>0, T k is the training time of the kth round, r k That is, reward feedback, which is used to guide the learning process and goal-oriented behavior of the intelligent agent.

6. The heterogeneous federated learning optimization method based on resource constraints according to claim 1 is characterized in that: After the deep reinforcement learning model outputs the total training intensity of a round, it is distributed to each client through a deterministic algorithm. The optimization goal of the deterministic algorithm is: in, Represents client c i The number of training iterations in the kth training round, B k represents the budget used to motivate clients to participate in training in the kth training round, T k is the training time of the kth round, α′ represents the hyperparameter used to balance the importance of budget consumption and training delay; The constraints are: Among them C k represents the set of clients selected by the server to participate in training in the kth training round, B k′ represents the budget consumed in the k′th training round, represents a positive integer, represents the global budget of the federated learning task, τ k represents the total training intensity in the kth training round; in For client c i Compensation for the computational resource budget consumed by a local training iteration in the kth training round, represents the global budget of the federated learning task, τ k represents the total training intensity in the kth training round; the role of α′ is to balance the importance of the two objectives.

7. The heterogeneous federated learning optimization method based on resource constraints according to claim 6 is characterized in that: The deterministic algorithm is as follows: The set of clients C selected by the server to participate in training in the kth training round k Each client c i As the slowest client, set the number of local training times. The training time of the slowest client satisfies: where c i′ Represents any client participating in training, Represents client c i′ training time, Indicates the training time of the slowest client; Since each client participating in training should be assigned at least one local training time, the slowest client c i The number of local training times should satisfy: in, Represents client c i The number of training iterations in the kth training round, τ k represents the total training intensity in the kth training round; Enumerate the number of local training iterations of the slowest client value, The range is [1,τ k -|C k |+1], in the selected After that, other clients participating in the training meet the following conditions: in Represents client c i′ The number of local training rounds in the kth training round, Indicates the slowest client c i The computation time of a local training, Indicates the slowest client c i The communication time, Represents client c i′ The communication time, Represents client c i′ The computation time of a local training; for The upper bound of .

8. The heterogeneous federated learning optimization method based on resource constraints according to claim 7 is characterized in that: Client c i′ Number of local training runs in the kth training round The upper bound of The following inequality is satisfied: where τ k represents the total training intensity in the kth training round; like The above two inequalities are not satisfied, indicating that C k \{c i }, there is a client that is assigned a local training number less than 1, or the local training number assigned to the client cannot be τ k After allocation, there is no feasible solution, and the next step is required. An attempt; where {c i } means only client c is included i The collection of C k \{c i } means that except for the slowest client c i All clients participating in the training except if If the above two inequalities are satisfied, it means that there is a feasible solution, and the remaining training intensity needs to be distributed to the remaining clients C participating in the training k \{c i }; Through the greedy strategy, C k \{c i }Clients are sorted from small to large according to the computing resources consumed per training iteration. First, each client is guaranteed to be allocated at least one training iteration. The largest number of training iterations is allocated to the front client. At the same time, Repeatedly enumerate in the range [1,τ k -|C k |+1] within And all the clients participating in the training are the slowest client c i , get the minimum B k +α′T k This is the optimal solution for the target.

9. A system for implementing the resource-constrained heterogeneous federated learning optimization method according to claim 1, characterized in that: include: Heterogeneous device clients: including smartphones, IoT devices, and edge devices. Each device performs federated learning local model training locally and shares model updates while ensuring data privacy. Training intensity allocation setting: At the beginning of each round of training, the server allocates the number of local training iterations required for this round of training to each client participating in the training based on the deep reinforcement learning model and deterministic algorithm; Deep reinforcement learning module: including storing the client's historical training information, updating the deep reinforcement learning model, and outputting actions at the beginning of a training round, and converting them into the total training intensity of all participating clients in a round; Deterministic algorithm module: After the deep reinforcement learning model outputs the total training intensity, it allocates the training intensity of each client according to the client's training time and unit consumption budget, and calculates the budget consumption; Global update and aggregation module: After training the local model, the client will upload the local model update and deep reinforcement learning status information obtained from the training to the server. The server will aggregate the global model and regularly update the policy network and value network of the deep reinforcement learning model.

10. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement a heterogeneous federated learning optimization method based on resource constraints as described in any one of claims 1 to 8 when executing the computer program.

Images