A resource-constrained heterogeneous federated learning optimization method and system

Abstract

This invention discloses a resource-constrained heterogeneous federated learning optimization method and system, comprising the following steps: training a machine learning model through federated learning; the server initializes or aggregates the global model; the server allocates training intensity to clients selected for training; the server allocates the total training intensity of all clients in training by training a deep reinforcement learning model, using a deterministic allocation algorithm to allocate the training intensity of each participating client; after completing local training, the client transmits the trained model parameters and deep reinforcement learning state to the server, and updates the deep reinforcement learning model based on historical training information. This invention, under the constraint of server-side computational resource budget compensation, balances the computational latency, communication latency, computational resources, and model convergence speed of heterogeneous devices in federated learning, improving training speed while reducing computational resource budget consumption and training latency.

Description

Technical Field

[0001] This invention belongs to the field of heterogeneous federated learning optimization, specifically relating to a resource-constrained heterogeneous federated learning optimization method and system. Background Technology

[0002] Since Google introduced Federated Learning (FL), this distributed learning framework has become a popular research area. Federated Learning enables different devices to collaborate on knowledge while protecting their own privacy. In today's privacy-conscious internet environment, Federated Learning alleviates data silos to some extent and enhances the generalization performance of models.

[0003] Instead of collecting data on a centralized server, Flight Through Flow (FL) supports distributed model training. The classic FL framework involves multiple rounds of communication and training. In each round, the client retrieves the global model from the server and trains it locally using its local dataset. After local training, the client returns the model parameter gradients to the central server, which then uses these gradients to refine the global model. This process is repeated until the target accuracy is achieved. Since the raw data does not leave the client during training, FL effectively protects data privacy and reduces the burden of wireless communication.

[0004] In the entire Federated Learning (FL) system, with the increasing number and development of mobile devices, the performance differences between them are becoming increasingly significant. There is a strong desire to improve the accuracy and training time of the global model in federated learning. To date, much research and effort has been devoted to improving FL performance, including increasing model accuracy, reducing communication costs, and minimizing training latency. Most FL solutions operate under synchronous conditions. Synchronous aggregation means that the central server needs to wait for all or a specified number of clients to complete local training and upload their parameters before aggregating the global model. This method makes the training time for each round dependent on the slowest client, significantly extending the duration of each communication round. Considering the hardware and network conditions of the clients, the differences in training time become even greater. Some clients with limited training capabilities or poor network conditions will slow down FL training, causing them to fall behind.

[0005] To alleviate these problems, asynchronous model aggregation (FL) was proposed, which allows the global model to be aggregated when local parameter updates are received from any client. While the asynchronous approach reduces latency, it can lead to unstable model aggregation and gradient lag, reducing training accuracy. Other studies have attempted to mitigate network bottlenecks through distributed communication, but this approach can impact system stability and reliability.

[0006] Another strategy for addressing the fallback problem is to allocate customized and diversified training intensities to clients based on their training efficiency. Specifically, for clients with limited training efficiency, the server allocates fewer local training intensities (i.e., fewer local training iterations), while for more efficient clients, more local training intensities can be allocated. The aim of this approach is to bring the training times of all clients closer together, reducing server latency. However, differences in training intensities can create unfairness among clients and reduce their motivation to participate. To incentivize client participation, FL task initiators offer monetary rewards to participants. Most work assumes a limited budget per training round. However, in most cases, the budget is typically constrained throughout the entire training process, rather than within a single round. Therefore, the focus should shift to the global budget allocation for training.

[0007] A recent study, AMBLE (Adjusting mini-batch and local epoch for federated learning with heterogeneous devices, Juwon Park, Daegun Yoon, Sangho Yeo, Sangyoon 0h), proposed an adaptive method that dynamically adjusts the mini-batch size and the number of local iterations during training. It employs a linear learning rate scaling strategy to ensure balanced gradient updates, thereby improving model convergence speed and accuracy. However, this technique has a limitation: it only considers device performance and does not take into account budget constraints and client-side energy consumption, which lacks incentive for participants. Summary of the Invention

[0008] The main objective of this invention is to overcome the aforementioned deficiencies in the prior art and propose a resource-constrained heterogeneous federated learning optimization method. This method optimizes federated learning models for heterogeneous devices by considering the allocation of global and single-round budgets based on client heterogeneity, within the constraints of computational resource budget. This method takes into account the hardware and network differences in computing and communication capabilities of clients, minimizing global model loss and training latency in FL tasks, and allocating the global budget to appropriate training rounds.

[0009] The present invention is achieved by at least one of the following technical solutions.

[0010] A resource-constrained heterogeneous federated learning optimization method includes the following steps:

[0011] S1. Determine the parameter server task and initialize the global model parameters of federated learning and the deep reinforcement learning model;

[0012] S2. A deep reinforcement learning model is used to learn and optimize task information, and a proximal policy optimization algorithm is used to train the deep reinforcement learning model.

[0013] S3. The parameter server outputs the total training intensity of the participating clients through the deep reinforcement learning model, and allocates the training intensity of each participating client according to the deterministic algorithm.

[0014] S4. The client trains the model using local data, updates the parameters of the trained local model and sends them to the parameter server, and also sends the information status of the deep reinforcement learning model to the parameter server.

[0015] S5. The parameter server updates the deep reinforcement learning model based on historical information and aggregates the parameters uploaded by each client to perform a global model update.

[0016] Furthermore, in step S1, the client hardware settings for federated learning specifically include:

[0017] Define a set of clients C = {c1, c2, ..., c...} i c N}, assuming the i-th client c i The computation time t for each local training iteration i,cmp for:

[0018]

[0019] in Indicates the i-th client c i The CPU loop frequency in the k-th training round, where μi represents the number of CPU loops executing 1 bit of training data, and b i c represents the number of bits required for one local training iteration; N Indicates the Nth client;

[0020] The client's communication time includes the time to download the global model and the time to upload and update the local model parameters. Considering only the upload time as the communication time, the client's communication time in the k-th training round is c. i Communication time for:

[0021]

[0022] Where ξ represents the size of the local model parameters on the client side. Indicates communication bandwidth;

[0023] Client c i Training time in the k-th training round for:

[0024]

[0025] in, Indicates client c i The number of training iterations in the k-th training round determines the training time T of the k-th round. k for:

[0026]

[0027] Where C k This represents the set of clients selected by the server to participate in training during the k-th training round.

[0028] Furthermore, in step S1, the server training task for federated learning specifically includes:

[0029] The server receives and updates the model parameters from the client, and performs aggregation. The client c... i The local model loss function value F i (w) is:

[0030]

[0031] in To represent client c i In local dataset D i The loss function for the j-th example, where w represents the parameters of the global model in federated learning, and the loss function of the global model is defined as:

[0032]

[0033] The training task objective is defined as follows:

[0034]

[0035] Where K represents the total number of rounds of federated learning training, w K F(w) represents the model parameters of the global model in the Kth round. K B represents the training loss function value of the global model in the Kth round. k T represents the budget used to incentivize client participation in training during the k-th training round. k This represents the training time for the k-th round;

[0036] Constraints and limitations:

[0037]

[0038] in For client c i Compensation for computational resources consumed in one local training iteration during the k-th training round. This represents the overall budget for federal learning tasks. Represents positive integers. Indicates client c i The number of training iterations in the k-th training round, Client c i In the training time of the k-th training round, C k This represents the set of clients selected by the server to participate in training during the k-th training round.

[0039] Furthermore, in step S2, the near-end policy optimization algorithm is to collect policy π. k′ =π(θ) k′ The algorithm learns the advantage by taking a partial set of trajectories from θ and then using an advantage estimation algorithm, where θ k′ π represents the network parameters of the policy network in the deep reinforcement learning model at the k′-th step of training. k′ and π(θ) k′ ) represents the policy of the policy network at step k′;

[0040] The advantage estimation algorithm defines the difference between the expected cumulative reward of action a and the total expected cumulative reward:

[0041]

[0042] in, Let represent the attractiveness of action 'a'; 's' represent the current state of the deep reinforcement learning model; and π represent the pi. θ This represents the policy of the current policy network. This indicates that the policy network takes action a in state s, following policy π. θ The expected cumulative reward starting from the current moment. This indicates that the policy network, in state s, follows policy π. θ Expected cumulative reward at that time Represents the mathematical expectation;

[0043] After obtaining the estimated advantage, calculate and update the strategy:

[0044]

[0045] Where θ k′+1 Let θ represent the policy network parameters at step k′+1, i.e., θ k′+1 , Let represent the loss function of the near-end policy optimization algorithm; where Defined as:

[0046]

[0047] in The model represents the current policy π. k′In the generated trajectory distribution, the expected value of a certain function, where T represents the time from time 0 to the present, and r... t (θ) represents the importance sampling ratio. This represents the estimated value of the advantage estimation function, where ∈ is a hyperparameter used to control the magnitude of change between the old and new strategies.

[0048] Furthermore, the state, action, and reward information of the deep reinforcement learning model are as follows:

[0049] State: The state of the deep reinforcement learning model needs to record the loss function value and training time of the (k-1)th round of training. It needs to consider the remaining global budget of the server. The state record shows the ratio of the used budget to the original global budget. k′ represents the k′-th training round in federated learning, B k′ This represents the budget consumption for the k′-th training round; This represents the overall budget for federal learning tasks;

[0050] The state is defined as:

[0051] s k-1 ={F k-1 Q k-1 T k-1 ,k-1};

[0052] Where s k-1 F represents the state in round (k-1). k-1 T represents the loss function value of the global model in the previous round; k-1 Q represents the training time for the (k-1)th round. k-1 This represents the ratio of the budget used in round (k-1) to the original global budget.

[0053] Action: Given the current state, the action of the deep reinforcement learning model agent is determined by the policy network. The total training intensity 'a' of all clients in the current round is 'a'. k for:

[0054] a k =τ k ;

[0055] Where τ k This represents the total training intensity in the k-th training round;

[0056] Reward: Formula for the reward function:

[0057] r k =α(F k-1 -F k )-β(Q k -Q k-1 )-T k ;

[0058] Where α and β are constants, α > 0 and β > 0, T k Let r be the training time for the kth round. k This refers to rewards and feedback, used to guide the learning process and goal-oriented behavior of intelligent agents.

[0059] Furthermore, after the deep reinforcement learning model outputs the total training intensity for one round, it is distributed to each client using a deterministic algorithm. The optimization objective of the deterministic algorithm is:

[0060]

[0061] in, Indicates client c i The number of training iterations in the k-th training round, B k T represents the budget used to incentivize client participation in training during the k-th training round. k Let be the training time for the kth round, and α′ be a hyperparameter used to balance the importance of budget consumption and training latency;

[0062] The constraints are as follows:

[0063]

[0064] Where C k B represents the set of clients selected by the server to participate in training in the k-th training round. k′ This represents the budget consumed in the k′-th training round. Represents positive integers. τ represents the global budget for the federated learning task. k This represents the total training intensity in the k-th training round;

[0065] in For client c i Compensation for computational resources consumed in one local training iteration during the k-th training round. τ represents the global budget for the federated learning task. k This represents the total training intensity in the k-th training round; α′ is used to balance the importance of the two objectives.

[0066] Furthermore, the deterministic algorithm is as follows:

[0067] The set C of clients selected by the server to participate in training in the k-th training round k In the middle, each client c i As the slowest client, the number of local training iterations is set, and the training time of the slowest client satisfies the following:

[0068]

[0069] Where c i′ This represents any client participating in the training. Indicates client c i′ Training time, This represents the training time for the slowest client;

[0070] Since each client participating in training should be allocated at least one local training attempt, it is assigned to the slowest client, c. i The number of local training iterations should meet the following requirements:

[0071]

[0072] in, Indicates client c i The number of training iterations in the k-th training round, τ k This represents the total training intensity in the k-th training round;

[0073] Enumerate the number of local training iterations for the slowest client. value, The range is [1, τ] k -|C k |+1], in the selected Afterwards, the other participating clients satisfy the following:

[0074]

[0075] in Indicates client c i′ The number of local training iterations in the k-th training round. Indicates the slowest client c i The computation time for one local training session. Indicates the slowest client c i Communication time, Indicates client c i′ Communication time, Indicates client c i′ The computation time for one local training session; for The upper boundary.

[0076] 8. The resource-constrained heterogeneous federated learning optimization method according to claim 7, characterized in that, client c i′ Number of local training iterations in the k-th training round The upper realm The following inequalities must be satisfied:

[0077]

[0078] Where τk This represents the total training intensity in the k-th training round;

[0079] like If the above two inequalities are not satisfied, it means that C k \{c i In the context of}, there exists a client that is allocated fewer than 1 local training iteration, or the number of local training iterations allocated to the client is insufficient to reduce τ. k After allocation, there is no feasible solution, and we need to proceed to the next step. The attempt; where {c i} indicates that it only includes client c i The set of C k \{c i} indicates that, except for the slowest client c i All clients participating in the training, except for those mentioned above;

[0080] if If the above two inequalities are satisfied, it means that a feasible solution exists, and the remaining training intensity should be distributed to the remaining clients C participating in the training. k \{c i In the middle; through a greedy strategy, C k \{c i The clients are sorted in ascending order of computational resources consumed per training iteration. First, it is ensured that each client is allocated at least one training iteration, and the client at the top is allocated the largest number of training iterations. Simultaneously, the following conditions must be met: Repeated enumeration in the range [1, τ] k -|C k |+1] within And all clients participating in the training are designated as the slowest client c. i To obtain the smallest B k +α′T k This is the optimal solution for the objective.

[0081] A system for implementing the resource-constrained heterogeneous federated learning optimization method includes:

[0082] Heterogeneous device clients include smartphones, IoT devices, and edge devices. Each device performs local federated learning model training and shares model updates while ensuring data privacy.

[0083] Training intensity allocation settings: At the beginning of each training round, the server allocates the number of local training iterations required for this round of training to each client participating in the training, based on the deep reinforcement learning model and deterministic algorithm.

[0084] The deep reinforcement learning module includes storing the client's historical training information, updating the deep reinforcement learning model, outputting actions at the beginning of the training round, and converting them into the total training intensity of all participating clients in one round.

[0085] Deterministic Algorithm Module: After the deep reinforcement learning model outputs the total training intensity, it allocates the training intensity to each client based on the client's training time and unit consumption budget, and calculates the budget consumption.

[0086] Global update and aggregation module: After the client completes the training of the local model, it uploads the trained local model update and deep reinforcement learning state information to the server. The server will aggregate the global model and periodically update the policy network and value network of the deep reinforcement learning model.

[0087] A computer device, the computer device including a memory and a processor;

[0088] The memory is used to store computer programs;

[0089] The processor is configured to execute the computer program and, in executing the computer program, implement a resource-constrained heterogeneous federated learning optimization method as described in any one of claims 1 to 8.

[0090] Compared with the prior art, the present invention has the following beneficial effects:

[0091] (1) This invention proposes a resource-constrained heterogeneous federated learning optimization method. This invention considers the impact of device heterogeneity, global budget compensation constraints and training latency on the federated learning system, and focuses on optimizing model accuracy while reducing training latency and computational resource consumption.

[0092] (2) To address the problems caused by the dynamic nature of practical FL systems, this invention proposes a deep reinforcement learning model. The deep reinforcement learning (DRL) agent adaptively determines the total training intensity of the client in each round of training, thereby ensuring the convergence speed and convergence effect of the model.

[0093] (3) This invention proposes a deterministic algorithm, which aims to allocate a specific local training intensity to each client participating in the training and manage the budget of the current training round. Through theoretical analysis and practice, the proposed deterministic algorithm effectively minimizes training latency and computational resource budget compensation in each training round. Attached Figure Description

[0094] Figure 1 This is a flowchart illustrating a resource-constrained heterogeneous federated learning optimization method according to an embodiment of the present invention.

[0095] Figure 2 This is a flowchart of the deterministic algorithm according to an embodiment of the present invention;

[0096] Figure 3 This is a comparison of the model accuracy over time between embodiments of the present invention and other algorithms;

[0097] Figure 4 This is a comparison of the model accuracy of the embodiments of the present invention with that of other algorithms as a function of budget consumption. Detailed Implementation

[0098] To better understand the specific embodiments of the present invention, the invention will be described in detail below with reference to the accompanying drawings and examples. The present invention aims to propose a method and system for optimizing heterogeneous federated learning. The present invention allocates the number of local training iterations that each client should train through deep reinforcement learning models and deterministic algorithm design.

[0099] This embodiment of a resource-constrained heterogeneous federated learning optimization method system includes:

[0100] Heterogeneous device clients: Client devices have different computing and storage capabilities. They can be smartphones, IoT devices, edge devices, etc. Each device trains the model locally and shares model updates while ensuring data privacy.

[0101] Training intensity allocation settings: At the beginning of each training round, the server allocates the number of local training iterations required for this round of training to each client participating in the training, based on the deep reinforcement learning model (DRL model) and the deterministic algorithm.

[0102] The Deep Reinforcement Learning (DRL) module stores historical training information for each client, including model loss values, the proportion of budget consumed, and the budget time for the current round. It updates the DRL network, outputs actions at the start of each training round, and converts these actions into the total training intensity for all participating clients in that round.

[0103] Deterministic Algorithm Module: After the DRL model outputs the total training intensity, it allocates the training intensity to each client based on the client's training time and unit consumption budget, and calculates the budget.

[0104] Global Updates and Aggregation: After training the local model, the client uploads the trained model update and DRL state information to the server. The server will aggregate the global model and periodically update the policy network and value network of the DRL.

[0105] like Figure 1 As shown in this embodiment, a resource-constrained heterogeneous federated learning optimization method includes the following steps:

[0106] S1. The parameter server determines the task and initializes the global model parameters and DRL model for federated learning:

[0107] Assuming a federated learning system (the classic FL algorithm FedAvg), the parameters of the global model in federated learning are w, and a set of clients C = {c1, c2, ..., c...} is defined. i c N}, assuming the i-th client c i With local dataset D i The dataset size is |D i |, Client c i The loss function is defined as:

[0108]

[0109] Accordingly, the loss function of the global model can be defined by the N clients participating in the training:

[0110]

[0111] Where F i (w) represents client c i The local model loss function value, F(w) represents the global model loss function;

[0112] A classic FL optimization objective is to find the optimal solution parameter w* that minimizes the loss function value:

[0113]

[0114] A federated learning system consists of a central server and a set of clients, where the client set is C = {c1, c2, ..., c...} N}, for any client c i ∈C, assuming that in the k-th training round, client c i The CPU cycle frequency is The number of CPU operations to execute one bit of training data is μ i The number of bits in one training iteration is b. i And the communication bandwidth is Then client c i The computation time consumed per unit training iteration is:

[0115]

[0116] in Indicates the i-th client c i CPU loop frequency in the k-th training round, μ i This represents the number of CPU loops required to execute 1 bit of training data, bi c represents the number of bits required for one local training iteration; N This indicates the meaning of the Nth client;

[0117] Since download time is negligible compared to upload time, only upload time is considered as communication time. In the k-th training round, client c i The communication time can be expressed as:

[0118]

[0119] Where ξ represents the size of the local model parameters on the client side. Indicates communication bandwidth;

[0120] To incentivize client participation, the task publisher needs to determine a total budget, which will then be used to incentivize client participation in training. Represent the global budget of the FL task, and use B k Define the budget used to incentivize client participation in the k-th training round. To define client c i The number of training iterations (i.e., training intensity). Based on the above definition, client c i The training time for the k-th training round can be obtained by the following formula:

[0121]

[0122] in, Indicates client c i The number of training iterations in the k-th training round is given by the training time in the k-th training round.

[0123]

[0124] Where C k This represents the set of clients selected by the server to participate in training in the k-th training round;

[0125] Client c i The resource consumption in the k-th training round consists of computational energy consumption and communication energy consumption. Communication energy consumption is independent of the client's computing power and training intensity, and is easily supplemented or ignored. We primarily consider the client's computational resource consumption. Assume the client's computational consumption per unit training iteration is... In one round of training, client c i The computational energy consumption is at least

[0126] S2. The Deep Reinforcement Learning (DRL) model is used to learn and optimize task information. The Proximal Policy Optimization (PPO) algorithm is used to train a random policy in an on-policy manner, and the sampling action is explored based on the latest version of the random policy.

[0127] The deep reinforcement learning model algorithm used in this invention is the PPO algorithm. The PPO algorithm trains a stochastic policy on-policy and explores sampled actions based on the latest version of the stochastic policy, primarily used to solve continuous control tasks. The core idea of ​​PPO is to improve training stability by limiting the gradient of each policy update, preventing the policy from updating too quickly or with too large an amplitude. It introduces a new optimization objective function and uses clipping to limit the changes between the old and new policies.

[0128] The main process of PPO is to collect a set of trajectories based on a strategy, such as π. k′ =π(θ) k′ ) is the strategy at step k′, where θ k′ π represents the network model parameters of the policy network in the deep reinforcement learning model at the k′-th step of training. k′ and π(θ) k′ Both ) represent the policy of the policy network at step k′. PPO learns the advantage through an advantage estimation algorithm, where the advantage is defined as... This refers to the difference between the expected cumulative reward of a specific action *a* and the total expected cumulative reward, describing the attractiveness of the action. *s* represents the current state of the deep reinforcement learning model, and *π* represents the value of the π value. θ This represents the policy of the current policy network. This indicates that the policy network takes action a in state s, following policy π. θ The expected cumulative reward starting from the current moment, and This indicates that the policy network, in state s, follows policy π. θ Expected cumulative reward at that time This represents the expected value. After obtaining the estimated advantage, the calculation strategy is updated as follows:

[0129]

[0130] Where θ k′+1 Let θ represent the policy network parameters at step k′+1, i.e., θ k′+1 , Let represent the loss function of the PPO algorithm; where Defined as:

[0131]

[0132] in The model represents the current policy π.k′ In the generated trajectory distribution, the expected value of a certain function, where T represents the time from time 0 to the present, and r... t (θ) represents the importance sampling ratio. This represents the estimated value of the advantage estimation function, where ∈ is a hyperparameter used to control the magnitude of change between the old and new strategies;

[0133] The strategy will employ K′-step minibatch stochastic gradient descent (SGD) updates.

[0134] S3. The parameter server outputs the total training intensity of the participating clients through the deep reinforcement learning model, and allocates the training intensity of each participating client according to the deterministic algorithm.

[0135] Since it is very difficult to mathematically express and calculate the accuracy of the loss function before training, it is very difficult to design a deterministic algorithm to optimize the task objective. This method proposes to use a deep reinforcement learning model to adaptively learn the training performance of the client and determine the total training intensity of all clients in each round of communication based on historical data.

[0136] The computation of the DRL model is performed on a central server, where the DRL model consists of a policy network and a value network. At the start of the k-th training round, the policy network receives the current state s. k-1 And assess the impact of future rewards, then the central server executes action a. k After the k-th round of training, the client uploads the model parameters and DRL state information, and the DRL agent receives the state s from the client. k and reward r k This information is then used for the next round of training. The goal of the DRL model is to maximize the expected cumulative discount reward R = ∑ k γ k-1 r k , where γ is a future discount factor between 0 and 1.

[0137] In this method, given state s k The DRL agent determines actions through a policy network; action a k Defined as the total training intensity τ of the clients participating in training in one round. k , which is the number of local training iterations performed on the client side in training round k, expressed by the formula:

[0138] a k =τ k ;

[0139] Where τ kThis represents the total training intensity in the k-th training round; the purpose of this part is to balance the accuracy of the global model and the consumption of training resources.

[0140] The deterministic algorithm allocates the training intensity to each participating client:

[0141] In the k-th round of training, the server randomly selects a set C of clients from all clients. k Participate in training. The total training intensity τ is output in the DRL model. k After that, it is necessary to provide the client with set C. k Each client in the process is assigned a specific training intensity. Given τ k When all client data is evenly distributed, the training loss and accuracy of the current training round can be guaranteed to a certain extent. In addition to shortening the training time and minimizing the budget used in the current training round, the following local training intensity allocation problem is defined:

[0142]

[0143] Where α′ is a hyperparameter used to balance the importance of budget consumption and training latency;

[0144] The constraints are as follows:

[0145]

[0146] Where C k B represents the set of clients selected by the server to participate in training in the k-th training round. k′ This represents the budget consumed in the k′-th training round. Represents a positive integer.

[0147] The goal of the above formula is to minimize the weighted sum of the training time and the budget used in the current training round, where α′ is a parameter used to balance the importance of these two objectives. In the constraints, the first two constraints state that the budget used should not exceed the remaining global budget, while the latter two constraints restrict the total training intensity of the client to be at least greater than 0, and the sum to equal the total training intensity, i.e., τ. k .

[0148] The specific steps of the deterministic algorithm are as follows:

[0149] In the k-th training round, the client set C k Selected to participate in training. The training time for the current training round is determined by C. k The slowest client is determined by enumerating all possible slowest clients. Specifically, C... k Each client c in i As the slowest client, enumerate the possible training intensities. With a fixed c... iAs the slowest client, the following formula must hold true:

[0150]

[0151] Training intensity is assigned to each client according to a greedy rule, where c i′ This represents any client participating in the training. Indicates client c i′ Training time, This indicates the training time for the slowest client.

[0152] Because of C k Each client should be allocated at least one local training iteration, therefore, the number of iterations allocated to c is... i The number of local training iterations should satisfy:

[0153]

[0154] In the range [1, τ] k -|C k Enumerate all possible possibilities within |+1] Value. After fixing Then, assume c i It is the slowest client, for C k Other clients c i′ It can be deduced that:

[0155]

[0156] in Indicates client c i′ The number of local training iterations in the k-th training round. Indicates the slowest client c i The computation time for one local training session. Indicates the slowest client c i Communication time, Indicates client c i′ Communication time, Indicates client c i′ The computation time for one local training session;

[0157] use to indicate The upper boundary.

[0158] In C k In the middle, it has been assumed If it is the maximum training time, then The values ​​of feasible solutions must satisfy the following inequalities:

[0159]

[0160]

[0161] Where τ k This represents the total training intensity in the k-th training round. Perform conditional checks on the above two inequalities.

[0162] If the above conditions cannot be met, it means that in C k \{c i In the given training intensity τ, at least one client is assigned less than 1 local training iteration, or the number of local training iterations that can be assigned to a client is not greater than the given total training intensity τ. k In this case, client c i There is no feasible solution for the assignment, therefore the next step is required. An attempt. {c i} indicates that it only includes client c i The set, \ indicates exclusion, C k \{c i} indicates that, except for the slowest client c i All clients participating in the training, except for those mentioned above.

[0163] if If the above conditions are met, it means that client c i Allocation It can be guaranteed that there is at least one feasible solution such that τ k It can be distributed to all clients participating in training, and it guarantees c i It is C k The slowest client. Allocate the remaining training intensity to C. k \{c i}, and use This indicates the remaining number of local training iterations allocated to the client.

[0164] Because of C k \{c i Each client in the} must be trained at least once, for Initial allocation Then we have the following formula:

[0165]

[0166] This indicates that the remaining training intensity is allocated using a greedy rule; based on the client's computational energy consumption, the intensity is allocated to client C. k \{c i Sort from smallest to largest, that is in Indicates that, except for client c i Other clients ci′ The computational energy consumed in a single local training session.

[0167] The remaining training intensity is allocated using a greedy rule. Consider C one by one k \{c i For example, for the l-th client If the remaining training intensity is still greater than Allocate the maximum training intensity to it, that is Otherwise, allocate the remaining training intensity values. The distribution formula is:

[0168]

[0169] Then update in Indicates the nth client The assigned training intensity.

[0170] Repeat the above allocation process until all remaining training intensity has been allocated or the budget has been exhausted.

[0171] In the range [1, τ] k -|C k Try possible within |+1] Value, find B k +α′T k The optimal solution for the minimum value.

[0172] Using the above method, try C k Each client in the algorithm acts as the slowest client, recording the case with the smallest objective function value. The algorithm flow is as follows: Figure 2 As shown.

[0173] S4. The client trains the model using local data and then transmits the updated training model parameters and deep reinforcement learning information to the server.

[0174] S5. After the server allocates the exact training intensity, the client performs training, updates the model parameters and uploads them to the server for aggregation, and simultaneously transmits the state information of deep reinforcement learning to the server.

[0175] The objective of DRL is to maximize the expected cumulative discount reward R = ∑ k γ k-1 r k γ is a discount factor for future rewards, ranging from 0 to 1. In the heterogeneous federated learning system mentioned in this invention, the DRL state that needs to be recorded is the loss function value and training time in the (k-1)th round. In addition, the remaining global budget also needs to be considered, which can be represented by the ratio of the used budget to the original global budget, i.e.:

[0176]

[0177] Where k′ represents the k′-th training round in federated learning, B k ′ represents the budget consumption of the k′-th training round;

[0178] Therefore, the state after the (k-1)th round of training is defined as:

[0179] s k-1 ={F k-1 Q k-1 T k-1 ,k-1};

[0180] Where F k-1 This represents the global model loss function value from the previous round.

[0181] After the FL system executes the algorithm, three aspects need to be considered: the model training loss value, the training time, and the budget used in the current training round. The reward function can be expressed as:

[0182] r k =α(F k-1 -F k )-β(Q k -Q k-1 )-T k ;

[0183] Where α > 0 and β > 0 are both positive constants, used to balance the loss function of the current training round, the proportion of the budget used, and the training time.

[0184] The server updates the deep reinforcement learning model based on historical information and aggregates the global model:

[0185] Deep reinforcement learning (DRL) models consist of a policy network and an evaluation network. The server updates and adjusts the DRL model after collecting a certain number of policy trajectories. Simultaneously, the server's global federated learning model aggregates parameter updates uploaded from the client using the following formula:

[0186]

[0187] Next, the server will determine whether the global budget has been exhausted or the model performance has met the requirements. If neither condition is met, the process will continue to the next training round. Otherwise, the training of the current reinforcement learning epoch will end, and the training of the next reinforcement learning epoch will begin, so that the reinforcement learning agent can collect enough policy trajectories for adjustment and updates.

[0188] For deep reinforcement learning (DRL) models, the external environment is a federated learning model. The DRL model needs to record the state information within this environment and update it after each training round. Feedback and adjustments are made through interaction with the federated learning environment. Throughout the system, client hardware and network conditions are heterogeneous, resulting in varying training latency and computational resource consumption. The system considers minimizing the global budget and latency during training and improving the convergence efficiency of the global model.

[0189] As a preferred embodiment, simulation experiments were conducted on three public real-world datasets: MNIST, Fashion-MNIST, and CIFAR-10. The MNIST and Fashion-MNIST (F-MNIST) datasets are two similar datasets, both consisting of a training set of 60,000 samples and a test set of 10,000 samples. The test set is divided into 10 classes, and the sample images consist of 28×28 grayscale pixels. The difference between the two datasets is that MNIST uses handwritten digits, while F-MNIST uses clothing. The CIFAR-10 dataset consists of a training set of 50,000 samples and a test set of 10,000 samples, also divided into 10 different classes, and the images consist of 32×32 RGB pixels.

[0190] For the MNIST and F-MNIST datasets, the CNN model was trained using two 5×5 convolutional layers. The first layer had 10 output channels, and the second layer had 20 output channels. Each layer was followed by a 2×2 max pooling layer, then two fully connected layers and an output layer with 10 units. The model for the F-MNIST dataset had an additional dropout layer between the two fully connected layers, and the learning rate was 0.005.

[0191] For the CIFAR dataset, the CNN model was trained using two 5×5 convolutional layers. The first layer had 6 output channels, and the second layer had 16 output channels. Each layer was followed by a 2×2 max pooling layer, then three fully connected layers and one output layer with 10 units. The learning rate of the model was set to 0.05.

[0192] As one embodiment, this embodiment sets the total number of clients N = 100, and each client has the same number of data samples. Similar to the traditional algorithm FedAvg, the central server randomly selects 10 clients to participate in training in each training round. The total training intensity range in the k-th training round is set to [1 × |C k |,5×|C k |]. For each client c iThe CPU loop frequency range is set to [1.0, 5.0] GHz, and the communication time for each client is randomly set within [0.5, 20] seconds. The number of CPU loops executed per bit is μ. i It is set to 40 times / second, and the effective capacitance coefficient of the chipset is set to 0.15.

[0193] For the hyperparameters related to DRL, the learning rates of the actor and critic networks were set to 0.00005, and the reward discount factor was set to γ ​​= 0.95.

[0194] As an example, this embodiment compares with two baseline algorithms, FedAvg and AMBLE, and refers to the optimization method provided by this invention as BFL-HTI with DRL. FedAvg is a classic algorithm proposed by Google, which allocates the same training intensity to each client in each round of communication; the AMBLE algorithm adaptively adjusts the local mini-batch training size and the number of local training iterations for heterogeneous clients. Figure 3 The results show the comparative effects of the present invention on model accuracy over training time. Figure 3 (a) illustrates that, on the MNIST dataset, the accuracy of the model of this invention increases faster than other models (FedAvg, AMBLE). Figure 3 (b) shows that, on the F-MNIST dataset, the accuracy of the model of this invention increases faster than other models. Figure 3 (c) shows that the model of the present invention has a significant advantage in accuracy and convergence speed on the CIFAR-10 dataset. Figure 4 This shows the comparative results of the present invention in terms of model accuracy as a function of budget consumption. Figure 4 (a) illustrates that, on the MNIST dataset, the model of this invention consumes the least amount of budget. Figure 4 (b) illustrates that, on the F-MNIST dataset, the model of this invention consumes the least amount of budget. Figure 4 (c) indicates that, on the CIFAR-10 dataset, the model budget consumption of this invention is significantly lower than that of other models. The results show that the differences in test accuracy among different algorithms are most significant in the early stages of training, and the optimization method proposed in this invention can accelerate model convergence and reduce training time, achieving the target accuracy with minimal global budget. This verifies the effectiveness and superiority of this invention and provides a feasible optimization scheme for practical heterogeneous federated learning systems.

[0195] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A resource-constrained heterogeneous federated learning optimization method, characterized in that, Includes the following steps: S1. Determine the parameter server task and initialize the global model parameters of federated learning and the deep reinforcement learning model; S2. A deep reinforcement learning model is used to learn and optimize task information, and a proximal policy optimization algorithm is used to train the deep reinforcement learning model. S3. The parameter server outputs the total training intensity of the participating clients through the deep reinforcement learning model, and allocates the training intensity of each participating client according to the deterministic algorithm. S4. The client trains the model using local data, updates the parameters of the trained local model and sends them to the parameter server, and also sends the information status of the deep reinforcement learning model to the parameter server. S5. The parameter server updates the deep reinforcement learning model based on historical information and aggregates the parameters uploaded by each client to perform a global model update. After the deep reinforcement learning model outputs the total training intensity for one round, it is distributed to each client using a deterministic algorithm. The optimization objective of the deterministic algorithm is: in, Indicates client c i The number of training iterations in the k-th training round, B k T represents the budget used to incentivize client participation in training during the k-th training round. k Let α be the training time for the kth round. ′ Hyperparameters representing the importance of balancing budget consumption and training latency; The constraints are as follows: Where C k B represents the set of clients selected by the server to participate in training in the k-th training round. k′ Indicates the k-th ′ The budget consumed per training round Represents positive integers. For client c i Compensation for computational resources consumed in one local training iteration during the k-th training round. This represents the overall budget for federal learning tasks; The deterministic algorithm is as follows: The set C of clients selected by the server to participate in training in the k-th training round k In the middle, each client c i As the slowest client, the number of local training iterations is set, and the training time of the slowest client satisfies the following: Where c i′ This represents any client participating in the training. Indicates client c i′ Training time, This represents the training time for the slowest client; Since each client participating in training should be allocated at least one local training attempt, it is assigned to the slowest client, c. i The number of local training iterations should meet the following requirements: in, Indicates client c i The number of training iterations in the k-th training round, τ k This represents the total training intensity in the k-th training round; Enumerate the number of local training iterations for the slowest client. value, The range is [1, τ] k -|C k |+1], in the selected Afterwards, the other participating clients satisfy the following: in Indicates client c i′ The number of local training iterations in the k-th training round. Indicates the slowest client c i The computation time for one local training session. Indicates the slowest client c i Communication time, Indicates client c i′ Communication time, Indicates client c i′ The computation time for one local training session; for The upper boundary.

2. The resource-constrained heterogeneous federated learning optimization method according to claim 1, characterized in that, In step S1, the client hardware settings for federated learning specifically include: Define a set of clients C = {c1, c2, ..., c...} i ,…,c N }, assuming the i-th client c i The computation time t for each local training iteration i,cmp for: in Indicates the i-th client c i CPU loop frequency in the k-th training round, μ i This represents the number of CPU loops required to execute 1 bit of training data, b i c represents the number of bits required for one local training iteration; N Indicates the Nth client; The client's communication time includes the time to download the global model and the time to upload and update the local model parameters. Considering only the upload time as the communication time, the client's communication time in the k-th training round is c. i Communication time for: Where ξ represents the size of the local model parameters on the client side. Indicates communication bandwidth; Client c i Training time in the k-th training round for: in, Indicates client c i The number of training iterations in the k-th training round determines the training time T of the k-th round. k for: Where C i This represents the set of clients selected by the server to participate in training during the k-th training round.

3. The resource-constrained heterogeneous federated learning optimization method according to claim 1, characterized in that, In step S1, the server training task for federated learning specifically includes: The server receives and updates the model parameters from the client, and performs aggregation. The client c... i The local model loss function value F i (w) is: in To represent client c i In local dataset D i The loss function for the j-th example, where w represents the parameters of the global model in federated learning, and the loss function of the global model is defined as: The training task objective is defined as follows: Where K represents the total number of rounds of federated learning training, w K F(w) represents the model parameters of the global model in the Kth round. K B represents the training loss function value of the global model in the Kth round. k T represents the budget used to incentivize client participation in training during the k-th training round. k This represents the training time for the k-th round; Constraints and limitations: in For client c i Compensation for computational resources consumed in one local training iteration during the k-th training round. This represents the overall budget for federal learning tasks. Represents positive integers. Indicates client c i The number of training iterations in the k-th training round, Client c i In the training time of the k-th training round, C k This represents the set of clients selected by the server to participate in training during the k-th training round.

4. The resource-constrained heterogeneous federated learning optimization method according to claim 1, characterized in that, In step S2, the near-end policy optimization algorithm collects policy π. k′ =π(θ) k′ The algorithm learns the advantage by taking a partial set of trajectories from θ and then using an advantage estimation algorithm, where θ k′ This represents the policy network of the deep reinforcement learning model at the k-th training stage. ′ The network parameters of step π k′ and π(θ) k′ ) indicates the k-th ′ The strategy of the step-policy network; The advantage estimation algorithm defines the difference between the expected cumulative reward of action a and the total expected cumulative reward: in, Let represent the attractiveness of action 'a'; 's' represent the current state of the deep reinforcement learning model; and π represent the pi. θ This represents the policy of the current policy network. This indicates that the policy network takes action a in state s, following policy π. θ The expected cumulative reward starting from the current moment. This indicates that the policy network, in state s, follows policy π. θ Expected cumulative reward at that time Represents the mathematical expectation; After obtaining the estimated advantage, calculate and update the strategy: Where θ k′+1 Let θ represent the policy network parameters at step k′+1, i.e., θ k′+1 , Let represent the loss function of the near-end policy optimization algorithm; where Defined as: in The model represents the current policy π. k′ In the generated trajectory distribution, the expected value of a certain function, where T represents the time from time 0 to the present, and r... t (θ) represents the importance sampling ratio. This represents the estimated value of the advantage estimation function, where ∈ is a hyperparameter used to control the magnitude of change between the old and new strategies.

5. The resource-constrained heterogeneous federated learning optimization method according to claim 1, characterized in that, The state, action, and reward information of the deep reinforcement learning model are as follows: State: The state of the deep reinforcement learning model needs to record the loss function value and training time of the (k-1)th round of training. It needs to consider the remaining global budget of the server. The state record shows the ratio of the used budget to the original global budget. k ′ In federated learning, the k-th... ′ One training round, B k′ Indicates the kth ′ Budget consumption per training round; This represents the overall budget for federal learning tasks; The state is defined as: s k-1 ={F k-1 ,Q k-1 ,T k-1 ,k-1}; Where s k-1 F represents the state in round (k-1). k-1 T represents the loss function value of the global model in the previous round; k-1 Q represents the training time for the (k-1)th round. k-1 This represents the ratio of the budget used in round (k-1) to the original global budget. Action: Given the current state, the action of the deep reinforcement learning model agent is determined by the policy network. The total training intensity 'a' of all clients in the current round is 'a'. k for: a k =t k ; Where τ k This represents the total training intensity in the k-th training round; Reward: Formula for the reward function: r k =α(F k-1 -F k )-β(Q k -Q k-1 )-T k ; Where α and β are constants, α>0 and β>0, T k Let r be the training time for the kth round. k This refers to rewards and feedback, used to guide the learning process and goal-oriented behavior of intelligent agents.

6. The resource-constrained heterogeneous federated learning optimization method according to claim 1, characterized in that, Client c i′ Number of local training iterations in the k-th training round The upper realm The following inequalities must be satisfied: Where τ k This represents the total training intensity in the k-th training round; like If the above two inequalities are not satisfied, it means that C k \{c i In the context of}, there exists a client that is allocated fewer than 1 local training iteration, or the number of local training iterations allocated to the client is insufficient to reduce τ. k After allocation, there is no feasible solution, and we need to proceed to the next step. The attempt; where {c i } indicates that it only includes client c i The set of C k \{c i } indicates that, except for the slowest client c i All clients participating in the training, except for those mentioned above; if If the above two inequalities are satisfied, it means that a feasible solution exists, and the remaining training intensity should be distributed to the remaining clients C participating in the training. k \{c i In the middle; through a greedy strategy, C k \{c i The clients are sorted in ascending order of computational resources consumed per training iteration. First, it is ensured that each client is allocated at least one training iteration, and the client at the top is allocated the largest number of training iterations. Simultaneously, the following conditions must be met: Repeatedly listed in the range [1, τ] k -|C k |+1] within And all clients participating in the training are designated as the slowest client c. i To obtain the smallest B k +α ′ T k This is the optimal solution for the objective.

7. A system for implementing the resource-constrained heterogeneous federated learning optimization method as described in claim 1, characterized in that, include: Heterogeneous device clients include smartphones, IoT devices, and edge devices. Each device performs local federated learning model training and shares model updates while ensuring data privacy. Training intensity allocation settings: At the beginning of each training round, the server allocates the number of local training iterations required for this round of training to each client participating in the training, based on the deep reinforcement learning model and deterministic algorithm. The deep reinforcement learning module includes storing the client's historical training information, updating the deep reinforcement learning model, outputting actions at the beginning of the training round, and converting them into the total training intensity of all participating clients in one round. Deterministic Algorithm Module: After the deep reinforcement learning model outputs the total training intensity, it allocates the training intensity to each client based on the client's training time and unit consumption budget, and calculates the budget consumption. Global update and aggregation module: After the client completes the training of the local model, it uploads the trained local model update and deep reinforcement learning state information to the server. The server will aggregate the global model and periodically update the policy network and value network of the deep reinforcement learning model.

8. A computer device, characterized in that, The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, in executing the computer program, implement a resource-constrained heterogeneous federated learning optimization method as described in any one of claims 1 to 6.

Images