Heterogeneous federated learning method based on reinforcement learning model and joint optimization algorithm

Abstract

This invention discloses a heterogeneous federated learning method based on reinforcement learning models and joint optimization algorithms. This method, used in scenarios with given heterogeneous data client pricing and computing power, calculates the contribution of client-side weights to the global model through reinforcement learning. Combined with a dynamic programming algorithm that considers both total time and contribution, it achieves shorter model training time and better final model performance. The method includes the following steps: training a machine learning model using federated learning; the client-side sends model weights back to the server; the server calculates the contribution of the client-side models to the global model using the reinforcement learning model, records the model's performance on the client-side test set and the training time of each model; then, based on a hierarchical dynamic programming algorithm, selects a client set that minimizes the objective function containing both total contribution and the longest training time.

Description

Technical Field

[0001] This invention belongs to the field of overhead and quality optimization in federated learning system design, specifically relating to a heterogeneous federated learning method based on reinforcement learning models and joint optimization algorithms. Background Technology

[0002] With the rapid development of the Internet of Things (IoT) and social networking sites, the number of smart devices has exploded in the past few years, generating an unprecedented amount of data every day. Traditionally, this data is usually offloaded to cloud servers for processing, but this centralized processing method faces many challenges: First, large-scale data transmission leads to a sharp increase in server load; second, under the constraint of limited network bandwidth, the solution of transmitting all data from local devices to remote servers for processing is impractical in real-world applications; more importantly, data privacy and security issues during data transmission are becoming increasingly prominent, making it difficult to meet the data privacy protection requirements of laws such as the Data Security Law and the Personal Information Protection Law.

[0003] Given these key factors, the paradigm for data storage and analytics is undergoing a significant shift from cloud-based centralized processing to distributed and on-device processing. The core enabling technology for this shift is edge computing, which provides computing resources at the network edge, enabling IoT devices to offload complex computing tasks or applications locally. Edge computing effectively addresses key issues such as insufficient computing power of IoT devices themselves, high latency when offloading to cloud servers, and significant data security risks.

[0004] Against the backdrop of the rapid development of edge computing and distributed machine learning, federated learning has emerged as an innovative distributed machine learning paradigm. Federated learning enables edge nodes or terminal devices to achieve privacy-preserving distributed learning by collaboratively training machine learning models without exposing the original data. A typical federated learning system employs a parameter server architecture, where terminal devices use local data to train models synchronized with the parameter server, uploading only model parameter updates rather than the original data, thus achieving collaborative learning while protecting data privacy.

[0005] Despite the many advantages of federated learning, it still faces several technical challenges in practical applications:

[0006] Data heterogeneity challenge: Data from edge devices often exhibits non-independent identically distributed (Non-IID) characteristics, and the data distribution of different devices may vary significantly, leading to model training bias and performance degradation.

[0007] Communication efficiency bottleneck: Federated learning requires frequent exchange of model parameters between devices and servers. When the number of participating devices is large or network conditions are poor, communication overhead becomes a major factor restricting training efficiency. Especially for highly dynamic real-time scenarios, such as real-time traffic prediction in intelligent transportation systems, communication latency can severely affect the system's usability.

[0008] Heterogeneity of computing resources: There are significant differences in computing power, storage space and power supply of edge devices. Some devices may become "lagging behind" in the training process due to limited resources, thus slowing down the overall training progress.

[0009] The trade-off between privacy and security: Although federated learning does not directly share raw data, model parameter updates can still potentially leak sensitive information. Existing privacy-enhancing techniques, such as homomorphic encryption and differential privacy, while improving security, also introduce additional computational overhead and model performance degradation.

[0010] Dynamic Environment Adaptability: In highly dynamic scenarios such as the Industrial Internet of Things (IIoT) and the Internet of Vehicles (IoV), devices may frequently join or leave the training process, and the network topology and link quality may change continuously. This places higher demands on the stability and adaptability of federated learning. The existing general federated learning standard implementation, FedAvg (Communication-Efficient Learning of Deep Networks), maintains a global objective machine learning model on a server and randomly selects a subset of clients (the proportion controlled by parameter C) to participate in training each round. Each client performs multiple rounds of stochastic gradient descent (SGD) on its local dataset, with parameters including the number of local epochs E and the local batch size B. After updates, the server performs a weighted average of the client models based on the amount of data. This approach provides a single dimension for evaluating data contribution and cannot accurately measure the client's contribution to the global model, resulting in poor performance in scenarios with heterogeneous client data (i.e., non-IID data distribution) and heterogeneous systems (differences in computing power and communication capabilities).

[0011] Subsequent work, the FedProx architecture [Federated Optimization in Heterogeneous Networks], added consideration to the aforementioned heterogeneity beyond regular training. Its approach was to add a regularization term measuring the similarity between the client-side training model and the global model in the target loss function, thereby constraining the offset of the client-side training relative to the global model. The drawback of this approach lies in the limitations of its analytical form: since the offset relative to the global model cannot accurately determine the quality of the global model itself, it performs poorly in scenarios with higher levels of data heterogeneity.

[0012] Furthermore, due to the practical challenges of multi-party participation and balancing communication costs and resource bidding in federated learning systems, some works have attempted to construct effective incentive and contribution calculation mechanisms based on mathematical models such as game theory. For example, [Joint Participation Incentive and Network Pricing] constructed a three-stage inductive equilibrium calculation based on Stackelberg game theory, achieving a system design that incentivizes participants and effectively reduces communication costs. However, its evaluation of client contributions relies on historical data rather than quantifiable client contributions, making it difficult to guarantee the final training effect of the model.

[0013] In summary, current research on federated learning largely focuses on the challenges of heterogeneous problems and the design of incentive mechanisms for multiple participants, with few works comprehensively considering both aspects and providing a feasible specific system design. In reality, the "fallback" phenomenon during training (caused by communication latency or insufficient computing power) significantly impacts the overall training progress of federated learning, especially in applications with high real-time requirements, where this issue is particularly critical. Therefore, there is an urgent need for a federated learning optimization method that comprehensively considers communication efficiency, computational resource allocation, and model performance to accelerate the training process and improve overall system efficiency, particularly in highly dynamic edge computing environments. Summary of the Invention

[0014] The main objective of this invention is to overcome the aforementioned deficiencies in the prior art by providing a heterogeneous federated learning method and system based on a reinforcement learning model and a joint optimization algorithm. Modeling of federated learning on heterogeneous datasets: The present invention is achieved by at least one of the following technical solutions.

[0015] The heterogeneous federated learning method based on reinforcement learning models and joint optimization algorithms includes the following steps: (1) In a federated learning system, the client trains the target model and sends the weights of the trained target model back to the server. (2) The server calculates the contribution value of the client-side target model training to the global target model training based on the client contribution comprehensive evaluation system, calculates the new round of global target model based on the contribution value, and records the classification effect of the new round of global target model on the test set of the server and the training time of each client visual classification model. (3) Based on the step-by-step dynamic programming algorithm, select a client set that minimizes the total training time and maximizes the total contribution.

[0016] Furthermore, in step (1), the target model is a reinforcement learning model. Training the reinforcement learning model involves embedding federated learning into the reinforcement learning environment for interaction, including: Configure federated learning training tasks: Initialize neural network model parameters, edge device model parameters, and server global model parameters; Design a federated learning reinforcement learning environment: Abstract the reinforcement learning environment, encapsulate the standard federated learning architecture as the interaction interface of the reinforcement learning environment, and define the quadruple data returned by the reinforcement learning environment at each state transition step to evaluate the decision effect. The quadruple data includes the state returned by the decision agent in reinforcement learning, the calculated value of the reinforcement learning reward function, the action taken, i.e. the selected optimal target client, and the state returned by the decision agent in the next step.

[0017] Furthermore, the reinforcement learning model is an Actor-Critic reinforcement learning model built on the agent's policy network and value network. The Critic model serves as a scoring model, generating corresponding scores and probabilities from the Actor model. Contribution factors are calculated according to a certain formula, and corresponding weights are generated based on the normalization of contribution factors.

[0018] Furthermore, the training of the Actor-Critic reinforcement learning model includes the following steps: 1) The Actor model outputs the selection probability of all clients based on the state of the previous round, and selects clients randomly or according to the selection probability output. 2) The Critic model takes the selected client and the state of the previous round as input and outputs the contribution score of the client combination under the current weight state. Based on the score of the client combination, the server adjusts the contribution of the client combination in the current round according to the Critic model after each decision and calculates the contribution value of the client after participating in the new round of training. 3) Perform a round of federated learning training on the clients selected in step 1), combine them to obtain a new set of weights, calculate the contribution factor based on the contribution value of this round and the probability of the corresponding client output by the Actor model, calculate the corresponding weight coefficient of the client weight after mapping, calculate the target model weight obtained by client aggregation, and calculate the new round of global target model weight, update the state accordingly, and obtain a new state. 4) Validate the target model weights obtained from client-side training on the test set to obtain the average accuracy on the test set. Based on the average accuracy of the previous round, obtain the accuracy improvement and calculate the reward for this round. Store the latest state, the old state of the previous round, the reward, and the selected client as a quadruple. Randomly sample from the stored quadruples and calculate the loss of the Actor model and the Critic model based on the quadruples. Then, use the loss of the Actor model and the loss of the Critic model for backpropagation to train the Actor model and the Critic model.

[0019] Furthermore, the corrected formula for step 2) is: ;

[0020] in and These are weighting coefficients. The mean, For the value assessment of the Critic model in round t, For the selected client, This is the value sequence value of the client in the previous round.

[0021] Furthermore, the comprehensive evaluation system for client contributions includes the model's training quality and the total training time; For a single federated learning session, the total training time ,in This represents the training time measured on the client side and the total communication time between the client and the server during a round of federated learning. , It is the time required for each client to send a task from the server to the new weight being sent back to the server; This represents a constraint metric, which is indirectly constrained by the training quality of the federated learning model.

[0022] Furthermore, the training quality of a certain client-side model can be expressed by the following formula:

[0023] in For the training quality of the model, This represents the probability output by the Actor model in this round. The contribution value of the client in this round, calculated by the server. , This is the scoring threshold, used to constrain the scoring range of the model. This refers to the client serial number selected in each round.

[0024] Furthermore, the step-by-step dynamic programming algorithm includes the following steps: Initialization: First, sort the clients in ascending order by their last participation time, then sort them in descending order by the quality contribution value of the clients maintained by the server; State transition: After sorting, fix the current client with the longest time consumption in each round, and perform state transition according to the state equation. Calculate the corresponding objective function value according to different client combinations, and record the maximum value and the selected client combination. Optimal solution extraction: The final solution is , ,in Let K be the state variable, and K be the total number of selected clients. For the server-side budget in each round, For the total budget, The maximum time consumed in the selected client combination. , The time penalty coefficient is used to find the client combination corresponding to the final solution, which is the final goal of this round of federated training.

[0025] A computer device according to the present invention includes a memory and a processor, the memory being electrically connected to the processor, the memory storing a computer program, which, when executed by the processor, causes the processor to implement the method described herein.

[0026] The present invention provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor implements the method described herein.

[0027] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention designs a scheme that does not use user data, but only uses user model weights to deduce the distribution of user data and then evaluate the contribution of user models, thereby accelerating federated learning to achieve usable overall time efficiency and improving the training quality of the final model while ensuring privacy and security.

[0028] (2) This invention is the first to fully utilize the characteristics of reinforcement learning models: it perceives based on different task model structures to generate targeted Actor-Critic models; and it is the first to use the output meaning of the Actor-Critic model as the basis for model contribution and measurement, thereby achieving full utilization of data. (3) The overall architecture proposed in this invention is the first to optimize the training quality and training time of the model based on the actual benefits of the training quality of the model learned by the reinforcement learning model. The optimal training quality and training speed of the model are achieved by constructing an optimization algorithm. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating a heterogeneous federated learning method based on a reinforcement learning model and a joint optimization algorithm, as an example.

[0030] Figure 2 This is a comparison chart of the accuracy of training with different federated learning baselines in the example.

[0031] Figure 3 This is a comparison chart showing the overall time taken to train different federated learning baselines in the implementation examples. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] This invention studies the scheduling optimization problem in federated learning and proposes a federated learning training optimization method based on reinforcement learning and optimization algorithms. Reinforcement learning is used to evaluate the stable data contribution across heterogeneous datasets, and considering contribution, cost, and time comprehensively, the overall efficiency and final model performance of the federated learning system can be improved by proactively identifying higher-quality edge devices. An Actor-Critic model is used to identify the training quality of the model for edge devices, and optimization algorithms are used to perform overall optimization of time and model training quality based on this quality.

[0034] like Figure 1 As shown in this embodiment, a heterogeneous federated learning method based on a reinforcement learning model and a joint optimization algorithm is used to contribute to the global model by means of reinforcement learning computational edge weights in a given scenario of heterogeneous data client pricing and computing power. Combined with a dynamic programming algorithm that considers both total time and contribution, it achieves shorter model training time and better final model performance. The method includes the following steps: S1. Design a reinforcement learning environment for federated learning decision agents: embed federated learning into the reinforcement learning environment for interaction, and build a reward model based on the test set prepared by the server.

[0035] Design the learning environment for the federated learning decision agent and train it, specifically including: Configure the federated learning training task: After determining the target model (e.g., a neural network model CNN for visual classification), initialize the model parameters; in round t, the client device... n The model parameters are The server's global model parameters are ; Design a federated learning reinforcement learning environment: Abstract the reinforcement learning environment env, encapsulate the standard federated learning architecture as an interaction interface, and define the four-tuple data returned by the env at each state transition (step) to evaluate the decision performance: ,in and It is the state returned by the reinforcement learning of the decision-making agent, which consists of the client state weights and the global state at time t: ; in Representative client The local model weights in round t, and This indicates the currently used global aggregation weight. The action space A is defined as the client index set {0,1,2,…,K}, where K is the given client size. Essentially, it executes topk filtering, exploration, or optimization strategies to select a subset of devices to participate in federated updates.

[0036] Reinforcement learning calculates rewards based on environmental feedback according to a pre-defined reward function. Defined as: , ,in In other words, accuracy refers to the classification accuracy of the aggregated global target model on the server-side test set. The natural base, The action to be taken is defined as the client index set {1,2,…,K}, which essentially involves executing a topk filtering or exploration strategy to select a subset of devices to participate in the federated update. Here, it refers to the optimal target client number selected. ,in The average accuracy rate. For the exponential term, when the accuracy increases by a certain amount... When growth is positive, the exponential effect amplifies its marginal returns non-linearly. Mathematical analysis shows that this function... When the first derivative is greater than 0, it is significantly greater than that of a linear function. This forces reinforcement learning agents to more actively seek leaps in accuracy, thereby alleviating the convergence lag problem caused by the slow increase in accuracy in traditional federated learning. This is the adjustment coefficient.

[0037] S2. For specific training tasks, perceive and design the corresponding agent's policy network and value network to construct an Actor-Critic reinforcement learning model; train the Actor-Critic reinforcement learning model to obtain the Critic model as a scoring model.

[0038] This embodiment designs an agent based on the Actor-Critic approach to automatically perceive the weight structure of a given CNN model and learn online based on the state returned by reinforcement learning. Specifically, it includes: For a given CNN model task: let We represent the general structural information of the model, Indicates client i The parameters of the target visual classification model are used. (Convolutional Layer, denoted by conv) represents the model weights of a specific convolutional layer in the object visual classification model, where... The structure adopts a general image processing structure, which is in the form of batch, channel, width, and height, represented as follows: The four-dimensional tensor, i.e. In this case, the state information in each round of reinforcement learning can be represented as: ,in, This is the model parameter information of the nth edge model after the t-th round of training. This is the global model parameter information aggregated after the client's selection is completed. Let's call it the set of weights (Selected Weights, i.e., Sw) from the previous round of model selection. The weighted aggregation function is Its aggregation weights are determined by the subsequent reinforcement learning agent, then:

[0039] ( , In this context, p and k represent the indices of the given weight in the client set, i.e., the p-th and k-th elements in the (t-1)-th round. The weight 'a' is defined as a continuous set of weights of the same size as S, mapped to the 0-1 interval. The corresponding score is generated by the Critic model in the Actor-Critic architecture, and the corresponding probability information, also known as the contribution factor, is calculated by the Actor model. The corresponding weight is generated based on the normalization of the contribution factor.

[0040] For a given Actor-Critic reinforcement learning model architecture: the model consists of an Actor model and a Critic model. Its structure is generated autonomously based on a given CNN model. A small test set is prepared on the server side. Let the Actor model be Ac and the Critic model be Cr. The structure generation function is func, and its structure generation can be described as follows:

[0041] Where We represents the weight structure information of the CNN model. For a CNN neural network model, one implementation is to perform PCA compression on the weights of the CNN convolutional layers, calculate the corresponding output dimension, and generate the corresponding Actor and Critic neural network models.

[0042] The standard training steps for a reinforcement learning round are as follows: (1) The state in the previous round was ,in The last item in the state is the weight aggregation result of the previous round, which is calculated by weighting the target model weights of the clients participating in the training based on the quality factor in the previous round. The Actor model is responsible for the transformation from state to action, based on the previous round's state. Given the input, output the selection probability of all clients. ,in It is a probability value mapped to the 0-1 interval, with the sum of Prob being 1. It is randomly selected based on the task exploration configuration, or the probabilities are taken from largest to smallest. The client is described as follows: The formula for selecting a client based on the output probability of the Actor network is:

[0043] Let be the set of client indices selected in round t. By default, the Actor model selects the k items with the highest output selection probability. At the same time, an ε-greedy strategy is introduced in this process (ε=0.4 in the experiment) to balance the needs of exploration and development. That is, with a probability of 0.4, the algorithm will randomly select the set of clients participating in training in this round from all participating clients to ensure that the algorithm does not get trapped in a local optimum too early.

[0044] (2) At the same time, the Critic model performs value assessment, that is, based on the state of the previous round. And the selected client (i.e., the action to be taken in this round). The input is the client combination under the current weight state. The contribution score is expressed as follows: ,

[0045] in Value, in this context, is the value assessment of the action taken by the Critic model in round t. The scoring threshold is used to constrain the scoring range of the model, allowing a certain amount of negative weights to be used to mark the corresponding combinations that bring negative global returns.

[0046] The server maintains a sequence of contribution values ​​for each client. The contribution of the client combination in each round is adjusted based on the Critic model after each decision, and the contribution value of the client after the new round of training is calculated. The contribution value is constrained to an interval ( To prevent gradient explosion, the value within m,m) is specified. Specifically, the server records a new mean for each client participating in the current training round. ,right , Where α and β are weighting coefficients, meaning the value estimate for this round is determined by a weighted average of the value from the previous round and the value from this round, and the value sequence items for other clients who did not participate in this round are also considered. .

[0047] (3) Perform a round of federated learning training on the clients selected in step (1) and combine them to obtain a new set of weights. According to the selected client And the calculated value of the corresponding client in this round The contribution factor is calculated based on the selected client. Based on the contribution value in this round and the probability of the corresponding client output by the Actor model. Calculated, that is The corresponding contribution factor After mapping, the corresponding weighting coefficients for the client weights are calculated. ,Right now: ,in The model weights for the j-th client in the t-th round are calculated as follows: The contribution factor for the corresponding client in each round. The target model weights obtained by training and aggregating them for the client. Calculate the global target model weights for the next round, based on the weights of the corresponding client-side model in this round. Update the state accordingly to obtain the new state. .

[0048] (4) To evaluate the performance of the target global model, a test dataset is reserved on the server side. This test dataset maintains the same sample distribution as the original dataset, or is a portion of the performance verification dataset used in a real-world scenario. The calculated... Validate on the test set on the server to obtain the average accuracy on the test set. According to the previous round The accuracy improvement was obtained. Calculate the reward for this round. Meanwhile, based on the score of the Critic model calculated in step (2) According to the new situation in this round Calculated based on the Critic model Construct the advantage function for each client selected in the previous round. ,in The depreciation factor is used to calculate the following: , ,in To enhance the time discount coefficient for rewards in learning scenarios, The actions of the aforementioned Critic model are scored, and actor_loss and critic_loss are the loss values ​​used to update the Actor model and Critic model, respectively.

[0049] S3. Based on the Critic scoring model as the model contribution, and based on the model contribution, training time of each client, and price cost of each client, a comprehensive evaluation system for client contribution is constructed, specifically including: Constructing a model evaluation system: The evaluation system for federated learning includes the model's training quality Q and the total training time T. In one round of federated learning, the training time and communication time of the edge models are calculated and denoted uniformly as follows: ,in This refers to the time required for each client to send a task from the server to the server receiving the new weights. For each round of training in federated learning, the time cost is determined by the client that finishes last. This is the expression for the time consumed in a single round. Therefore, the total time consumed in a complete federated learning training process can be expressed as:

[0050] Meanwhile, the total training time for federated learning is determined by the time cost of each training round. The total training time for a single federated learning session is related to the total number of epochs required to achieve the target accuracy on the test set. , The constraint metric can be indirectly constrained by the training quality of the federated learning model: selecting a higher quality client can speed up the model's progress to the target accuracy, i.e., reduce the number of training rounds.

[0051] Training quality factor of a client model The probability of the client producing a relatively high-quality model in the current training round t is jointly determined by the Critic model and the Actor model: The Critic model outputs the predicted final return for the selected client combination. Ultimately, following the aforementioned process, the feedback is sent to the historical value sequence maintained by the server on the client side. For a trained model, the training quality of the model... This can be expressed as a formula: , These are the client probabilities output by the Actor model and the action values ​​output by the Critic model, respectively. After training, based on the aforementioned global client value sequence maintained by the server side... The value corresponding to the client in this round is calculated using a weighted average, based on the selection probability of all clients. The client set is obtained by selecting from the top k items or through random exploration. Simultaneously calculate the sum of the contribution factors for that round of actions. .

[0052] S4. Based on the comprehensive client evaluation system, design a client selection algorithm that minimizes the total training time and maximizes the total training contribution, including the following steps: After completing the initial training steps of reinforcement learning, a stepwise dynamic programming algorithm based on a given evaluation metric is used to replace the original top-k client selection method to solve the metric optimization problem, including: Construct an index function to express the above quality as an optimization objective. For each round of federated learning, the following function needs to be maximized:

[0053] in It is the time penalty coefficient. This refers to the time cost of a single round of training.

[0054] Considering system construction, each client will bid on its own model in each round, denoted as . This fixes the server's budget for each round. Therefore, the above optimization objective is transformed into a budgeted optimization problem for each round: Each round selects a set of clients. This ensures that after the final training stops: Maximum, and guaranteed in each round .

[0055] Algorithm Design: Noting that with a fixed final time, the problem is actually transformed into an optimal planning problem with a budget, which can be solved using dynamic programming. State definition: Define state variables ), indicating that within the framework of this invention, in each round of federated learning training, in the previous Within each client, select a total budget not exceeding m And make the aforementioned quality optimization objectives The largest client combination The function value, and the corresponding state transition equation, are:

[0056] in For the aforementioned client data calculated based on the Actor and Critic models and weighted according to historical sequences, at the current round t, the data is maintained by the server. k quality factor, Offer a bid for it, since it has already been sorted. To estimate training time, here, ,in and Probability estimates from the Actor and Critic models.

[0057] Objective function: as described above The maximum value of the function after adding constraints can be restated in each round of the optimization problem as follows:

[0058] in The set of clients selected for the current round. This represents the time penalty coefficient.

[0059] Step-by-step dynamic programming algorithm steps: Initialization: The client is initialized based on the last participation time. Sort in ascending order, and above that, sort by... Sort in descending order; initialize boundary conditions, i.e. (Q=0 when there is no client).

[0060] State Transition: After sorting, in each round, the client with the longest current execution time is fixed, and state transitions are performed according to the aforementioned state equation, calculating its corresponding state transition. The function's value records the current state. Selected client combination.

[0061] Optimal solution extraction: The final solution is Where K is the total number of selected clients, To determine the specific budget and find the corresponding client combination, which is the ultimate goal of this round of participation in federated training.

[0062] Based on the above algorithm and overall reinforcement learning selection architecture, design the system startup scheme and model initialization method, and determine the relevant parameter settings and initialization scheme: For the reinforcement learning process described in step S2, taking a CNN network image classification task as an example, the initial weight initialization of the neural network uses random initialization. For the first round of initialization, all clients need to participate in training and record the training time for subsequent advancement. During training, a random exploration algorithm with a probability of 0.4 is used, allowing the Actor model to provide sufficient action space exploration. As an example, the end marker of a reinforcement learning episode is set to an accuracy of 0.95 on the test set, and the upper limit of epochs is set to 50. If the accuracy is not achieved within the expected number of rounds, the corresponding... Marked as negative.

[0063] For the specific inference process of the model described in step S4, the model needs to perform initial training in basic reinforcement learning rounds, usually 3-5 rounds, to avoid performance bias caused by excessive randomness in the initialization of the Actor-Critic model. After the initial training, the aforementioned optimal selection algorithm based on contribution evaluation is introduced to correct the reinforcement learning model for the optimization objective, thereby achieving overall optimization of the training process in subsequent rounds.

[0064] In one embodiment, federated learning is used to train a visual classification model (CNN model). Here, we take training a two-layer CNN neural network on the Cifar10 dataset for a classification task as an example: defined in a compact feature space. Above and a set of tags , Let be the total number of tags. This represents a specific labeled sample. ;make Represents the prediction function. Let represent a probability vector for each sample corresponding to a different label, where This represents the probability of selecting each label. The parameters of the prediction function, i.e., the parameters of the trained model; function Generate a probability vector for each sample, and the loss function of the prediction function. It can be defined as cross-entropy:

[0065] in To predict whether a sample belongs to the first... The probability of a class This represents the true probability distribution that the training objective needs to learn.

[0066] The learning problem is to solve the following optimization problem:

[0067] In federated learning, let there be a total of The first edge device; Each device has a data distribution following of This is a data sample; it is a sample from this device. The joint distribution. In each round In the process, several edge devices are selected, and each device downloads the current global model weight parameters from the server. And perform the following stochastic gradient descent (SGD) training locally:

[0068] in The number of sample categories in this classification dataset. Model parameters generated from local training on client devices. This represents the global model parameters sent by the server when the edge device starts local training. For learning rate, This represents the gradient produced by the loss function. Represented by cross-entropy , ( x ) is the predicted sample Belongs to the The probability of class label.

[0069] The specific execution process is as follows: Based on the task definition, select a target model with a two-layer CNN neural network (including convolutional layers and pooling layers), and then... The defined task-aware function generates a reinforcement learning Actor model and a Critic model for the target task based on a given structure. For a given CNN network, the task-aware function `func` is designed to collect the model weights of two convolutional layers and pooling layers. After PCA compression, the dimensions are calculated to generate two neural networks with corresponding dimensions, which become the Actor model and the Critic model, respectively. Due to the small size of the target network, it is randomly initialized, with 5 epochs of reinforcement learning initially. Then, a client-side optimization selection strategy based on contribution metrics is introduced, with a training limit of 50 epochs to obtain the results.

[0070] In the first round of training, all client devices participate in model training. The server maintains and records the time taken for each device from sending weights to returning results. In each round, the server sends a message to each client device k. , The model version to be sent (initially 0, incremented by 1 each time global model aggregation is executed). The server maintains and records the time taken from sending the model parameters to recycling them, and updates the data in each round.

[0071] The client device that receives the parameters trains the model parameters on a local dataset using the stochastic gradient descent (SGD) algorithm for at least a given intensity (usually 3-5 iterations), thus obtaining the locally trained model parameters. The formula for each iteration of training is: , This represents the gradient produced by the loss function with respect to the model parameters. The learning rate. Finally, The package is sent to the server, with parameters representing the parameters of the trained local model. The version of the global model on which this device training is based. For the first round of training, the global weights of the target model were randomly initialized. Due to the lack of knowledge about the client information, a full average aggregation was used on the global model in the first round, and a validation round was completed on a small test set on the server to record the current accuracy. The training time for the client is calculated, and a common contribution value is initialized for all clients.

[0072] After the first round, the model enters the standard reinforcement learning round. For a standard reinforcement learning training round, the server obtains the global state space based on the training of the model in the previous round and the current global model. :

[0073] in The weight passed from the client to the server for maintenance. This represents the current global weight after aggregation is complete.

[0074] The Actor model is based on the global state space. Give the probability of selecting the corresponding client in the current training round. An ε-greedy strategy is introduced (actually set ε=0.4). The server randomly selects K clients or... Select the top K clients with the highest probabilities (i.e., greedy strategy and topk strategy), as the client participating in the next round of training, the Critic model uses the state information given in this round. and action information Receive a score for the action. The score is then evenly distributed across all participating clients. The server updates the client-side scoring strategy, taking historical information into account and weighting the previous and current scores. , ,in and The coefficient represents the score of clients that did not participate in this round of training.

[0075] After receiving the task information from the server, each selected client completes a federated learning training cycle. The server receives the weights from the model and calculates the contribution factor of the selected client based on the outputs of the Actor and Critic models and the historical contribution values ​​maintained by the server. Based on contribution factors Perform weighted aggregation:

[0076] At this point, a new global model is obtained. Simultaneously, the global state is updated based on the newly trained edge model weights. The global model is validated on the server-side test dataset to obtain the reward for this round of modeling, and the reinforcement learning model is updated accordingly, first based on the average classification accuracy of this round. Average classification accuracy compared to the previous round Calculate the reward for this round: ,

[0077] Based on the scores calculated in the previous round by the Critic model Meanwhile, the Actor and Critic models adjust according to the new state of this round. Generate new states and new actions, and calculate And update the scores of all associated clients in this round. Based on the score differences, construct an advantage function for selecting each client relative to the previous round:

[0078] in To reinforce the time discount factor in learning, the Actor model loss (actor_loss) used to update the Actor model and the Critic model loss (critic_loss) used to update the Critic model are calculated accordingly:

[0079]

[0080] Then, backpropagation is used with actor_loss and critic_loss to update the corresponding Actor and Critic models.

[0081] After the standard reinforcement learning rounds are completed, the optimization policy round begins (until the model reaches the target accuracy or exceeds the preset number of rounds, at which point model training ends). The optimization client-selected policy is then used for reinforcement learning updates. The basic reinforcement learning process remains unchanged, but the top-k policy of the client selection is transformed into a dynamic programming algorithm based on the objective optimization function. In a round of federated learning, the training time and communication time of the edge models are calculated and denoted uniformly as follows: ,in This is the time required for each client k to send its task from the server to the server receiving the new weights. For each round of training in federated learning, the time cost is determined by the client that finishes last. After the initial round and the standard reinforcement learning round, the server maintains historical data on the client's training time. and value sequence Based on this, the objective function is constructed as follows:

[0082] Considering the client's bids in each round and a fixed budget per round, the client selection process based on maximum contribution and minimum latency is established as a constrained optimization problem:

[0083] Note that in each round If the state is fixed, the problem can be solved by dynamic programming, and the state transition equation is as follows:

[0084] Due to the existence of the corresponding equation, the decision problem is defined as a multi-round dynamic programming problem with multiple rounds of choices.

[0085] For a basic decision-making process, the first step is to analyze the client based on... Sort in ascending order, then based on the current round The algorithm sorts the clients in descending order and then starts from the beginning. For each selection round, given the number of participating clients, it fixes the slowest participating client (the last client in the current sequence due to the sorting). Then, it processes the clients within the fixed range for that selection round, obtaining the combination that maximizes the objective function according to the state transition equation. Finally, from all the optimal combinations across rounds, it selects the best combination as the action for that round, and then proceeds normally to the subsequent reinforcement learning process.

[0086] This operational example compares several widely used federated learning baselines, including the classic FedAVG, FAVOR, and FedProx. Some comparison results are as follows: Figure 2 , Figure 3As shown, FedWA is a specific implementation of the current embodiment. On the Cifar10 dataset, this embodiment achieves the same accuracy in the same rounds ( Figure 2 ) and overall time consumption ( Figure 3 The indicators have achieved better results.

[0087] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A heterogeneous federated learning method based on reinforcement learning models and joint optimization algorithms, characterized in that, Includes the following steps: (1) In a federated learning system, the client trains the target model and sends the weights of the trained target model back to the server. (2) The server calculates the contribution value of the client-side target model training to the global target model training based on the client contribution comprehensive evaluation system, calculates the new round of global target model based on the contribution value, and records the classification effect of the new round of global target model on the test set of the server and the training time of each client visual classification model. (3) Based on the step-by-step dynamic programming algorithm, select a client set that minimizes the total training time and maximizes the total contribution.

2. The heterogeneous federated learning method based on reinforcement learning model and joint optimization algorithm according to claim 1, characterized in that, In step (1), the target model is a reinforcement learning model. Training the reinforcement learning model involves embedding federated learning into the reinforcement learning environment for interaction, including: Configure federated learning training tasks: Initialize neural network model parameters, edge device model parameters, and server global model parameters; Design a federated learning reinforcement learning environment: Abstract the reinforcement learning environment, encapsulate the standard federated learning architecture as the interaction interface of the reinforcement learning environment, and define the quadruple data returned by the reinforcement learning environment at each state transition step to evaluate the decision effect. The quadruple data includes the state returned by the decision agent in reinforcement learning, the calculated value of the reinforcement learning reward function, the action taken, i.e. the selected optimal target client, and the state returned by the decision agent in the next step.

3. The heterogeneous federated learning method based on reinforcement learning model and joint optimization algorithm according to claim 1, characterized in that, The reinforcement learning model is an Actor-Critic reinforcement learning model built on the agent's policy network and value network. The Critic model serves as a scoring model, generating corresponding scores and probabilities from the Actor model. Contribution factors are calculated according to a certain formula, and corresponding weights are generated based on the normalization of contribution factors.

4. The heterogeneous federated learning method based on reinforcement learning model and joint optimization algorithm according to claim 3, characterized in that, The training of the Actor-Critic reinforcement learning model includes the following steps: 1) The Actor model outputs the selection probability of all clients based on the state of the previous round, and selects clients randomly or according to the selection probability output. 2) The Critic model takes the selected client and the state of the previous round as input and outputs the contribution score of the client combination under the current weight state. Based on the score of the client combination, the server adjusts the contribution of the client combination in the current round according to the Critic model after each decision and calculates the contribution value of the client after participating in the new round of training. 3) Perform a round of federated learning training on the clients selected in step 1), combine them to obtain a new set of weights, calculate the contribution factor based on the contribution value of this round and the probability of the corresponding client output by the Actor model, calculate the corresponding weight coefficient of the client weight after mapping, calculate the target model weight obtained by client aggregation, and calculate the new round of global target model weight, update the state accordingly, and obtain a new state. 4) Validate the target model weights obtained from client-side training on the test set to obtain the average accuracy on the test set. Based on the average accuracy of the previous round, obtain the accuracy improvement and calculate the reward for this round. Store the latest state, the old state of the previous round, the reward, and the selected client as a quadruple. Randomly sample from the stored quadruples and calculate the loss of the Actor model and the Critic model based on the quadruples. Then, use the loss of the Actor model and the loss of the Critic model for backpropagation to train the Actor model and the Critic model.

5. The heterogeneous federated learning method based on reinforcement learning model and joint optimization algorithm according to claim 4, characterized in that, The corrected formula for step 2) is: ； in and These are weighting coefficients. The mean, For the value assessment of the Critic model in round t, For the selected client, This is the value sequence value of the client in the previous round.

6. The heterogeneous federated learning method based on reinforcement learning model and joint optimization algorithm according to claim 4, characterized in that, The comprehensive evaluation system for client contributions includes the model's training quality and the total training time; For a single federated learning session, the total training time ,in This represents the training time measured on the client side and the total communication time between the client and the server during a round of federated learning. , It is the time required for each client to send a task from the server to the new weight being sent back to the server; This represents a constraint metric, which is indirectly constrained by the training quality of the federated learning model.

7. The heterogeneous federated learning method based on reinforcement learning model and joint optimization algorithm according to claim 6, characterized in that, The training quality of a client-side model can be expressed by the following formula: in For the training quality of the model, This represents the probability output by the Actor model in this round. The contribution value of the client in this round calculated by the server. , This is the scoring threshold, used to constrain the scoring range of the model. This refers to the client serial number selected in each round.

8. The heterogeneous federated learning method based on reinforcement learning model and joint optimization algorithm according to claim 1, characterized in that, The step-by-step dynamic programming algorithm includes the following steps: Initialization: First, sort the clients in ascending order by their last participation time, then sort them in descending order by the quality contribution value of the clients maintained by the server; State transition: After sorting, fix the current client with the longest time consumption in each round, and perform state transition according to the state equation. Calculate the corresponding objective function value according to different client combinations, and record the maximum value and the selected client combination. Optimal solution extraction: The final solution is , ,in Let K be the state variable, and K be the total number of selected clients. For the server-side budget in each round, For the total budget, The maximum time consumed in the selected client combination. , The time penalty coefficient is used to find the client combination corresponding to the final solution, which is the final goal of this round of federated training.

9. A computer device comprising a memory and a processor, the memory being electrically connected to the processor, the memory storing a computer program, characterized in that: When the computer program is executed by the processor, it causes the processor to implement the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor implements the method as described in any one of claims 1 to 8.

Images