Federal reinforcement learning method and system based on hessian assistance and policy gradient in heterogeneous environment
By introducing Hessian assistance and policy gradient into federated reinforcement learning, the problems of client drift and policy gradient fluctuation in heterogeneous environments are solved, achieving more stable training and higher global policy performance.
Patent Information
- Application Number
- CN202610555560.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-24
- Publication Date
- 2026-07-28
AI Technical Summary
In heterogeneous environments, federated reinforcement learning suffers from problems such as significant client drift, large fluctuations in policy gradient estimation, and insufficient global training stability.
The federated reinforcement learning method using Hessian-assisted correction and policy gradient is adopted. The server initializes the global model parameters and control variables, the client performs parameter interpolation and dual-trajectory sampling, constructs Hessian-assisted correction terms and heterogeneous correction gradient estimates, performs local updates, and uploads the parameter update and control variable update. The server aggregates the data and iteratively outputs the global policy model.
It reduces client drift caused by heterogeneous environments, decreases policy gradient estimation fluctuations, improves training stability and global policy performance, and enhances sample utilization efficiency and convergence speed.
Smart Images
Figure CN122472133A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the technical fields of federated learning, reinforcement learning and distributed intelligent decision-making, and mainly relates to a federated reinforcement learning method and system based on Hessian assistance and policy gradient in heterogeneous environments. Background Technology
[0002] Reinforcement learning learns decision-making strategies through continuous interaction between intelligent agents and their environment, and it has broad application prospects in tasks such as robot control, resource scheduling, autonomous driving, and network optimization. Traditional reinforcement learning mostly employs centralized training methods, requiring the unified collection of interaction data before model updates. For problems involving multi-terminal, multi-scenario, or multi-institutional collaboration, this approach often suffers from high data aggregation costs, large communication overhead, and difficulties in privacy protection.
[0003] Federated learning offers a feasible approach to the aforementioned problems, where each participant retains its data and environment locally, uploading only model update information, which is then centrally aggregated by a server. Introducing federated learning into reinforcement learning can, to some extent, avoid the problem of centralized storage of original interactive data, thus possessing significant application value.
[0004] However, federated reinforcement learning differs significantly from traditional federated supervised learning. Clients in reinforcement learning typically need to sample trajectories independently in their respective environments, and these environments are often inconsistent. For example, the state transition patterns, reward functions, initial state distributions, and trajectory distributions of different clients may differ, and these differences are directly reflected in the direction of the local policy gradient. After multiple steps of local training, the client's parameter updates are prone to deviating from the more consistent global update direction, resulting in significant client drift. This makes simple averaging aggregation on the server side difficult to achieve ideal results.
[0005] On the other hand, policy gradient methods rely on trajectory sampling to estimate gradients. Due to the strong randomness of trajectory rewards, first-order policy gradient estimation typically suffers from large variance and significant fluctuations in update direction. When such methods are directly applied to heterogeneous federated scenarios, local sampling noise and environmental differences can overlap, easily leading to instability in the training process, decreased convergence speed, and even repeated fluctuations in global policy performance. Summary of the Invention
[0006] This invention addresses the problems of significant client drift, large fluctuations in policy gradient estimation, and insufficient global training stability in existing federated reinforcement learning technologies for heterogeneous environments. It provides a federated reinforcement learning method and system based on Hessian assistance and policy gradient in heterogeneous environments, comprising a server and multiple clients. The server initializes global model parameters, global gradient updates, and global control variables and broadcasts them to the clients. Each client uses its current global model parameters as its local update starting point, obtaining the main trajectory and the Hessian-estimated auxiliary trajectory through parameter interpolation and dual-trajectory sampling. A Hessian assistance correction term is constructed at the interpolated parameters and combined with a control variable correction term to form a local gradient estimate. After multiple local updates, the client uploads the parameter and control variable updates. The server aggregates the uploaded results from each client, updates the global model parameters and global control variables, and outputs the global policy model parameters after iterative iteration, completing the federated reinforcement learning process. This invention, without sharing the original environment data and original interaction trajectories, can reduce client drift caused by heterogeneous environments, decrease policy gradient estimation fluctuations, and improve the training stability and global policy performance of federated reinforcement learning.
[0007] To achieve the above objectives, the technical solution adopted by the present invention is: a federated reinforcement learning system based on Hessian assistance and policy gradient in a heterogeneous environment, including a unique number of servers and a non-unique number of clients. Each client interacts with its own local reinforcement learning environment, and the server is responsible for broadcasting global parameters and aggregating the results uploaded by the clients.
[0008] The server initializes global model parameters, global gradient update amount and global control variables and broadcasts the current information data to the clients. After each client completes local update and uploads information data, the server aggregates the information data uploaded by each client, updates the global model parameters and global control variables, and outputs the global policy model parameters after iterative looping to complete federated reinforcement learning.
[0009] The client takes the current global model parameters as the starting point for local updates, obtains the main trajectory and Hessian estimated auxiliary trajectory through parameter interpolation and dual trajectory sampling, constructs Hessian auxiliary correction terms at the interpolation parameters, and forms local gradient estimates by combining control variable correction terms. After completing the local update, it uploads the parameter update amount and control variable update amount to the server.
[0010] To achieve the above objectives, the present invention also adopts the following technical solution: a federated reinforcement learning method based on Hessian assistance and policy gradient in heterogeneous environments, comprising the following steps:
[0011] S1: The server initializes global model parameters, global gradient update amounts, and global control variables; each client initializes its local control variables.
[0012] S2: At the beginning of each round of communication, the server broadcasts the current global model parameters, global gradient update amount, and global control variables to the client;
[0013] S3: The client uses the current global model parameters as the starting point for local updates, and obtains the main trajectory and the Hessian estimated auxiliary trajectory through parameter interpolation and dual trajectory sampling; the main trajectory is obtained by the client through local parameter sampling, and the Hessian estimated auxiliary trajectory is obtained by the client through parameter interpolation and mixing.
[0014] S4: Based on the Hessian estimated auxiliary trajectory obtained in step S3, the client constructs a Hessian auxiliary correction term, calculates the policy gradient estimate based on the main trajectory, and constructs a heterogeneous correction gradient estimate in conjunction with the control variables to complete the local policy update; the Hessian auxiliary correction term... Specifically:
[0015]
[0016] in, This indicates the offset of the current local parameter relative to the global parameter in the previous round. This represents the Hessian matrix estimation function. Indicates the interpolation mixing parameters, This indicates that the client uses interpolation parameters. Hessian-estimated auxiliary trajectory obtained through interaction with the environment;
[0017] S5: After the client updates its local policy in step S4, it updates the local control variables and calculates the parameter update amount and control variable update amount, and then uploads them to the server.
[0018] S6: The server aggregates the results uploaded by each client, updates the global model parameters and global control variables, and outputs the global policy model parameters after reaching the preset communication rounds or meeting the preset convergence conditions, thus completing the federated reinforcement learning.
[0019] As an improvement to this invention, the server can choose to have all clients participate or only select a portion of clients participate in each round of training. When only a portion of clients participate in training, the aggregation formula... This indicates the number of clients that actually participated in the aggregation in this round.
[0020] Compared with the prior art, the present invention has the following beneficial effects:
[0021] (1) The present invention introduces a second-order Hessian auxiliary correction term in the local update stage, so that the parameter update not only depends on the first-order gradient, but also can sense the local curvature change near the parameter path, thereby reducing the training oscillation caused by the high variance of the policy gradient.
[0022] (2) By setting global control variables and local control variables and adding heterogeneous correction terms in local gradient estimation, this invention can suppress systematic shifts between different environments and alleviate client drift problems.
[0023] (3) This invention unifies momentum update, dual trajectory sampling, Hessian-assisted estimation and control variable correction in the same federated reinforcement learning framework, which can improve training stability, sample utilization efficiency and global policy performance. Attached Figure Description
[0024] Figure 1 This is a flowchart of the steps of the method of the present invention;
[0025] Figure 2 This is a flowchart illustrating the local update process on the client side in the system of this invention.
[0026] Figure 3 This is a schematic diagram illustrating the interaction between the server and multiple heterogeneous clients in the system of this invention;
[0027] Figure 4 This is a schematic diagram illustrating the combined operation of Hessen-assisted correction and control variable correction in the method of the present invention. Detailed Implementation
[0028] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.
[0029] Example 1
[0030] A federated reinforcement learning system based on Hessian assistance and policy gradient in a heterogeneous environment includes a unique number of servers and a non-unique number of clients. Each client interacts with its own local reinforcement learning environment. The server is responsible for broadcasting global parameters and aggregating the results uploaded by the clients. In each training round, the server can choose to have all clients participate or only select some clients to participate.
[0031] The server initializes global model parameters, global gradient update amount and global control variables and broadcasts the current information data to the clients. After each client completes local update and uploads information data, the server aggregates the information data uploaded by each client, updates the global model parameters and global control variables, and outputs the global policy model parameters after iterative looping to complete federated reinforcement learning.
[0032] The client: such as Figure 2As shown, the local update starts with the current global model parameters. The main trajectory and Hessian estimated auxiliary trajectory are obtained through parameter interpolation and dual trajectory sampling. Hessian auxiliary correction terms are constructed at the interpolation parameters. Combined with the control variable correction terms, a local gradient estimate is formed. After the local update is completed, the parameter update and control variable update are uploaded to the server.
[0033] like Figure 3 As shown, the local reinforcement learning environments of each client exhibit heterogeneity, which is manifested in at least one or more differences in state transition patterns, reward mechanisms, initial state distributions, or trajectory distributions. Due to these differences, each client often updates its parameters in different directions during local training, leading to client drift during the federated aggregation process. The client's local policy update is based on the cumulative discounted reward. That is, after obtaining trajectory samples through local environment interaction, the client adjusts its policy parameters according to the impact of actions at each time step in the trajectory on subsequent rewards. The server does not directly receive the trajectory samples themselves, but indirectly approximates the global policy improvement direction through the update amounts uploaded by each client.
[0034] Federated reinforcement learning methods based on Hessian assistance and policy gradients implemented using the above system in heterogeneous environments, such as Figure 1 As shown, it includes the following steps:
[0035] Step S1: The server initializes global model parameters, global gradient update amounts, and global control variables, while each client initializes its local control variables.
[0036] Server initializes global model parameters Global gradient update amount and global control variables Each client initializes its local control variables. The server initializes global model parameters using a neural network, trains the initialized model to obtain the global gradient update amount needed for the first update, and initializes control variables to a zero vector for both the server and each client.
[0037] Step S2: At the beginning of each round of communication, the server broadcasts the current global model parameters, global gradient update amount, and global control variables to the client.
[0038] Training requires a total of iterations In each iteration, the server encrypts the current global model parameters and global control variables and sends them to each client via the network at the beginning of each iteration, so that each client can sample and update the parameters.
[0039] Step S3: The client uses the current global model parameters as the starting point for local updates, and obtains the main trajectory and Hessian estimated auxiliary trajectory through parameter interpolation and dual trajectory sampling.
[0040] Regarding the first During the round, the client uses the current local parameters Compared with the previous round of global parameters Interpolation coefficients between samples Construct interpolation blending parameters:
[0041]
[0042] This interpolation parameter is located between the previous global position and the current local position, and is used to characterize the local structural information as the parameter changes along the update path.
[0043] Subsequently, the client utilizes the current local parameters respectively and interpolation mixing parameters Sampling main trajectory Heissen estimated auxiliary trajectory The main trajectory is used to calculate the first-order policy gradient, while the Hessian-estimated auxiliary trajectory is used to estimate the local curvature correction. By sampling the two types of trajectories separately, the accumulated bias caused by a single trajectory simultaneously undertaking multiple estimation tasks can be reduced. Step S4: As... Figure 4 As shown, the client constructs the Hessian auxiliary correction term based on the Hessian estimated auxiliary trajectory, calculates the policy gradient estimate based on the main trajectory, and constructs the heterogeneous correction gradient estimate in combination with the control variables to complete the local multi-step policy update.
[0044] Client in interpolation mixing parameters Constructing Hessian auxiliary correction term
[0045]
[0046] In the formula, The Hessian matrix estimation function represents the offset of the current local parameters relative to the global parameters of the previous round. The Hessian matrix is used to characterize the local curvature change in the offset direction, representing the policy performance. As the local update gradually deviates from the global parameters, the Hessian auxiliary correction term compensates for the change in update direction, thus preventing local training from relying solely on the first-order gradient for further offsetting. The Hessian matrix estimation function is expressed as follows:
[0047]
[0048] in,
[0049]
[0050] In the formula, A policy performance metric used to characterize the cumulative reward along the trajectory, which links the log probability of the action at each time step to the corresponding discounted reward in the subsequent time step.
[0051] The Hessian matrix estimation function utilizes second-order information to describe the local variation trend during parameter updates in greater detail. Simultaneously, the Hessian auxiliary correction term is implemented using the Hessian vector product method to reduce the storage and computational overhead of directly constructing the complete Hessian matrix. The client calculates the strategy gradient estimate based on the main trajectory.
[0052]
[0053] This formula indicates that the gradient contribution at each moment in the trajectory is weighted by its subsequent cumulative discounted returns, thus reflecting the degree of influence of the current action on subsequent returns. Indicates the state Select action The probability of policy parameters The gradient of is the core term in the policy gradient method.
[0054] In step S4, in addition to policy gradient estimation and Hessian-assisted correction terms, the global control variables maintained on the server side are utilized. Local control variables maintained by the client Construct heterogeneous correction gradient estimates together:
[0055]
[0056] in, The momentum coefficient, For the gradient estimation of heterogeneous correction in the previous inner loop, the initialization at the beginning of each inner loop is the global update direction of the previous round. , This is a supplementary correction item for Heisenberg. This is the gradient estimation of the policy corresponding to the current main trajectory. This is a heterogeneity correction term.
[0057] The first half of the heterogeneous correction gradient estimation described above preserves the historical global direction and introduces local curvature correction, while the second half uses control variables to suppress local update offsets caused by environmental heterogeneity. Compared with relying solely on local gradients or simply performing federated averaging, the heterogeneous correction gradient estimation in this invention is more conducive to improving the consistency between local training and global aggregation, thereby further improving training stability.
[0058] Furthermore, the client performs a local model update according to the following formula:
[0059]
[0060] in, This is the local learning rate.
[0061] Step S5: After the client completes the local update, it updates the local control variables and calculates the parameter update amount and control variable update amount, and then uploads them to the server.
[0062] Client completed After the first local update, update the local control variables according to the following formula.
[0063]
[0064] This formula uses the local first-order gradient estimation of the global parameters in this round to describe the current local gradient of the client, aiming to grasp the current local gradient shift of the client.
[0065] Then calculate
[0066]
[0067] in, This represents the amount of parameter updates made by the client in this round. This is the update amount for local control variables.
[0068] The client only needs to upload the above update results, without uploading the original state, actions, reward sequence or complete interaction trajectory. Therefore, the method of the present invention can avoid sharing the original environment data and the original interaction trajectory, which can reduce client drift caused by heterogeneous environment, reduce policy gradient estimation fluctuation, protect client privacy, and reduce communication costs.
[0069] Step S6: The server aggregates the results uploaded by each client, updates the global model parameters and global control variables, and outputs the global strategy model parameters after reaching the preset communication rounds or meeting the preset convergence conditions.
[0070] The server aggregates the upload results from each client and obtains...
[0071]
[0072] Update global model parameters and global control variables
[0073]
[0074] in, This is the global aggregation step size.
[0075] The above aggregation method utilizes the parameter offsets of all clients to correct the global strategy, and simultaneously updates the global control variables, enabling the next round of local training to obtain a new calibration benchmark. When repeating... Training stops once the preset convergence condition is met, and the final parameters are output. .
[0076] In summary, the method proposed in this invention can train reinforcement learning models in heterogeneous environments with a smaller sample size and lower communication costs, utilizing second-order information and control variables, thereby improving training stability and convergence rate.
[0077] Test case
[0078] This test case applies to a distributed multi-agent reinforcement learning scenario. The system consists of one server and... It consists of several clients. Each client corresponds to an independent intelligent agent environment. Data between clients is isolated, and no raw data is exchanged; communication is only with the server regarding model updates. Each client corresponds to a similar but heterogeneously distributed reinforcement learning environment, such as distributed robot navigation or autonomous driving scenarios.
[0079] This test case uses the Python deep learning framework PyTorch to build a parameterized deep neural network and uses it to initialize the server's global policy model parameters. , At the same time, set global hyperparameters, such as the global number of iteration rounds. Local iteration count Local learning rate Global learning rate Momentum coefficient Discount Factor Etc. Specifically, the hyperparameter configuration for this test case is as follows: global iteration rounds. Local iteration count Local learning rate A dynamic decay strategy is adopted, with an initial value of 3e-4 and a global decay coefficient of 0.99 in each iteration; the global learning rate is... The initial value is 1e-3, and the global iteration decay coefficient is 0.995 per round; momentum coefficient Fixed at 0.1; Discount factor trajectory length The global policy model employs a 3-layer fully connected neural network structure: the input layer dimension is consistent with the environment state space dimension; hidden layer 1 has a dimension of 64 and uses the ReLU activation function; hidden layer 2 has a dimension of 64 and uses the ReLU activation function; the output layer dimension is consistent with the environment action space dimension, using either the Softmax activation function / discrete action space or the Tanh activation function / continuous action space to output the probability distribution of actions or continuous action values.
[0080] At the start of each iteration, the server sends the global parameters and global control variables for that round to each client. Each client independently receives the global information and completes multi-step iterative updates locally according to steps S3 and S4 in Example 1. After each iteration, the difference is returned to the server. This completes the process. The optimal model parameters are output after the first global iteration.
[0081] This test case evaluates the practical performance of the proposed method against the reinforcement learning benchmark LunarLander lunar landing mission. For this benchmark, a problem is considered solved if the reward value is greater than 180 for 10 consecutive rounds. This test case compares the method with three baseline methods: FedPG-BR, SVRPG, and GPOMDP. The experiment was independently repeated 5 times with different random seeds and policy initialization conditions. All methods shared these conditions to ensure fair comparison, and the results are based on the average of the 5 independent runs.
[0082] This test case uses the final convergence reward and the average reward as evaluation criteria. The final convergence reward is the average reward over the last 10 rounds of the experiment. A higher final convergence reward indicates better method performance. The average reward is the average of the rewards throughout the experiment. A higher average reward indicates faster convergence and higher sample efficiency. The results of this test case are shown in the table below:
[0083]
[0084] As can be seen from the table above, the algorithm of this invention outperforms the baseline algorithm in both final convergence reward value and average reward value. This indicates that the algorithm of this invention has both a faster convergence rate and higher final performance. Therefore, the algorithm of this invention is superior to the baseline algorithm in heterogeneous environments.
[0085] In summary, the client of this invention obtains the main trajectory and the Hessian estimated auxiliary trajectory through parameter interpolation and dual-trajectory sampling during the local update phase. A Hessian auxiliary correction term is introduced at the interpolation parameters to mitigate training oscillations caused by the high variance of the policy gradient. By adding a heterogeneous correction term to the local gradient estimation, systematic shifts between different environments can be suppressed. This invention, without sharing the original environment data and original interaction trajectories, can reduce client drift caused by heterogeneous environments, decrease policy gradient estimation fluctuations, and improve training stability, sample utilization efficiency, and global policy performance.
[0086] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.
Claims
1. A federated reinforcement learning system based on Hessian assistance and policy gradient in a heterogeneous environment, characterized by: This includes a unique number of servers and a non-unique number of clients. The server initializes global model parameters, global gradient update amount and global control variables and broadcasts the current information data to the clients. After each client completes local update and uploads information data, the server aggregates the information data uploaded by each client, updates the global model parameters and global control variables, and outputs the global policy model parameters after iterative looping to complete federated reinforcement learning. The client takes the current global model parameters as the starting point for local updates, obtains the main trajectory and Hessian estimated auxiliary trajectory through parameter interpolation and dual trajectory sampling, constructs Hessian auxiliary correction terms at the interpolation parameters, and forms local gradient estimates by combining control variable correction terms. After completing the local update, it uploads the parameter update amount and control variable update amount to the server.
2. A federated reinforcement learning method based on Hessian assistance and policy gradient in heterogeneous environments, using the system as described in claim 1, characterized in that, Includes the following steps: S1: The server initializes global model parameters, global gradient update amounts, and global control variables; each client initializes its local control variables. S2: At the beginning of each round of communication, the server broadcasts the current global model parameters, global gradient update amount, and global control variables to the client; S3: The client uses the current global model parameters as the starting point for local updates, and obtains the main trajectory and the Hessian estimated auxiliary trajectory through parameter interpolation and dual trajectory sampling; the main trajectory is obtained by the client through local parameter sampling, and the Hessian estimated auxiliary trajectory is obtained by the client through parameter interpolation and mixing. S4: The client constructs the Hessian auxiliary correction term based on the Hessian estimated auxiliary trajectory obtained in step S3, calculates the policy gradient estimate based on the main trajectory, and constructs the heterogeneous correction gradient estimate in combination with the control variables to complete the local policy update. The Hessen auxiliary correction item Specifically: ； in, This indicates the offset of the current local parameter relative to the global parameter in the previous round. This represents the Hessian matrix estimation function. Indicates the interpolation mixing parameters, Indicates the interpolation parameters Hessian-estimated auxiliary trajectory obtained through interaction with the environment; S5: After the client updates its local policy in step S4, it updates the local control variables and calculates the parameter update amount and control variable update amount, and then uploads them to the server. S6: The server aggregates the results uploaded by each client, updates the global model parameters and global control variables, and outputs the global policy model parameters after reaching the preset communication rounds or meeting the preset convergence conditions, thus completing the federated reinforcement learning.
3. The federated reinforcement learning method based on Hessian assistance and policy gradient in heterogeneous environments as described in claim 2, characterized in that: In step S1, the server initializes the global model parameters using the neural network model, trains the initialized neural network model, and obtains the global gradient update amount required for the first update. The initialization control variables of the server and each client are zero vectors.
4. The federated reinforcement learning method based on Hessian assistance and policy gradient in heterogeneous environments as described in claim 2, characterized in that: The interpolation mixing parameters in step S3 are specifically as follows: In the In round-robin communication, the client In the In the next inner loop, the current local parameters are used. Compared with the previous round of global parameters Sampling interpolation coefficients Construct interpolation hybrid parameters : ； in , , ,in Indicates the number of communication rounds. Indicates the number of agents. This represents the number of times the system loops within the intelligent system. The client utilizes current local parameters. Generate trajectories through interaction with the environment: ； Each step of the action From local parameters Parameterized strategy Sampled, The current state of the environment. The reward is the feedback from the environment after the agent interacts with the environment. Indicates the number of interactions. Indicates the trajectory length; the client uses interpolation mixing parameters. Sampling to obtain Hessen's estimated auxiliary trajectory .
5. The federated reinforcement learning method based on Hessian assistance and policy gradient in heterogeneous environments as described in claim 4, characterized in that: The Hessian matrix estimation function in step S4 Specifically: ； in, Representing the trajectory With parameters The probability distribution under the strategy, The policy performance metric representing the cumulative reward along the trajectory: ； in, Indicates by parameters Decision strategy In state Choose below The probability, Indicates a single-step instant reward. This represents the discount factor.
6. The federated reinforcement learning method based on Hessian assistance and policy gradient in heterogeneous environments as described in claim 5, characterized in that: In step S4, the client calculates the policy gradient estimate based on the main trajectory: ； in, Indicates the state Select action The probability of policy parameters The gradient.
7. The federated reinforcement learning method based on Hessian assistance and policy gradient in heterogeneous environments as described in claim 6, characterized in that: In step S4, based on the global control variables maintained on the server side... Local control variables maintained by the client To jointly construct heterogeneous correction gradient estimates : ； in, The momentum coefficient, For the gradient estimation of heterogeneous correction in the previous inner loop, the initialization at the beginning of each inner loop is the global update direction of the previous round. , This is a supplementary correction item for Heisenberg. This is the gradient estimation of the policy corresponding to the current main trajectory. This is a heterogeneous correction term; Client based on Perform local model updates; where, This is the local learning rate.
8. The federated reinforcement learning method based on Hessian assistance and policy gradient in heterogeneous environments as described in claim 7, characterized in that: In step S5, the client updates the local control variables specifically as follows: ； in, Indicates the number of local updates. This indicates the use of global parameters in this round. and the main trajectory sampled in the first round of the client's internal loop. Calculate the policy gradient; Client-side calculation of parameter update volume Specifically: ； in, This represents the amount of parameter updates made by the client in this round. This is the update amount for local control variables.
9. The federated reinforcement learning method based on Hessian assistance and policy gradient in heterogeneous environments as described in claim 6, characterized in that: In step S6, the server aggregates the upload results from each client: ； in, The gradients required for updating the global model parameters in this round of server updates. The difference needed to update the global control variables for this round of server updates; The server updates global model parameters and global control variables: ； in, This is the global aggregation step size; For the updated global model parameters, The server will broadcast the updated global control variables in the next round. and To all clients; When repeat Training stops after each round, and the final parameters are output. , thus completing reinforcement learning.