A Multi-Agent Reinforcement Learning Policy Optimization Method Based on Intrinsic Reward Networks

Abstract

This invention discloses a multi-agent reinforcement learning policy optimization method based on an intrinsic reward network, belonging to the field of reinforcement learning technology. The method includes: constructing a reinforcement learning simulation scenario containing multiple agents and initializing model parameters, including decision network parameters, intrinsic reward network parameters, and state value network parameters; acquiring environmental rewards, global observation data, and local observation data for each agent, and obtaining the execution actions and intrinsic rewards for each agent; constructing loss functions for the decision network, intrinsic reward network, and state value network respectively, and optimizing the model parameters. This invention generates intrinsic rewards for each agent through a parameterized intrinsic reward network to incentivize agents to perform diverse actions, thereby improving the adaptability of each agent in a multi-agent system.

Description

Technical Field

[0001] This invention relates to the field of computer reinforcement learning, and more specifically to a multi-agent reinforcement learning strategy optimization method based on intrinsic reward networks. Background Technology

[0002] Many real-world problems, such as drones, traffic light control, autonomous driving, and resource management, can be modeled as cooperative multi-agent systems where the goal is to maximize the rewards achieved by the team of agents through cooperation. Multi-agent reinforcement learning holds great promise for handling such tasks.

[0003] The commonly used centralized training and distributed execution framework improves the agent's coordination ability by allowing the agent to access the global state during training. Many methods within the CTDE framework have studied how to enable the agent to learn the optimal policy. VDN and QMkX equivalent decomposition methods, as well as policy gradient algorithms such as MAPPO, have achieved good results on mainstream experimental platforms.

[0004] However, the monotonicity constraint required by value decomposition methods limits the expressive power of the network, and the joint action space grows exponentially with the number of agents. Furthermore, policy gradient algorithms require extensive sampling, leading to low training efficiency. Current algorithms often only utilize a uniform team reward provided by the environment, failing to consider the differences between agents. Recent algorithms have attempted to allocate rewards to agents using parameterized networks, but these efforts have not yielded satisfactory results.

[0005] Therefore, how to allocate personalized rewards to agents within an agent team is a question worth exploring. Summary of the Invention

[0006] In view of this, the present invention provides a multi-agent reinforcement learning policy optimization method based on intrinsic reward networks, which is used to at least solve some of the technical problems in the background art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] This invention discloses a multi-agent reinforcement learning policy optimization method based on intrinsic reward networks, comprising the following steps:

[0009] Model building and parameter initialization: Build a reinforcement learning simulation scenario with multiple agents and initialize the model parameters, including decision network parameters, intrinsic reward network parameters, and state value network parameters;

[0010] Data Acquisition: Acquire environmental rewards, global observation data, and local observation data for each agent from the environment of the reinforcement learning simulation scenario; input the local observation data into the decision network to output the action to be performed by each agent; input the local observation data and the action to be performed by each agent into the intrinsic reward network to output the intrinsic reward of the currently performed action; input the global observation data into the state value network to output the global value.

[0011] Construct the model's loss function and optimize the model parameters:

[0012] The loss of the decision network is calculated based on environmental rewards, the intrinsic reward of the current action, and global value. The parameters of the decision network are then optimized using a gradient descent algorithm so that the output of the decision network is close to the target output.

[0013] The loss of the state value network is calculated based on the environmental reward and the intrinsic reward of the current action, and the parameters of the state value network are optimized by the gradient descent algorithm so that the output of the state value network is close to the target output.

[0014] The loss of the intrinsic reward network is calculated based on the global value and the intrinsic reward of the current action, and the parameters of the intrinsic reward network are optimized by gradient descent algorithm so that the output of the intrinsic reward network is close to the target output.

[0015] Furthermore, the intelligent agent includes an autonomous vehicle, in which case,

[0016] Global observation data includes the location of all autonomous vehicles, pedestrians, obstacles, road and traffic light status information in the current reinforcement learning simulation scenario;

[0017] The local observation data for each agent includes: the position of each autonomous vehicle in the reinforcement learning simulation scenario, the planned driving trajectory, and the status information of other vehicles, pedestrians, obstacles, roads, and traffic lights that the autonomous vehicle can observe;

[0018] Each agent needs to perform actions including: acceleration, deceleration, steering, and stopping.

[0019] Furthermore, the intelligent agent includes traffic lights in a transportation network, where...

[0020] Global observation data includes all available data within the current transportation network;

[0021] The local observation data for each agent includes: the number, location, and route of vehicles and pedestrians around each traffic light;

[0022] Each agent needs to perform actions including switching between traffic lights and the corresponding light-on time.

[0023] Furthermore, the intelligent agent comprises each joint in the intelligent robot, at which point,

[0024] Global observation data includes all data that intelligent robots can perceive;

[0025] The local observation data for each intelligent agent includes: the orientation information that each joint of the intelligent robot can perceive;

[0026] The actions that each intelligent agent needs to perform include: the actions that each joint of the intelligent robot needs to perform.

[0027] Furthermore, the policy network consists of a multilayer perceptron module, an RNN module, and a classification distribution module connected in sequence.

[0028] Furthermore, the intrinsic reward network consists of a multilayer perceptron module, an RNN module, and a linear function approximator connected in sequence.

[0029] Furthermore, the state value network consists of a multilayer perceptron module, an RNN module, and a linear function approximator connected in sequence.

[0030] Furthermore, the loss function of the decision network is constructed based on environmental rewards, the intrinsic reward of the current action, and global value. Specifically, the loss function includes the following:

[0031] ;

[0032] ;

[0033] in, Decision network parameters that need updating; This represents the decision network parameters used for sampling; Indicates the number of samples collected; This represents the total number of agents in the environment; Number the sample; Number the intelligent agents; For intelligent agents Advantage estimates are calculated based on environmental rewards, intrinsic rewards, and generalized advantage estimates. This represents the ratio of the decision network outputs under different decision network parameters; and These are preset hyperparameters; Let be the clipping function, representing when Output ,when Output ; This represents the entropy used to calculate the approximate distribution. The probability distribution of each action in the action set output by the decision network; For decision network parameters The output at that time indicates when the decision network parameters are At that time, to observe history When taken as input, the decision network outputs the action to be executed. The probability of; For decision network parameters The output at that time indicates when the decision network parameters are At that time, to observe history When taken as input, the decision network outputs the action to be executed. The probability of.

[0034] Furthermore, the loss function of the state value network is constructed based on environmental rewards and the intrinsic reward of the current action, specifically including the following loss functions:

[0035] ;

[0036] ;

[0037] in, Represents the state-value network parameters; Indicates the number of samples collected; This represents the total number of agents in the environment; The agent's ID number; and These are the outputs of the value network at the current time step and the next time step, respectively. Let be the clipping function, representing when Output ,when Output ; It represents the sum of environmental rewards and the intrinsic rewards of the currently performed action; For environmental rewards; It is the intrinsic reward output by the intrinsic reward network. These are the preset hyperparameters.

[0038] Furthermore, a loss function for the intrinsic reward network is constructed based on the global value and the intrinsic reward of the current action, specifically including the following loss functions:

[0039] ;

[0040] in, These are intrinsic reward network parameters; Indicates the number of samples collected; This represents the total number of agents in the environment; Number the sample; Number the intelligent agents; These are preset hyperparameters; The intrinsic reward of agent k is the output of the intrinsic reward network; The intrinsic reward network parameters are: The intrinsic reward output by the intrinsic reward network; and These represent the parameters of the state-value network. The output of the value network for the current time step and the next time step. and These are the global observations of agent k at the current time step and the next time step, respectively. The variance function represents the computational agent's intrinsic reward.

[0041] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a multi-agent reinforcement learning policy optimization method based on an intrinsic reward network, which has the following beneficial effects:

[0042] This invention generates intrinsic rewards for each agent through a parameterized intrinsic reward network, which incentivizes the agent to perform diverse actions and improves the adaptability of each agent in a multi-agent system. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of the overall process framework of the multi-agent reinforcement learning strategy optimization method based on intrinsic reward networks provided by the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] This invention discloses a multi-agent reinforcement learning policy optimization method based on an intrinsic reward network, comprising the following steps:

[0047] Model building and parameter initialization: Build a reinforcement learning simulation scenario with multiple agents and initialize the model parameters, including decision network parameters, intrinsic reward network parameters, and state value network parameters;

[0048] Data Acquisition: Acquire environmental rewards, global observation data, and local observation data for each agent from the environment of the reinforcement learning simulation scenario; input the local observation data into the decision network to output the action to be performed by each agent; input the local observation data and the action to be performed by each agent into the intrinsic reward network to output the intrinsic reward of the currently performed action; input the global observation data into the state value network to output the global value.

[0049] Construct the model's loss function and optimize the model parameters:

[0050] The loss of the decision network is calculated based on environmental rewards, the intrinsic reward of the current action, and global value. The parameters of the decision network are then optimized using a gradient descent algorithm so that the output of the decision network is close to the target output.

[0051] The loss of the state value network is calculated based on the environmental reward and the intrinsic reward of the current action, and the parameters of the state value network are optimized by the gradient descent algorithm so that the output of the state value network is close to the target output.

[0052] The loss of the intrinsic reward network is calculated based on the global value and the intrinsic reward of the current action, and the parameters of the intrinsic reward network are optimized by gradient descent algorithm so that the output of the intrinsic reward network is close to the target output.

[0053] The overall flowchart of the multi-agent reinforcement learning policy optimization method based on intrinsic reward networks disclosed in this invention is shown below. Figure 1 In this invention, the policy network consists of a multilayer perceptron module, an RNN module, and a classification distribution module connected in sequence. The intrinsic reward network consists of a multilayer perceptron module, an RNN module, and a linear function approximator connected in sequence. The state value network consists of a multilayer perceptron module, an RNN module, and a linear function approximator connected in sequence.

[0054] In practice, the multilayer perceptron module in the neural network can be formed by connecting the Linear module, ReLU module and LayerNorm module provided by PyTorch, with the hidden layer set to 128; the RNN module is the GRU module provided by PyTorch; the classification distribution module is also provided by PyTorch.

[0055] In this invention, environmental rewards refer to feedback from the external environment, typically defined directly by the task objective. Environmental rewards are usually global, reflecting the performance of the entire system or the achievement of its objectives. Intrinsic rewards refer to rewards generated by each agent itself, used to encourage specific behaviors or exploratory actions. Intrinsic rewards are usually local, related to the learning process of a single agent, and can help agents better explore the environment, learn new skills, or improve existing strategies.

[0056] More specifically, the loss function of the decision network is constructed based on environmental rewards, the intrinsic reward of the current action, and global value. This loss function includes the following:

[0057] ;

[0058] ;

[0059] in, Decision network parameters that need updating; This represents the decision network parameters used for sampling; Indicates the number of samples collected; This represents the total number of agents in the environment; Number the sample; Number the intelligent agents; For intelligent agents Advantage estimates are calculated based on environmental rewards, intrinsic rewards, and generalized advantage estimates. This represents the ratio of the decision network outputs under different decision network parameters; and These are preset hyperparameters; Let be the clipping function, representing when Output ,when Output ; This represents the entropy used to calculate the approximate distribution. The probability distribution of each action in the action set output by the decision network; For decision network parameters The output at that time indicates when the decision network parameters are At that time, to observe history When taken as input, the decision network outputs the action to be executed. The probability of; For decision network parameters The output at that time indicates when the decision network parameters are At that time, to observe history When taken as input, the decision network outputs the action to be executed. The probability of.

[0060] In the above loss function, the formula for calculating the advantage estimator is as follows:

[0061] ;in, and Here are the preset hyperparameters, and t is the time step of the current sample. This represents the weighted sum of environmental rewards and the intrinsic rewards of the current action. and These are the first and second steps of the value network after the current time step. and Output at each time step.

[0062] The loss function of the state value network is constructed based on environmental rewards and the intrinsic reward of the current action, specifically including the following loss functions:

[0063] ;

[0064] ;

[0065] in, Represents the state-value network parameters; Indicates the number of samples collected; This represents the total number of agents in the environment; Number the sample; Number the intelligent agents; and These are the outputs of the value network at the current time step and the next time step, respectively. Let be the clipping function, representing when Output ,when Output ; This represents the weighted sum of environmental rewards and the intrinsic rewards of the currently performed action, used to represent the total reward received by agent k; For environmental rewards; It is the intrinsic reward output by the intrinsic reward network. and These are the preset hyperparameters.

[0066] The loss function of the intrinsic reward network is constructed based on the global value and the intrinsic reward of the current action, specifically including the following loss functions:

[0067] ;

[0068] in, These are intrinsic reward network parameters; Indicates the number of samples collected; This represents the total number of agents in the environment; Number the sample; Number the intelligent agents; These are preset hyperparameters; The intrinsic reward of agent k is the output of the intrinsic reward network; The intrinsic reward network parameters are: The intrinsic reward output by the intrinsic reward network; and These represent the parameters of the state-value network. The output of the value network for the current time step and the next time step. and These are the global observations of agent k at the current time step and the next time step, respectively. The variance function represents the computational agent's intrinsic reward.

[0069] In this invention, the loss function is the method of calculating loss. Gradient descent adjusts the network parameters to minimize the loss calculated according to the loss function. Loss function designers often use the difference between the network output and the target output as the loss, and minimize this difference to make the network output meet expectations.

[0070] Specifically, in practice, the data acquired from the environment and the output data of the neural network are first loaded into the data loader in batches of 400, so that the model parameters can be updated using gradient descent during training. Orthogonal initialization is used during neural network initialization. In the policy network loss, hyperparameters... Set to 0.01, hyperparameter Set to 0.2; in the intrinsic reward network loss, the hyperparameter is... Set to 0.5; in the state-value network loss, the hyperparameter is... Set it to 0.01; finally, fit the model parameters with a learning rate of 5e-4 and stochastic gradient descent until training stops after 10e7 time steps in the training set, and save the model's weight parameter file.

[0071] The multi-agent reinforcement learning strategy optimization method provided by this invention can be widely applied to all multi-agent scenarios that meet the conditions, so as to improve the adaptability of each agent in the multi-agent scenario. The specific application of this method is illustrated below with different embodiments.

[0072] Example 1

[0073] In this embodiment, a multi-agent reinforcement learning policy optimization method based on intrinsic reward networks is applied to an autonomous driving scenario. Here, the reinforcement learning simulation scenario includes an autonomous driving simulation scenario, with each agent representing each autonomous vehicle. In this scenario, global observation data includes the positions of all autonomous vehicles, pedestrians, obstacles, roads, and traffic light status information within the autonomous driving simulation scenario. Local observation data for each agent includes: the autonomous vehicle's own position in the reinforcement learning simulation scenario, its planned driving trajectory, and the status information of other vehicles, pedestrians, obstacles, roads, and traffic lights that the autonomous vehicle can observe. The actions each agent needs to perform include: acceleration, deceleration, steering, and stopping. The environmental reward for the autonomous driving scenario can be determined based on the arrival time of all vehicles at their destination, the optimization degree of path selection, and safety. The intrinsic reward can be the reward given when each autonomous vehicle performs a certain action.

[0074] The multi-agent reinforcement learning strategy optimization method disclosed in this invention can be used to optimize autonomous driving scenarios, thereby improving the adaptive capabilities of each autonomous vehicle in the autonomous driving scenario and enhancing the safety of autonomous driving vehicles.

[0075] Example 2

[0076] In this embodiment, a multi-agent reinforcement learning strategy optimization method based on intrinsic reward networks is applied to the field of intelligent traffic light control. Here, the agents are each traffic light in the traffic network, and the global observation data includes all available data in the current traffic network. Each agent's local observation data includes: the number, location, and route of vehicles and pedestrians around each traffic light; and the actions each agent needs to perform include: switching between traffic lights and the corresponding lighting time. The environmental reward for traffic light control is determined based on the overall smoothness of vehicle and pedestrian flow in the traffic network; the intrinsic reward is the reward given to each traffic light after it performs a certain action.

[0077] The multi-agent reinforcement learning strategy optimization method of this invention can be used to optimize traffic light scenarios, thereby improving the adaptive capability of traffic light control and thus improving the smoothness of traffic network flow.

[0078] Example 3

[0079] In this embodiment, a multi-agent reinforcement learning strategy optimization method based on intrinsic reward networks is applied to the motion domain of an intelligent robot. Here, the agent is each joint in the intelligent robot. Global observation data includes all data that the intelligent robot can perceive; local observation data for each agent includes: orientation information that each joint of the intelligent robot can perceive; and actions that each agent needs to perform, including: actions that each joint of the intelligent robot needs to perform, such as rotation and movement. Environmental rewards are determined based on indicators such as the overall smoothness of the robot's actions; intrinsic rewards are the rewards given to each joint of the intelligent robot after performing a certain action.

[0080] The multi-agent reinforcement learning strategy optimization method of the present invention can be used to optimize the motion behavior of intelligent robots, thereby improving the motion adaptability of each joint of the intelligent robot and thus improving the overall motion process and coordination of the intelligent robot.

[0081] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0082] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-agent reinforcement learning policy optimization method based on intrinsic reward networks, characterized in that, Includes the following steps: Model building and parameter initialization: Build a reinforcement learning simulation scenario containing multiple agents and initialize the model parameters, including decision network parameters, intrinsic reward network parameters, and state value network parameters; the agents include autonomous vehicles, traffic lights in a traffic network, or each joint in an intelligent robot; Data Acquisition: Acquire environmental rewards, global observation data, and local observation data for each agent from the environment of the reinforcement learning simulation scenario; input the local observation data into the decision network to output the action to be performed by each agent; input the local observation data and the action to be performed by each agent into the intrinsic reward network to output the intrinsic reward of the currently performed action; input the global observation data into the state value network to output the global value. Construct the model's loss function and optimize the model parameters: The loss of the decision network is calculated based on environmental rewards, the intrinsic reward of the current action, and global value. The parameters of the decision network are then optimized using a gradient descent algorithm so that the output of the decision network is close to the target output. The loss of the state value network is calculated based on the environmental reward and the intrinsic reward of the current action, and the parameters of the state value network are optimized by the gradient descent algorithm so that the output of the state value network is close to the target output. The loss of the intrinsic reward network is calculated based on the global value and the intrinsic reward of the current action, and the parameters of the intrinsic reward network are optimized by gradient descent algorithm so that the output of the intrinsic reward network is close to the target output.

2. The multi-agent reinforcement learning policy optimization method based on intrinsic reward networks according to claim 1, characterized in that, When the intelligent agent includes an autonomous vehicle... Global observation data includes the location of all autonomous vehicles, pedestrians, obstacles, road and traffic light status information in the current reinforcement learning simulation scenario; The local observation data for each agent includes: the position of each autonomous vehicle in the reinforcement learning simulation scenario, the planned driving trajectory, and the status information of other vehicles, pedestrians, obstacles, roads, and traffic lights that the autonomous vehicle can observe; Each agent needs to perform actions including: acceleration, deceleration, steering, and stopping.

3. The multi-agent reinforcement learning policy optimization method based on intrinsic reward networks according to claim 1, characterized in that, When the intelligent agent includes traffic lights in a transportation network. Global observation data includes all available data within the current transportation network; The local observation data for each agent includes: the number, location, and route of vehicles and pedestrians around each traffic light; Each agent needs to perform actions including switching between traffic lights and the corresponding light-on time.

4. The multi-agent reinforcement learning policy optimization method based on intrinsic reward networks according to claim 1, characterized in that, When the intelligent agent includes each joint in an intelligent robot... Global observation data includes all data that intelligent robots can perceive; The local observation data for each intelligent agent includes: the orientation information that each joint of the intelligent robot can perceive; The actions that each intelligent agent needs to perform include: the actions that each joint of the intelligent robot needs to perform.

5. The multi-agent reinforcement learning policy optimization method based on intrinsic reward networks according to claim 1, characterized in that, The decision network consists of a multilayer perceptron module, an RNN module, and a classification distribution module connected in sequence.

6. The multi-agent reinforcement learning policy optimization method based on intrinsic reward networks according to claim 1, characterized in that, The intrinsic reward network consists of a multilayer perceptron module, an RNN module, and a linear function approximator connected in sequence.

7. The multi-agent reinforcement learning policy optimization method based on intrinsic reward networks according to claim 1, characterized in that, The state value network consists of a multilayer perceptron module, an RNN module, and a linear function approximator connected in sequence.

8. The multi-agent reinforcement learning policy optimization method based on intrinsic reward networks according to claim 1, characterized in that, The loss function of the decision network is constructed based on environmental rewards, the intrinsic reward of the current action, and global value. Specifically, it includes the following loss functions: Where θ represents the decision network parameters that need to be updated; θ′ represents the decision network parameters used for sampling; B represents the number of samples; n represents the total number of agents in the environment; i is the sample number; k is the agent number; A k，i The advantage estimator for agent k is calculated based on environmental rewards, intrinsic rewards, and generalized advantage estimation; q k，i σ represents the ratio of the decision network output under different decision network parameters; σ and ∈ are preset hyperparameters; clip(q) k，i (1-∈, 1+∈) is the clipping function, representing the clipping function when q k，i <1-∈, output 1-∈, when q k，i >1+∈, output 1+∈; S[π η (τ k，i ] represents the entropy of the approximate distribution, π(τ) k，i π represents the probability distribution of each action in the action set output by the decision network; θ (a k，i |τ k，i ) represents the output of the decision network with decision network parameters θ, indicating the output based on observed history τ when the decision network parameters are θ. k，i When taken as input, the decision network outputs the action 'a'. k，i The probability of π θ′ (a k，i |τ k，i ) represents the output of the decision network with decision network parameters θ′, indicating the output of the observed history τ when the decision network parameters are θ′. k，i When taken as input, the decision network outputs the action 'a'. k，i The probability of.

9. The multi-agent reinforcement learning policy optimization method based on intrinsic reward networks according to claim 1, characterized in that, The loss function of the state value network is constructed based on environmental rewards and the intrinsic reward of the current action, specifically including the following loss functions: in, The parameters of the state value network are: B represents the number of samples; n represents the total number of agents in the environment; i is the sample number; k is the agent number; V and V' are the outputs of the state value network at the current time step and the next time step, respectively; clip(V, V'-ε, V'+ε) is the clipping function, which means that when V < 1-∈, the output is 1-∈, and when V > 1+∈, the output is 1+∈. This represents the weighted sum of environmental rewards and the intrinsic rewards of the currently performed action, used to represent the total reward received by agent k; For environmental rewards; λ is the intrinsic reward output by the intrinsic reward network, and ∈ are preset hyperparameters.

10. The multi-agent reinforcement learning policy optimization method based on intrinsic reward networks according to claim 1, characterized in that, The loss function of the intrinsic reward network is constructed based on the global value and the intrinsic reward of the current action, specifically including the following loss functions: Where η is the intrinsic reward network parameter; B represents the number of samples sampled; n represents the total number of agents in the environment; i is the sample number; k is the agent number; and δ is the preset hyperparameter. The intrinsic reward of agent k is the output of the intrinsic reward network. This represents the intrinsic reward output by the intrinsic reward network when the intrinsic reward network parameter is η; and These represent the parameters of the state-value network. The output of the value network for the current time step and the next time step, s k，i and s′ k，i represents the global observations of agent k at the current time step and the next time step, respectively; Std represents the variance function used to calculate the agent's intrinsic reward.

Images