Multi-agent reinforcement learning strategy optimization method based on internal reward network

Abstract

The invention discloses a multi-agent reinforcement learning strategy optimization method based on an internal reward network, and belongs to the technical field of reinforcement learning, and the method comprises the steps: building a reinforcement learning simulation scene containing a plurality of agents, and carrying out the initialization of model parameters, the model parameters comprise decision network parameters, internal reward network parameters and state value network parameters; obtaining an environment reward, global observation data and local observation data of each agent, and obtaining an execution action and an internal reward of each agent; and respectively constructing loss functions of the decision network, the internal reward network and the state value network, and optimizing model parameters. According to the method, the internal rewards are generated for each agent through the parameterized internal reward network and are used for exciting the agents to execute diversified actions, and the adaptive capacity of each agent in the multi-agent system is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer reinforcement learning, and more specifically to a multi-agent reinforcement learning strategy optimization method based on an intrinsic reward network. Background Art

[0002] Many real-world problems, such as drones, traffic light control, autonomous driving, and resource management, can be modeled as a cooperative multi-agent system, where the goal is to maximize the reward obtained by the team of agents through the cooperation of a group of agents. Multi-agent reinforcement learning has great potential in handling such tasks.

[0003] The commonly used centralized training and decentralized execution framework improves the coordination ability of agents by allowing them to access global states during training. Many methods study how to enable agents to learn optimal strategies under the framework of CTDE. VDN and QMkX equivalent decomposition methods and policy gradient algorithms such as MAPPO have achieved good results on mainstream experimental platforms.

[0004] However, the monotonicity constraints that the value decomposition method needs to meet limit the network's expressive power, and the joint action space will also grow exponentially with the increase in the number of agents, while the policy gradient algorithm requires more sampling, resulting in low training efficiency. In addition, current algorithms can often only use the unified team reward given by the environment, and do not take into account the differences between agents. Some recent algorithms have tried to use parameterized networks to assign rewards to agents, but have not achieved good results.

[0005] Therefore, how to assign personalized rewards to agents in an agent team is a question worth exploring. Summary of the invention

[0006] In view of this, the present invention provides a multi-agent reinforcement learning strategy optimization method based on an intrinsic reward network, which is used to at least solve some of the technical problems in the background technology.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] The present invention discloses a multi-agent reinforcement learning strategy optimization method based on an intrinsic reward network, comprising the following steps:

[0009] Model building and parameter initialization: Building a reinforcement learning simulation scenario containing multiple agents and initializing model parameters, including decision network parameters, intrinsic reward network parameters, and state value network parameters;

[0010] Data acquisition: Obtain environmental rewards, global observation data, and local observation data of each agent from the environment of the reinforcement learning simulation scenario; input the local observation data into the decision network, and output the action that each agent needs to perform; input the local observation data and the action that each agent needs to perform into the intrinsic reward network, and output the intrinsic reward of the current action; input the global observation data into the state value network, and output the global value;

[0011] Construct the loss function of the model and optimize the model parameters:

[0012] The loss of the decision network is calculated based on the environmental reward, the intrinsic reward of the current action, and the global value, and the decision network parameters are optimized through the gradient descent algorithm so that the output of the decision network is close to the target output;

[0013] The loss of the state value network is calculated based on the environmental reward and the intrinsic reward of the current action, and the state value network parameters are optimized through the gradient descent algorithm so that the output of the state value network is close to the target output;

[0014] The loss of the intrinsic reward network is calculated based on the global value and the intrinsic reward of the current action, and the parameters of the intrinsic reward network are optimized through the gradient descent algorithm so that the output of the intrinsic reward network is close to the target output.

[0015] Furthermore, the intelligent agent includes an autonomous driving vehicle. In this case,

[0016] Global observation data includes the position of all autonomous vehicles, pedestrians, obstacles, roads, and traffic light status information in the current reinforcement learning simulation scenario;

[0017] The local observation data of each agent includes: the position of each autonomous vehicle in the reinforcement learning simulation scenario, the planned driving trajectory, and the status information of other vehicles, pedestrians, obstacles, roads, and traffic lights that the autonomous vehicle can observe;

[0018] The actions that each agent needs to perform include: acceleration, deceleration, turning, and stopping.

[0019] Furthermore, the intelligent agent includes a traffic light in a traffic network.

[0020] Global observation data include all data available in the current transportation network;

[0021] The local observation data of each agent includes: the number, location and forward route of vehicles and pedestrians around each traffic light;

[0022] The actions that each agent needs to perform include: switching between red and green lights and the corresponding lighting time.

[0023] Furthermore, the intelligent body includes each joint in the intelligent robot. In this case,

[0024] Global observation data includes all data that the intelligent robot can perceive;

[0025] The local observation data of each intelligent agent includes: the orientation information that each joint of the intelligent robot can perceive;

[0026] The actions that each intelligent agent needs to perform include: the actions that each joint of the intelligent robot needs to perform.

[0027] Furthermore, the strategy network is composed of a multi-layer perceptron module, an RNN module and a classification distribution module which are connected in sequence.

[0028] Furthermore, the intrinsic reward network is composed of a multi-layer perceptron module, an RNN module and a linear function approximator connected in sequence.

[0029] Furthermore, the state value network is composed of a multi-layer perceptron module, an RNN module and a linear function approximator connected in sequence.

[0030] Furthermore, the loss function of the decision network is constructed based on the environmental reward, the intrinsic reward of the current action, and the global value, including the following loss functions:

[0031] ;

[0032] ;

[0033] in, Decision network parameters that need to be updated; Represents the decision network parameters used for sampling; Indicates the number of samples taken; Represents the total number of agents in the environment; Number the sample; Number the agents; For intelligent agents Advantage estimates calculated from environmental rewards, intrinsic rewards, and generalized advantage estimates; Represents the ratio of decision network outputs under different decision network parameters; and are the preset hyperparameters; is the clipping function, which means when , output ,when , output ; represents the entropy of the calculated approximate distribution, The probability distribution of each action in the action set output by the decision network; The decision network parameters are The output when , means that when the decision network parameters are When observing history As input, the decision network outputs the execution action probability; The decision network parameters are The output when , means that when the decision network parameters are When observing history As input, the decision network outputs the execution action probability.

[0034] Furthermore, the loss function of the state value network is constructed based on the environmental reward and the intrinsic reward of the current action, including the following loss functions:

[0035] ;

[0036] ;

[0037] in, Represents the state value network parameters; Indicates the number of samples taken; Represents the total number of agents in the environment; is the number of the agent; and are the outputs of the state value network for the current time step and the next time step respectively; is the clipping function, which means when , output ,when , output ; Represents the sum of the environmental reward and the intrinsic reward of the current action; Rewards for the environment; is the intrinsic reward output by the intrinsic reward network, are the preset hyperparameters.

[0038] Furthermore, the loss function of the intrinsic reward network is constructed based on the global value and the intrinsic reward of the current action, including the following loss functions:

[0039] ;

[0040] in, are the intrinsic reward network parameters; Indicates the number of samples sampled; Represents the total number of agents in the environment; Number the sample; Number the agents; are the preset hyperparameters; The intrinsic reward of agent k output by the intrinsic reward network; Denote the intrinsic reward network parameters as The intrinsic reward output by the intrinsic reward network; and Respectively represent the network parameters in the state value The output of the state value network for the current time step and the next time step, and are the global observations of agent k at the current time step and the next time step respectively; Represents the variance function for computing the agent's intrinsic reward.

[0041] It can be seen from the above technical solution that, compared with the prior art, the present invention discloses a multi-agent reinforcement learning strategy optimization method based on an intrinsic reward network, which has the following beneficial effects:

[0042] The present invention generates intrinsic rewards for each agent through a parameterized intrinsic reward network to motivate the agent to perform diverse actions and improve the adaptability of each agent in the multi-agent system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0044] Figure 1 Schematic diagram of the overall process framework of the multi-agent reinforcement learning strategy optimization method based on the intrinsic reward network provided by the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] The embodiment of the present invention discloses a multi-agent reinforcement learning strategy optimization method based on an intrinsic reward network, comprising the following steps:

[0047] Model building and parameter initialization: Building a reinforcement learning simulation scenario containing multiple agents and initializing model parameters, including decision network parameters, intrinsic reward network parameters, and state value network parameters;

[0048] Data acquisition: Obtain environmental rewards, global observation data, and local observation data of each agent from the environment of the reinforcement learning simulation scenario; input the local observation data into the decision network, and output the action that each agent needs to perform; input the local observation data and the action that each agent needs to perform into the intrinsic reward network, and output the intrinsic reward of the current action; input the global observation data into the state value network, and output the global value;

[0049] Construct the loss function of the model and optimize the model parameters:

[0050] The loss of the decision network is calculated based on the environmental reward, the intrinsic reward of the current action, and the global value, and the decision network parameters are optimized through the gradient descent algorithm so that the output of the decision network is close to the target output;

[0051] The loss of the state value network is calculated based on the environmental reward and the intrinsic reward of the current action, and the state value network parameters are optimized through the gradient descent algorithm so that the output of the state value network is close to the target output;

[0052] The loss of the intrinsic reward network is calculated based on the global value and the intrinsic reward of the current action, and the parameters of the intrinsic reward network are optimized through the gradient descent algorithm so that the output of the intrinsic reward network is close to the target output.

[0053] The overall flow chart of the multi-agent reinforcement learning strategy optimization method based on intrinsic reward network disclosed in the present invention is referenced Figure 1 In the present invention, the policy network is composed of a multi-layer perceptron module, an RNN module, and a classification distribution module connected in sequence. The intrinsic reward network is composed of a multi-layer perceptron module, an RNN module, and a linear function approximator connected in sequence. The state value network is composed of a multi-layer perceptron module, an RNN module, and a linear function approximator connected in sequence.

[0054] In the specific operation process, the multilayer perceptron module in the neural network can be connected by the Linear module, ReLU module and LayerNorm module provided by PyTorch, and the hidden layer is set to 128; the RNN module is the GRU module provided by PyTorch; the classification distribution module is also provided by PyTorch.

[0055] In this invention, environmental rewards refer to feedback from the external environment, which is usually directly defined by the task objectives. Environmental rewards are usually global and reflect the performance of the entire system or the completion of the goal. Intrinsic rewards refer to rewards generated by each agent itself, which are used to encourage certain specific behaviors or exploratory behaviors. Intrinsic rewards are usually local and related to the learning process of a single agent. Intrinsic rewards can help agents better explore the environment, learn new skills or improve existing strategies.

[0056] More specifically, the loss function of the decision network is constructed based on the environmental reward, the intrinsic reward of the current action, and the global value, including the following loss functions:

[0057] ;

[0058] ;

[0059] in, Decision network parameters that need to be updated; Represents the decision network parameters used for sampling; Indicates the number of samples taken; Represents the total number of agents in the environment; Number the sample; Number the agents; For intelligent agents Advantage estimates calculated from environmental rewards, intrinsic rewards, and generalized advantage estimates; Represents the ratio of decision network outputs under different decision network parameters; and are the preset hyperparameters; is the clipping function, which means when , output ,when , output ; represents the entropy of the calculated approximate distribution, The probability distribution of each action in the action set output by the decision network; The decision network parameters are The output when , means that when the decision network parameters are When observing history As input, the decision network outputs the execution action probability; The decision network parameters are The output when , means that when the decision network parameters are When observing history As input, the decision network outputs the execution action probability.

[0060] In the above loss function, the calculation formula of the advantage estimator is as follows:

[0061] ;in, and is the preset hyperparameter, t is the time step of the current sample, represents the weighted sum of the environmental reward and the intrinsic reward of the currently executed action, and They are the value network’s first and The output of time steps.

[0062] The loss function of the state value network is constructed based on the environmental reward and the intrinsic reward of the current action, including the following loss functions:

[0063] ;

[0064] ;

[0065] in, Represents the state value network parameters; Indicates the number of samples taken; Represents the total number of agents in the environment; Number the sample; Number the agents; and are the outputs of the state value network for the current time step and the next time step respectively; is the clipping function, which means when , output ,when , output ; represents the weighted sum of the environment reward and the intrinsic reward of the currently executed action, which is used to represent the total reward received by agent k; Rewards for the environment; is the intrinsic reward output by the intrinsic reward network, and are the preset hyperparameters.

[0066] The loss function of the intrinsic reward network is constructed based on the global value and the intrinsic reward of the current action, including the following loss functions:

[0067] ;

[0068] in, are the intrinsic reward network parameters; Indicates the number of samples sampled; Represents the total number of agents in the environment; Number the sample; Number the agents; are the preset hyperparameters; The intrinsic reward of agent k output by the intrinsic reward network; Denote the intrinsic reward network parameters as The intrinsic reward output by the intrinsic reward network; and Respectively represent the network parameters in the state value The output of the state value network for the current time step and the next time step, and are the global observations of agent k at the current time step and the next time step respectively; Represents the variance function for computing the agent's intrinsic reward.

[0069] In the present invention, the loss function is a way of calculating the loss. The gradient descent method adjusts the network parameters to make the loss calculated according to the loss function as small as possible. The loss function designer often takes the gap between the network output and the target output as the loss, and minimizes the gap to make the network output meet expectations.

[0070] Specifically, in actual operation, the data obtained from the environment and the data output by the neural network are first loaded into the data loader in the form of a batch size of 400, so that the model parameters can be updated using the gradient descent method during the training process. The orthogonal initialization method is used when initializing the neural network. In the policy network loss, the hyperparameter Set to 0.01, hyperparameter Set to 0.2; in the intrinsic reward network loss, the hyperparameter Set to 0.5; in the state value network loss, the hyperparameter Set to 0.01; finally, fit the model parameters using the stochastic gradient descent method with a learning rate of 5e-4 until the training is stopped after 10e7 time steps in the training set, and save the model's weight parameter file.

[0071] The multi-agent reinforcement learning strategy optimization method provided by the present invention can be widely applied to all eligible multi-agent scenarios to improve the adaptability of each agent in the multi-agent scenario. The specific application of this method is described in detail below with different embodiments.

[0072] Example 1

[0073] In this embodiment, a multi-agent reinforcement learning strategy optimization method based on an intrinsic reward network is applied to an autonomous driving scenario. At this time, the reinforcement learning simulation scenario includes an autonomous driving simulation scenario, and each agent is each autonomous driving vehicle. In this scenario, the global observation data is the position, pedestrians, obstacles, roads and traffic light status information of all autonomous driving vehicles in the autonomous driving simulation scenario; the local observation data of each agent includes: the position of each autonomous driving vehicle in the reinforcement learning simulation scenario, the planned driving trajectory, and the other vehicles, pedestrians, obstacles, roads, and traffic light status information that the autonomous driving vehicle can observe; the actions that each agent needs to perform include: acceleration, deceleration, steering and stopping. The environmental reward of the autonomous driving scenario can be formulated according to the time for all vehicles to reach the destination, the degree of optimization of the path selection and the safety; the intrinsic reward can be the reward given when each autonomous driving vehicle performs a certain action.

[0074] By optimizing the autonomous driving scenario through the multi-agent reinforcement learning strategy optimization method disclosed in the present invention, the adaptive capability of each autonomous driving vehicle in the autonomous driving scenario can be improved, thereby improving the safety of autonomous driving vehicles.

[0075] Example 2

[0076] In this embodiment, the multi-agent reinforcement learning strategy optimization method based on the intrinsic reward network is applied to the intelligent control field of traffic lights. The agent at this time is each traffic light in the traffic network, and the global observation data is all the data that can be obtained in the current traffic network; the local observation data of each agent includes: the number, location and forward route of vehicles and pedestrians around each traffic light; the actions that each agent needs to perform include: switching between traffic lights and the corresponding lighting time. The environmental reward for traffic light control is determined according to the overall smoothness of vehicle and pedestrian flow in the traffic network; the intrinsic reward is the reward given after each traffic light performs a certain action.

[0077] Optimizing traffic light scenarios using the multi-agent reinforcement learning strategy optimization method of the present invention can enhance the adaptive capability of traffic light control and thereby improve the flow of traffic in the traffic network.

[0078] Example 3

[0079] In this embodiment, the multi-agent reinforcement learning strategy optimization method based on the intrinsic reward network is applied to the motion field of the intelligent robot. At this time, the agent is each joint in the intelligent robot, and the global observation data includes all the data that the intelligent robot can perceive; the local observation data of each agent includes: the orientation information that each joint of the intelligent robot can perceive; the actions that each agent needs to perform include: the actions that each joint of the intelligent robot needs to perform, such as rotation and movement. The environmental reward is determined according to indicators such as the overall smoothness of the robot's execution of the action; the intrinsic reward is the reward given to each joint of the intelligent robot after performing a certain action.

[0080] Optimizing the motion behavior of an intelligent robot using the multi-agent reinforcement learning strategy optimization method of the present invention can enhance the motion adaptability of each joint of the intelligent robot, thereby improving the overall motion flow and coordination of the intelligent robot.

[0081] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0082] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-agent reinforcement learning strategy optimization method based on intrinsic reward network, characterized in that: The following steps are involved: Model building and parameter initialization: Building a reinforcement learning simulation scenario containing multiple agents and initializing model parameters, including decision network parameters, intrinsic reward network parameters, and state value network parameters; Data acquisition: Obtain environmental rewards, global observation data, and local observation data of each agent from the environment of the reinforcement learning simulation scenario; input the local observation data into the decision network, and output the action that each agent needs to perform; input the local observation data and the action that each agent needs to perform into the intrinsic reward network, and output the intrinsic reward of the current action; input the global observation data into the state value network, and output the global value; Construct the loss function of the model and optimize the model parameters: The loss of the decision network is calculated based on the environmental reward, the intrinsic reward of the current action, and the global value, and the decision network parameters are optimized through the gradient descent algorithm so that the output of the decision network is close to the target output; The loss of the state value network is calculated based on the environmental reward and the intrinsic reward of the current action, and the state value network parameters are optimized through the gradient descent algorithm so that the output of the state value network is close to the target output; The loss of the intrinsic reward network is calculated based on the global value and the intrinsic reward of the current action, and the parameters of the intrinsic reward network are optimized through the gradient descent algorithm so that the output of the intrinsic reward network is close to the target output.

2. The multi-agent reinforcement learning strategy optimization method based on intrinsic reward network according to claim 1, characterized in that: The intelligent agent includes an autonomous vehicle. Global observation data includes the position of all autonomous vehicles, pedestrians, obstacles, roads, and traffic light status information in the current reinforcement learning simulation scenario; The local observation data of each agent includes: the position of each autonomous vehicle in the reinforcement learning simulation scenario, the planned driving trajectory, and the status information of other vehicles, pedestrians, obstacles, roads, and traffic lights that the autonomous vehicle can observe; The actions that each agent needs to perform include: acceleration, deceleration, turning, and stopping.

3. The multi-agent reinforcement learning strategy optimization method based on intrinsic reward network according to claim 1, characterized in that: The agent includes traffic lights in a traffic network. Global observation data include all data available in the current transportation network; The local observation data of each agent includes: the number, location and forward route of vehicles and pedestrians around each traffic light; The actions that each agent needs to perform include: switching between red and green lights and the corresponding lighting time.

4. The multi-agent reinforcement learning strategy optimization method based on intrinsic reward network according to claim 1, characterized in that: The intelligent body includes each joint in the intelligent robot. Global observation data includes all data that the intelligent robot can perceive; The local observation data of each intelligent agent includes: the orientation information that each joint of the intelligent robot can perceive; The actions that each intelligent agent needs to perform include: the actions that each joint of the intelligent robot needs to perform.

5. The multi-agent reinforcement learning strategy optimization method based on intrinsic reward network according to claim 1, characterized in that: The decision network consists of a multi-layer perceptron module, an RNN module and a classification distribution module connected in sequence.

6. The multi-agent reinforcement learning strategy optimization method based on intrinsic reward network according to claim 1, characterized in that: The intrinsic reward network consists of a multi-layer perceptron module, an RNN module, and a linear function approximator connected in sequence.

7. The multi-agent reinforcement learning strategy optimization method based on intrinsic reward network according to claim 1, characterized in that: The state value network is composed of a multi-layer perceptron module, an RNN module and a linear function approximator connected in sequence.

8. The multi-agent reinforcement learning strategy optimization method based on intrinsic reward network according to claim 1, characterized in that: The loss function of the decision network is constructed based on the environmental reward, the intrinsic reward of the current action, and the global value, including the following loss functions: ；； ； in, Decision network parameters that need to be updated; Represents the decision network parameters used for sampling; Indicates the number of samples taken; Represents the total number of agents in the environment; Number the sample; Number the agents; For intelligent agents Advantage estimates calculated from environmental rewards, intrinsic rewards, and generalized advantage estimates; Represents the ratio of decision network outputs under different decision network parameters; and are the preset hyperparameters; is the clipping function, which means when , output ,when , output ; represents the entropy of the calculated approximate distribution, The probability distribution of each action in the action set output by the decision network; The decision network parameters are The output when , means that when the decision network parameters are When observing history As input, the decision network outputs the execution action probability; The decision network parameters are The output when , means that when the decision network parameters are When observing history As input, the decision network outputs the execution action probability.

9. The multi-agent reinforcement learning strategy optimization method based on intrinsic reward network according to claim 1, characterized in that: The loss function of the state value network is constructed based on the environmental reward and the intrinsic reward of the current action, including the following loss functions: ； ； in, Represents the state value network parameters; Indicates the number of samples taken; Represents the total number of agents in the environment; Number the sample; Number the agents; and are the outputs of the state value network for the current time step and the next time step respectively; is the clipping function, which means when , output ,when , output ; represents the weighted sum of the environment reward and the intrinsic reward of the currently executed action, which is used to represent the total reward received by agent k; Rewards for the environment; is the intrinsic reward output by the intrinsic reward network, and are the preset hyperparameters.

10. The multi-agent reinforcement learning strategy optimization method based on intrinsic reward network according to claim 1, characterized in that: The loss function of the intrinsic reward network is constructed based on the global value and the intrinsic reward of the current action, including the following loss functions: ； in, are the intrinsic reward network parameters; Indicates the number of samples taken; Represents the total number of agents in the environment; Number the sample; Number the agents; are the preset hyperparameters; The intrinsic reward of agent k output by the intrinsic reward network; Denote the intrinsic reward network parameters as The intrinsic reward output by the intrinsic reward network; and Respectively represent the network parameters in the state value The output of the state value network for the current time step and the next time step, and are the global observations of agent k at the current time step and the next time step respectively; Represents the variance function for computing the agent's intrinsic reward.

Images