Multi-agent reinforcement learning method and system based on value decomposition

Abstract

The invention discloses a multi-agent reinforcement learning method and system based on value decomposition. The method comprises the steps of obtaining a state St of an environment at a current t moment, an action which can be obtained by an initial observation value of each agent and a reward r corresponding to the action; for each agent, calculating a value function Qi (taui) observed by each action based on the local information taui through an evaluation-agent network; utilizing a random-agent network to obtain a reward value function Qi (tau) of each agent based on the global information tau; using the target-agent network to calculate a loss function and update parameters; decomposing a reward value function Qi (tau) of each agent based on the global information tau by using a competition-agent network; and adding decomposition results to obtain a joint reward value function Qtot (tau, a) based on global information tau, and completing training. According to the method, double extraction is carried out on the logical topological relation among multiple agents, and the learning efficiency and the strain capacity of the agents are improved in a complex heterogeneous part observable scene.

Description

technical field [0001] The invention relates to a multi-party cooperative processing technology, in particular to a multi-agent reinforcement learning method and system based on value decomposition. Background technique [0002] Nowadays, multi-agent reinforcement learning (MARL) is a hot topic. MARL has broad prospects in solving many complex real-world problems such as sensor networks, robot swarm coordination, and autonomous vehicles. However, in practical applications, MARL faces Partial observability and stability are the two main challenges. First, when the agent interacts with the environment, the agent cannot observe and make decisions from a global perspective, so it cannot learn the global optimal strategy, and can only observe the information within its own field of vision. Secondly, in a multi-agent environment, agents interact with each other, and each agent makes corresponding actions according to its own local observations, which may affect other agents, beca...

AI Technical Summary

Problems solved by technology

[0007] In a large-scale collaborative environment, a problem arises in the process of value function decomposition: the size of the joint state-action space of the value function grows exponentially with the number of agents, which makes value decomposition fast and efficient. becomes more difficult, convergence times are often not guaranteed

[0008] (1) Due to the complexity of the multi-agent environment, the agent needs to spend a lot of time to explore a state that is beneficial to itself or the system in the early stage of exploration. The number of exploration spaces will increase with the increase of the number of agents. In some rewards In a sparse multi-agent scene, it is very likely that the positive feedback of the reward will not be obtained for a long time, the agent cannot effectively perceive the scene information and make correct decisions, and the convergence time is difficult to guarantee

[0009] (2) When the agent performs an action according to the strategy, if the misestimation of several suboptimal joint actions exceeds the better estimate of a single optimal joint action, the reward corresponding to the optimal action will be underestimated, making the agent Choose actions with suboptimal value, which causes the agent's action evaluation to fall into a local optimal cycle and cannot decide the best action, prolonging the time for the agent to make action decisions

Images