A multi-agent reinforcement learning training method and system

Abstract

The application relates to a multi-agent reinforcement learning training method and system, which comprises the following steps: a local strategy training stage, in which an agent makes an action by using local observation information; a global strategy training stage, in which a global strategy of the agent uses the local strategy as an action module for interacting with an environment, the global strategy takes global state of the environment as input, encodes the global information in a hidden space, and uses a neural network to fit the global state by using local observation information of all agents; the local strategy makes a suitable action in the environment according to the local observation information and the output of the global strategy; and a local strategy optimization stage, in which the global strategy, the local strategy and a fitting model obtained in the previous two stages are used to optimize the existing local strategy, so that an agent with better effect is finally obtained. The application can improve the speed and accuracy of multi-agent reinforcement learning training.

Description

Technical Field

[0001] This invention relates to agent reinforcement learning technology, specifically to a multi-agent reinforcement learning training method and system. Background Technology

[0002] Reinforcement learning (RL) is a field within machine learning that focuses on learning "what to do (i.e., how to map the current situation into actions) to maximize numerical reward signals." The agent is not told what actions to take; instead, it must discover through trial and error which actions yield the greatest reward. Reinforcement learning is a computational approach for understanding and automating goal-oriented learning and decision-making problems. It emphasizes that the agent learns through direct interaction with its environment, without requiring supervised signals to be imitated or a complete model of the surrounding environment.

[0003] Deep reinforcement learning uses deep neural networks (DNNs) to approximate the optimal policy, but DNNs typically require fixed-dimensional feature vectors (state representations) as input. However, due to the uncertainty of the population size, the dimension of the system state representation changes dynamically. This contradiction limits the scalability of deep reinforcement learning algorithms, and the learned optimal policy often cannot adapt to the dynamic changes in variable population sizes.

[0004] Multi-agent reinforcement learning applies the ideas and algorithms of deep reinforcement learning to the learning and control of multi-agent systems, and is an important method for developing swarm intelligence. However, most existing multi-agent algorithms adopt a centralized training and distributed execution (CTDE) approach. While this improves the scalability of multi-agent models, it simplifies and neglects the communication problem between agents. Communication between agents often contains a wealth of information, which can be used to achieve better performance. Alternatively, fixing the scope of information propagation between agents during centralized training can cause the algorithm to lose scalability during distributed execution. Summary of the Invention

[0005] The purpose of this invention is to solve the aforementioned technical problems of existing multi-agent algorithms, thereby improving the speed and accuracy of multi-agent reinforcement learning training.

[0006] To achieve the above objectives, in one aspect, the present invention provides a multi-agent reinforcement learning training method, the method comprising:

[0007] During the local policy training phase, the agent's local policy uses its own local observation information to make actions;

[0008] During the global policy training phase, the agent's global policy uses local policies as action modules for interacting with the environment. The global policy takes the global state of the environment as input, encodes global information in the latent space, and uses the local observation information of all agents to fit the global state using a neural network. The local policies combine their respective local observation information and the output of the global policy to make appropriate actions in the environment that meet the requirements of the environment.

[0009] In the local policy optimization stage, the global and local policies obtained in the first two stages, as well as the fitted model, are used to optimize the existing local policies, ultimately resulting in a better-performing agent.

[0010] On the other hand, the present invention provides a multi-agent reinforcement learning training system, comprising:

[0011] The local policy training module is used to enable the agent to make actions based on its local observation information.

[0012] The global policy training module is used to train the global policy of the agent. The global policy uses local policies as action modules to interact with the environment. The global policy takes the global state of the environment as input and encodes the global information in the latent space. At the same time, it uses the local observation information of all agents to fit the global state using a neural network. The local policies combine their local observation information and the output of the global policy to make appropriate actions in the environment that meet the requirements of the environment.

[0013] The local policy optimization module is used to optimize the existing local policies based on the global and local policies obtained in the first two stages and the fitted model, ultimately resulting in a better agent.

[0014] The beneficial effects of this invention are as follows: 1) It enables faster training of multi-agent reinforcement learning algorithms. 2) The semi-centralized reinforcement learning algorithm structure provides better scalability. 3) Encoding local observation information of agents using a fitted model eliminates the limitation on algorithm scalability imposed by the number of agents, achieving the goal of transferring the model to different scales. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of a multi-agent reinforcement learning training method provided in an embodiment of the present invention;

[0016] Figure 2 This is a schematic diagram of another multi-agent reinforcement learning training method provided in an embodiment of the present invention;

[0017] Figure 3 This is a schematic diagram of another multi-agent reinforcement learning training method provided in an embodiment of the present invention;

[0018] Figure 4 This is a schematic diagram of the structure of a multi-agent reinforcement learning training system provided in an embodiment of the present invention;

[0019] Figure 5 This is a diagram illustrating the rewards for the first stage of model training.

[0020] Figure 6 This is a diagram illustrating the rewards for the second stage of model training.

[0021] Figure 7 This is a diagram illustrating the rewards for model training in the third stage. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] The multi-agent reinforcement learning training method disclosed in this invention includes a local policy training phase, a global policy training phase, and a local policy optimization phase. The entire agent is divided into two layers: a local policy layer and a global policy layer.

[0024] Figure 1 This is a schematic diagram of a multi-agent reinforcement learning training method provided in an embodiment of the present invention. Figure 1 As shown, the multi-agent reinforcement learning training method includes the following steps:

[0025] Step S101, Local Policy Training Phase: The agent's local policy uses its own local observation information to make actions.

[0026] Specifically, during local policy training, only the local policy layer of the agent participates in the training; the output of the global policy layer does not affect the result of the local policy. The agent only uses its own local observation information to make actions, without any cooperative behavior between agents. This local observation information includes the actions and states of other agents within the visible range, as well as its own state. Different agents have different actions; for example, an agent controlling a drone needs to control the drone's related behaviors. In the StarCraft game environment, the agent controls the actions of units in the game.

[0027] Step S102, Global Policy Training Phase: The agent's global policy utilizes local policies as action modules for interacting with the environment. The global policy takes the global state of the environment as input, encoding global information in the latent space. This involves mapping the global information to a high-dimensional space, known as the latent space, using a neural network. Simultaneously, the local observation information of all agents is used to fit the global state using a neural network. This maps both global and local observation information to the latent space, minimizing the distance between the two quantities. Local policies combine their local observation information with the output of the global policy to perform appropriate actions that meet environmental requirements. The global policy can adopt an Actor-Critic (AC) structure.

[0028] Step S103, Local Policy Optimization Stage: Optimize the existing local policy based on the global and local policies and the fitted model obtained in the first two stages.

[0029] Specifically, the local policy corrects the output action based on the global information encoded by the global policy. The local policy then updates and strengthens the policy based on the corrected action, local observation information, and reward information, ultimately resulting in a better-performing agent. The reward information varies in different environments, and it incentivizes the agent to learn and converge. Taking the StarCraft environment as an example, the reward value is calculated based on the controlled agent's health and the damage dealt to the opponent, weighted by the reward amount.

[0030] In the execution of this embodiment of the invention, the impact of inter-agent communication on the training strategy will be ignored. Global policy updates utilize the local observation information of the agents. For a given number of agents, a fixed length of global state information can be obtained by fitting a model, thus enabling the reuse of training data across agent groups of different sizes.

[0031] Figure 2 This is a schematic diagram of another multi-agent reinforcement learning training method provided in an embodiment of the present invention. Figure 2 As shown, the training method consists of three stages. In the first stage, the agent fits the global state information and reward information of the environment to its own state observation information and executable actions as input. In the second stage, the global policy considers all actions using the state information encoded by all agents and provides appropriate action state values ​​for each agent. Each agent executes actions using the obtained action state values. In the third stage, each agent optimizes its local policy under the guidance of the global policy, enabling the agent to utilize the information of other agents from its own local observations.

[0032] Figure 3 This is a schematic diagram of another multi-agent reinforcement learning training method provided in an embodiment of the present invention; as shown below. Figure 3 As shown, S represents global state information, C represents control information transmitted from the global policy to the local policy, and O1, O2, ..., O n The local observation information represents different agents, and a1, a2, ..., an represents the actions given by the local policies. In the central policy (i.e., the global policy), multiple agents use global information to update the policy, and the global policy gives corresponding commands for different local executions. The commands contain the encoding of the global information and the expectation of each agent's action. In the local policy, agents (Agent 1, Agent 2, ..., Agent n) perform actions based on their local observation information and the command encoding given by the central model, and select the optimal action for the current state.

[0033] The beneficial effects of this invention are as follows: 1) It provides faster training speed for multi-agent reinforcement learning algorithms. 2) The semi-centralized reinforcement learning algorithm structure offers better scalability. 3) Encoding local observation information of agents using a fitted model overcomes the limitation of the number of agents on the algorithm's scalability, achieving the goal of transferring the model to different scales.

[0034] Accordingly, embodiments of the present invention provide a multi-agent reinforcement learning training system structure.

[0035] Figure 4 This is a schematic diagram of a multi-agent reinforcement learning training system provided in an embodiment of the present invention; as shown below. Figure 4 As shown, the multi-agent reinforcement learning training system is characterized by comprising: a local policy training module 410, a global policy training module 420, and a local policy optimization module 430.

[0036] Local policy training module 410 is used for the agent to make actions based on its local observation information;

[0037] The global policy training module 420 is used for the agent's global policy to use local policies as action modules for interacting with the environment. The global policy takes the global state of the environment as input, encodes the global information in the latent space, and uses the local observation information of all agents to fit the global state using a neural network. The local policies combine their own local observation information and the output of the global policy to make appropriate actions in the environment that meet the requirements of the environment. The local observation information includes the actions and states of other agents within the visible range, as well as the state of the agent itself.

[0038] The local policy optimization module 430 is used to optimize the existing local policy based on the global and local policies obtained in the first two stages and the fitted model, and finally obtain a better agent.

[0039] The local policy training module 410 is specifically used for the agent to fit the global state information and reward information of the environment based on its own state observation information and actions as input.

[0040] The global policy training module 420 is specifically used by the central model to consider all actions using the state information encoded by all agents, to provide appropriate action state values ​​for each agent, and for each agent to perform actions using the obtained action state values.

[0041] The local policy optimization module 430 is specifically used to correct the output action based on the global information encoding of the global policy output, and to update and strengthen the policy based on the corrected action, local observation and reward information, so as to obtain a better agent.

[0042] In one embodiment, such as experimenting with the above method under 8m conditions in a StarCraft II environment, the following results can be obtained. During the initial training of the local policy, the model performs poorly because action decisions are made only based on local observation information (e.g., Figure 5 (As shown).

[0043] During global policy training, when the global state is used to provide global encoded information to guide local policies, the performance of the agent swarm rapidly improves to a better level (e.g., Figure 6 (As shown). Therefore, the guidance information of the global policy is very important for the performance of the agent population.

[0044] When the local strategy is further optimized, it quickly reaches a very good performance level (e.g., Figure 7 (As shown).

[0045] It is obvious that many variations of the invention described herein can be made without departing from the true spirit and scope of the invention. Therefore, all changes that are obvious to those skilled in the art should be included within the scope of the claims. The scope of protection of this invention is defined only by the claims.

Claims

1. A multi-agent reinforcement learning training method applied in a game environment, characterized in that, Comprise: The local strategy training phase, the local strategy of the agent uses its local observation information to make actions; The agent is used to control the action of a unit in a game environment; In the local strategy training phase, the agent trains the local strategy of the agent according to the state observation information, action and reward information as input; At the same time, the global state information of the environment is encoded and fitted in the latent space using the local observation information of all agents; In the game environment, the reward information is calculated according to the weighted reward value of the controlled agent's life value and the damage to the opponent; The global strategy training phase, the global strategy of the agent uses the local strategy as the action module for interacting with the environment, and the global strategy takes the global state of the environment as input to encode the global information in the latent space, and uses neural network to fit the global state using the local observation information of all agents; The local strategy combines its own local observation information and the output of the global strategy to make actions in the environment that meet the requirements of the environment; In the global strategy training phase, the global strategy controls all agent local strategies using all agent encoded state information, and each agent uses the obtained action state value to perform actions; The local strategy optimization phase, using the global strategy and local strategy obtained in the previous two phases and the fitting model, optimizes the existing local strategy, and finally obtains a better agent; In the local strategy optimization phase, the local strategy corrects the output action according to the global information encoded by the global strategy output, and the local strategy updates and strengthens the strategy according to the corrected action, local observation and reward information, and finally obtains a better agent.

2. The method of claim 1, wherein, The local observation information includes the actions and states of other agents within the visible range, and the state of itself. 3.A multi-agent reinforcement learning training system applied in a game environment, characterized in that, Comprise: The local strategy training module is used for the local strategy of the agent to make actions using its own local observation information; The agent is used to control the action of a unit in a game environment; In the local strategy training phase, the agent trains the local strategy of the agent according to the state observation information, action and reward information as input; At the same time, the global state information of the environment is encoded and fitted in the latent space using the local observation information of all agents; In the game environment, the reward information is calculated according to the weighted reward value of the controlled agent's life value and the damage to the opponent; The global strategy training module is used for the global strategy of the agent to use the local strategy as the action module for interacting with the environment, and the global strategy takes the global state of the environment as input to encode the global information in the latent space, and uses neural network to fit the global state using the local observation information of all agents; The local strategy combines its own local observation information and the output of the global strategy to make actions in the environment that meet the requirements of the environment; In the global policy training stage, the global policy controls all the local policies of the agents using the state information coded by all the agents, and each agent uses the action state value given by the global policy to perform an action; The local policy optimization module is configured to optimize the local policy based on the global policy and the local policy obtained in the previous two stages and the fitting model, and finally obtain a better local policy of the agent. In the local policy optimization stage, the local policy corrects the output action according to the global information coded by the global policy output by the global policy, and updates and strengthens the policy according to the corrected action, local observation and reward information, and finally obtains a better agent.

4. The system of claim 3, wherein, The local observation information includes the actions and states of other agents within the visible range and the state of the agent itself.

Images