A multi-agent reinforcement learning training method based on rotational symmetry

Abstract

The application provides a multi-agent reinforcement learning training method based on rotational symmetry, and belongs to the technical field of multi-agent reinforcement learning.The method uses the global symmetry of a multi-agent system, performs rotational transformation based on original samples to obtain enhanced samples, uses the enhanced samples to calculate a symmetric loss, improves the loss function of multi-agent reinforcement learning, adds the symmetric loss to the original loss function to provide additional information for the training of a neural network, and adds the enhanced samples to an experience cache pool to participate in the training of the next round of multi-agent reinforcement learning.The application improves sample utilization, improves the convergence speed and final convergence effect of a multi-agent reinforcement learning model, is a plug-and-play method, can be embedded into any multi-agent reinforcement learning algorithm, and has very wide application.

Description

Technical Field

[0001] This invention belongs to the field of multi-agent reinforcement learning technology, and relates to a multi-agent reinforcement learning training method based on rotational symmetry. Background Technology

[0002] Various modern artificial intelligence challenges can be formulated as multi-agent reinforcement learning (MARL) problems. With the development of MARL, significant achievements have been made in various task scenarios, such as multiplayer games, traffic control, and multi-robot applications. However, most reinforcement learning (RL) methods require large amounts of data to train agents. In simulations, data efficiency is crucial for achieving rapid experimental iterations due to rendering speed limitations. While parallel and accelerated simulation environments enable agents to solve complex tasks within a reasonable timeframe, agent training in real-world environments is constrained by physical limitations. Therefore, improving the sample efficiency of existing MARL methods is essential for both theoretical research and practical application.

[0003] Improving sample efficiency in MARL (Multi-Agent Reinforcement Learning) refers to reducing the number of environmental interactions to better learn the agent's behavioral policies. Recent literature has proposed many methods to address the data inefficiency problem in deep reinforcement learning. These methods can be broadly categorized into two types: data augmentation-based methods and network architecture design-based methods. Specifically, data augmentation has been widely used in recent research to generate more data samples in single-agent reinforcement learning, but most proposed solutions focus on image-based data augmentation. For example, a data augmentation technique called RAD (Reinforcement Learning Algorithm) performs operations such as cropping and flipping on observed image data, improving the data efficiency of single-agent reinforcement learning. Therefore, despite its importance, data augmentation methods in MARL have been largely unexplored. In multi-agent reinforcement learning, agents execute actions based on the neural network policy according to their current state at each time step. The complex interactions between agents make it difficult to simply use data augmentation methods from computer vision. Therefore, while data augmentation-based methods can generate a sufficient number of samples, they significantly increase the computational cost of multi-agent reinforcement learning algorithms. Another approach, based on network architecture design, typically aims to design specialized network architectures that implicitly embed prior knowledge relevant to the given task. Multi-agent homomorphic networks integrate the global symmetry of multi-agent systems into neural network models, thereby improving data efficiency. Despite their significant effectiveness, these methods require careful design of the network architecture. Furthermore, because the underlying invariants are difficult to implicitly encode into the neural network architecture, these methods suffer from poor generality and are difficult to scale to more complex tasks. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies and address the problem of low sample efficiency in multi-agent reinforcement learning, this invention proposes a multi-agent reinforcement learning training method based on rotational symmetry. By improving the loss function of the multi-agent reinforcement learning algorithm and utilizing the global symmetry of the multi-agent system, additional information is provided for the training of the neural network, enabling the multi-agent reinforcement learning algorithm to achieve higher data efficiency and better convergence during training, thus effectively promoting the training of the multi-agent reinforcement learning algorithm.

[0005] This invention provides a multi-agent reinforcement learning training method based on rotational symmetry, comprising the following steps:

[0006] Step 1: Initialize the policy network parameters, action value network parameters, and symmetric loss weights of the multi-agent system.

[0007] Step 2: Obtain the current state of each agent in the environment, use the current policy network of each agent to obtain the action vector of each agent, and execute their respective action vectors; obtain the reward after the action is executed and the agent state after the action is executed; generate the original sample (s,a,r,s').

[0008] Step 3: Obtain a sufficient number of raw samples and store them in the experience cache pool.

[0009] Step 4: Randomly collect samples from the experience cache pool and perform rotation transformation on the collected samples.

[0010] Step 5: Use the samples collected in Step 3 to train the policy network and the action value network respectively, and use the loss function of the new action value network based on rotational symmetry to update the action value function.

[0011] Step 6: In the decision-making phase, each agent selects actions based on local observations and its own policy function and outputs them to the environment, thereby interacting with the environment.

[0012] In step 3, based on the global symmetry of the multi-agent system, an enhanced sample is obtained by performing a rotation transformation on the original sample. Let the obtained original sample be (s,a,r,s'), where s is the current state of the multi-agent in the environment, a is the action vector of the agent obtained by the policy network of the multi-agent, and r and s' are the reward obtained by each agent after executing its own action vector and the state of the multi-agent in the environment after executing the action, respectively.

[0013] The rotation matrix is ​​obtained based on the pre-set rotation angle β. Multiplying s, a, and s' by the rotation matrix T yields the state and action after rotation by angle β. Let L[s] and K[s] be obtained respectively.s [a], L[s'], then the enhanced sample (L[s], K) is obtained. s [a],r,L[s']).

[0014] In step 5, a symmetric loss based on rotational symmetry is calculated. This symmetric loss is added to the original action value network loss of the multi-agent reinforcement learning algorithm to obtain a new loss function L(φ) for the action value network, as follows:

[0015] L(φ)=L M (φ)+c m L sym (φ)

[0016] Where φ represents the parameters of the action value network, and L M (φ) represents the loss of the original action-value network in the multi-agent reinforcement learning algorithm used, c m For symmetric loss weights, L sym (φ) represents the symmetry loss based on rotational symmetry;

[0017] L sym (φ) is calculated as follows:

[0018]

[0019] in, For experience cache pool; It is the expected cumulative discounted reward that a multi-agent agent can obtain by performing action a in state s, using policy π. The multi-agent system employs policy π, and in state L[s], executes action K. s [a] The expected cumulative discount reward that can be obtained.

[0020] The advantages of the multi-agent reinforcement learning training method based on rotational symmetry of the present invention are as follows:

[0021] (1) Traditional multi-agent reinforcement learning algorithms ignore the inherent properties of the system itself. The method of this invention is based on the rotational symmetry of the multi-agent system. It transforms the samples generated by the interaction between the agent and the environment to calculate the symmetric loss, improves the loss function of multi-agent reinforcement learning, adds symmetric loss on the basis of the original loss function, improves the sample utilization rate, and improves the convergence speed and final convergence effect of the multi-agent reinforcement learning algorithm.

[0022] (2) Compared with data augmentation methods that store additional samples in the experience cache pool, the method of the present invention does not need to add the augmented samples after symmetry transformation to the experience cache pool during each training process. Instead, it directly uses the symmetry-transformed samples for calculation when calculating the loss function, without increasing the number of additional samples.

[0023] (3) Since this invention is an improvement on the loss function in the training process of multi-agent reinforcement learning, it does not require any assumptions about the design of a specific MARL algorithm. It is a plug-and-play method and can be embedded into any multi-agent reinforcement learning algorithm, thus having a wide range of applications. Attached Figure Description

[0024] Figure 1 A visual example of global symmetry for multi-agent systems;

[0025] Figure 2 This is a block diagram of a multi-agent reinforcement learning training method based on rotational symmetry according to the present invention.

[0026] Figure 3 This is a flowchart of a multi-agent reinforcement learning training method based on rotational symmetry according to the present invention. Detailed Implementation

[0027] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0028] Existing mainstream MARL algorithms neglect the symmetry of multi-agent systems, while the most common symmetry in the multi-agent system of this invention is as follows: Figure 1 The global symmetry is shown. Figure 1 This represents a typical multi-agent navigation task, where circles represent agents, crosses represent target points, and the agents' task is to navigate to the target points. During reinforcement learning training, agent policies π1, π2, ..., π are used. N Interacting with the environment, each interaction generates a sample (s, a, r, s'), where s represents the state of all agents, i.e., s = (s1, s2, ..., s'). N ), s i Let a represent the state of agent i. Let a represent the actions of all agents, a = (a1, a2, ..., a...). N ), s' represents agent i performing action a. i The next state that can be reached is s. i The set of ', i = 1, 2, 3, ..., N; r represents the agent's reward r from environmental feedback. i The set of i, i = 1, 2, 3, ..., N. Figure 1 Let's take Agent 1 as an example to illustrate global symmetry. At each moment, Agent 1 can observe the position information of other agents. and target point location information (g x ,g y ), This represents the difference in coordinates between agent i and agent j in the x-direction. G represents the difference in coordinates between agent i and agent j in the y-direction. x and g y These represent the coordinate differences between the current agent and the target point in the x and y directions, respectively. The state of agent 1. For agent 1 in system state s, the policy is to perform the action a1 of moving downwards. After performing the action, the state changes to s1′ and the agent receives a reward r from the environment, i.e., it obtains a sample (s, a, r, s'). Rotating the entire system by 90° yields... Figure 1 In the upper right part, the system state is L[s]. For L[s], the strategy of agent 1 should be to move to the left, which is exactly the action K obtained by rotating 90° clockwise from action a1. s [a1]. Simultaneously, the state s' of agent 1 after executing the action is transformed into L[s'] by rotation, and the action K is executed in state L[s]. s The state corresponding to [a1] is the same.

[0029] Inspired by symmetry phenomena in the multi-agent learning (MARL) field, this invention proposes a loss function for MARL training. This loss function provides additional information for neural network training, guiding the direction of MARL training. When the update direction of the neural network violates system symmetry, it can adjust the update direction in a timely manner, preventing training collapse and achieving better results with fewer samples, thus improving the data efficiency of MARL algorithms. This invention's rotationally symmetric-based MARL training method is a plug-and-play approach applicable to most MARL algorithms, requiring no specific assumptions for MARL algorithm design. Figure 2 and Figure 3 As shown, the present invention will be described in six steps below.

[0030] Step 1: Initialize the policy network parameters, action value network parameters, and symmetric loss weights of the multi-agent system.

[0031] Initialize policies π1,π2,...,π for N agents. N , where π i It is to state s i Mapped to action a i The function, i.e., a i =π i (s i Specifically, the parameter used is θ. i The neural network represents the strategy π i Initialization includes: initializing the action value network. in This is the expected cumulative discounted reward that a multi-agent system can obtain by performing action a in state s, using policy π. A neural network with parameters φ is used to represent the action-value network. Initialize the experience replay pool D, initially empty. Initialize the symmetric loss weights c. m Set it to 0.5.

[0032] Step 2: Obtain the current state of each agent in the environment, use the current policy network of each agent to obtain the action vector of each agent, and execute their respective action vectors; obtain the reward after executing the action and the agent state after executing the action.

[0033] Using agent policies π1, π2, ..., π N Interacting with the environment, generating samples (s, a, r, s'), where s represents the state of all agents, i.e., s = (s1, s2, ..., s'). N ), where a represents the actions of all agents, a = (a1, a2, ..., a...). N ), s' represents agent i performing action a. i The next state that can be reached is s. i The set of ', i = 1, 2, 3, ..., N; r represents the agent's reward r from environmental feedback. i The set of i, i = 1, 2, 3, ..., N.

[0034] Step 3: Obtain enough data and store it in the experience cache pool.

[0035] Obtain a sufficient number of original samples (s, a, r, s') and store them in the experience replay pool D. Then, randomly collect M groups of samples (s, a, r, s') from the experience replay pool. k ,a k ,r k ,s′ k ), where k = 1, 2, ..., M.

[0036] Step 4: Perform a rotation transformation on the collected samples.

[0037] Design rotational transformations L[s] and K based on the system's global symmetry. s [a], where L[s] refers to the rotation transformation of the global state s of the system, and K s [a] refers to the rotation transformation performed on action a when the global state is s. Subsequently, rotation transformation L[s], K is performed on the samples (s,a,r,s') generated by the agent's interaction with the environment. s [a], obtain the enhanced sample (L[s],K s [a],r,L[s']), where L[s'] is the state of the original sample s' after performing a rotation transformation.

[0038] Define matrix Any two-dimensional vector can be multiplied by matrix T to obtain a vector rotated by an angle β. If the system state is a two-dimensional vector s, and the agent i's policy is π... i The corresponding action is a = π i The data (s,a) generated by the rotation matrix T can be rotated using the rotation matrix T.

[0039] The formula for calculating the rotation operation on state s is as follows:

[0040]

[0041] The formula for calculating the rotation operation of action a is as follows:

[0042]

[0043] The augmented data obtained using the above method does not require interaction between the agent and the environment.

[0044] The augmented data obtained can be stored in the experience cache pool to participate in the next round of training.

[0045] Step 5: Update the agent policies π1, π2, ..., π using a multi-agent reinforcement learning algorithm. N The multi-agent reinforcement learning algorithm is used to update the policy network and action-value network using the samples collected in step 3.

[0046] The update process is described using the typical multi-agent reinforcement learning algorithm MADDPG as an example.

[0047] The loss function of the policy network is:

[0048]

[0049] in Let the loss function be the policy network. This is an experience cache pool.

[0050] For action value networks Calculate the loss function of the original MADDPG algorithm:

[0051]

[0052]

[0053] Where L MADDPG (φ) represents the loss of the global action value network. For the experience cache pool, y represents The target value is γ, which is a discount factor typically set to 0.99.

[0054] Calculate the symmetric loss based on rotational symmetry:

[0055]

[0056] Action value network The loss function is:

[0057] L(φ)=L MADDPG (φ)+c m L sym (φ)

[0058] Where c m The weight for symmetric loss is typically set to 0.5.

[0059] Symmetry is inherent in multi-agent systems, and traditional multi-agent reinforcement learning algorithms learn this property through samples generated by the interactions between agents and the environment. This invention transforms the symmetry of multi-agent systems into a symmetric loss L. sym (φ) indicates that the agent is penalized by the loss term when the learned policy does not conform to symmetry. The modified loss function provides more information to guide the agent's training. Essentially, it directly tells the agent this property, eliminating the need for the agent to learn from data.

[0060] Repeating steps 4 and 5 will perform multiple rounds of training on the multi-agent policy network and action-value network. If augmentation samples are stored in the experience cache, they will also be sampled in the next round of training to train the multi-agent reinforcement learning network.

[0061] Step 6: After training is completed, the policy network parameters are determined. In the decision-making phase, the action value network is removed. The agent will select actions to output to the environment based on the local observation input to its own policy network, thereby interacting with the environment. This is no different from the conventional approach and will not be elaborated here.

[0062] Thus, this invention realizes a method to accelerate the training of multi-agent reinforcement learning by calculating symmetric loss without additional interaction with the environment, and this method can be embedded into any mainstream multi-agent reinforcement framework.

Claims

1. A multi-agent reinforcement learning training method based on rotational symmetry, characterized in that, include: (1) Based on the global symmetry of the multi-agent system, perform rotation transformation on the original samples to obtain enhanced samples; Suppose we obtain the original sample Where s is the current state of the multi-agent system in the environment, a is the action vector of the agent obtained by the multi-agent policy network, and r and Each agent receives a reward after executing its own action vector, and the state of the multi-agent in the environment after executing the action is also recorded. According to the preset rotation angle Obtain the rotation matrix , s, a, Multiply each by the rotation matrix T to obtain the rotation. Let the state and action after the angle be obtained respectively. , , Then, enhanced samples are obtained. ; Among them, the state The formula for calculating the rotation operation is: ; Action The formula for calculating the rotation operation is: ; (2) When training the agent policy network and action value network, calculate the symmetric loss based on rotational symmetry. Add the symmetric loss to the original action value network loss of the multi-agent reinforcement learning algorithm to obtain the loss function of the new action value network. ,as follows: in, For the parameters of the action value network, The loss of the original action-value network representing the multi-agent reinforcement learning algorithm used. For symmetric loss weights, Represents the symmetry loss based on rotational symmetry; The calculation is as follows: in, For experience cache pool; It is a multi-agent strategy In state s, the expected cumulative discount reward that can be obtained by performing action a; Multi-agent strategy In state At that time, perform the action The expected cumulative discount rewards that can be obtained.

2. The method according to claim 1, characterized in that, The multi-agent reinforcement learning training method includes the following steps: Step 1: Initialize the policy network parameters, action value network parameters, and symmetric loss weights of the multi-agent system; Step 2: Obtain the current state of each agent in the environment, use the current policy network of each agent to obtain the action vector of each agent, execute the respective action vector, obtain the reward after the action is executed and the agent state after the action is executed, and generate the original sample. ; Step 3: Obtain the set number of raw samples and store them in the experience cache pool; Step 4: Randomly collect samples from the experience buffer pool and perform rotation transformation on the collected samples; Step 5: Use the samples collected in Step 3 to train the policy network and the action value network respectively, and use the loss function of the new action value network to update the action value network. Step 6: After training is completed, in the decision-making phase, each agent selects actions based on local observations and its own policy function and outputs them to the environment, thus interacting with the environment.

3. The method according to claim 2, characterized in that, Step 4 involves storing the enhanced samples obtained after the rotation transformation into the experience cache pool to participate in the next round of training. Steps 4 and 5 are executed in each round of training.

Images