A multi-agent reinforcement learning method for decoupling exploration and exploitation strategies

Abstract

The application provides a multi-agent reinforcement learning method for decoupling exploration and utilization strategies, comprising the following steps: S1, initializing an exploration strategy and a utilization strategy; S2, sampling interaction with an environment using the exploration strategy, and updating the exploration strategy; S3, putting the sampling samples of the exploration strategy into a buffer; S4, training the utilization strategy using the data in the buffer, and updating the utilization strategy based on cumulative returns; and S5, using the trained utilization strategy for decision-making in a multi-agent environment. The application can better balance the exploration and utilization trade-off by decoupling the exploration and utilization strategies. Through decoupling, the agent can flexibly adjust the exploration degree according to the situation, avoid excessive exploration interference with the behavior of other agents, and achieve better collaboration and task completion effect.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and more specifically, to a multi-agent reinforcement learning method that decouples exploration and exploitation strategies. Background Technology

[0002] In recent years, with the development of deep learning, multi-agent reinforcement learning has been widely researched and applied in various fields such as autonomous driving, smart grids, robot control, and multiplayer games. These fields share the common characteristic of requiring multiple agents to collaborate to complete complex tasks. This necessitates that agents understand the behavior of other agents, coordinate actions, and share knowledge and information. However, this also brings a series of challenges, among which the exploration and utilization of multi-agent scenarios is a key aspect.

[0003] In reinforcement learning, problem exploration is a key challenge. Agents need to explore unknown states and actions while attempting to maximize long-term rewards in order to better understand the environment and formulate better strategies. The core of this problem lies in how to balance exploration and exploitation—that is, finding a balance between trying new actions and utilizing known actions. To address this, intrinsic rewards can be introduced to incentivize agents to explore unknown environments.

[0004] Intrinsic reward is a special reward signal in reinforcement learning, used to encourage agents to actively explore unknown states and actions during the learning process in order to better understand the environment and acquire more information. Burda Y and Edwards H (2018) proposed stochastic network distillation, defining intrinsic exploration reward as the error between the state features predicted by the predictor network and the state features of the target network. Ju-Bong Kim (2022) proposed quantifying the novelty of the agent's state by the difference between the current state and the previous state in multi-agent scenarios. Although intrinsic reward can motivate agents to explore to some extent, it also has some instability and sensitivity issues, limiting its effectiveness in complex multi-agent environments. First, the use of intrinsic reward may lead to non-stationary reward signals. This is because intrinsic reward is usually designed to gradually decrease as exploration progresses to encourage agents to explore more in the initial stage. This decreasing pattern may lead to unstable changes in the reward signal, thus affecting the agent's learning process. Second, the exploration of intrinsic reward is very sensitive to the decay rate on which it depends throughout the training process. The choice of decay rate may affect the balance between exploration and exploitation by the agent, thus affecting the learning effect. However, finding a suitable decay rate is often a complex problem that requires a lot of trial and error and adjustments. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-agent reinforcement learning method that decouples exploration and utilization strategies.

[0006] The present invention aims to address the problems in existing technologies where intrinsic reward exploration leads to non-stationary rewards and the intrinsic reward's dependence on decay rate is highly sensitive.

[0007] Compared with the prior art, the technical solution of the present invention and its beneficial effects are as follows:

[0008] A multi-agent reinforcement learning method decoupling exploration and exploitation strategies includes: S1, initializing an exploration strategy and an exploitation strategy; S2, using the exploration strategy to perform interactive sampling with the environment and updating the exploration strategy; S3, placing the sampled data of the exploration strategy into a buffer; S4, training the exploitation strategy using the data in the buffer and updating the exploitation strategy based on cumulative rewards; S5, the agents using the trained exploitation strategy to make decisions in the multi-agent environment.

[0009] As a further improvement, in step S1, the exploration strategy is: The goal of the exploration strategy is to conduct thorough exploration in a multi-agent environment in order to discover unknown information; the utilization strategy is... The goal of the exploitation strategy is to optimize the long-term performance of the agent by maximizing the cumulative reward; where n represents the number of agents in the multi-agent system.

[0010] As a further improvement, step S2 includes: S21, obtaining the global state s of the multi-agent environment. t Each agent, based on the global state s t Actions are sampled from the exploration policy network using the softmax function to generate a probability distribution of actions; the actions of all agents are combined into a joint action a for the multi-agent system. t S22, the multi-agent system executes a joint action. t And interact with the environment to proceed to the next time step; in a single time step, each agent interacts with the environment to obtain the next global state s. t+1 and external rewards S23, Calculate the intrinsic reward based on the global state prediction error. S24, using the intrinsic reward Update the exploration strategy network π β .

[0011] As a further improvement, in step S23, the intrinsic reward Calculated using MSE, it is expressed as: in, For the predicted state, s t This represents the original global state.

[0012] As a further improvement, in step S24, the policy network π is explored. β , is represented as:

[0013]

[0014] in, To maximize the cumulative reward of intrinsic incentives.

[0015] As a further improvement, in step S3, the sampling samples of the exploration strategy include: the global state s of the multi-agent environment. t Joint actions of multi-agent systems a t External rewards obtained and the next global state s t+1 That is, the sampling samples of the exploration strategy.

[0016] As a further improvement, step S4 includes: S41, randomly selecting a batch of samples from the buffer, the samples including the global state of the multi-agent environment, joint actions generated by the exploration strategy, and external rewards obtained from the environment; S42, using the samples as input data to update the exploitation policy network; predicting the probability distribution of the corresponding joint actions through the trained exploitation policy network, and minimizing the loss using optimization algorithms such as gradient descent; S43, repeating steps S41 and S42, updating the exploitation policy network multiple times using different batches of samples, fine-tuning the weights of the exploitation policy network, and improving the performance of the exploitation strategy.

[0017] As a further improvement, in step S42, the policy network π is utilized. e , is represented as:

[0018]

[0019] in, To maximize the cumulative return from external rewards.

[0020] As a further improvement, step S5 includes: the agent using the trained exploitation strategy to select and execute actions in a multi-agent environment, conducting tests and obtaining results.

[0021] The beneficial effects of this invention are as follows:

[0022] This invention decouples exploration and exploitation strategies, enabling agents to better balance the trade-offs between exploration and exploitation. Through decoupling, agents can flexibly adjust the degree of exploration as needed, avoiding excessive exploration from interfering with the behavior of other agents, thus achieving better collaboration and task completion.

[0023] This invention employs a phased training approach, involving exploration and utilization strategies. This allows the agent to experiment more freely with different actions under the exploration strategy, while focusing more on optimization and fine-tuning under the utilization strategy. This phased training method reduces volatility and instability in the overall training process, enabling the agent to learn and adapt more reliably.

[0024] This invention effectively addresses the trade-off between exploration and exploitation in multi-agent systems, thereby enabling agents to learn and adapt to complex environments more effectively and improve the performance of multi-agent systems. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of a multi-agent reinforcement learning method that decouples exploration and utilization strategies, provided by an embodiment of the present invention.

[0026] Figure 2 This is a schematic diagram of the algorithm architecture provided in an embodiment of the present invention.

[0027] Figure 3 This is a schematic diagram of an application scenario provided by an embodiment of the present invention.

[0028] Figure 4 This is a schematic diagram of the test results provided in an embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0030] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0031] Reference Figure 1 and Figure 2 As shown, a multi-agent reinforcement learning method that decouples exploration and exploitation strategies includes:

[0032] S1, Initialize the exploration and utilization strategies;

[0033] S2, use the exploration strategy to perform interactive sampling with the environment, and update the exploration strategy;

[0034] S3, Place the sampled data from the exploration strategy into the buffer;

[0035] S4, the exploitation strategy is trained using the data in the buffer and updated based on the cumulative reward;

[0036] S5, the agent makes decisions in a multi-agent environment using the trained exploitation strategy.

[0037] During training, the agent first uses an exploration strategy to sample in a multi-agent environment, puts the sampling results into a buffer, then uses the sampled samples to train the strategy, and finally uses the trained strategy to make decisions.

[0038] In step S1

[0039] The exploration strategy is as follows: The goal of the exploration strategy is to conduct thorough exploration in a multi-agent environment in order to discover unknown information;

[0040] The exploitation strategy is as follows: The goal of the exploitation strategy is to optimize the long-term performance of the agent by maximizing cumulative rewards;

[0041] Where n represents the number of agents in a multi-agent system.

[0042] The input to the policy network for exploration and the policy network for utilization is the environment state, and the output is the probability distribution of actions.

[0043] Using a dual reward mechanism, the reward signal for exploring the policy is the intrinsic reward that encourages the agent to try new actions, while the reward signal for utilizing the policy is the cumulative reward that multiple agents obtain in the environment.

[0044] Step S2 includes:

[0045] S21, For each agent, obtain the global state s of the multi-agent environment based on its current individual state. t Each agent, based on the global state s t Actions are sampled from the exploration policy network using the softmax function to generate a probability distribution of actions; the actions of all agents are combined into a joint action a for the multi-agent system. t That is, a set of coordinated actions;

[0046] S22, the multi-agent system executes joint action a t And interact with the environment to proceed to the next time step; in a single time step, each agent interacts with the environment to obtain the next global state s. t+1 and external rewards To provide data support for subsequent learning;

[0047] S23, To encourage the agent to explore more unknown areas, an intrinsic reward is calculated based on the global state prediction error. Intrinsic rewards, as a signal of curiosity, encourage agents to try new actions in order to better understand their environment;

[0048] S24, using the intrinsic reward Update the exploration strategy network π β By using the gradient descent optimization algorithm, the parameters of the neural network are adjusted to better balance exploration and exploitation, thereby maximizing the cumulative reward of intrinsic value.

[0049] By interacting with the environment, sampling is performed using an exploration strategy. At each time step, the global state of the multi-agent environment is acquired, and each agent generates a joint action by sampling from its exploration strategy based on this global state. The multi-agents execute these joint actions to enter the next global state and receive an external reward. Furthermore, an intrinsic reward is calculated using the global state and joint actions to guide the exploration strategy. The intrinsic reward is calculated based on the state prediction error and helps balance exploration and exploitation. The exploration strategy is updated using gradient descent to maximize the intrinsic reward.

[0050] In step S23, to guide exploration, an intrinsic reward is calculated. This intrinsic reward is based on the state prediction error and aims to encourage the agent to try new actions in unknown areas to better explore the environment. Calculated using MSE, it is expressed as:

[0051]

[0052] in, For the predicted state, s t This represents the original global state.

[0053] In step S24

[0054] Explore the policy network π β , is represented as:

[0055]

[0056] in, To maximize the cumulative reward of intrinsic incentives.

[0057] In step S3, the sampling samples of the exploration strategy include: the global state s of the multi-agent environment. t Joint actions of multi-agent systems a t External rewards obtained and the next global state s t+1 ;

[0058] That is, the sampling samples of the exploration strategy.

[0059] To further decouple the exploration and exploitation strategies, sampled data from the exploration strategy is placed in a buffer for use in subsequent exploitation strategy training. These samples include states, joint actions, and acquired external rewards in the multi-agent environment. Storing these samples in the buffer helps stabilize the training process, reduces sample correlation, and thus better supports the learning of the exploitation strategy.

[0060] Step S4 includes:

[0061] S41, randomly select a batch of samples from the buffer, including the global state of the multi-agent environment, joint actions generated by the exploration strategy, and external rewards obtained from the environment, to better guide the training to utilize the strategy;

[0062] S42, using samples as input data, update the exploitation policy network; predict the probability distribution of the corresponding joint action through the trained exploitation policy network, and use optimization algorithms such as gradient descent to minimize the loss; use the gradient descent optimization algorithm to maximize the cumulative reward of external rewards and gradually improve the performance of the exploitation policy.

[0063] S43, Repeat steps S41 and S42, updating the exploitation policy network multiple times using different batches of samples. During each iteration, fine-tune the weights of the exploitation policy network to gradually improve the performance of the exploitation policy. This process allows the agent to learn from experience and gradually improve its exploitation policy, enabling it to collaborate and make decisions more effectively in multi-agent environments to maximize cumulative rewards.

[0064] During the training phase of the policy, a batch of samples is randomly sampled from the experience buffer. These samples include the global state and joint actions, along with their associated extrinsic rewards. Using these samples, the policy-enabled neural network is trained to make better decisions based on existing knowledge and experience. The training process uses the policy gradient descent algorithm to maximize the cumulative reward.

[0065] The strategy involves sampling from the buffer and using the sampled data. Conduct training to maximize cumulative rewards.

[0066] In step S42, the sampled samples are used as input to apply strategy π. e To perform the update, utilize the policy network π. e , is represented as:

[0067]

[0068] in, To maximize the cumulative return from external rewards.

[0069] Step S5 includes:

[0070] The agent uses the trained exploitation strategy to select and execute actions in a multi-agent environment, performs tests, and obtains results.

[0071] After the exploit policy is trained, the trained exploit policy neural network and corresponding weight files are used to make practical decisions in a multi-agent environment. Each agent can sample and generate actions from the exploit policy based on the current environmental state.

[0072] The principle of the multi-agent reinforcement learning method for decoupling exploration and utilization strategies provided by this invention is as follows:

[0073] Initialize the exploration and exploitation strategies. Each agent has its own exploration and exploitation strategies; the exploration strategy is used to actively explore the environment, and the exploitation strategy is used to optimize performance.

[0074] By interacting with the multi-agent environment, the agents sample using exploration strategies; based on the current global state, they select actions from their respective exploration strategies; these joint actions enable the multi-agent system to enter a new state and obtain external rewards. Intrinsic rewards are calculated to guide the exploration strategies toward unknown directions;

[0075] To further separate exploration and exploitation, samples generated by the exploration strategy are stored in a buffer; these samples include states, joint actions, and external rewards, which help in the subsequent training of exploitation strategies.

[0076] Samples are randomly sampled from the experience buffer; these samples are used to train the exploitation policy; the goal of the exploitation policy is to maximize the overall reward by optimizing existing knowledge, so that the agent can make more accurate decisions.

[0077] After training, the trained exploitation strategy is used to make actual decisions; the agent selects actions from the exploitation strategy to execute based on the current environmental state.

[0078] Figure 3 In an application scenario where the method of this invention is used, the goal of n agents is to get as close as possible to the random locations occupied by their corresponding n landmarks, while avoiding mutual conflicts. The agents can move in four directions or remain stationary. Figure 3 In the task shown, n is 4.

[0079] Figure 4 The results show the test results of the method of this invention in a multi-agent application scenario. Training and testing were performed on a single 2080Ti graphics card. The results demonstrate that the method achieves a 97.78% success rate in occupying three landmarks after 25 steps with three agents, and a 69.17% success rate in occupying four landmarks after 25 steps with four agents.

[0080] The above embodiments are only used to explain the technical solutions of the present invention and are not intended to limit it. Those skilled in the art should understand that any modifications and equivalent substitutions that do not depart from the spirit and scope of the present invention should fall within the protection scope of the claims of the present invention.

Claims

1. A multi-agent reinforcement learning method that decouples exploration and exploitation strategies, characterized in that, include: S1, Initialize the exploration and utilization strategies; S2, use the exploration strategy to perform interactive sampling with the environment, and update the exploration strategy; S3, Place the sampled data from the exploration strategy into the buffer; S4, the exploitation strategy is trained using the data in the buffer and updated based on the cumulative reward; S5, the agent makes decisions in a multi-agent environment using the trained exploitation strategy; Step S2 includes: S21, Obtain the global state of the multi-agent environment. Each agent, based on the global state Actions are sampled from the exploration strategy using the softmax function to generate a probability distribution of actions; the actions of all agents are then combined into a joint action of the multi-agent system. ; S22, the multi-agent system executes a joint action. And interact with the environment to proceed to the next time step; in a single time step, each agent interacts with the environment to obtain the next global state. and external rewards ; S23, Calculate the intrinsic reward based on the global state prediction error. ; S24, using the intrinsic reward Update the exploration strategy ; In step S23, intrinsic reward Calculated using MSE, it is expressed as: in, For the predicted state, This represents the original global state. In step S24, the exploration strategy is... , is represented as: in, To maximize the cumulative return of intrinsic rewards; Step S4 includes: S41, randomly select a batch of samples from the buffer, the samples include the global state of the multi-agent environment, joint actions generated by the exploration strategy, and external rewards obtained from the environment; S42, using samples as input data, update the exploitation strategy; predict the probability distribution of the corresponding joint action through the trained exploitation strategy, and use the gradient descent optimization algorithm to minimize the loss; S43, Repeat steps S41 and S42, update the exploitation strategy multiple times using different batches of samples, fine-tune the weights of the exploitation strategy, and improve the performance of the exploitation strategy; In step S42, the strategy is used. , is represented as: in, To maximize the cumulative return from external rewards.

2. The multi-agent reinforcement learning method for decoupling exploration and utilization strategies according to claim 1, characterized in that, In step S1, The exploration strategy is as follows: The goal of the exploration strategy is to conduct thorough exploration in a multi-agent environment in order to discover unknown information; The exploitation strategy is as follows: The goal of the exploitation strategy is to optimize the long-term performance of the agent by maximizing cumulative rewards; Where n represents the number of agents in a multi-agent system.

3. The multi-agent reinforcement learning method for decoupling exploration and utilization strategies according to claim 1, characterized in that, In step S3, the sampling samples of the exploration strategy include: the global state of the multi-agent environment. Coordinated actions of multi-agent systems External rewards obtained and the next global state ; That is, the sampling samples of the exploration strategy. .

4. The multi-agent reinforcement learning method for decoupling exploration and utilization strategies according to claim 1, characterized in that, Step S5 includes: The agent uses the trained exploitation strategy to select and execute actions in a multi-agent environment, performs tests, and obtains results.

Images