A multi-agent reinforcement learning method based on option-value decomposition

Abstract

The application discloses a multi-agent reinforcement learning method based on option-value decomposition, through a multi-agent option-value framework (OMIX), defines the action strategy of each agent as an option, the agent can select an option to imitate the action strategy of the corresponding agent, stops imitation through an option termination function, which can integrate other agent strategies, increase the exploration range and sample utilization rate; meanwhile, CTDE is used to realize multi-agent cooperation, global information is used in the training process, and is dispersedly arranged in the execution process; an optimistic heuristic method is integrated into option learning to prevent the agent from falling into local optimization in a sparse reward environment.

Description

Technical Field

[0001] This invention relates to the field of deep reinforcement learning technology, and specifically to a multi-agent reinforcement learning method based on option-value decomposition. Background Technology

[0002] Multi-agent reinforcement learning (MARL) is an important branch of reinforcement learning. Existing techniques mainly fall into two categories: the CTDE framework and the hierarchical MARL framework. These existing techniques suffer from the following main drawbacks:

[0003] The existing technology has the following main drawbacks:

[0004] (1) Low sample utilization. For large-scale multi-agent systems (such as those with more than dozens of agents), the state-action space of CTDE methods (such as QMIX and VDN) grows exponentially with the number of agents, resulting in complex training process and low sample utilization efficiency. These methods are difficult to converge within a reasonable time, and the space required to store these samples increases dramatically.

[0005] (2) Training is difficult in sparse reward or delayed reward environments. In long-term collaborative tasks, agents need to go through a long sequence of actions to obtain rewards. It is difficult for agents to obtain timely and effective feedback from the current action, resulting in slow learning speed or premature convergence in complex environments, getting stuck in local optima and failing to explore potentially better sequential actions. Existing hierarchical methods attempt to alleviate this problem by decomposing sub-goals, but often require manually defining sub-goals or relying on domain-specific hyperparameters (such as termination conditions), which limits generality and scalability.

[0006] (3) Communication and information dependency problem: Existing hierarchical methods assume a completely decentralized scenario, but require explicit communication or public information broadcasting between agents. This may be impractical or introduce additional overhead in real applications, increasing model complexity and the risk of overfitting.

[0007] In summary, existing technologies suffer from drawbacks such as low training efficiency, high difficulty in collaboration, and susceptibility to local optima when dealing with multi-agent collaboration, especially when facing sparse rewards and large-scale agents. Summary of the Invention

[0008] Purpose of the Invention: The purpose of this invention is to provide a multi-agent reinforcement learning method based on option-value decomposition. Through the Multi-Agent Option-Value Framework (OMIX), each agent's action policy is defined as a series of options. An agent can choose an option to mimic the action policy of its corresponding agent, and the mimicry stops through an option termination function. This approach can integrate the policies of other agents, increasing the exploration scope and sample utilization. Simultaneously, CTDE is used to achieve multi-agent collaboration, utilizing global information during training and distributed deployment during execution. An optimistic heuristic is integrated into option learning to prevent agents from getting trapped in local optima under sparse reward environments, thus solving the problems of high training complexity, learning difficulties under sparse rewards, and susceptibility to local optima in existing technologies.

[0009] Technical solution: The multi-agent reinforcement learning method based on option-value decomposition described in this invention includes the following steps:

[0010] (1) Define an option set for each agent, which enables the agent to select and temporarily mimic the behavior strategies of other agents;

[0011] (2) Initialize the neural network parameters of each agent, the parameters of the centralized hybrid network, and the experience replay buffer;

[0012] (3) The agent interacts with the environment based on the currently selected option and its corresponding internal action strategy, and stores the data generated by the interaction into the experience replay buffer;

[0013] (4) Introduce an optimistic heuristic mechanism into the action selection process of each agent, and bias the current action selection probability by maintaining and utilizing the reward statistics of historical actions, so as to encourage exploration and avoid the strategy from getting stuck in local optima.

[0014] (5) Manage the start and end process of each agent's option. When the agent's current option ends according to its termination condition, the agent reselects a new option.

[0015] (6) Using data sampled from the experience replay buffer, the option value functions of each agent are nonlinearly monotonically fused through a centralized hybrid network to calculate the team joint value function, and thereby train and update the parameters of the option value network of each agent and the hybrid network.

[0016] (7) Based on the calculated team joint value function, update the parameters of the option internal action policy network and option termination function network of each agent using the policy gradient method;

[0017] (8) Update the target network parameters regularly to stabilize the training process.

[0018] Furthermore, in step (1), each option in the option set corresponds to the ability to mimic the behavioral strategy of another specific agent in the system.

[0019] Furthermore, in step (3), the agent selects the currently executed option using an exploratory strategy based on its own option value function.

[0020] Furthermore, in step (4), the optimistic heuristic mechanism specifically involves maintaining the historical maximum reward value and the frequency of obtaining the maximum reward for different actions of the agent in different states, and calculating an optimistic factor based on this, and incorporating the optimistic factor into the calculation of the action selection probability, so that the probability of actions with better historical performance being selected in the future increases.

[0021] Furthermore, in step (5), the termination of each option is determined by the termination function corresponding to the option. When the value of the termination function exceeds a preset threshold, the current option terminates.

[0022] Furthermore, in step (6), the centralized hybrid network is constrained to have non-negative weights to ensure that the fusion of the value functions of each agent's options is monotonic, thereby ensuring the consistency between the decentralized execution strategy and the centralized training objective.

[0023] Furthermore, in step (6), when calculating the team joint value function, multiple scenarios are considered, including the absence of agent termination options in the next state and the partial termination options of agents followed by the selection of new options.

[0024] Furthermore, in step (7), a term based on the advantage function and a policy entropy regularization term used to encourage exploration are introduced to calculate the gradient of the network parameters of the update option termination function.

[0025] Furthermore, in step (8), the online parameters of the agent's option value network and centralized hybrid network are periodically copied to the corresponding target network.

[0026] Furthermore, it also includes: the entire training process adopts a centralized training and distributed execution framework, during training, global state information is used to optimize all network parameters, and during execution, each agent makes independent decisions based only on its own local observations and selected options.

[0027] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: In complex cooperative scenarios with multiple global optima (such as penalty games), existing non-hierarchical and hierarchical algorithms are prone to getting stuck in severe local optima due to cooperation failures between agents. This invention, by integrating optimistic heuristics into its in-option policy, guides agents to favor actions that have previously yielded high rewards, thereby stably learning globally optimal joint policies. In sparse reward environments such as "cooperative navigation," where rewards are only provided upon task completion, this invention, through an end-to-end option-value decomposition framework, synchronously and collaboratively learns high-level option policies and low-level action policies based on the same reward signal, achieving more stable and efficient credit allocation and policy optimization, thus converging to better performance faster under sparse reward conditions. Other hierarchical algorithms need to learn high and low-level network structures simultaneously, leading to unstable training. This invention, by introducing option-time abstraction, decomposes decision-making into high-level option selection and low-level action execution, effectively reducing the learning difficulty. Meanwhile, its hybrid network based on monotonicity constraints for option value decomposition ensures that while using global state information for centralized training, it can derive efficient distributed execution strategies, thereby achieving better coordination capabilities and overall performance than pure hierarchical methods and pure value decomposition methods in complex multi-agent collaborative tasks. Attached Figure Description

[0028] Figure 1 This is a flowchart of the present invention;

[0029] Figure 2 This is the core architecture of the OMIX framework of this invention. Detailed Implementation

[0030] This invention provides a multi-agent reinforcement learning method based on option-value decomposition. This technical solution achieves efficient and robust multi-agent collaboration by combining the CTDE option-value framework, value decomposition, and optimistic heuristics. It includes the following steps:

[0031] Step 1: Initialize all components and parameters necessary for the entire OMIX framework.

[0032] Step 1-1: Define the set of options available to each agent, with the number of elements in the option set equal to the number of other agents.

[0033] Step 1-2: Initialize the neural network parameters for each agent, including: action policy parameters within the options, termination function parameters within the options, and value network parameters within the options.

[0034] Steps 1-3: Initialize the parameters of the centralized hybrid network and its target network.

[0035] Steps 1-4: Create an experience replay buffer to store historical data (states, actions, rewards, next state, etc.) generated by the agent's interaction with the environment.

[0036] Steps 1-5: Set training hyperparameters, including the sampling batch size of the replay buffer and the update frequency of the target network.

[0037] Step 2: Agent interacts with the environment and collects data. In each training round, the agent interacts with the environment according to the current option policy and the action policy within the option, and stores the interaction data in the buffer.

[0038] Step 2-1: Reset the environment and set the initial state.

[0039] Step 2-2: For each agent, according to - A greedy strategy selects an option based on its own option value function.

[0040] Steps 2-3: For each agent, select a specific action to execute based on its current action policy and in conjunction with an optimistic heuristic (see Step 4 for details).

[0041] Steps 2-4: After all agents have performed the joint action, the environment returns the team reward and the next global state.

[0042] Steps 2-5: Store the tuple of this interaction (current state, options and actions of each agent, team reward, next state) into the experience replay buffer.

[0043] Step 3: Optimistic heuristic update of action policy within options. To prevent the agent from getting stuck in local optima, an optimistic heuristic bias is introduced when the agent selects an action.

[0044] Step 3-1: Use a hash function to discretize the high-dimensional continuous state space into hash codes to facilitate tracking the statistics of each state-action pair.

[0045] Step 3-2: For the intelligent agent Each state-action pair maintains two values: (The maximum reward obtained so far by performing this action in this state) and (The frequency is the number of times the action is performed in this state to obtain the maximum reward.)

[0046] Step 3-3: Calculate the optimism factor = And normalize it to the (0,1) interval.

[0047] Steps 3-4: Action strategy within the agent's options In the Softmax output, this optimism factor is added as a bias to form a new action selection probability distribution:

[0048]

[0049] Where τ is a temperature parameter used to control the weights of the optimistic heuristic.

[0050] Step 4: Manage the activation and deactivation of each agent option.

[0051] Step 4-1: At each time step, the agent performs an action according to the action policy within its options until the value of the option's termination function exceeds a threshold.

[0052] Step 4-2: When the option is terminated, the agent will... - The greedy strategy selects a new option and begins executing the strategy within that new option.

[0053] Step 5: Training the value network and hybrid network. Sample data from the experience replay buffer and train the option value network and hybrid network in an end-to-end manner.

[0054] Step 5-1: Calculate the value function of the action performed by each agent under the corresponding option based on the sampled data:

[0055]

[0056] Step 5-2: Calculate the option value function for each agent based on the action selection probability and the action value function:

[0057]

[0058] Step 5-3: Through a hybrid network, monotonically mix the option value functions of each agent to obtain the joint option value for reaching the goal. and update the joint option value function. The weights of the hybrid network are generated by the supernetwork and are constrained by non-negativity to ensure monotonicity.

[0059]

[0060] in It is the set of agents that will terminate the current option in the next state. It is the exponentiation operator except for the empty set. It is the probability that no agent will terminate the current option in the next state.

[0061]

[0062] in It's a team reward.

[0063] Step 5-4: Update the option value network parameters and the parameters of the centralized hybrid network for each agent by minimizing the mean squared error loss function. The loss function is:

[0064]

[0065] in This is the batch size of the replay buffer samples. .

[0066] Step 6: Training the policy network and termination network. Using the joint value function calculated in Step 5, update the policies of each agent through the policy gradient method.

[0067] Step 6-1: Calculate the gradient of the policy option for each agent:

[0068]

[0069] Step 6-2: Calculate the gradient of the termination function for each agent's option:

[0070]

[0071] in It is the dominance function.

[0072] Meanwhile, to encourage exploration, a regularization term for policy entropy is added to the gradient update. .

[0073] Step 7: Target network update. Periodically copy the parameters of the online network to the target network to stabilize the training process.

[0074] Step 7-1: After every K training steps, copy the parameters of the agent's option value network and hybrid network to the corresponding target network.

[0075] Figure 2 The document showcases the core architecture of the OMIX framework, a multi-agent hierarchical reinforcement learning model based on option-value decomposition. This framework employs a paradigm of centralized training and distributed execution, topped by a centralized hybrid network that receives the global state S and option-value functions from all agents. And obtain the joint option value function. The weights of the hybrid network are constrained to be non-negative to ensure the monotonicity of the value function decomposition.

[0076] At the bottom of the framework is a distributed agent option-value network, typically composed of MLP and GRU modules, which are responsible for processing their own local observations and option information, and outputting the option value function for each agent. Simultaneously, it outputs the actions within each agent's options. Each agent performs local computations, and the results are fed into a central hybrid network for global joint value estimation, thereby achieving effective credit allocation during the training phase and ultimately learning an efficient cooperative strategy.

Claims

1. A multi-agent reinforcement learning method based on option-value decomposition, characterized in that, Includes the following steps: (1) Define an option set for each agent, which enables the agent to select and temporarily mimic the behavior strategies of other agents; (2) Initialize the neural network parameters of each agent, the parameters of the centralized hybrid network, and the experience replay buffer; (3) The agent interacts with the environment based on the currently selected option and its corresponding internal action strategy, and stores the data generated by the interaction into the experience replay buffer; (4) Introduce an optimistic heuristic mechanism into the action selection process of each agent, and bias the current action selection probability by maintaining and utilizing the reward statistics of historical actions, so as to encourage exploration and avoid the strategy from getting stuck in local optima. (5) Manage the start and end process of each agent's option. When the agent's current option ends according to its termination condition, the agent reselects a new option. (6) Using data sampled from the experience replay buffer, the option value functions of each agent are nonlinearly monotonically fused through a centralized hybrid network to calculate the team joint value function, and thereby train and update the parameters of the option value network of each agent and the hybrid network. (7) Based on the calculated team joint value function, update the parameters of the option internal action policy network and option termination function network of each agent using the policy gradient method; (8) Update the target network parameters regularly to stabilize the training process.

2. The multi-agent reinforcement learning method based on option-value decomposition according to claim 1, characterized in that, In step (1), each option in the option set corresponds to the ability to mimic the behavior strategy of another specific agent in the system.

3. The multi-agent reinforcement learning method based on option-value decomposition according to claim 1, characterized in that, In step (3), the agent selects the currently executed option using an exploratory strategy based on its own option value function.

4. The multi-agent reinforcement learning method based on option-value decomposition according to claim 1, characterized in that, In step (4), the optimistic heuristic mechanism is as follows: maintain the historical maximum reward value and the frequency of obtaining the maximum reward for different actions of the agent in different states, and calculate an optimistic factor based on this, and incorporate the optimistic factor into the calculation of the action selection probability, so that the probability of the action with better historical performance being selected in the future increases.

5. A multi-agent reinforcement learning method based on option-value decomposition according to claim 1, characterized in that, In step (5), the termination of each option is determined by the termination function corresponding to the option. When the value of the termination function exceeds the preset threshold, the current option terminates.

6. A multi-agent reinforcement learning method based on option-value decomposition according to claim 1, characterized in that, In step (6), the centralized hybrid network is constrained to have non-negative weights to ensure that the fusion of the value functions of each agent's options is monotonic, thereby ensuring the consistency between the decentralized execution strategy and the centralized training objective.

7. A multi-agent reinforcement learning method based on option-value decomposition according to claim 1, characterized in that, In step (6), when calculating the team joint value function, multiple scenarios are considered, including no agent termination option in the next state, and some agents termination options followed by new selections.

8. A multi-agent reinforcement learning method based on option-value decomposition according to claim 1, characterized in that, In step (7), a term based on the advantage function and a policy entropy regularization term used to encourage exploration are introduced to calculate the gradient of the network parameters of the update option termination function.

9. A multi-agent reinforcement learning method based on option-value decomposition according to claim 1, characterized in that, In step (8), the online parameters of the agent's option value network and centralized hybrid network are periodically copied to the corresponding target network.

10. A multi-agent reinforcement learning method based on option-value decomposition according to claim 1, characterized in that, It also includes: the entire training process adopts a centralized training and distributed execution framework. During training, global state information is used to optimize all network parameters, and during execution, each agent makes independent decisions based only on its own local observations and selected options.

Images