A multi-agent intrinsic motivation framework for joint intrinsic motivation exploration

Abstract

The present invention discloses a multi-agent intrinsic motivation framework for joint intrinsic motivation exploration, and proposes a joint intrinsic motivation exploration algorithm JIME; JIME includes two components: an encoder for compressing multi-agent joint observations and a decoder for predicting joint actions of agents; first, the environment state is input into the encoder to obtain the generated state, then input into the decoder to obtain the predicted action, and finally the loss function is calculated, and the parameters of the encoder and decoder are updated through the back propagation algorithm to minimize the total loss and obtain the final JIME. The present invention provides a new intrinsic motivation exploration method, which can enhance the QMIX algorithm by integrating two types of rewards, and can solve the multi-agent intrinsic motivation framework of joint intrinsic motivation exploration of intelligent agent learning strategies for given tasks.

Description

Technical Field

[0001] The present invention addresses the challenge of sparse rewards in multi-agent reinforcement learning (MARL), particularly in scenarios where coordinated actions between agents are required, and provides a multi-agent intrinsic motivation framework for joint intrinsic motivation exploration. Background Art

[0002] Past methods include uncertainty-guided exploration and intrinsic motivation-guided exploration, but these methods have problems such as the difficulty of acquiring domain knowledge and the accuracy of learning models. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a new intrinsic motivation exploration method, which enhances the QMIX algorithm by integrating two reward types, and is capable of solving a multi-agent intrinsic motivation framework for joint intrinsic motivation exploration of intelligent agent learning strategies for a given task.

[0004] The objective of the present invention is achieved through the following technical solutions: a multi-agent intrinsic motivation framework for joint intrinsic motivation exploration, and a joint intrinsic motivation exploration algorithm, namely JIME, is proposed; JIME includes two components: an encoder for compressing multi-agent joint observations and a decoder for predicting joint actions of agents; the encoder extracts and compresses information in the environment state s, and the decoder restores the decision a made by the agent; the encoder includes a 3-layer fully connected neural network, and the activation function is Relu; the decoder includes a 3-layer fully connected neural network, and the activation function is Relu;

[0005] In each training iteration, we first transform the environment state Input to the encoder and get and represents the mean value of agent n mapped to z at time step t, represents the covariance of the mapping of agent n to z at time step t; using the reparameterization technique from and generate Then Input to the decoder to get the predicted action Finally, the loss function is calculated, and the parameters of the encoder and decoder are updated through the back-propagation algorithm to minimize the total loss and obtain the final JIME.

[0006] The loss function of JIME is as follows:

[0007]

[0008] Among them, θ is the parameter of JIME, and are sampled from interactive samples, which are the original environment states and actions; For the encoder, is the output of the decoder, is a normal distribution.

[0009] The beneficial effects of the present invention are as follows: the present invention proposes a new intrinsic motivation exploration method, which can solve the intelligent agent learning strategy of a given task by integrating two types of rewards to enhance the QMIX algorithm; the present invention focuses on additional rewards based on the current state and rewards of the subject, based on the observed environmental changes, forcing the agent to choose actions different from past events, and optimizing their joint strategy to promote enhanced exploration. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a schematic diagram of the JIME structure. DETAILED DESCRIPTION

[0011] This paper proposes a multi-agent intrinsic motivation framework called Joint Intrinsic Motivation Exploration (JIME) to improve cooperation by incentivizing strategic behaviors characterized by collective novelty among agents. The method provided by this paper plays a key role in solving tasks that require complex coordination to achieve optimal strategic execution, highlighting the importance of incorporating intrinsic motivation mechanisms in MARL systems to promote effective cooperation among agents.

[0012] This paper proposes a multi-agent intrinsic motivation framework called Joint Intrinsic Motivation Exploration (JIME) to improve the cooperation between agents in a multi-agent cooperative environment. is the observed state of the agent (collected images or sensor data).

[0013] The innovation points of the present invention are as follows: 1. A new intrinsic motivation exploration method is proposed, which enhances the QMIX algorithm by integrating two types of rewards, aiming to promote intelligent agent learning strategies that can solve given tasks; 2. Focusing on additional rewards based on the current state and reward of the subject, based on the observed environmental changes, forcing the agent to choose actions different from past events, and optimizing their joint strategies to promote enhanced exploration; (1) According to the specific environment, the environmental state faced by the agent and the measurement of decision-making are clarified; (2) The intrinsic motivation reward model is designed so that it depends not only on the current immediate reward, but also on the agent's state and past action choices; (3) The agent interacts with the environment to obtain trajectory data; (4) The intrinsic motivation reward model and the agent decision model are trained simultaneously based on the interaction data; (5) When evaluating the performance of the agent, if the requirements are met, the training is exited; if the requirements are not met, (3) and (4) are repeated.

[0014] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0015] A multi-agent intrinsic motivation framework for joint intrinsic motivation exploration, the Joint Intrinsic Motivation Exploration (JIME) algorithm is proposed to motivate MARL agents to explore the joint observation space.

[0016] The joint intrinsic motivation exploration algorithm, namely JIME, consists of two components: an encoder for compressing joint observations of multiple agents and a decoder for predicting joint actions of agents. The encoder extracts and compresses information in the environment state s so that it can be expressed using less information z; the decoder learns the decision a made by the agent from z to verify the validity of the compressed information z; the encoder contains 3 layers of fully connected neural networks, the activation function is Relu, the input dimension is the dimension of the environment state, and the output dimension is the state distribution of the dimension z. The encoder outputs a Gaussian distribution parameterized z. The decoder contains 3 layers of fully connected neural networks, the activation function is Relu, the input dimension is the dimension of the compressed information z, and the output dimension is the dimension of the action taken by the agent. The structure of JIME is as follows Figure 1 shown.

[0017] In each training iteration, we first transform the environment state Input to the encoder and get and Using the reparameterization trick and generate Then Input to the decoder to get the predicted action Finally, the loss function is calculated, and the parameters of the encoder and decoder are updated through the back-propagation algorithm to minimize the total loss and obtain the final JIME.

[0018] In JIME, Compressed into a potential space and represents the mean value of agent n mapped to z at time step t, represents the covariance of the mapping of agent n to z at time step t; sampling is done using the reparameterization trick The decoder takes the latent representation As input, and predict the target value

[0019] The optimization objective of JIME involves two subprojections: Specifically, KL-divergence measures and The compression degree of the value predictor network is the mean square error; therefore, the loss function of JIME is as follows:

[0020]

[0021] Among them, θ is the parameter of JIME, and are sampled from interactive samples, which are the original environment states and actions; For the encoder, is the output of the decoder, is a normal distribution.

[0022] KL-divergence is also used to calculate the intrinsic reward of the agent This makes it compatible with any MARL algorithm that follows the CTDE paradigm. These algorithms typically use a centralized value function that looks at joint observations to predict the value of the agent's actions. By associating the rewards provided by JIME with new configurations in the joint observation space, the centralized value function induces the agent to actively search for these configurations.

[0023] A generalized MARL cooperative environment is used for testing; in the rel_overgen environment, two agents are rewarded based on their positions. By incorporating intrinsic motivation rewards into the multi-agent reinforcement learning algorithm, additional incentive signals are provided to encourage the agents to explore more effectively and discover better strategies.

[0024] Observation: At each time step, the agent observes its own position as a hot vector

[0025] Action: Two agents can move on a discrete one-dimensional axis, and the agents can choose between three actions: move in one direction or the other or maintain the appropriate position;

[0026] Reward: They receive a reward corresponding to their combined position: The goal of the agents is to maximize their total reward, but the model does not always strike a balance between the global optimal reward and the individual optimal reward. We divide the distribution model of intrinsic rewards into central rewards and local rewards: in the former, all agents share a single intrinsic reward signal, while in the latter, each agent independently receives intrinsic rewards based on its own observations. Central distribution implies coordinated behavior among agents, while local distribution can promote exploratory strategies independent of other agents, thereby better adapting to various environments and promoting the learning of optimal policies. For our proposed JIME, we used an encoder-decoder framework with states as input and actions as the optimization objective of the model.

[0027] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A multi-agent intrinsic motivation training method combined with intrinsic motivation exploration, characterized in that: The Joint Intrinsic Motivation Exploration algorithm, or JIME, is proposed. JIME consists of two components: an encoder for compressing joint observations of multiple agents and a decoder for predicting joint actions of agents. The encoder extracts and compresses the information in the environment state s, and the decoder restores the decision a made by the agent. The encoder contains a 3-layer fully connected neural network with the activation function Relu. The decoder contains a 3-layer fully connected neural network with the activation function Relu. In each training iteration, we first transform the environment state Input to the encoder and get and Input Data is the observed state of the agent, including the collected images or sensor data; represents the mean value of the mapping of agent n to z at time step t. The compressed information z is obtained by extracting and compressing the information in the environment state s through the encoder; represents the covariance of agent n mapped to z at time step t; Use the reparameterization technique from and generate Then Input to the decoder to get the predicted action Finally, the loss function is calculated, and the parameters of the encoder and decoder are updated through the back-propagation algorithm to minimize the total loss and obtain the final JIME. The loss function of JIME is as follows: Among them, θ is the parameter of JIME, and are sampled from interactive samples, which are the original environment states and actions; For the encoder, is the output of the decoder, is a normal distribution.

Images