A multi-agent collaborative exploration method based on information gain to construct intrinsic individual rewards.

Abstract

This invention proposes a multi-agent collaborative exploration method based on information gain to construct intrinsic individual rewards. The method includes: acquiring agent state and action information; creating a multi-agent reinforcement learning environment and initializing the parameters of a dynamic model; constructing a CVAE as the dynamic model, compressing the state space into the latent space, and predicting the state distribution; optimizing the model parameters using reconstruction loss and KL divergence loss to obtain a trained CVAE model; estimating the prior and posterior entropies of agent state transitions using the trained model and calculating the information gain; designing an exploration frequency factor and calculating intrinsic individual rewards in conjunction with the information gain; and generating utilization and exploration strategies for the agents based on intrinsic and extrinsic rewards to guide the agents in selecting actions to perform and completing the task objective. The method proposed in this invention helps agents explore unknown regions, significantly improving their exploration capabilities and task completion rate.

Description

Technical Field

[0001] This invention belongs to the field of multi-agent reinforcement learning technology, and particularly relates to a multi-agent collaborative exploration method based on information gain to construct intrinsic individual rewards. Background Technology

[0002] In multi-agent reinforcement learning, multiple agents typically learn and make decisions within a shared environment. The goal of reinforcement learning is to obtain the highest possible cumulative reward through interaction with the environment. However, in some tasks, rewards in the environment may be sparse, meaning agents may not receive reward feedback for extended periods, making it difficult for them to determine which actions are beneficial and thus impacting the learning process. To help agents explore these environments more effectively, researchers often introduce intrinsic rewards—reward signals that do not depend on external environmental feedback—to motivate agents to explore different state spaces.

[0003] However, in multi-agent environments, while intrinsic rewards can promote exploration, they also introduce another problem: credit assignment. When multiple agents participate in a task, it is difficult to explicitly attribute the contribution to each agent because their actions are highly coupled. This is especially true when using global intrinsic rewards, making it even harder to determine which agent's action caused a change in the environment. Therefore, even given a reward signal, correctly attributing the reward to each agent's behavior is a complex problem. In traditional single-agent reinforcement learning, an agent's behavior directly affects its reward, making the attribution problem relatively simple. However, in multi-agent systems, agent behaviors are often interdependent, requiring consideration of multiple factors such as each agent's role, task allocation, and cooperation patterns. Individual contribution as an intrinsic exploration scaffold guides agent exploration by evaluating each agent's contribution to changes in the global system state. In this way, agents not only focus on their immediate rewards but also understand how their actions have a long-term impact on the entire system, significantly improving their exploration capabilities in sparse reward environments and accelerating the learning process in multi-agent collaborative tasks. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention trains multi-agent systems in both global and local environments. By calculating the amount of information gained by the predictive model of a state after the agent explores a new state, it guides the agent to prioritize exploring unknown or highly uncertain states. Information gain combined with an exploration frequency factor is used as an intrinsic individual reward to measure the amount of new information gained by the agent through exploring a state-action pair. If the model's predictive uncertainty is significantly reduced due to exploration, then the state-action pair is worth exploring. This further improves the exploration efficiency of multi-agent systems, enabling guided agent learning in complex environments and unseen regions.

[0005] To achieve the above-mentioned technical objectives, the present invention provides the following technical solution:

[0006] A multi-agent collaborative exploration method based on information gain to construct intrinsic individual rewards specifically includes the following steps:

[0007] S1. Obtain the current state s of the agent. t The action a at the current moment t and the state s at the next moment t+1 Create a multi-agent reinforcement learning environment and initialize the parameters of the dynamic model;

[0008] S2. Combine Generative Adversarial Network (GAN) to construct Conditional Variational Autoencoder (CVAE). Use CVAE as a dynamic model to compress the state space into the potential space and predict the state distribution of the agent in the next time step.

[0009] S3. Optimize the model parameters of the CVAE constructed in step S2 using reconstruction loss and KL divergence loss to ensure the representational ability of the latent space and distribution regularization, and obtain a trained CVAE model.

[0010] S4. Using the trained CVAE model, estimate the prior entropy and posterior entropy of the agent's state transition from the current time step to the next time step, and calculate the information gain.

[0011] S5. Design an exploration frequency factor and combine it with the information gain obtained in step S4 to calculate the intrinsic individual reward, guiding the agent to prioritize actions that reduce uncertainty.

[0012] S6. Based on intrinsic and extrinsic rewards, generate utilization and exploration strategies for the agent. The utilization and exploration strategies work together to guide the agent in selecting the action to be performed and completing the task objective.

[0013] Furthermore, step S1 specifically includes:

[0014] S11. Create and configure the reinforcement learning environment to obtain the shape of the observation space;

[0015] S12. Create a global baseline network and a local baseline network for the dynamic model; the global baseline network takes the global state and the actions of all agents in the multi-agent environment as input to learn the potential rules of the overall state transition of the agents; while the local baseline network is trained only on the local state and actions of the current agent, so as to independently predict the state transition of the agent.

[0016] S13. Collect multi-step experience data of the agents, including the current state s of each agent. t The action a at the current moment t The state s at the next moment t+1 ;

[0017] S14. Create and initialize the dynamic model, embedding and normalizing the high-dimensional state and action information to prepare it for training.

[0018] Furthermore, step S2 specifically includes:

[0019] S21, Input s t a t s t+1 The concatenated vectors are used, and a multilayer perceptron (MLP) is employed as the encoder to progressively extract features from the input data, outputting the distribution parameters of the latent space: mean μ and standard deviation σ. Furthermore, the posterior distribution of the latent space is defined to satisfy...

[0020] S22. Sample the latent state z from the latent space generated by the encoder. To ensure the differentiability of the sampling process, a reparameterization technique is used to obtain the latent state. It represents the potential pattern of state transition from the current moment to the next moment;

[0021] S23. Construct a decoder using GAN; GAN includes a generator and a discriminator; input is the current state s. t and the action a at the current moment t The data is input into the generator, which processes the input data through multiple fully connected layers and activation functions to extract features and learn the underlying patterns of state transitions. The generator ultimately outputs the predicted distribution of the next state. Where μ' is the mean of the predicted state distribution at the next time step, and σ' is the standard deviation of the predicted state distribution at the next time step; the discriminator then performs feedback optimization on the predicted state distribution at the next time step to ultimately achieve the decoder output a prediction of the state distribution at the next time step that closely resembles the real situation.

[0022] Furthermore, step S3 specifically includes:

[0023] S31, Design Reconstruction Loss L reconstructionThis makes the decoder predict the state distribution P(s) for the next time step. t+1 |z,s t ,a t (As close as possible to the actual state of the next moment) t+1 The generator's training process is optimized through feedback from the discriminator, enhancing the generator's ability to simulate the real state distribution; the reconstruction loss is defined as the negative log-likelihood, expressed by the formula:

[0024]

[0025] Wherein, Q(z|s) t ,a t ,s t+1 Let be the posterior distribution of the potential state z; E represents the expectation.

[0026] S32, Design KL divergence loss L KL So that the posterior distribution of the latent state z is Q(z|s) t ,a t ,s t+1 The distribution is close to its prior distribution P(z|s). t ,a t This regularizes the latent space distribution, preventing overfitting and ensuring that the structure of the latent space can fully capture the potential patterns of state transitions; the formula for KL divergence loss is expressed as:

[0027] L KL =D KL [Q(z|s t ,a t ,s t+1 )||P(s t |a t )];

[0028] Among them, D KL (.) is the formula for calculating KL divergence;

[0029] Substituting the normal distribution into the formula, we obtain the formula for the KL divergence loss as follows:

[0030]

[0031] Where, μ i , σ i Let z be the mean and standard deviation of the posterior distribution of the potential state z.

[0032] S33. Define the total loss L of the CVAE model. CVAE The weighted sum of reconstruction loss and KL divergence is expressed by the formula:

[0033] L CVAE =Lreconstruction +γL KL ;

[0034] Wherein, γ is a balance parameter used to adjust the weight between reconstruction loss and KL divergence;

[0035] S34. Minimize the total loss L using gradient descent. CVAE The model parameters of CVAE are optimized to obtain a trained CVAE model.

[0036] Furthermore, step S4 specifically includes:

[0037] S41. Use the global baseline network based on the current state s t To estimate the overall state transitions of the agent; the Global Limit Network considers the dynamics of the entire multi-agent reinforcement learning environment and generates global state transition predictions that do not depend on the actions of specific agents; then the prior network in CVAE estimates the state s at the current time step. t Based on the output of the global baseline network, the prior distribution P(s) of the state transition of the dynamic model is further inferred. t+1 |s t );

[0038] S42. Combine the local baseline network and the posterior network to estimate a more accurate state transition posterior distribution; the local baseline network incorporates the current state s. t and the actions of the intelligent agent a t The specific impact of the action on the state transition is evaluated, while the posterior network updates the state transition estimate based on the output of the local baseline network and the current state; finally, the posterior distribution P(s) of the state transition of the dynamic model is obtained. t+1 |s t ,a t That is, the action a taken by the agent at the current moment. t The subsequent state change;

[0039] S43. Based on P(s) obtained in steps S41 and S42 t+1 |s t ), P(s t+1 |s t ,a t ) Calculate the prior entropy H(s) respectively t+1 |s t ) and posterior entropy H(s) t+1 |s t ,a t The formula is expressed as:

[0040]

[0041] Where e is the natural constant; d is the total number of dimensions of the agent's states, with each dimension representing a state in the latent space; σ prior,i σ represents the standard deviation of the prior distribution of state transitions in dimension j; posterior,j The standard deviation of the posterior distribution of the state transition in dimension j is represented by .

[0042] S44. Based on the prior entropy and posterior entropy calculated in step S43, the definition of information gain is obtained, expressed by the formula:

[0043] IG(s t ,a t )=H(s t+1 |s t )-H(s t+1 |s t ,a t );

[0044] Using information gain IG(s) t ,a t ) represents the action a at the current moment. t The larger the value of a in reducing the uncertainty of the state at the next time step, the better. t It is beneficial for exploring unknown states.

[0045] Furthermore, step S5 specifically includes:

[0046] S51, Based on state-action pairs (s t ,a t The exploration frequency during the training process is introduced by the exploration frequency factor ω(s). t ,a t This is to encourage agents to explore less frequently accessed state-action pairs; the formula for calculating the exploration frequency factor is:

[0047]

[0048] Wherein, N(s) t ,a t ) is a state-action pair (s t ,a t The historical access frequency is δ, which is a hyperparameter that controls the degree of penalty for historical access frequency.

[0049] S52. Intrinsic individual reward is defined by combining information gain and exploration frequency factor, expressed as follows:

[0050] r int (s t ,a t )=ω(s t ,a t )·IG(s t ,at );

[0051] Where, r int (s t ,a t ) is intrinsic individual reward, IG(s) t ,a t ) is information gain;

[0052] S53. Normalize the intrinsic individual reward to the [0,1] interval, and use the normalized intrinsic individual reward as the selection index for the agent's exploration strategy; the normalization is expressed by the formula:

[0053]

[0054] Where max(r) int ), min(r int ) represent the maximum and minimum values ​​of the intrinsic individual reward for all state-action pairs of the agent, respectively; r int '(s t ,a t () represents the normalized intrinsic individual reward.

[0055] Furthermore, step S6 specifically includes:

[0056] S61. Set the exploitation strategy A strategy is a policy used by an agent to complete a task. It involves selecting the action to perform by evaluating the extrinsic reward corresponding to each known action; that is, choosing the action that maximizes the extrinsic reward. π i This represents the utilization strategy of the i-th agent;

[0057] S62, Set exploration strategy The exploration strategy selects actions to be performed by evaluating the intrinsic individual reward corresponding to each unknown action, prioritizing actions with higher intrinsic individual rewards, i.e., actions that significantly reduce environmental uncertainty; i This represents the exploration strategy of the i-th agent;

[0058] S63, Agent's Behavioral Strategy b i The action selection process is That is, the final action u i It is sampled from behavioral strategies, which involve a trade-off between exploration and exploitation strategies, expressed by the formula:

[0059]

[0060] in, Indicates that by exploring strategy v i Sampling prioritizes actions that offer high intrinsic individual rewards. Indicates the use of the target policy π i Sampling prioritizes actions with high extrinsic rewards; α is a hyperparameter that controls the balance between exploration and exploitation strategies.

[0061] S64. After each exploration action, the exploration strategy is further optimized based on the state and movement trajectory; the optimization objective of the exploration strategy is set to maximize intrinsic individual reward and strategy entropy, expressed by the formula:

[0062]

[0063] Among them, J i (ξ) is the objective function; ξ is the parameter of the exploration strategy. For exploring strategy v i The intrinsic individual reward is below; E represents the expected value; H(·|τ) i ,s) represents the entropy of the exploration strategy, τ i s and s represent the given motion trajectory and state, used to encourage exploration strategy diversity, and β is a hyperparameter that controls the weights for entropy regularization.

[0064] Based on the above technical solution, the method proposed in this invention has at least the following beneficial effects:

[0065] The method proposed in this invention introduces intrinsic individual rewards based on information gain and exploration frequency factors to help the agent better understand the impact of its actions on the environment, focusing on valuable action choices and avoiding ineffective exploration during its learning process. Simultaneously, by combining extrinsic and intrinsic individual rewards, and balancing the use of strategies and exploration strategies, the method significantly improves the agent's learning efficiency and decision-making ability.

[0066] The method proposed in this invention exhibits good adaptability in different environments. The collaborative exploration mechanism based on intrinsic individual rewards and extrinsic rewards enables the agent to quickly adapt to different complex environments and achieve effective policy learning and optimization, thereby improving the model's versatility and robustness. By setting rewards for intrinsic individuals, the agent is encouraged to explore more during training, optimizing the reward sparsity problem in complex environments. This allows the agent to obtain effective policies more quickly and avoid getting trapped in local optima, thus improving training efficiency and the performance of the final model.

[0067] The method proposed in this invention effectively solves the problems of low exploration efficiency, long training time and poor adaptability in traditional methods, and provides an efficient, flexible and highly adaptable exploration method, which provides strong support for the application of intelligent agents in complex environments. Attached Figure Description

[0068] Figure 1 This is an overall flowchart of the method proposed in this invention;

[0069] Figure 2 This is a schematic diagram of the experimental environment 2s3z in an embodiment of the present invention;

[0070] Figure 3 This is a schematic diagram of the experimental environment 3-vs-1 with Keeper in an embodiment of the present invention;

[0071] Figure 4 The test win rate curve is shown in the experimental environment 2s3z.

[0072] Figure 5 The graph shows the average training score curve under the experimental environment 3-vs-1 with Keeper. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this invention clearer, the following description is provided in conjunction with the appendix. Figure 1-5 The present invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0074] Although the steps in this invention are arranged by reference numerals, this is not intended to limit the order of the steps. Unless the order of the steps is explicitly stated or the execution of a step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items.

[0075] like Figure 1 As shown, the multi-agent collaborative exploration method proposed in this invention, based on information gain to construct intrinsic individual rewards, learns the environmental state transition rules through a dynamic model (such as an autovariable encoder (CVAE)) and calculates the contribution of actions to reducing environmental state uncertainty, thereby achieving efficient exploration. This invention first trains the CVAE to learn the prior and posterior distributions of the environmental states, and calculates the information gain by the difference between the two, which is used to measure the exploration value of each action. Intrinsic individual rewards are generated by combining information gain with historical access frequency, and these rewards, combined with extrinsic rewards, guide the agent's behavioral decisions, thus greatly improving exploration efficiency.

[0076] The method proposed in this invention specifically includes the following steps:

[0077] S1. Obtain the current state s of the agent. t The action a at the current moment t and the state s at the next moment t+1 Create a multi-agent reinforcement learning environment and initialize the parameters of the dynamic model;

[0078] In a preferred embodiment, step S1 specifically includes:

[0079] S11. Create and configure the reinforcement learning environment to obtain the shape of the observation space;

[0080] S12. Create a global baseline network and a local baseline network for the dynamic model; the global baseline network takes the global state and the actions of all agents in the multi-agent environment as input to learn the potential rules of the overall state transition of the agents; while the local baseline network is trained only on the local state and actions of the current agent, so as to independently predict the state transition of the agent.

[0081] S13. Collect multi-step experience data of the agents, including the current state s of each agent. t The action a at the current moment t The state s at the next moment t+1 ;

[0082] S14. Create and initialize the dynamic model, embedding and normalizing the high-dimensional state and action information to prepare it for training.

[0083] S2. Combine Generative Adversarial Network (GAN) to construct Conditional Variational Autoencoder (CVAE). Use CVAE as a dynamic model to compress the state space into the potential space and predict the state distribution of the agent in the next time step.

[0084] In a preferred embodiment, step S2 specifically includes:

[0085] S21, Input s t a t s t+1 The concatenated vectors are used, and a multilayer perceptron (MLP) is employed as the encoder to progressively extract features from the input data, outputting the distribution parameters of the latent space: mean μ and standard deviation σ. Furthermore, the posterior distribution of the latent space is defined to satisfy... The existence of the latent space allows high-dimensional input data to be mapped to low dimensions, preserving important features while making the input data more compact.

[0086] S22. Sample the latent state z from the latent space generated by the encoder. To ensure the differentiability of the sampling process, a reparameterization technique is used to obtain the latent state. It represents the potential pattern of state transition from the current moment to the next moment;

[0087] S23. Construct a decoder using GAN; GAN includes a generator and a discriminator; input is the current state s. t and the action a at the current moment tThe data is input into the generator, which processes the input data through multiple fully connected layers and activation functions to extract features and learn the underlying patterns of state transitions. The generator ultimately outputs the predicted distribution of the next state. Where μ' is the mean of the predicted state distribution at the next time step, and σ' is the standard deviation of the predicted state distribution at the next time step; the discriminator then performs feedback optimization on the predicted state distribution at the next time step to ultimately achieve the decoder output a prediction of the state distribution at the next time step that closely resembles the real situation.

[0088] S3. Optimize the model parameters of the CVAE constructed in step S2 using reconstruction loss and KL divergence loss to ensure the representational ability of the latent space and distribution regularization, and obtain a trained CVAE model.

[0089] In this embodiment, the optimization process is performed in the discriminator of the GAN model. The discriminator evaluates the difference between the state distribution generated by the generator and the real state, and provides feedback signals indicating the direction in which the generator needs to be optimized. The generator adjusts its parameters according to the output of the discriminator, gradually generating prediction results that are closer to the real state, thereby enhancing the generator's ability to simulate the real state distribution. In this application, reconstruction loss and KL divergence loss are used as the evaluation criteria for optimization. Therefore, as a preferred embodiment, step S3 specifically includes:

[0090] S31, Design Reconstruction Loss L reconstruction This makes the decoder predict the state distribution P(s) for the next time step. t+1 |z,s t ,a t (As close as possible to the actual state of the next moment) t+1 The generator's training process is optimized through feedback from the discriminator, enhancing the generator's ability to simulate the real state distribution; the reconstruction loss is defined as the negative log-likelihood, expressed by the formula:

[0091]

[0092] Wherein, Q(z|s) t ,a t ,s t+1 Let be the posterior distribution of the potential state z; E represents the expectation.

[0093] S32, Design KL divergence loss L KL So that the posterior distribution of the latent state z is Q(z|s) t ,a t ,s t+1 The distribution is close to its prior distribution P(z|s). t ,a tThis regularizes the latent space distribution, preventing overfitting and ensuring that the structure of the latent space can fully capture the potential patterns of state transitions; the formula for KL divergence loss is expressed as:

[0094] L KL =D KL [Q(z|s t ,a t ,s t+1 )||P(s t |a t )];

[0095] Among them, D KL (.) is the formula for calculating KL divergence;

[0096] Substituting the normal distribution into the formula, we obtain the formula for the KL divergence loss as follows:

[0097]

[0098] Where, μ i , σ i Let z be the mean and standard deviation of the posterior distribution of the potential state z.

[0099] S33. Define the total loss L of the CVAE model. CVAE The weighted sum of reconstruction loss and KL divergence is expressed by the formula:

[0100] I CVAE =L reconstruction +γL KL ;

[0101] Here, γ is a balance parameter used to adjust the weight between reconstruction loss and KL divergence; it flexibly adjusts the balance between generation task and latent space regularization so that the model can generate accurate samples and has a good generation space structure.

[0102] S34. Minimize the total loss L using gradient descent. CVAE The model parameters of CVAE are optimized to obtain a trained CVAE model.

[0103] S4. Using the trained CVAE model, which has both good representation ability and regularized distribution characteristics, estimate the prior entropy and posterior entropy of the agent's state transition from the current time step to the next time step, and calculate the information gain.

[0104] In a preferred embodiment, step S4 specifically includes:

[0105] S41. Use the global baseline network based on the current state s tTo estimate the overall state transitions of the agent; the Global Limit Network considers the dynamics of the entire multi-agent reinforcement learning environment and generates global state transition predictions that do not depend on the actions of specific agents; then the prior network in CVAE estimates the state s at the current time step. t Based on the output of the global baseline network, the prior distribution P(s) of the state transition of the dynamic model is further inferred. t+1 |s t This prior distribution reflects the inherent dynamic changes of the environment, helping the agent understand how the environment evolves without specific actions.

[0106] S42. Combine the local baseline network and the posterior network to estimate a more accurate state transition posterior distribution; the local baseline network incorporates the current state s. t and the actions of the intelligent agent a t The specific impact of the action on the state transition is evaluated, while the posterior network updates the state transition estimate based on the output of the local baseline network and the current state; finally, the posterior distribution P(s) of the state transition of the dynamic model is obtained. t+1 |s t ,a t That is, the action a taken by the agent at the current moment. t The subsequent state change;

[0107] S43. Based on P(s) obtained in steps S41 and S42 t+1 |s t ), P(s t+1 |s t ,a t ) Calculate the prior entropy H(s) respectively t+1 |s t ) and posterior entropy H(s) t+1 |s t ,a t The formula is expressed as:

[0108]

[0109] Where e is the natural constant; d is the total number of dimensions of the agent's states, with each dimension representing a state in the latent space; σ prior,i σ represents the standard deviation of the prior distribution of state transitions in dimension j; posterior,j The standard deviation of the posterior distribution of the state transition in dimension j is represented by .

[0110] S44. Based on the prior entropy and posterior entropy calculated in step S43, the definition of information gain is obtained, expressed by the formula:

[0111] IG(s t ,a t )=H(s t+1 |st )-H(s t+1 |s t ,a t );

[0112] Using information gain IG(s) t ,a t ) represents the action a at the current moment. t The larger the value of a in reducing the uncertainty of the state at the next time step, the better. t It facilitates the exploration of unknown states, thereby guiding intelligent agents to conduct effective exploration.

[0113] S5. Design an exploration frequency factor and combine it with the information gain obtained in step S4 to calculate the intrinsic individual reward, guiding the agent to prioritize actions that reduce uncertainty.

[0114] In a preferred embodiment, step S5 specifically includes:

[0115] S51, Based on state-action pairs (s t ,a t The exploration frequency during the training process is introduced by the exploration frequency factor ω(s). t ,a t This is to encourage agents to explore less frequently accessed state-action pairs; the formula for calculating the exploration frequency factor is:

[0116]

[0117] Wherein, N(s) t ,a t ) is a state-action pair (s t ,a t The historical access frequency is δ, which is a hyperparameter that controls the degree of penalty for historical access frequency.

[0118] S52. Intrinsic individual reward is defined by combining information gain and exploration frequency factor, expressed as follows:

[0119] r int (s t ,a t )=ω(s t ,a t )·IG(s t ,a t );

[0120] Where, r int (s t ,a t ) is intrinsic individual reward, IG(s) t ,a tThis is information gain; thus, frequently accessed state-action pairs will receive smaller rewards, while infrequently accessed state-action pairs will receive larger rewards.

[0121] S53. Since the range of information gain fluctuates greatly under different tasks and scales, the intrinsic individual reward needs to be normalized and mapped to the [0,1] interval. The normalized intrinsic individual reward is used as the selection index for the agent's exploration strategy. The normalization is expressed by the formula:

[0122]

[0123] Where max(r) int ), min(r int ) represent the maximum and minimum values ​​of the intrinsic individual reward for all state-action pairs of the agent, respectively; r int '(s t ,a t (Normalized intrinsic individual reward)

[0124] S6. Based on intrinsic and extrinsic rewards, generate utilization and exploration strategies for the agent. The utilization and exploration strategies work together to guide the agent to select the action to be performed and complete the task objective.

[0125] In a preferred embodiment, step S6 specifically includes:

[0126] S61. Set the exploitation strategy A strategy is a policy used by an agent to complete a task. It selects the action to perform by evaluating the extrinsic reward corresponding to each known action; that is, it selects the action that maximizes the extrinsic reward. π i This represents the utilization strategy of the i-th agent;

[0127] In this embodiment, the QMIX algorithm is used to calculate the extrinsic reward, specifically as follows:

[0128] S611: The state s of each agent i at the current moment. t The following calculates the action 'a' to be selected in this state. t The expected return afterward, i.e., the dynamic-action value function Q. i (s t ,a t This value is typically estimated using historical experience or current strategies;

[0129] S612: The state-action value function Q of each agent... i (s t ,a t ) merge into a single global state-action value function Q tot (s t ,at ), and use it as an external reward; Q tot (s t ,a t The formula is obtained by weighted summation and is expressed as follows:

[0130]

[0131] Where, m i These are the weights of each agent, used to reflect the importance or contribution of each agent in the system;

[0132] 613: Each agent i selects the optimal action in the policy based on external rewards to maximize the overall reward, as expressed by the formula:

[0133]

[0134] S62, Set exploration strategy The exploration strategy selects actions to be performed by evaluating the intrinsic individual reward corresponding to each unknown action, prioritizing actions with higher intrinsic individual rewards, i.e., actions that significantly reduce environmental uncertainty; i This represents the exploration strategy of the i-th agent;

[0135] S63, Agent's Behavioral Strategy b i The action selection process is That is, the final action u i It is sampled from behavioral strategies, which involve a trade-off between exploration and exploitation strategies, expressed by the formula:

[0136]

[0137] in, Indicates that by exploring strategy v i Sampling prioritizes actions that offer high intrinsic individual rewards. Indicates the use of the target policy π i Sampling prioritizes actions with high extrinsic rewards; α is a hyperparameter that controls the balance between exploration and exploitation strategies.

[0138] S64. After each exploration action, the exploration strategy is further optimized based on the state and movement trajectory; the optimization objective of the exploration strategy is set to maximize intrinsic individual reward and strategy entropy, expressed by the formula:

[0139]

[0140] Among them, J i (ξ) is the objective function; ξ is the parameter of the exploration strategy, which adjusts the randomness or diversity of the exploration strategy through entropy regularization; For exploring strategy v i The intrinsic individual reward is below; E represents the expected value; H(·|τ) i ,s) represents the entropy of the exploration strategy, τ i s and s represent the given motion trajectory and state, used to encourage exploration strategy diversity, and β is a hyperparameter that controls the weights for entropy regularization.

[0141] Thus, the multi-agent collaborative exploration method proposed in this invention, which constructs intrinsic individual rewards based on information gain, builds a collaborative exploration mechanism for agents by combining intrinsic individual rewards with extrinsic rewards (i.e., simultaneously considering both utilization strategies and exploration strategies, taking into account both known and unknown environments), thereby making the agents' exploration more effective. The following specific experimental examples verify the effectiveness of the method proposed in this invention.

[0142] Experimental Examples

[0143] This experimental example sets up two different experimental environments, such as Figure 2 , 3 As shown. Figure 2 The StarCraft Multi-Agent Challenge (SMAC) environment is a multi-agent reinforcement learning environment developed by Blizzard Entertainment and DeepMind based on the real-time strategy game StarCraft II. The SMAC environment includes 23 maps, such as... Figure 2 The image shown is one of the land maps. Figure 2 The actual gameplay footage from S3Z shows a heterogeneous symmetrical map where both sides consist of two Stalkers and three Zealots. Each unit is controlled by an independent agent trained only within its own limited field of vision, while the opposing units are uniformly controlled by a hand-coded built-in AI. This built-in AI has ten levels, ranging from increasingly powerful levels 1 to 9, with the strongest being level A. This is a two-unit battle where the model needs to learn one or more strategies to defeat the enemy.

[0144] Figure 3The Google Research Football (GRF) environment shown is a football game environment released by Google. It provides a physics-based 3D football simulation environment where agents control players, learning how to pass and combine between players, and how to break through the opponent's defense to score. This is a very challenging reinforcement learning task because in a football game scenario, the agent needs to maintain a connection and balance between real-time control (such as learned concepts of passing and dribbling) and a high level of strategy. These scenarios are different; some scenarios may require the agent to score with an empty goal, while others require the agent to combine with teammates to break through the opponent's defensive line to score. The 3-vs-1 with Keeper environment simulates a 3-on-1 football match, with 3 attacking agents (attacking players) and 1 defending agent (goalkeeper). The attacking team's goal is to attack through passing, combination play, and shooting, while the defending team's goal is to prevent the attacking team's shots as much as possible and protect their own goal.

[0145] This experimental example uses a sparse reward setting in SMAC and GRF environments. In 2s3z, the reward value increases by 200 when all enemy units are defeated, which is the ultimate reward for completing the task and encourages agents to cooperate in eliminating all enemies. The reward value increases by 10 when an enemy unit is defeated, which incentivizes the behavior of gradually eliminating enemy units. When an allied unit is defeated, the reward value decreases by 5, which is a penalty mechanism to encourage the protection of allies. In 3-vs-1 with Keeper, the reward value for our team's score increases by 100, which is the target reward for the task and incentivizes agents to cooperate to achieve the final goal. The reward value decreases by 1 when the opponent scores, and the reward value decreases by 1 when the ball or an allied player returns to their own half of the court, which is a penalty signal to prevent agents from ignoring defensive strategies and choosing to delay or engage in ineffective exploration behaviors. Here, rewards are only given when key events occur, and there are no reward signals during the intermediate process, which increases the difficulty of learning and requires agents to explore and cooperate more efficiently. The deep reinforcement learning model parameters are set as follows: action embedding dimension is 4; intrinsic reward frame learning rate is 0.0001; the optimizer is Adam; gradient clipping value is 0.1; the exploration network learning rate is set to 0.01 for the 2s3z task and 0.001 for the 3-vs-1 with Keeper task; the exploration-exploitation balance coefficient α is set to 0.1 in the 2s3z task, 0.2 in the early stages of training for the 3-vs-1 with Keeper task, and gradually reduced to 0.05 as training progresses; the intrinsic reward entropy weight β is set to 0.1 in the 2s3z task and 0.05 in the 3-vs-1 with Keeper task.

[0146] The agent was trained using both the traditional QMIX algorithm and the method proposed in this invention. The training results are as follows: Figure 4 , Figure 5 As shown in the figure. The horizontal axis represents the number of training steps, and the vertical axis represents the test win rate and the average training score, respectively. The dashed line represents the method proposed in this invention, and the solid line represents the QMIX algorithm. From Figure 4 As can be seen, when the number of steps reaches 0.5M (500,000), the test win rate of the method proposed in this invention begins to rise significantly, and begins to converge at 1M steps, maintaining the highest team win rate until the end of training. In contrast, the curve of the traditional QMIX algorithm begins to rise at 1M steps and is relatively slow. Figure 5 The average score of this invention begins to rise around 1M steps and gradually converges around 2M steps. From the animation effects of the environment rendering, it can be seen that when the method proposed in this invention reaches convergence, the paths taken by the agent are all shortest paths, while the traditional QMIX algorithm has a very low task completion rate. In summary, the proposed method significantly improves the convergence speed, enhances the exploration rate of the environment, and has a good effect on exploration.

[0147] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0148] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A multi-agent collaborative exploration method based on information gain and internal individual reward construction, characterized in that, Based on the multi-agent cooperative exploration scene under the real-time strategy game, the real-time strategy game includes StarCraft II and Google Soccer, specifically including the following steps: S1, obtaining a state of the agent at a current time , an action at the current time , and a state at a next time , creating a multi-agent reinforcement learning environment and initializing parameters of a dynamic model; specifically comprising: S11, create and configure a reinforcement learning environment, the reinforcement learning environment includes: 2s3z heterogeneous symmetric combat environment in the multi-agent confrontation scene of the real-time strategy game StarCraft II or 3-vs-1 with Keeper environment in the multi-agent cooperative scene of Google Soccer; Obtain the shape of the observation space; S12, create a global baseline network and a local baseline network of a dynamic model; The global baseline network takes the global state in the multi-agent environment and the actions of all agents as input, and learns the potential law of the state transition of the whole agent; And the local baseline network is trained based on the local state and action of the current agent, so as to independently predict the state transition of the agent; The agent includes two Stalkers and three Zealots in the 2s3z heterogeneous symmetric combat environment; Or 3 offensive players and 1 goalkeeper in the 3-vs-1 with Keeper environment; S13, collect multi-step experience data of the agent, including the state of each agent at the current time , the action at the current time , the state at the next time ; S14, create and initialize a dynamic model, embed and normalize high-dimensional state information and action information, and prepare for training of the dynamic model; S2, construct a conditional variational autoencoder CVAE combined with a generative adversarial network GAN, use the CVAE as a dynamic model, compress the state space to the latent space, and predict the state distribution of the agent at the next time; S3, use the reconstruction loss and the KL divergence loss to optimize the model parameters of the CVAE constructed in step S2, ensure the representation ability and distribution regularization of the latent space, and obtain the trained CVAE model; S4, use the trained CVAE model to estimate the prior entropy and posterior entropy of the state transition of the agent from the current time to the next time, and calculate the information gain; S5, design an exploration frequency factor, and calculate the intrinsic individual reward combined with the information gain obtained in step S4, to guide the agent to preferentially select actions that reduce uncertainty; S6, generate the utilization strategy and the exploration strategy of the agent according to the intrinsic individual reward and the external reward, and guide the agent to select the action to be executed by the cooperative action of the utilization strategy and the exploration strategy, and complete the task goal.

2. The multi-agent cooperative exploration method for constructing intrinsic individual rewards based on information gain according to claim 1, characterized in that, Step S2 specifically includes: S21, Input , , The concatenated vectors are used, and a multilayer perceptron (MLP) is used as the encoder to progressively extract features from the input data, outputting the mean distribution parameters of the latent space. and standard deviation Furthermore, the posterior distribution of the latent space is defined to satisfy... ; S22, sampling a latent state from the latent space generated by the encoder To ensure the sampling process is differentiable, a reparameterization trick is adopted to sample the latent state which represents the latent law of state transition from the current time to the next time S23, constructing a decoder using a GAN; the GAN includes a generator and a discriminator; inputting the state at the current time and the action at the current time into the generator, the generator processes the input data through multiple fully connected layers and activation functions, extracts features therefrom, and learns the underlying rules of state transition, and finally outputs the predicted state distribution at the next time: wherein is the mean of the predicted state distribution at the next time, is the standard deviation of the predicted state distribution at the next time; the discriminator further optimizes the predicted state distribution at the next time to ultimately achieve the prediction of the state distribution at the next time that is close to the true situation.

3. The multi-agent cooperative exploration method for constructing intrinsic individual rewards based on information gain according to claim 2, characterized in that, Step S3 specifically includes: S31, design reconstruction loss make the decoder predict the next time state distribution as close to the real next time state as possible the training process of the generator is optimized through the feedback of the discriminator, and the generation ability of the generator for the real state distribution is strengthened; the reconstruction loss is defined as a negative log likelihood, and the formula is represented as: ； wherein is the posterior distribution of latent states denotes expectation.​ S32、design KL divergence loss so that the posterior distribution of the latent state close to its prior distribution , so that the latent space distribution is regularized, preventing the model from overfitting, and ensuring that the structure of the latent space can fully capture the potential law of state transition; the formula of KL divergence loss is: ； wherein, is the KL divergence calculation formula; Again, the normal distribution is brought in, and the formula of the KL divergence loss is: ； wherein , are the mean and standard deviation of the posterior distribution of the latent state. S33, defining a total loss of the CVAE model The weighted sum of reconstruction loss and KL divergence is expressed by the formula: ； wherein, is a balancing parameter used to adjust the weight between the reconstruction loss and the KL divergence; S34, minimizing the total loss by gradient descent method The model parameters of the CVAE are optimized, and finally a trained CVAE model is obtained.

4. The multi-agent collaborative exploration method for constructing intrinsic individual rewards based on information gain according to claim 1, characterized in that, Step S4 specifically includes: S41, using the global baseline network to estimate the state transition of the whole agent according to the state at the current moment ; the global limit network considers the dynamics of the whole multi-agent reinforcement learning environment, generates a global state transition prediction that does not depend on the specific action of the agent; and then the prior network in the CVAE further infers the prior distribution of the state transition of the dynamic model based on the output of the current moment and the global baseline network ; S42, estimate a more accurate state transition posterior distribution by combining the local baseline network and the posterior network; the local baseline network evaluates the specific impact of the action on the state transition and the agent's action based on the current state, while the posterior network updates the estimate of the state transition based on the output of the local baseline network and the current state; finally, the posterior distribution of the state transition of the dynamic model is obtained , i.e. the state change that occurs after the agent takes the action at the current time ​ S43, obtained from steps S41 and S42 , Calculate the prior entropy respectively and posterior entropy The formula is expressed as: ； ； wherein, is a natural constant; is the total number of dimensions of the agent state, each dimension representing a state in the latent space; represents j is the standard deviation of the prior distribution of the dimension state transition; represents j is the standard deviation of the posterior distribution of the dimension state transition; S44, according to the prior entropy and the posterior entropy calculated in step S43, obtain the definition of information gain, and the formula is: ； with information gain an action at the current time the greater the value, the greater the contribution to reducing uncertainty in the state at the next time favors exploration of unknown states.

5. The multi-agent collaborative exploration method for constructing intrinsic individual rewards based on information gain according to claim 1, characterized in that, Step S5 specifically includes S51, based on state-action pair Exploration frequency in the training process, introduce exploration frequency factor To promote the agent to explore less visited state-action pairs; the calculation formula of the exploration frequency factor is: ； wherein, is a history visit frequency of state-action pair is a history visit frequency of state-action pair is a hyper-parameter, controlling the degree of punishment on the history visit frequency; S52, define the intrinsic individual reward combined with the information gain and the exploration frequency factor, and the formula is: ； wherein, is the intrinsic individual reward, is the information gain; S53, normalize the intrinsic individual reward to interval, taking the normalized intrinsic individual reward as the selection index of the agent exploration strategy; the normalization formula is: ； wherein, , are the maximum and minimum values, respectively, of the intrinsic individual reward for the respective state-action pair of the agent; is the normalized intrinsic individual reward.

6. The multi-agent cooperative exploration method for constructing intrinsic individual rewards based on information gain according to claim 5, characterized in that, Step S6 specifically is: S61、set the use policy , as the policy of the agent for completing the task, the action to be executed is selected by evaluating the extrinsic reward corresponding to each known action, that is, the action that maximizes the extrinsic reward is selected, represent the use policy of the first agent; S62, Set exploration strategy The exploration strategy selects the action to be performed by evaluating the intrinsic individual reward corresponding to each unknown action, and prioritizes actions with large intrinsic individual rewards, that is, actions that can significantly reduce environmental uncertainty. Indicates the first The exploration strategy of each agent; S63, behavior policy of the agent The action selection process for i.e., the final action is sampled from a behavior policy that balances exploration and exploitation, expressed as: ； wherein, denotes sampling from the exploration policy sampling, prefer actions with high intrinsic individual reward, denotes sampling from the target policy sampling, prefer actions with high extrinsic reward, is a hyperparameter controlling the balance between the exploration policy and the exploitation policy; S64, after each execution of the exploration action, the exploration strategy is optimized again according to the state and the motion trajectory; The optimization goal of the exploration strategy is to maximize the intrinsic individual reward and the strategy entropy, and the formula is: ； where, is the objective function; parameters of the exploration policy, is the exploration policy under the intrinsic individual reward; denotes expectation; denotes the entropy of the exploration policy, and s denotes the given trajectory and state, for encouraging exploration policy diversity, is a hyper-parameter, controlling the weight of the entropy regularization.

Images