Multi-agent sparse reward exploration method and system based on causal reasoning

Abstract

The invention discloses a multi-agent sparse reward exploration method and system based on causal reasoning, and belongs to the field of multi-agent cooperation. The causal effect of a multi-agent sparse reward task is fully captured through two important components: 1) providing an anti-fact-based exploration target extraction module to extract causal information in an environment; and 2) an internal reward modification module based on the subspace is provided, and the information in the exploration target is utilized to guide the intelligent agent to explore the subspace which is more important for obtaining the reward, so that the exploration efficiency is improved.

Description

Technical Field

[0001] This invention belongs to the field of multi-agent cooperation, specifically relating to a multi-agent sparse reward exploration method and system based on causal reasoning. Background Technology

[0002] Large-scale, complex, continuous collaborative decision-making problems pose significant challenges to human decision-making capabilities in fields such as industry, traffic control, and network routing. The lack of effective solutions leads to substantial resource waste. Improving the execution efficiency and accuracy of decision-making algorithms has become a crucial issue. Solutions to this problem can be broadly categorized into rule-based control methods and multi-agent reinforcement learning methods. Due to the complexity of real-world environments, manually designing control rules and dense rewards for agents often leads to getting trapped in local optima. Exploring rewards has become a key challenge for the practical application of multi-agent algorithms.

[0003] Multi-agent exploration aims to collect reward signals from the environmental state space and is the cornerstone of agent collaboration. Related research mainly focuses on introducing intrinsic rewards to compensate for the sparsity of extrinsic rewards, including guiding agents to choose influential behaviors, explore novel states, and explore diverse trajectories. Based on the methods and criteria for calculating intrinsic rewards, existing multi-agent exploration methods can be broadly divided into two categories: self-exploration and global exploration.

[0004] Self-exploration focuses on introducing intrinsic goals at the individual agent level. Specifically, Zheng et al. (Zheng L, Chen J, Wang J, et al. Episodic multi-agent reinforcement learning with curiosity-driven exploration[J]. Advances in Neural Information Processing Systems, 2021, 34: 3757-3769.) use the prediction error of the individual Q-value as an intrinsic reward; Zhang et al. (Zhang S, Cao J, Yuan L, et al. Self-Motivated Multi-Agent Exploration[C] / / Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems. 2023: 476-484.) expand the exploration space of each agent using state edge matching; Kim et al. (Kim W, Sung Y. An adaptive entropy-regularization framework for multi-agent reinforcement learning[C] / / International Conference on MachineLearning. PMLR, 2023: (16829-16852.) The necessity of exploration for each agent is assessed, and the optimal target entropy is determined to drive maximum entropy exploration. The social influence approach captures the dependencies between agents by measuring the impact of an agent's behavior on the behavior of other agents. Wang et al. (Wang T, Wang J, Wu Y, et al. Influence-based multi-agent exploration[J]. arXiv preprint arXiv:1910.05512, 2019.) quantified the impact of an agent's behavior on the transfer dynamics and expected rewards of other agents.Li et al. (Li C, Wang T, Wu C, et al. Celebrating diversity in shared multi-agent reinforcement learning[J]. Advances in Neural Information Processing Systems, 2021, 34: 3991-4002.) promote diverse individualized behaviors by utilizing the mutual information between agent identity and its local trajectory.

[0005] Global exploration, on the other hand, emphasizes encouraging a comprehensive exploration of the state space. For example, Mahajan et al. (Mahajan A, Rashid T, Samvelyan M, et al. Maven: Multi-agent variational exploration[J]. Advances in neural information processing systems, 2019, 32.) generated diverse global behaviors by maximizing the mutual information between global trajectories and latent variables. Yang et al. (Yang M, Yang Y, Lu Z, et al. Hierarchical multi-agent skill discovery[J]. Advances in Neural Information Processing Systems, 2024, 36.) proposed a skill discovery method that develops team-level skills, promoting diverse trajectories from both individual and global perspectives. Chen et al. (Chen X, Liu X, Zhang S, et al. Goal Consistency: An Effective Multi-Agent Cooperative Method for MultistageTasks[C] / / IJCAI. 2022: 172-178.) employed a goal-oriented multistage model to enhance goal cognition, helping agents understand tasks at the goal level and thus achieving cooperative global exploration. Iqbal et al. (Iqbal S, Sha F. Coordinated exploration via intrinsic rewards for multi-agent reinforcement learning[J]. arXiv preprint arXiv:1905.12127, 2019.) and Chitnis et al. (Chitnis R, Tulsiani S, Gupta S, et al. Intrinsic motivation for encouraging synergistic behavior[J]. arXiv preprint arXiv:2002.05189, 2020.) respectively motivated collective behavior in multi-agent systems by creating heuristic intrinsic rewards.

[0006] In summary, self-exploration offers a simple and scalable approach, but may lead to a lack of coordination in exploration behavior among agents. In contrast, global exploration helps promote cooperative behavior, but faces challenges such as exploration space explosion, especially in large-scale complex tasks involving a large number of agents. Subspace exploration, as a novel solution, addresses this problem by reducing the exploration space.

[0007] However, existing subspace methods such as CMAE (Liu IJ, Jain U, Yeh RA, et al. Cooperative exploration for multi-agent deep reinforcement learning[C] / / International conference on machine learning. PMLR, 2021: 6826-6836.) and FOX (JoY, Lee S, Yeom J, et al. FoX: Formation-aware exploration in multi-agent reinforcement learning[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(12): 12985-12994.) rely on projection and forming techniques to map high-dimensional state spaces to low-dimensional latent spaces. These simple projections lack semantic information and perform poorly in complex multi-agent cooperative tasks. SAME (Xu P, Zhang J, Yin Q, et al. Subspace-aware exploration for sparse-reward multi-agent tasks[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(10): 11717-11725.) uses the optimistic principle in the face of uncertainty to assess the importance of each subspace, but this is only an approximation based on priors and does not fundamentally explore the impact of different subspaces on rewards or exploration efficiency.

[0008] In industrial applications, such as collaborative sorting in warehousing and logistics, the warehouse environment is vast and contains a massive number of state dimensions, while effective reward signals are extremely sparse and delayed. When faced with such a high-dimensional search space full of interfering information, traditional methods often fall into blind exploration due to a lack of understanding of the core task objectives, making it difficult for multi-robot systems to converge an efficient sorting collaboration strategy within a limited time.

[0009] The aforementioned problems prompt us to delve into the underlying reasons why existing subspace exploration methods are unable to solve the sparse reward problem from the perspective of causality, thus providing a theoretical guarantee for proving the rationality of the proposed structural causal model. Summary of the Invention

[0010] To address the issue of low exploration efficiency under sparse reward conditions in current methods, this invention re-examines the learning paradigm of multi-agent exploration methods from a causal perspective. It proposes a multi-agent sparse reward exploration method and system based on causal reasoning, which fully captures the causal effects of multi-agent sparse reward tasks through two key components: 1) a counterfactual-based exploration target extraction module to extract causal information from the environment; and 2) a subspace-based intrinsic reward modification module that utilizes information from the exploration target to guide the agent to explore subspaces more important for obtaining rewards, thereby improving exploration efficiency.

[0011] To achieve the above objectives, the technical solution of the present invention includes the following:

[0012] A multi-agent sparse reward exploration method based on causal reasoning, the method comprising: The global state space is divided into Subspace ; Construct an exploration target extraction module, which is used to weight the subspace according to the structural causal model. The structured causal model is updated so that actions generated by the action distribution that depends on the policy network are replaced with actions generated by the action counterfactual distribution that depends on prior selection. Construct an intrinsic reward modification module, which is used to modify the reward based on the subspace weights. For this subspace The original intrinsic reward Weighting is performed to generate a total intrinsic reward, and a combined reward is obtained based on the total intrinsic reward and the environmental extrinsic reward. The exploration target extraction module and the intrinsic reward modification module are embedded into the multi-agent reinforcement learning baseline architecture to form the MARL model; Train the MARL model until the policy performance converges.

[0013] Furthermore, the subspace weights are adjusted based on the structural causal model. Updates will be made, including: Global state in the actual trajectory of technology Distribution ; Choose a counterfactual value Replace global state The first in Each component To obtain the replaced state ; For the replaced state copy After that, joint actions will be taken. Each component is replaced by uniform random sampling to obtain counterfactual intervention trajectory data. ; Counterfactual intervention trajectory data Input a traditional reward model, and output a predicted reward for each data point. ; Combine global state Distribution And action counterfactual distribution, defining counterfactual values. With predicted rewards direct causal relationship ;in, Indicates intervention; Statistical subspace Overall Predictive Reward Distribution of Counterfactual Intervention Trajectory Data ; Based on direct causation Compared with the overall predicted reward distribution Obtain subspace weights .

[0014] Furthermore, the subspace weights ;in, Representing subspace China's anti-factual value Quantity, This indicates the calculation of KL divergence.

[0015] Furthermore, the original intrinsic reward ;in, It is the global state. The Middle The number of times each component is counted. This is a hyperparameter.

[0016] Furthermore, the method also includes: The target task is completed based on the trained MARL model; wherein, the target task includes various multi-agent tasks, such as collaborative sorting tasks in warehousing and logistics.

[0017] Furthermore, the process of completing collaborative sorting tasks in warehousing logistics based on the trained MARL model includes: Obtain the global state of the storage environment and the global state It is divided into a self-state subspace reflecting the pose information of the intelligent agent, a cargo association subspace reflecting the relative relationship between the intelligent agent and the target cargo, a cargo guidance subspace reflecting the location information of the unloading area, and an obstacle avoidance subspace reflecting the distribution of environmental obstacles. Calculate the average causal effect of each subspace on the sparse reward triggered by the successful task, and identify subspaces with weights. The cargo-related subspace and unloading guidance subspace that exceed the set threshold are the target exploration areas; When the state component of the cargo-associated subspace indicates that the agent does not hold cargo and is outside the grasping range, it is subject to the weight of the corresponding subspace. Guided by the system, the intelligent agent performs a movement action that approaches the coordinates of the goods, and performs a goods grabbing action when it enters the grabbing range; When goods are already in hand and the object enters the sensing range of the unloading guidance subspace, it is subject to the corresponding subspace weight. Guided by the system, the intelligent agent executes the unloading action of aligning with the unloading point and placing the goods. It monitors its own state subspace and obstacle avoidance subspace. If there is a collision risk, it adjusts the intrinsic reward according to the weight to trigger the avoidance action, thereby ensuring the stability of the strategy output by eliminating causal confounding factors caused by historical trajectory conflicts. Each agent updates its strategy based on the combined rewards generated by the intrinsic reward modification module, and finally outputs the joint action sequence that optimizes the overall sorting efficiency.

[0018] A multi-agent sparse reward exploration system based on causal reasoning, the system comprising: The space partitioning module is used to divide the global state space into... Subspace ; The first construction module is used to construct an exploration target extraction module, which is used to weight the subspace according to a structural causal model. The structured causal model is updated so that actions generated by the action distribution that depends on the policy network are replaced with actions generated by the action counterfactual distribution that depends on prior selection. The second construction module is used to construct the intrinsic reward modification module, which is used to modify the subspace weights. For this subspace The original intrinsic reward Weighting is performed to generate a total intrinsic reward, and a combined reward is obtained based on the total intrinsic reward and the environmental extrinsic reward. The embedding module integrates the exploration target extraction module and the intrinsic reward modification module into the multi-agent reinforcement learning baseline architecture to form the MARL model. The model training module is used to train the MARL model until the policy performance converges.

[0019] An electronic device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the multi-agent sparse reward exploration method based on causal reasoning as described above.

[0020] A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the multi-agent sparse reward exploration method based on causal reasoning described above.

[0021] A computer program product, characterized in that, when the computer program product is run on a computer device, the computer device executes the multi-agent sparse reward exploration method based on causal reasoning described above.

[0022] Compared with the prior art, the present invention has at least the following beneficial effects.

[0023] (1) The subspace-based intrinsic reward modification module decomposes the global state space. By setting different exploration goals, important subspaces can be explored first, which improves the flexibility of exploration and reduces the overhead of maintaining the number of global state accesses.

[0024] (2) The counterfactual exploration target extraction module analyzes the structural causal model, obtains the causal relationship between each state subspace and the reward through counterfactual operations on the existing data, and uses it to construct exploration targets, effectively utilizing the causal relationship within the environment to improve exploration efficiency.

[0025] (3) This invention is a plug-and-play method that only adjusts the intrinsic reward received by the agent. It can be applied to various multi-agent value decomposition model architectures and improves the performance of the model in sparse reward exploration scenarios.

[0026] (4) For typical multi-agent sparse reward tasks such as warehousing and logistics sorting, this invention enables robots to automatically lock key causal subspaces in a large number of environmental variables. By focusing on these areas that contribute the most to the success of the task, this invention effectively filters out irrelevant interference in the environment, significantly reduces the robot's invalid operation paths, and thus greatly improves the sorting success rate and overall operation efficiency of multi-machine collaboration in complex dynamic environments. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating the implementation of the present invention. Detailed Implementation

[0028] The specific embodiments of the present invention will now be described in detail with reference to examples and the accompanying drawings. The examples described herein are for illustrative and explanatory purposes only and are not intended to limit the scope of the invention.

[0029] This invention designs a multi-agent sparse reward exploration method that introduces causal inference to discover causal knowledge in the environmental subspace, comprising the following steps: Before conducting a multi-agent collaborative task, a structural causal model (SCM) is first constructed to analyze the true causal relationships among the global state, agent actions, historical information, and reward signals in the task. Secondly, a multi-agent reinforcement learning environment is prepared for the task, containing specific definitions of the global state, agent actions, reward signals, initial state, and termination conditions, and the environmental dynamics model conforms to the actual task. The environment receives agent actions and provides feedback on the current reward and the next global state. Then, based on the constructed structural causal model, a counterfactual exploration target extraction module is proposed. This module extracts the next-stage exploration target based on data collected in the environment. Next, a subspace-based intrinsic reward modification module is designed, which guides the exploration direction of the intrinsic reward to the state space specified by the exploration target. Then, the exploration target extraction module and the intrinsic reward modification module are introduced together into the traditional multi-agent reinforcement learning baseline architecture as an adaptive exploration model. Finally, the intrinsic reward and the environmental extrinsic reward are merged to replace the original reward, the cumulative reward loss is calculated, and the adaptive exploration model is trained until convergence. Finally, the exploration target of the trained adaptive exploration model is extracted, the model is initialized, and the exploration target is initialized to the trained exploration target. The model is then trained again to test whether its ability to explore sparse rewards has been improved.

[0030] The following example illustrates a collaborative sorting task in warehouse logistics. In this scenario, multiple handling robots need to locate, pick up, and transport goods to a designated unloading area within a complex warehouse environment. Due to the vast warehouse space and the fact that rewards are only generated upon successful delivery, this task presents an extremely high level of challenge. The warehouse environment used in this embodiment includes the following definitions: Global state Includes the real-time coordinates of all robots. The data includes: battery level, whether the robot is carrying cargo, relative distance to the target cargo, coordinates of the unloading point, and distribution of surrounding obstacles (shelves or other robots).

[0031] intelligent agent actions The robot's movement (forward, backward, left, right), grasping operations, placement actions, and stationary standby.

[0032] Reward signal Extremely sparse reward. A +10 reward is given only when the robot successfully places the cargo in the correct unloading area; a -5 penalty is given if a collision occurs; the reward is 0 for all other moves.

[0033] The detailed steps are as follows: Step 1: Construct a structural causal model and analyze causal relationships.

[0034] Before undertaking multi-agent collaborative tasks, a structural causal model is first constructed. This is because traditional distributed partially observable Markov decision process models contain unobservable historical information variables. This leads to direct calculation of the state. To reward The causal effect has a confounding path, namely , Indicates an action.

[0035] To analyze the true causal relationships among global state, agent actions, historical information, and reward signals in this task, the structural causal model constructed in this invention severs the action through counterfactual intervention. Historical information and state The model assumes the following dependencies. Specifically, it makes the following assumptions: 1) The action distribution that originally relied on the policy network Replace with a counterfactual distribution of actions made through prior choices independent of historical information. That is, to perform an operation .

[0036] 2) Under this structure, analyze specific sub-states. For rewards When the direct causal effect is no longer subject to unobservable variables, This allows us to mitigate the confusion caused by observable data and thus calculate the true causal relationships based on observable data.

[0037] Step 2: Prepare the multi-agent reinforcement learning environment required for the task.

[0038] In this example, the system initializes the warehouse map, which includes obstacles, randomly distributed cargo points, and fixed unloading areas. Each step, the environment receives the combined actions of all robots and provides feedback on the current reward and the global state for the next step.

[0039] Step 3: Based on the constructed structural causal model, a counterfactual-based exploration target extraction module is proposed. This module extracts the next stage exploration target based on data collected in the environment.

[0040] Step 3.1: To calculate the value of the i-th subspace in the exploration target (e.g., self-state subspace, cargo association subspace, unloading guidance subspace, and obstacle avoidance subspace), first, the distribution of the global state s in the real trajectory is statistically analyzed. .

[0041] Step 3.2: Record the actual trajectory data ,in This is the global state. For joint actions of intelligent agents, The first in Each component is denoted as .choose An optional value at position (i.e., counterfactual value taking), copy each real trajectory data and use it. replace The state after replacement The expression is as follows: in, The number of dimensions of the global state.

[0042] Step 3.3: In order to simulate the actions in the SCM described in Step 1 Intervention operations (i.e.) For each replaced trajectory data copy Second-rate( The value of needs to be large enough to cover the action space or satisfy statistical significance, and is related to the subspace. The number of values ​​is not directly related, but rather used to approximate a uniform distribution through Monte Carlo sampling. (The combined action...) Each component is replaced by uniform random sampling to obtain counterfactual intervention trajectory data. As shown in the following formula: in Indicates the combined action after the substitution. For the number of agents, This represents the value space of the action of the j-th agent. This indicates a uniform distribution. This step applies the SCM cutting technique from step 1. The concept of a path.

[0043] Step 3.4: Input the counterfactual intervention trajectory data into the traditional reward model, and output a predicted reward for each data point. Based on the SCM analysis in step 1, the subspace values ​​are now calculated. The formula for the direct causal relationship with the reward (i.e., the probability of intervention) is as follows: in, It is a counterfactual distribution of actions chosen a priori. Representing state The probability distribution. This formula uses observable data. and uniform motion distribution It achieves an unbiased estimation of the true causal relationship.

[0044] Step 3.5: Apply steps 3.1 to 3.4 to each value of subspace i. Obtain the direct causal relationship for each value. For the counterfactual intervention trajectory data of this subspace, the overall predicted reward distribution is statistically analyzed again. The i-th component of the exploration target can then be calculated, defined as the average difference between the reward distribution of each value in subspace i and the overall reward distribution, measured by KL divergence, as shown in the following formula: in, denoted as the number of possible values ​​in subspace i.

[0045] Step 3.6: Applying steps 3.1 to 3.5 to each subspace i yields the complete exploration objective, as shown in the following formula. Step 4: Design a subspace-based intrinsic reward modification module that guides the exploration direction of intrinsic rewards to the state space specified by the exploration target (such as prioritizing the exploration of the "cargo association" and "unloading guidance" subspaces).

[0046] Step 4.1: For real trajectory data Regarding the state within it The original intrinsic reward of the subspace is calculated as follows: in It is the original intrinsic reward of subspace i. It is the count of the i-th component in state s. This is a hyperparameter.

[0047] Step 4.2: For each subspace reward, introduce the exploration objective to weight the reward, and obtain the total intrinsic reward at the current moment. .

[0048] Step 4.3: Obtain external rewards from the environment In this specific implementation scenario, Direct feedback from the environment, such as when a robot successfully places goods in the correct unloading area. Points, or a collision will result in... Punishment. Combining intrinsic rewards with extrinsic environmental rewards constitutes the actual reward: in Indicates external environmental rewards, This is a hyperparameter.

[0049] Step 4.4: Insert this module as a plugin into the traditional multi-agent reinforcement learning value decomposition baseline model, replacing the original reward item.

[0050] Step 5: Introduce the exploration target extraction module and the intrinsic reward modification module together into the traditional multi-agent reinforcement learning baseline architecture as an adaptive exploration model.

[0051] Step 6: Combine the intrinsic reward with the extrinsic reward to replace the original reward, calculate the cumulative reward loss, and train the adaptive exploration model until convergence.

[0052] Step 7: Extract the exploration target from the trained adaptive exploration model, initialize the model and initialize the exploration target to the trained exploration target, and train again to test whether the model's ability to explore sparse rewards has been improved.

[0053] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-agent sparse reward exploration method based on causal reasoning, characterized in that, The method includes: The global state space is divided into Subspace ; Construct an exploration target extraction module, which is used to weight the subspace according to the structural causal model. The structured causal model is updated so that actions generated by the action distribution that depends on the policy network are replaced with actions generated by the action counterfactual distribution that depends on prior selection. Construct an intrinsic reward modification module, which is used to modify the reward based on the subspace weights. For this subspace The original intrinsic reward Weighting is performed to generate a total intrinsic reward, and a combined reward is obtained based on the total intrinsic reward and the environmental extrinsic reward. The exploration target extraction module and the intrinsic reward modification module are embedded into the multi-agent reinforcement learning baseline architecture to form the MARL model; Train the MARL model until the policy performance converges.

2. The method according to claim 1, characterized in that, Subspace weights are determined based on a structural causal model. Updates will be made, including: Global state in the actual trajectory of technology Distribution ; Choose a counterfactual value Replace global state The first in Each component To obtain the replaced state ; For the replaced state copy After that, joint actions will be taken. Each component is replaced by uniform random sampling to obtain counterfactual intervention trajectory data. ; Counterfactual intervention trajectory data Input a traditional reward model, and output a predicted reward for each data point. ; Combine global state Distribution And action counterfactual distribution, defining counterfactual values. With predicted rewards direct causal relationship ;in, Indicates intervention; Statistical subspace Overall Predictive Reward Distribution of Counterfactual Intervention Trajectory Data ; Based on direct causation Compared with the overall predicted reward distribution Obtain subspace weights .

3. The method according to claim 2, characterized in that, The subspace weights ;in, Representing subspace China's anti-factual value Quantity, This indicates the calculation of KL divergence.

4. The method according to claim 1, characterized in that, The original intrinsic reward ;in, It is the global state. The Middle The number of times each component is counted. This is a hyperparameter.

5. The method according to claim 1, characterized in that, The method further includes: The target task is completed based on the trained MARL model; wherein, the target task includes various multi-agent tasks, such as collaborative sorting tasks in warehousing and logistics.

6. The method according to claim 1, characterized in that, The process of completing collaborative sorting tasks in warehousing and logistics based on the trained MARL model includes: Obtain the global state of the storage environment and the global state It is divided into a self-state subspace reflecting the pose information of the intelligent agent, a cargo association subspace reflecting the relative relationship between the intelligent agent and the target cargo, a cargo guidance subspace reflecting the location information of the unloading area, and an obstacle avoidance subspace reflecting the distribution of environmental obstacles. Calculate the average causal effect of each subspace on the sparse reward triggered by the successful task, and identify subspaces with weights. The cargo-related subspace and unloading guidance subspace that exceed the set threshold are the target exploration areas; When the state component of the cargo-associated subspace indicates that the agent does not hold cargo and is outside the grasping range, it is subject to the weight of the corresponding subspace. Guided by the system, the intelligent agent performs a movement action that approaches the coordinates of the goods, and performs a goods grabbing action when it enters the grabbing range; When goods are already in hand and the object enters the sensing range of the unloading guidance subspace, it is subject to the corresponding subspace weight. Guided by the system, the intelligent agent executes the unloading action of aligning with the unloading point and placing the goods. It monitors its own state subspace and obstacle avoidance subspace. If there is a collision risk, it adjusts the intrinsic reward according to the weight to trigger the avoidance action, thereby ensuring the stability of the strategy output by eliminating causal confounding factors caused by historical trajectory conflicts. Each agent updates its strategy based on the combined rewards generated by the intrinsic reward modification module, and finally outputs the joint action sequence that optimizes the overall sorting efficiency.

7. A multi-agent sparse reward exploration system based on causal reasoning, characterized in that, The system includes: The space partitioning module is used to divide the global state space into... Subspace ; The first construction module is used to construct an exploration target extraction module, which is used to weight the subspace according to a structural causal model. The structured causal model is updated so that actions generated by the action distribution that depends on the policy network are replaced with actions generated by the action counterfactual distribution that depends on prior selection. The second construction module is used to construct the intrinsic reward modification module, which is used to modify the subspace weights. For this subspace The original intrinsic reward Weighting is performed to generate a total intrinsic reward, and a combined reward is obtained based on the total intrinsic reward and the environmental extrinsic reward. The embedding module integrates the exploration target extraction module and the intrinsic reward modification module into the multi-agent reinforcement learning baseline architecture to form the MARL model. The model training module is used to train the MARL model until the policy performance converges.

8. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the multi-agent sparse reward exploration method based on causal reasoning as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the multi-agent sparse reward exploration method based on causal reasoning as described in any one of claims 1-6.

10. A computer program product, characterized in that, When the computer program product is run on a computer device, the computer device performs the multi-agent sparse reward exploration method based on causal reasoning as described in any one of claims 1-6.

Images