Multi-agent sparse reward exploration method and system based on meta learning

Abstract

The invention discloses a multi-agent sparse reward exploration method and system based on meta-learning, and belongs to the field of multi-agent cooperation. According to the method, the important subspace of the multi-agent sparse reward task is adaptively searched through two important components: 1) an exploration target updating module based on meta-learning is proposed to dynamically search an exploration target suitable for the task; and 2) an internal reward modification module for balancing different subspaces is provided, and the intelligent agent is guided to efficiently explore the subspace which is more important for obtaining the reward, so that the exploration efficiency is improved.

Description

Technical Field

[0001] This invention belongs to the field of multi-agent cooperation, specifically relating to a multi-agent sparse reward exploration method and system based on meta-learning. Background Technology

[0002] Large-scale, complex, continuous collaborative decision-making problems pose significant challenges to human decision-making capabilities in fields such as industry, traffic control, and network routing. The lack of effective solutions leads to substantial resource waste. Improving the execution efficiency and accuracy of decision-making algorithms has become a crucial issue. Solutions to this problem can be broadly categorized into rule-based control methods and multi-agent reinforcement learning methods. Due to the complexity of real-world environments, manually designing control rules and dense rewards for agents often leads to getting trapped in local optima. Exploring rewards has become a key challenge for the practical application of multi-agent algorithms.

[0003] Multi-agent exploration aims to collect reward signals from the environmental state space and is the cornerstone of agent collaboration. Related research mainly focuses on introducing intrinsic rewards to compensate for the sparsity of extrinsic rewards, including guiding agents to choose influential behaviors, explore novel states, and explore diverse trajectories. Based on the methods and criteria for calculating intrinsic rewards, existing multi-agent exploration methods can be broadly divided into two categories: self-exploration and global exploration.

[0004] Self-exploration focuses on introducing intrinsic goals at the individual agent level. Specifically, Zheng et al. (Zheng L, Chen J, Wang J, et al. Episodic multi-agent reinforcement learning with curiosity-driven exploration[J]. Advances in Neural Information Processing Systems, 2021, 34: 3757-3769.) use the prediction error of the individual Q-value as an intrinsic reward; Zhang et al. (Zhang S, Cao J, Yuan L, et al. Self-Motivated Multi-Agent Exploration[C] / / Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems. 2023: 476-484.) expand the exploration space of each agent using state edge matching; Kim et al. (Kim W, Sung Y. An adaptive entropy-regularization framework for multi-agent reinforcement learning[C] / / International Conference on MachineLearning. PMLR, 2023: (16829-16852.) The necessity of exploration for each agent is assessed, and the optimal target entropy is determined to drive maximum entropy exploration. The social influence approach captures the dependencies between agents by measuring the impact of an agent's behavior on the behavior of other agents. Wang et al. (Wang T, Wang J, Wu Y, et al. Influence-based multi-agent exploration[J]. arXiv preprint arXiv:1910.05512, 2019.) quantified the impact of an agent's behavior on the transfer dynamics and expected rewards of other agents.Li et al. (Li C, Wang T, Wu C, et al. Celebrating diversity in shared multi-agent reinforcement learning[J]. Advances in Neural Information Processing Systems, 2021, 34: 3991-4002.) promote diverse individualized behaviors by utilizing the mutual information between agent identity and its local trajectory.

[0005] Global exploration, on the other hand, emphasizes encouraging a comprehensive exploration of the state space. For example, Mahajan et al. (Mahajan A, Rashid T, Samvelyan M, et al. Maven: Multi-agent variational exploration[J]. Advances in neural information processing systems, 2019, 32.) generated diverse global behaviors by maximizing the mutual information between global trajectories and latent variables. Yang et al. (Yang M, Yang Y, Lu Z, et al. Hierarchical multi-agent skill discovery[J]. Advances in Neural Information Processing Systems, 2024, 36.) proposed a skill discovery method that develops team-level skills, promoting diverse trajectories from both individual and global perspectives. Chen et al. (Chen X, Liu X, Zhang S, et al. Goal Consistency: An Effective Multi-Agent Cooperative Method for MultistageTasks[C] / / IJCAI. 2022: 172-178.) employed a goal-oriented multistage model to enhance goal cognition, helping agents understand tasks at the goal level and thus achieving cooperative global exploration. Iqbal et al. (Iqbal S, Sha F. Coordinated exploration via intrinsic rewards for multi-agent reinforcement learning[J]. arXiv preprint arXiv:1905.12127, 2019.) and Chitnis et al. (Chitnis R, Tulsiani S, Gupta S, et al. Intrinsic motivation for encouraging synergistic behavior[J]. arXiv preprint arXiv:2002.05189, 2020.) respectively motivated collective behavior in multi-agent systems by creating heuristic intrinsic rewards.

[0006] In summary, self-exploration offers a simple and scalable approach, but may lead to a lack of coordination in exploration behavior among agents. In contrast, global exploration helps promote cooperative behavior, but faces challenges such as exploration space explosion, especially in large-scale complex tasks involving a large number of agents. Subspace exploration, as a novel solution, addresses this problem by reducing the exploration space.

[0007] However, existing subspace methods such as CMAE (Liu IJ, Jain U, Yeh RA, et al. Cooperative exploration for multi-agent deep reinforcement learning[C] / / International conference on machine learning. PMLR, 2021: 6826-6836.) and FOX (JoY, Lee S, Yeom J, et al. FoX: Formation-aware exploration in multi-agent reinforcement learning[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(12): 12985-12994.) rely on projection and forming techniques to map high-dimensional state spaces to low-dimensional latent spaces. These simple projections lack semantic information and perform poorly in complex multi-agent cooperative tasks. SAME (Xu P, Zhang J, Yin Q, et al. Subspace-aware exploration for sparse-reward multi-agent tasks[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(10): 11717-11725.) uses the optimistic principle in the face of uncertainty to evaluate the importance of each subspace, but this is only an approximation based on priors and does not fundamentally explore the impact of different subspaces on rewards or exploration efficiency. The above problems have prompted researchers to think deeply about the significance and role of subspace weights in existing subspace exploration methods. They proposed that the core significance of subspace weights is to explicitly give clear exploration goals. Based on this, a multi-agent sparse reward exploration method based on meta-learning was constructed. It adaptively finds important subspaces of the environment through gradient updates. At the same time, an intrinsic reward module that balances different subspaces was designed to ensure the consistency between the actual exploration direction and the exploration goal, thereby improving the interpretability of subspace weights. Summary of the Invention

[0008] The problem addressed by this invention is the low exploration efficiency of current methods under sparse reward conditions. It re-examines the learning paradigm of multi-agent exploration methods and proposes a meta-learning-based multi-agent sparse reward exploration method and system. This method adaptively finds important subspaces for multi-agent sparse reward tasks through two key components: 1) a meta-learning-based exploration target update module to dynamically find suitable exploration targets for the task; and 2) an intrinsic reward modification module that balances different subspaces, guiding agents to efficiently explore subspaces more important for obtaining rewards, thereby improving exploration efficiency.

[0009] To achieve the above objectives, the technical solution of the present invention includes the following:

[0010] A multi-agent sparse reward exploration method based on meta-learning, the method comprising: The global state space is divided into Subspace ; Construct an intrinsic reward modification module, which is used to modify the reward based on the subspace weights. For this subspace The original intrinsic reward Weighting is performed to generate a total intrinsic reward, and a combined reward is obtained based on the total intrinsic reward and the environmental extrinsic reward. Construct a meta-learning exploration goal update module, which is used to update subspace weights based on the cumulative environmental external rewards over a lifecycle. Update; The intrinsic reward modification module and the meta-learning exploration goal update module are embedded into the multi-agent reinforcement learning baseline architecture to form the MARL model; Train the MARL model until the policy performance converges.

[0011] Furthermore, the original intrinsic reward ;in, It is a state The Middle The number of distinct sub-states for each component has been counted. For state The neutron states of each component The frequency of occurrence, It is a subspace A certain substate in.

[0012] Furthermore, the cumulative environmental extrinsic rewards based on the lifecycle are weighted in the subspace. Updates will be made, including: For each non-overlapping lifetime, count the lifetime from time [time]. Initial lifecycle cumulative rewards ; Calculate the cumulative reward based on the lifecycle under the current agent's policy. The objective function for subspace weights The gradient; The subspace weights are updated based on the gradient. To obtain the subspace weights for the next lifecycle. .

[0013] Furthermore, the cumulative lifecycle reward ;in, for Environmental external rewards at any time, total lifespan , Indicates the number of lifecycles. Indicates the length of a single lifecycle. express The reward discount factor for each moment.

[0014] Furthermore, the gradient ;in, It is the subspace weight. It is an objective function based on the cumulative reward over the lifecycle. These are the agent's policy parameters. The strategy parameter is The intelligent agent in the state Select action The probability, Represents the weights of the subspace gradient operator, Indicates policy parameters The expected value.

[0015] Furthermore, the intrinsic reward modification module and the meta-learning exploration goal update module are embedded into the multi-agent reinforcement learning baseline architecture to form a MARL model, including: Use the combined reward to replace the reward item in the multi-agent reinforcement learning baseline architecture; Within the lifecycle, strategy parameters Updated according to the multi-agent reinforcement learning baseline; wherein, the policy parameters The update goal is to maximize the use of combined reward calculations. Value or strategy return; After the lifecycle ends, the cumulative environmental external reward at the current moment is obtained, and the subspace weights are updated in conjunction with the meta-learning exploration goal update module. Update.

[0016] Furthermore, the method also includes: Based on the trained MARL model, complete tasks such as multi-agent cooperative path planning, multi-agent resource scheduling and allocation, or multi-agent target search and region coverage.

[0017] Furthermore, the process of completing the multi-agent cooperative path planning task based on the trained MARL model includes: Deploy the trained MARL model in the target environment; Without introducing meta-learning weight updates, the subspace weights are fixed. and strategy parameters ; In path planning scenarios, obtain the global state, including the positions of multiple agents, the target point, the distribution of obstacles, and environmental constraints. The global state is mapped into subspaces. The MARL model, trained using sub-state inputs, utilizes policy parameters. Output the action decisions of each agent; Based on the action decisions of each agent, their pose or discrete grid position is updated to form a multi-agent joint path sequence. During the path update process, multi-agent cooperative constraint processing is performed, which includes collision avoidance, path conflict resolution, and formation / communication connectivity maintenance. When all agents reach their respective target points, or the target area coverage reaches a threshold, or the maximum number of steps is reached, execution terminates and the path planning result is output to complete the multi-agent collaborative path planning task.

[0018] A multi-agent sparse reward exploration system based on meta-learning, the system comprising: The space partitioning module is used to divide the global state space into... Subspace ; The first construction module is used to construct the intrinsic reward modification module, which is used to modify the subspace weights. For this subspace The original intrinsic reward Weighting is performed to generate a total intrinsic reward, and a combined reward is obtained based on the total intrinsic reward and the environmental extrinsic reward. The second construction module is used to construct a meta-learning exploration goal update module, which uses subspace weights based on the cumulative environmental external rewards over a lifecycle. Update; The embedding module is used to embed the intrinsic reward modification module and the meta-learning exploration goal update module into the multi-agent reinforcement learning baseline architecture to form a MARL model. The model training module is used to train the MARL model until the policy performance converges.

[0019] An electronic device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the multi-agent sparse reward exploration method based on meta-learning as described above.

[0020] Compared with the prior art, the present invention has at least the following beneficial effects.

[0021] 1) The meta-learning-based exploration target update module updates the exploration target through gradient update based on the cumulative reward of the external environment, realizing lightweight adaptive search of the exploration target, effectively improving exploration efficiency while having a relatively low computational cost.

[0022] 2) The intrinsic reward modification module for balancing different subspaces achieves consistency between the reward-guided exploration direction and the exploration goal, enabling accurate and efficient exploration of important subspaces described by the exploration goal, thus improving exploration efficiency and interpretability.

[0023] 3) This invention is a plug-and-play method that only adjusts the intrinsic reward received by the agent. It can be applied to various multi-agent value decomposition model architectures and improves the performance of the model in sparse reward exploration scenarios. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the implementation of the present invention. Detailed Implementation

[0025] The specific embodiments of the present invention will now be described in detail with reference to examples and the accompanying drawings. The examples described herein are for illustrative and explanatory purposes only and are not intended to limit the scope of the invention.

[0026] This invention designs a multi-agent sparse reward exploration method that incorporates meta-learning search environment state space exploration objectives, comprising the following steps: Before conducting a multi-agent collaborative task, firstly, a multi-agent reinforcement learning environment required for the task is prepared. This environment includes specific definitions of the global state, agent actions, reward signals, initial state, and termination conditions, and the environment dynamics model conforms to the actual task. The environment receives agent actions and provides feedback on the current reward and the next global state. Secondly, the exploration objective—the reward weights of all subspaces—is initialized. An intrinsic reward modification module based on the environment state subspace is designed. This module constructs intrinsic rewards that balance different subspaces, guiding the exploration direction of intrinsic rewards towards the exploration objective. Then, a meta-learning exploration objective update module is designed to update the subspace reward weights based on the lifetime accumulated reward. This module updates the subspace weights using a gradient update method, correcting the exploration objective to guide the exploration direction of intrinsic rewards. Finally, the intrinsic reward modification module and the exploration objective update module are introduced together into the traditional multi-agent reinforcement learning baseline architecture as an adaptive exploration model. Next, the intrinsic reward and extrinsic reward are combined as the training reward for the agent's policy. The lifetime cumulative reward is calculated periodically using only the extrinsic reward, and the adaptive exploration model is trained until convergence. Finally, the exploration objective of the trained adaptive exploration model is extracted, the model is initialized, and the exploration objective is initialized to the trained objective. The model is then trained again to test whether its ability to explore sparse rewards has improved.

[0027] Step 1: Prepare the multi-agent reinforcement learning environment required for the task. The environment includes specific definitions of the global state, agent actions, reward signals, initial state, and termination conditions, and the environment dynamics model conforms to the actual task. The environment receives agent actions and executes them, providing feedback on the current reward and the global state for the next time step.

[0028] Step 2: Initialize the exploration goal—the reward weights of all subspaces, and design an intrinsic reward modification module based on the environment state subspace. This module constructs an intrinsic reward that balances different subspaces, guiding the exploration direction of the intrinsic reward towards the exploration goal.

[0029] Step 2.1: For real trajectory data , Indicates an action, and its state. The original intrinsic reward of the subspace is calculated as follows: in, It is a subspace The original intrinsic reward, It is a state The Middle The number of distinct sub-states for each component has been counted. For state The neutron states of each component The frequency of occurrence, It is a subspace A certain substate in, This is a hyperparameter.

[0030] Step 2.2: For each subspace reward, introduce the exploration objective to weight the reward, and obtain the total intrinsic reward at the current moment. As shown in the following formula: Step 2.3: Combine the total intrinsic reward with the extrinsic reward to form the actual reward received by the agent at the current moment, as shown in the following formula: in, Indicates external environmental rewards, This is a hyperparameter.

[0031] Step 2.4: Insert this module as a plugin into the traditional multi-agent reinforcement learning value decomposition baseline model, replacing the original reward item.

[0032] Step 3: Design a meta-learning exploration goal update module that updates the subspace reward weights based on the cumulative reward over the lifecycle. This module updates the subspace weights using a gradient update method to correct the exploration goal and guide the exploration direction of intrinsic rewards.

[0033] Step 3.1: For each non-overlapping lifecycle—continuous The trajectory is used to calculate the cumulative reward over its lifecycle, as shown in the following formula: in, Indicates the time from Accumulated rewards at the start of the lifecycle. The length of the lifecycle, i.e., continuous The length of the trajectory, The length of a single trajectory. As a discount factor for rewards, for External environmental rewards at any given moment.

[0034] Step 3.2: After each lifecycle ends, calculate the gradient of the objective function based on the cumulative reward over the lifecycle with respect to the subspace weights under the current agent's policy, as shown in the following formula: in, It is the subspace weight. It is an objective function based on the cumulative reward over the lifecycle. These are the agent's policy parameters. Is the agent in a state? Select action The probability, Represents the weights of the subspace gradient operator, Indicates policy parameters The expected value.

[0035] Step 3.3: Update each subspace based on the obtained gradients The weights are used to determine the exploration target for the next lifecycle, as shown in the following formula. in, This is a hyperparameter.

[0036] Step 4: After completing Step 2 (intrinsic reward module) and Step 3 (meta-learning update module), embed both into the multi-agent reinforcement learning baseline architecture to form a MARL model (multi-agent reinforcement learning model) with adaptive exploration capabilities.

[0037] Step 4.1: In the loss calculation process of the baseline algorithm, use the combined reward from step 2.3. The original reward items are replaced, making the new module seamlessly compatible with mainstream MARL algorithms.

[0038] Step 4.2: Within the lifecycle, strategy parameters The update follows the baseline approach, with the goal of maximizing the Q-value or policy reward calculated using combined rewards.

[0039] Step 4.3: Trigger the weight update process of Step 3 after the lifecycle ends: accumulate lifecycle rewards, update weights according to the meta-gradient formula, and form a two-layer optimization structure of "lifecycle update strategy + lifecycle update exploration target".

[0040] Step 5: Train using the complete model integrated in Step 4, and repeat the following process until the policy performance converges: Step 5.1: Policy Update (Inner Loop) Update strategy parameters based on combined rewards throughout the lifecycle. .

[0041] Step 5.2: Subspace weight update (outer loop) After the lifecycle ends, the weights are updated via meta-gradient. .

[0042] Step 5.3: Iterate Repeatedly execute the inner and outer loops to allow the strategy and exploration objectives to evolve together until the task rewards no longer improve or the preset conditions are met.

[0043] Step 6: After training converges, extract the final learned weight vector. As an exploration target, a two-phase verification was performed: Step 6.1: Configure the policy network Random initialization is used to exclude the influence of previous strategies on the evaluation.

[0044] Step 6.2: Fix the exploration weights, do not perform meta-learning updates, retrain the policy using only the intrinsic reward construction method from Step 2, and evaluate its exploration ability.

[0045] Step 6.3: Compare indicators such as the time of first triggering of external rewards, convergence speed, average reporting, and exploration trajectory coverage; if the performance is significantly improved after fixing the exploration target, it proves that the exploration target obtained by this invention has transferable exploration value.

[0046] Step 7: Apply the validated exploration objectives to the execution and deployment of the multi-agent collaborative path planning task.

[0047] In multi-agent cooperative path planning tasks, existing technologies typically employ fixed intrinsic rewards or uniform exploration strategies, making it difficult to adaptively allocate exploration value for different state subspaces. This results in low exploration efficiency, slow convergence, and a tendency to get trapped in local optima under sparse extrinsic rewards. This invention utilizes a meta-learning mechanism based on lifecycle-accumulated extrinsic rewards to weight each subspace. Adaptive updates are performed to dynamically focus intrinsic rewards on subspaces more likely to generate extrinsic rewards, thereby improving exploration efficiency and final strategy performance in sparse reward scenarios. Therefore, after verifying the fixed subspace exploration weights in step 6 to confirm their transferable exploration value, the present invention executes the following process: Step 7.1: Solidify the finally learned subspace weight vector As the exploration target parameter for the path planning task, it is combined with the parameters of the multi-agent policy network. And save it as a deployment model; Step 7.2: Load the deployment model in the multi-agent path planning environment, obtain the global state including the initial position of each agent, the target point position, the obstacle distribution, and map constraints, and map the global state to each subspace. The sub-states are used as model inputs; Step 7.3: Based on fixed exploration weights Construct a weighted intrinsic reward and combine it with extrinsic environmental rewards to form a combined reward, driving multi-agent agents to follow policy parameters. Each agent outputs its own action to generate a multi-agent joint path planning trajectory. Step 7.4: Perform cooperative constraint processing during path generation, which includes at least one of the following: collision avoidance among multiple agents, path conflict resolution, and communication connectivity maintenance; Step 7.5: Continuously update the pose or discrete grid position of each agent until all agents reach the corresponding target point, or the target area coverage reaches the preset threshold, or the maximum number of execution steps is reached, then terminate the planning process. Step 7.6: Output the final planned path and corresponding performance indicators for each agent. The performance indicators include at least one of the following: task success rate, average path length, average arrival time, time to trigger external reward for the first time, trajectory coverage, and number of effective explorations per unit time. Then, complete the deployment and application of the path planning task.

[0048] In summary, this invention forms an adaptive subspace exploration framework for sparse reward multi-agent tasks by explicitly dividing the global state space into multiple subspaces, independently constructing intrinsic rewards in each subspace, and combining this with a meta-learning weight update mechanism based on lifecycle-accumulated external rewards.

[0049] This invention can automatically identify key subspaces that are highly correlated with task rewards. By backpropagating external rewards to subspace reward weights via policy gradients over the lifecycle scale, this invention does not simply adjust weights based on the number of visits or training rounds, but explicitly characterizes the marginal contribution of each subspace to long-term task rewards. This allows the system to gradually increase the exploration intensity of key state dimensions (such as key agent positions, key object states, etc.) during training, while suppressing ineffective exploration of irrelevant dimensions.

[0050] This invention significantly reduces the effective exploration dimensionality of the joint state-action space, improving sample efficiency in sparse reward scenarios. In multi-agent environments, the joint state-action space expands exponentially with the number of agents, and directly conducting uniform exploration across the entire space easily leads to local optima or prolonged lack of effective external feedback. This invention subdivides the state into subspaces and assigns exploration weights at the subspace level, essentially performing an "importance-weighted subspace exploration" in a high-dimensional space. This concentrates exploration on the state components most sensitive to the task, thereby increasing the probability of reaching sparse rewards without increasing the number of environmental interactions.

[0051] This invention maintains the simplicity of the intrinsic reward module while enhancing the flexibility and transferability of the overall exploration strategy. In this invention, the original intrinsic reward of the subspace can be constructed using any monotonically decreasing frequency mapping function, without relying on specific maximum entropy projections or complex density estimation models. The dominant change in the exploration strategy focuses on the meta-learning update of the subspace weights, enabling the same intrinsic reward construction to be quickly adapted to different tasks and multi-agent environments by retraining the weight module, thus exhibiting good versatility.

[0052] This invention can operate stably in extremely sparse multi-agent tasks with strong cooperative constraints. It introduces the concept of a "lifecycle," aggregating the external rewards of multiple trajectories and using this as an evaluation signal for weight updates, thus alleviating the difficulty of gradient estimation caused by the extreme sparsity of single-step rewards. Combined with a multi-agent value decomposition framework, this invention can effectively distinguish the contribution of different subspaces to the overall team performance in cooperative tasks, experimentally demonstrating faster convergence speed and higher success rate.

[0053] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-agent sparse reward exploration method based on meta-learning, characterized in that, The method includes: The global state space is divided into Subspace ; Construct an intrinsic reward modification module, which is used to modify the reward based on the subspace weights. For this subspace The original intrinsic reward Weighting is performed to generate a total intrinsic reward, and a combined reward is obtained based on the total intrinsic reward and the environmental extrinsic reward. Construct a meta-learning exploration goal update module, which is used to update subspace weights based on the cumulative environmental external rewards over the lifecycle. Update; The intrinsic reward modification module and the meta-learning exploration goal update module are embedded into the multi-agent reinforcement learning baseline architecture to form the MARL model; Train the MARL model until the policy performance converges.

2. The method according to claim 1, characterized in that, The original intrinsic reward ;in, It is a state The Middle The number of distinct sub-states for each component has been counted. For state The neutron states of each component The frequency of occurrence It is a subspace A certain substate in.

3. The method according to claim 1, characterized in that, Lifecycle-based cumulative environmental external reward pair subspace weights Updates will be made, including: For each non-overlapping lifetime, the lifetime from time [time] is counted. Initial lifecycle cumulative rewards ; Calculate the cumulative reward based on the lifecycle under the current agent's policy. The objective function for subspace weights The gradient; The subspace weights are updated based on the gradient. To obtain the subspace weights for the next lifecycle. .

4. The method according to claim 3, characterized in that, The cumulative rewards over the lifetime ;in, for Environmental external rewards at any time, total lifespan , Indicates the number of lifecycles. Indicates the length of a single lifecycle. express The reward discount factor for each moment.

5. The method according to claim 3, characterized in that, The gradient ;in, It is the subspace weight. It is an objective function based on the cumulative reward over the lifecycle. These are the agent's policy parameters. The strategy parameter is The intelligent agent in the state Select action The probability, Represents the weights of the subspace gradient operator, Indicates policy parameters The expected value.

6. The method according to claim 1, characterized in that, The intrinsic reward modification module and the meta-learning exploration goal update module are embedded into the multi-agent reinforcement learning baseline architecture to form the MARL model, which includes: Use the combined reward to replace the reward item in the multi-agent reinforcement learning baseline architecture; Within the lifecycle, strategy parameters Updated according to the multi-agent reinforcement learning baseline; wherein, the policy parameters The update goal is to maximize the use of combined reward calculations. Value or strategy return; After the lifecycle ends, the cumulative environmental external reward at the current moment is obtained, and the subspace weights are updated in conjunction with the meta-learning exploration goal update module. Update.

7. The method according to claim 1, characterized in that, The method further includes: Based on the trained MARL model, complete tasks such as multi-agent cooperative path planning, multi-agent resource scheduling and allocation, or multi-agent target search and region coverage.

8. The method according to claim 7, characterized in that, The process of completing a multi-agent cooperative path planning task based on a trained MARL model includes: Deploy the trained MARL model in the target environment; Without introducing meta-learning weight updates, the subspace weights are fixed. and strategy parameters ; In path planning scenarios, obtain the global state, including the positions of multiple agents, the target point, the distribution of obstacles, and environmental constraints. The global state is mapped into subspaces. The MARL model, trained using sub-state inputs, utilizes policy parameters. Output the action decisions of each agent; Based on the action decisions of each agent, their pose or discrete grid position is updated to form a multi-agent joint path sequence. During the path update process, multi-agent cooperative constraint processing is performed, which includes collision avoidance, path conflict resolution, and formation / communication connectivity maintenance. When all agents reach their respective target points, or the target area coverage reaches a threshold, or the maximum number of steps is reached, execution terminates and the path planning result is output to complete the multi-agent collaborative path planning task.

9. A multi-agent sparse reward exploration system based on meta-learning, characterized in that, The system includes: The space partitioning module is used to divide the global state space into... Subspace ; The first construction module is used to construct the intrinsic reward modification module, which is used to modify the subspace weights. For this subspace The original intrinsic reward Weighting is performed to generate a total intrinsic reward, and a combined reward is obtained based on the total intrinsic reward and the environmental extrinsic reward. The second construction module is used to construct a meta-learning exploration goal update module, which uses subspace weights based on the cumulative environmental external rewards over a lifecycle. Update; The embedding module is used to embed the intrinsic reward modification module and the meta-learning exploration goal update module into the multi-agent reinforcement learning baseline architecture to form a MARL model. The model training module is used to train the MARL model until the policy performance converges.

10. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the multi-agent sparse reward exploration method based on meta-learning as described in any one of claims 1-8.

Images