Double-stage multi-agent cooperation method based on exploration reward molding

Abstract

The invention discloses a two-stage multi-agent cooperation strategy based on exploration reward molding, and belongs to the field of multi-agent reinforcement learning. Trajectory data (including environment states, rewards, rewards and actions) generated by interaction of the intelligent agent and the environment are stored in an experience buffer pool and are updated and maintained through increment. Subsequently, randomly sampling an environment state and a corresponding return from the experience pool, and constructing a conditional diffusion model by using the return as a condition to generate a high-return target state; thirdly, global environment states of different time steps in different trajectories are sampled, time structure mapping is learned, and states with similar time are mapped to hidden states with similar geometric space; according to the method, a double-end Q network is adopted, an exploration strategy is decoupled into a target exploration strategy and a behavior exploration strategy, reward functions corresponding to two stages respectively act on a decision-making network of an intelligent agent, and more effective exploration and collaboration are achieved.

Description

Technical Field

[0001] This invention belongs to the field of multi-agent reinforcement learning and computer program technology, and relates to a two-stage multi-agent collaborative method based on exploration reward shaping. Background Technology

[0002] In the field of multi-agent reinforcement learning, efficient exploration and collaborative decision-making are key to agents learning effective strategies in complex environments, and are widely used in scenarios such as drone swarming, intelligent traffic scheduling, and distributed robot collaboration. Traditional multi-agent exploration methods mainly rely on independent exploration or simple shared reward mechanisms. However, in high-dimensional state spaces, sparse reward environments, or dynamic interaction environments, these methods often face problems such as low exploration efficiency, unbalanced credit allocation, and slow policy convergence.

[0003] In recent years, value-decomposition-based multi-agent reinforcement learning methods have made significant progress in collaborative tasks. Their core idea is to optimize the global policy through centralized training and a distributed execution framework. However, these methods still have limitations in their exploration mechanisms: on the one hand, sparse reward environments make it difficult for agents to acquire effective learning signals, resulting in a lack of direction in exploration; on the other hand, traditional exploration strategies (such as ε-greedy) fail to fully consider the collaborative characteristics of multi-agent systems, leading to inconsistencies between individual behavior and the global goal. Furthermore, most existing methods adopt a single-stage exploration model, failing to distinguish between the different needs of goal-oriented exploration and environmental awareness exploration, making it difficult for agents to balance immediate exploration gains and the ultimate collaborative goal in long-term tasks.

[0004] Recent research has begun exploring more efficient exploration mechanism designs, attempting to enhance the targeting of task objectives while maintaining exploration diversity. One scholar proposed a hierarchical exploration framework that separates global goal exploration from local behavior exploration, using upper-layer networks to generate potential goals to guide the exploration direction of lower-layer strategies. This approach alleviates the inefficiency of random exploration to some extent, but the discretization of the goal space limits its applicability in continuous state spaces. Furthermore, other scholars have introduced a goal generation mechanism based on a diffusion model, generating more informative exploration goals by modeling the probability distribution of the state space; however, the adaptability of this method in dynamic environments still needs improvement.

[0005] Regarding the balance between exploration and collaboration, some scholars have innovatively proposed a method for maximizing collaborative exploration entropy. This method maintains exploration diversity by optimizing the joint policy entropy of the agent group, while simultaneously utilizing attention mechanisms to dynamically adjust the collaborative relationships among agents. Experiments show that this method achieves good results in some collaborative tasks, but in fully collaborative tasks, excessive focus on diversity may lead to deviation from the team's goal. To address this issue, some scholars have developed goal-aware diversity exploration algorithms that constrain policy diversity within a task-related subspace. However, such artificially imposed constraints may limit the agents' ability to discover better collaborative strategies.

[0006] While these studies have improved multi-agent exploration mechanisms from different perspectives, the following key challenges remain unresolved: how to achieve automatic generation and dynamic adjustment of exploration goals, avoiding reliance on manual design or fixed courses; how to balance individual exploration freedom with the needs of team collaboration, preventing strategy dispersion due to over-exploration; and how to design a universal exploration evaluation mechanism to enable autonomous judgment and switching between exploration phases. These challenges mean that existing methods still exhibit insufficient adaptability and low exploration efficiency when facing complex and ever-changing real-world application scenarios. Summary of the Invention

[0007] This invention addresses the problems of existing technologies by providing a two-stage multi-agent collaborative method based on exploration reward shaping.

[0008] A two-stage multi-agent cooperative method based on exploration reward shaping includes the following steps:

[0009] Step S1: Collect trajectory data generated by the interaction between the agent and the environment, including the current environmental state, current reward, environmental reward, actions, etc., and save it to the experience buffer pool. Update the experience pool using an incremental update method.

[0010] Step S2: Randomly sample the environment state, target state, and reward of the target state from the experience buffer pool, and construct a conditional diffusion model to generate a high-reward target state using the reward as the generation condition.

[0011] Step S3: Sample the global environment state at different time steps in different trajectories from the experience buffer pool, learn the temporal structure mapping, and map temporally similar states to geometrically similar hidden states.

[0012] Step S4: Calculate the L2 distance d1 between the current environment state and the target state in the latent space, the L2 distance d2 between the next time step and the target state in the latent space, and finally the reward function is d1-d2.

[0013] Step S5: Obtain the global environment state of two adjacent time steps and control the observation changes of each agent in the next time step, thereby decomposing the overall exploration reward into the sum of the influence of each agent on the target exploration.

[0014] Step S6: Add random noise to the environmental state of the next time step, and calculate the average distance between the current time step state and these noisy state sets in the latent space as a reward for behavioral exploration.

[0015] Step S7: Using a dual-head Q-network, the exploration strategy is decoupled into a goal exploration strategy and a behavior exploration strategy, and the reward functions of the two different stages are applied to the agent's decision network at their respective stages.

[0016] This invention proposes a two-stage multi-agent collaborative method based on exploration reward shaping, which significantly improves exploration efficiency and collaborative capabilities in the field of multi-agent reinforcement learning. This method effectively solves the problems of inefficiency and credit allocation imbalance in traditional methods under sparse rewards and dynamic interactive environments through an innovative two-stage exploration mechanism. Its core breakthrough lies in decoupling the exploration process into two stages: goal-oriented exploration and environmental cognition exploration, and designing corresponding reward functions for each stage. This allows the agent to quickly locate high-value areas and fully learn the dynamic characteristics of the environment. The combination of a conditional diffusion model and a temporal structure latent space mapping enables the automatic generation of high-reward goals and a geometric measurement of state evolution relationships, providing the agent with a clear and computable intrinsic exploration direction and significantly reducing ineffective exploration behavior.

[0017] In terms of collaborative optimization, this method accurately allocates global exploration rewards to individual agents through a contribution decomposition mechanism, ensuring consistency between local behaviors and team goals. Simultaneously, the behavior exploration reward design based on a noisy state set maintains exploration diversity while avoiding the risk of policy dispersion. This dynamic balancing mechanism enables agents to flexibly adapt to environmental changes in complex tasks while consistently optimizing strategies around collaborative goals. The introduction of a dual-head Q-network architecture further enhances the stability of policy learning, allowing goal exploration and behavior exploration to proceed in parallel without conflict.

[0018] Another significant advantage of this technology lies in its superior environmental adaptability. Through the synergistic effect of the diffusion model and latent space mapping, adaptive target generation in a continuous state space is achieved, eliminating the reliance on manually designed exploratory courses. The autonomous switching mechanism during the exploration phase further enhances the system's robustness in dynamic environments. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. As shown in the figures:

[0020] Figure 1 This is a flowchart of the multi-agent collaborative method described in an embodiment of the present invention.

[0021] Figure 2 This is an architecture diagram of the method for generating high-return target states based on a diffusion model, as described in an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram of the time structure mapping principle described in an embodiment of the present invention.

[0023] Figure 4 This is a diagram of the dual-head Q network architecture described in an embodiment of the present invention. Detailed Implementation

[0024] After discovering the aforementioned problems, the inventors of this application conducted in-depth research on existing multi-agent reinforcement learning exploration mechanisms. Existing multi-agent exploration methods typically focus on single-dimensional optimization strategies, such as curiosity-driven intrinsic rewards, experience replay mechanisms, or strategy diversity enhancement. These methods improve the learning efficiency of agents by optimizing exploration behavior in a specific dimension. However, in complex real-world environments, single-dimensional exploration strategies often struggle to balance global task requirements with local exploration efficiency. On one hand, because different task stages have different exploration needs, fixed-pattern exploration strategies make it difficult for agents to balance immediate exploration with the final goal in long-term tasks. On the other hand, in multi-agent collaborative scenarios, there is an inherent contradiction between individual exploration behavior and group collaborative goals, making it difficult to coordinate individual and team benefits during the exploration process. These two problems, when mapped to algorithm performance, manifest as a large deviation between the exploration direction and the task goal in the feature space, and a high similarity in exploration strategies among different agents, ultimately making it difficult to effectively distinguish valuable exploration behaviors from ineffective random exploration.

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. It should be noted that, without conflict, the embodiments and features in the embodiments of the present invention can also be combined with each other.

[0026] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In the description of this invention, the terms "first," "second," "third," "fourth," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Example 1: As Figure 1 , Figure 2 , Figure 3 and Figure 4 As shown, a two-stage multi-agent collaborative method based on exploration reward shaping decouples the agent's exploration process into two stages: goal exploration and behavior exploration. In the goal exploration stage, a diffusion model is used to generate high-value ideal state goals. By learning a time mapping, the temporal structure characteristics of the Markov decision process are captured, and goal exploration rewards are designed based on the time mapping to guide the agent swarm to conduct targeted exploration. To further improve collaborative efficiency, a local reward decomposition mechanism is introduced in the goal exploration stage to ensure that agents can reasonably adjust their behavior according to their respective contributions, thereby optimizing the collaborative effect of the multi-agent system. In the behavior exploration stage, a behavior exploration reward mechanism based on state uncertainty utilizes the dynamic characteristics of state transitions to measure the exploration degree of a state, guiding agents to prioritize in-depth exploration in high-uncertainty regions.

[0029] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0030] As a preferred embodiment of the present invention, the incremental update experience buffer pool method in step S1 includes:

[0031] Step S11: Define the trajectory data format.

[0032] Single trajectory data τ t Represented as a quintuple: τ t = t ,a t ,r t ,s t+1 ,R t >,

[0033] Where s t ​∈S, represents the environmental state at time t, where S is the state space; a t ∈A, represents the action executed by the agent at time t, where A is the action space; represents the immediate reward feedback from the environment, is the reward space; s t+1 ∈S, represents the next state at the moment after executing a t ; R t is for s t 's discounted return.

[0034] Step S12, initialize the experience buffer pool. Define the experience buffer pool D as a first-in-first-out queue with a fixed capacity: D = {τ1, τ2, …, τ N}, |D| = N max .

[0035] Where: N max is the maximum capacity of the experience pool, with a default value of 1000.

[0036] Step S13, incrementally update the experience pool.

[0037] For each new trajectory data τ1, perform the following operations:

[0038] D = D ∪ {τ t}, if |D| < N max ; D = D \ {τ oldest} ∪ {τ t}, if |D| = N max ,

[0039] Where τ oldest is the earliest stored trajectory data in the experience pool.

[0040] As a preferred embodiment of the present invention, the method for constructing the conditional diffusion model described in step S2 includes:

[0041] Step S21, data sampling and preprocessing. First, sample triple data <s t , s g , R g > from the experience buffer pool D, where s t and s g belong to the same trajectory, and 0 < t < g < T, where T is the termination time step in a trajectory, and R g is the discounted cumulative return of s g .

[0042] Step S22, in the conditional diffusion process, first, a forward diffusion process needs to be carried out: gradually add Gaussian noise to the latent vector z0, for a total of M steps:

[0043]

[0044] Where z0 is the initial latent vector, which is usually extracted by the encoder from the raw data, and z t This represents the latent vector after adding noise at step t. As t increases, z... t It will gradually transition from the original data distribution to an approximately pure noise distribution.

[0045] Noise scheduling coefficient β t This controls the intensity of noise addition at each step. The default value is usually set to 0.02, but in practice, more complex scheduling strategies, such as linear growth or cosine variation, may be used to balance the rate at which noise is introduced. β t The value of is between 0 and 1, ensuring that 1-β t It has a clear mathematical meaning. The number of diffusion steps M determines the total number of times noise is added. Usually, a sufficiently large number of steps (such as hundreds to thousands of steps) is needed to ensure the smoothness of the noise reduction, but too many steps will increase the computational cost.

[0046] Gaussian distribution mean It retains some information from the latent vector in the previous step, while the variance β t I controls the amplitude of the noise, where I is the identity matrix, indicating that the noise is added independently in each dimension. Through this gradual noisification process, the forward diffusion process eventually transforms z0 into z that is close to pure noise. M .

[0047] Step S23: When the forward diffusion time step reaches M, the reverse diffusion process needs to begin, with R... g As a condition, through Stepwise denoising to generate the target latent vector:

[0048]

[0049] Where z t R is the noisy latent vector at the current time step, where t represents the diffusion step number. g It is the cumulative reward condition that guides the generation process, and it is the maximum reward value in the experience buffer pool.

[0050] μ in the formula θ (z t ,t,R g ) is a neural network The predicted mean, which combines the current noise vector z t Embedded information of time step t and condition signal R g This is used to guide the direction of noise reduction. ∑ θ (z t,t) is the predicted covariance matrix, which is usually simplified to a time-step correlated scalar or diagonal matrix to control the randomness of denoising.

[0051] Step S24: When the reverse diffusion process reaches its final step, i.e., t=0, it needs to be decoded. Decode z0 into state pairs in the original state space: Where z0 is the pure latent variable after the reverse diffusion process is completed. It has removed all added noise and contains all the structured information required to generate the final output.

[0052] Traditional multi-agent reinforcement learning methods typically rely on random exploration or manually designed target states, resulting in low exploration efficiency and difficulty in discovering the globally optimal policy. In contrast, this invention generates ideal targets conditioned on high-reward states through a diffusion model, significantly improving the guidance of exploration. The introduction of the diffusion model also enables the system to automatically adapt to environmental changes, while traditional methods require readjusting target parameters in dynamic environments.

[0053] In a preferred embodiment of the present invention, step S3 involves learning a time structure mapping method, including:

[0054] Step S31, define the time structure mapping function: Its function is to map the states at different time steps in the original state space to the latent space, where S is the original state space. This is a hidden space. h = φ(s) t ), Where h is the embedding vector of the state at a certain time step in the original state space into the latent space. d represents the embedding dimension in the latent space.

[0055] Step S32, define the distance between different states in the latent space as the state-target value estimate: V(s) t ,s t+k )=‖φ(s t )-φ(s t+k )‖2. Where V(s) t ,s t+k ) represents state s t For state s t+k The value of is equivalent to the latent space embedding vector φ(s). t ) and φ(s t+k The Euclidean distance between ) is given by ‖φ(s t )-φ(s t+k )‖2 represents the L2 distance, or Euclidean distance, between two vectors.

[0056] Step S33 requires training the temporal structure mapping using a reinforcement learning paradigm. The goal of reinforcement learning is to optimize the Bellman equation error:

[0057] in It is the target value of the objective value function, usually calculated through an objective network, with fixed parameters to reduce fluctuations; V(s) t ,s g ) is the value function to be optimized, predicting the current state s. t To the target state s g Expected return; -1(s t ≠s g The meaning of ) is that if the current state is not the target state, the reward value is -1, otherwise it defaults to 0; γ is the discount factor, and 0≤γ<1, which balances the importance of current and future rewards; It is usually positively correlated with the time step τ, and its function is to amplify the error weights of long-term tasks.

[0058] Existing methods typically operate directly in the original state space, making it difficult to capture the complex temporal dependencies between states. In contrast, this invention learns a latent space representation with temporal structure, enabling agents to more naturally understand the intrinsic laws of state evolution. In typical multi-agent collaborative tasks, the latent space-based method exhibits more coherent behavioral strategies, better cooperation between agents, and smoother tactical transitions. Traditional methods, lacking explicit modeling of temporal relationships, often suffer from problems such as uncoordinated actions and broken strategies. Especially when facing unfamiliar environmental configurations, the latent space representation of this invention demonstrates a significant generalization advantage, allowing agents to quickly adapt and maintain the rationality of their strategies. This temporally aware representation not only improves the decision-making quality of single agents but also significantly enhances the overall collaborative performance of multi-agent systems. As a preferred embodiment of this invention, the target exploration reward function in step S4 is:

[0059] in, It is the reward for exploring the target at time step t; dis cur The current state s refers to t With the target state The distance in the latent space is calculated using the following formula: It is worth noting It is determined by the initial state s0 of the current trajectory; dis next This refers to the state s at the next time step. t+1 With the target state The distance in the latent space is calculated using the following formula:

[0060] Traditional reward designs, typically based on heuristic rules or sparse rewards in the original state space, struggle to provide effective policy optimization signals. This invention, employing a reward function constructed using latent space distance, demonstrates significant advantages in complex navigation tasks. Experiments show that the agent can more naturally plan paths that conform to environmental dynamics, avoiding the detours or deadlocks common in traditional methods. Particularly in long-cycle tasks, the latent space-based reward provides a more coherent optimization direction, leading to more stable policy convergence. Compared to distance metrics in the original state space, agents guided by latent space rewards exhibit superior exploration-exploitation balance, enabling rapid approach to the target while flexibly responding to environmental changes. This design also enables multi-agent systems to exhibit more consistent group behavior patterns in collaborative tasks.

[0061] As a preferred embodiment of the present invention, the method for decomposing the overall goal exploration reward based on the agent's local contribution evaluation in step S5 includes:

[0062] Step S51, define the global state and individual observations. The local observations of each agent collectively constitute the global state, specifically:

[0063] Global state

[0064] in Let N represent the local observation of the i-th agent at time step t, where N is the total number of agents and determines the cooperative complexity of the system.

[0065] Step S52: Construct a virtual local state. To quantify the contribution of a single agent, it is necessary to isolate the impact of its behavior on the global state. To generate the virtual state, the observations of other agents need to be fixed, and only the action result of the i-th agent needs to be updated:

[0066]

[0067] in This represents a new observation after the i-th agent performs an action. Simulate the global state after only the i-th agent takes action.

[0068] Step S53: Calculate individual contribution rewards. Individual contribution rewards. It is necessary to balance the value of local actions with the fairness of overall collaboration; the contribution reward formula is as follows:

[0069] in V(s) represents the local contribution reward for the i-th agent. t ,s g ) is state s t For the target state sg The value function, Simulate the global state after only the i-th agent takes action, that is, only the state dimension of the i-th agent comes from time step t+1, while the others remain at time step t; t represents the target exploration reward at time step t; N represents the total number of agents.

[0070] Traditional multi-agent systems often employ global team rewards, leading to ambiguity in individual contributions. This invention achieves precise credit allocation in goal-exploration tasks through a contribution-based reward decomposition mechanism. Experiments show that each agent can clearly identify the impact of its actions on the team reward, avoiding the problem of lazy agents caused by shared reward functions in traditional methods. Especially in dynamic task environments, the decomposed reward signals enable agents to quickly adjust their collaborative strategies, exhibiting more flexible task adaptability compared to global reward methods. This fine-grained credit allocation not only improves overall task efficiency but also encourages agents to develop more targeted professional skills.

[0071] In a preferred embodiment of the present invention, the reward for agent behavior exploration in step S6 is:

[0072] in, It is a reward for behavioral exploration, δ i For the next state s t+1 Explicit random perturbations are used to simulate environmental dynamics or the uncertainty of agent actions, and Where σ 2 δ i The variance of the distribution; M is the number of noise terms in the random sampling.

[0073] Traditional exploration strategies typically rely on random perturbations or fixed exploration rates, making it difficult to effectively cover key regions of the state space. This invention, by combining latent space temporal mapping with a frequency-aware exploration mechanism, exhibits more refined exploration capabilities in complex environments. Experiments show that the agent can proactively identify and focus on dynamic state regions with significant differences in latent space representation but low access frequency, thereby discovering key behavioral patterns that are easily overlooked by traditional methods. Compared to uniform exploration strategies, this method significantly enriches the agent's behavioral library while maintaining goal orientation, exhibiting stronger adaptability, especially under unexpected situations. This frequency-based perturbation screening mechanism avoids the inefficiency of blind exploration and ensures the diversity of micro-level behavioral strategies, providing a more comprehensive foundation for subsequent macro-level strategy optimization.

[0074] As a preferred embodiment of the present invention, the exploration strategy decoupling method based on a dual-head Q-network described in step S7 includes:

[0075] Step S71: Construct a dual-head Q-network architecture. First, a target exploration Q-network needs to be configured for each agent i. Q-Network for Behavioral Exploration

[0076] The target exploration Q-network receives local observations. i The Q-value of the target-guided action is output to quickly approach the global target; the behavior exploration Q-network receives the same local observations. i It outputs the Q-value of microscopic behaviors for fine-grained operation. This is achieved through independent parameters. and Achieve strategy decoupling to avoid coupling problems caused by the intertwining of exploration rewards.

[0077] Step S72: Design phase division and switching conditions. Define the target exploration phase (t∈[0,g]) and the behavior exploration phase (t∈[g,T]), where g is the time step of first reaching the vicinity of the target state. Generate a binary mask signal mask through global state detection. g If the group of agents does not reach the vicinity of the target area, mask g =0, enable target exploration Q network If the group of agents reaches the vicinity of the target area, mask g =1, switch to the Q-network for behavioral exploration

[0078] Step S73, based on the central hybrid network Q tot Collaborative training.

[0079] During the goal exploration phase, the various intelligent agents Input Hybrid Network Output joint Q value TD error optimization and During the behavior exploration phase, the various intelligent agents Input Hybrid Network Output joint Q value TD error optimization and in, It is a neural network, and the input is the value Q of the Q-network, which represents the goal exploration of each agent. g It is used to measure the value of state-action pairs during the goal exploration phase, and f ultimately outputs a joint Q-value. This is used to measure the value of the joint state-action pair of the agents during the goal exploration phase; during the goal exploration phase, the input f is the value Q output by the Q-network for each agent's behavior exploration. bIt is used to measure the value of state-action pairs during the behavior exploration phase, and f ultimately outputs a joint Q-value. Used to measure the value of joint state-action pairs of an agent group during the behavior exploration phase; and These are the network parameters of agent i's behavior exploration Q-network and goal exploration Q-network during the behavior exploration phase and goal exploration phase, respectively. and These are the network parameters of the hybrid network f during the behavior exploration phase and the goal exploration phase, respectively. In fact, these are the states of the same network parameters at different stages.

[0080] Step S74: Design a dynamic gradient isolation mechanism. During the target exploration phase, freeze the behavioral exploration Q-network. parameters Update only Freeze the target exploration Q network during the behavior exploration phase. parameters Update only

[0081] Traditional value decomposition methods typically employ a single Q-network architecture, which struggles to simultaneously address the dual needs of global goal orientation and local behavior optimization. This invention, through a staged dual Q-network design, exhibits superior learning characteristics in complex collaborative tasks. Experimental observations reveal that the early-stage goal-exploration Q-network quickly establishes the overall task-solving framework, while the later-stage behavior-exploration Q-network focuses on refined policy adjustments; their synergistic effect significantly improves training efficiency. Compared to a single Q-network structure, this framework avoids the problem of overly conservative early strategies and overcomes the drawback of getting trapped in local optima in later optimization. Particularly in dynamically changing task environments, the stage-switching mechanism of the dual Q-network enables the agent system to adaptively balance exploration and development, ultimately resulting in more stable and reliable group collaborative performance.

[0082] like Figure 1As shown, this invention proposes a two-stage multi-agent collaborative method based on exploration reward shaping. First, trajectory data generated by the agent's interaction with the environment is collected, including the current environment state, current reward, environment reward, and actions, and saved to an experience buffer. The experience buffer is updated incrementally. Next, the environment state, target state, and reward of the target state are randomly sampled from the experience buffer to construct a conditional diffusion model, using the reward as a generation condition to generate a high-reward target state. Then, the global environment state at different time steps in different trajectories is sampled from the experience buffer to learn a temporal structure mapping, mapping temporally similar states to geometrically similar latent states. Finally, the Euclidean distance d1 between the current environment state and the target state in the latent space is calculated, and the distance d1 between the current environment state and the target state at the next time step in the latent space is calculated. The Euclidean distance d2 is used, and the final reward function is d1-d2. The global environment state of two adjacent time steps is obtained, and the observation changes of each agent in the next time step are controlled, thereby decomposing the overall exploration reward into the sum of the influence of each agent on the target exploration. Random noise is added to the environment state of the next time step, and the average distance between the current time step state and these noisy state sets in the latent space is calculated as the behavior exploration reward. A dual-head Q-network is used to decouple the exploration strategy into a target exploration strategy and a behavior exploration strategy, and the reward functions of the two different stages are applied to the decision network of the agent in their respective stages. The Q-network refers to the value network used in reinforcement learning to measure the value of state-action pairs.

[0083] like Figure 2 The diagram shows an architecture for generating target states based on a diffusion model. This architecture describes a generative framework based on a conditional diffusion model, designed to learn from empirical data and generate state sequences related to high-reward objectives. First, the system samples triplet data (current state s) from an empirical buffer pool. t Target state s g and its corresponding cumulative return R g Subsequently, a dual-stream encoder ε is used to pair state pairs (s) t s g The latent space vector z0 is generated by encoding the latent space through a convolutional backbone network. The ε encoder extracts visual features by combining them with a convolutional backbone network and performs feature fusion through a multilayer perceptron (MLP). At the same time, it optimizes the reconstruction loss and reward prediction loss, so that the latent space can not only retain the spatiotemporal features of the original state, but also encode semantic information related to the reward.

[0084] During the diffusion phase, the system first performs a forward diffusion process, gradually adding Gaussian noise to the latent vector z0, and after T steps, obtains the noisy latent vector z. T Subsequently, during the reverse diffusion process, the system accumulates a reward R. g As a condition, utilize Denoising is performed gradually to gradually recover the target latent vector. The network encodes the reward condition into a condition vector c through a cross-modal attention mechanism. g Furthermore, by combining adaptive layer normalization (AdaGN) to dynamically adjust the feature distribution, the semantic controllability of the generation process is enhanced. Ultimately, The network predicts the noise residuals and calculates the mean and covariance, gradually reconstructing the low-noise latent vectors. When the back-diffusion reaches the final time step t=0, the system uses the decoder... Map the generated latent vector z′0 back to the original state space and output the reconstructed state pair.

[0085] like Figure 3 As shown, when the task is a pathfinding task, there are three agents in the original state space S, and their initial states are respectively... As time step t gradually increases from 0 to k, the state changes in the original state space. The agent gradually moves towards the target state As the agent approaches the target state, the joint state change of the agent at two time steps is mapped into the latent space Z, forming several line segments that approach the target state.

[0086] like Figure 4 As shown, each agent i is configured with a target exploration Q-network. Q-Network for Behavioral Exploration The target exploration Q-network accepts local observations. i Output the Q value of the target-oriented action. Used for rapid approach to the global objective; the behavioral exploration Q-network receives the same local observations. i It outputs the Q-value of microscopic behaviors for fine-grained operation. By independent parameters and To achieve policy decoupling and avoid coupling caused by the intertwining of exploration and rewards, each agent's objective exploration is decoupled into a Q-network. Q-Network for Behavioral Exploration As the central hybrid network Q tot =f(Q1,Q2,…,Q n The input to θ) is noteworthy; in addition to the target exploration Q-network, each agent i inputs the central hybrid network. Q-Network for Behavioral Exploration There are also rewards for exploring objectives. i g and

[0087] Example 2: Figure 1 , Figure 2 , Figure 3 and Figure 4As shown, a two-stage multi-agent collaborative method based on exploration reward shaping is proposed. This method decouples the exploration process of a multi-agent system into a goal exploration stage and a behavior exploration stage, shaping two different types of reward functions to encourage agents to perform specific exploration tasks at different stages. Specifically, in the reward design process of the goal exploration stage, it is first necessary to collect trajectory data generated by the agent's interaction with the environment, including the current environment state, current reward, environment reward, actions, etc., and save it to an experience buffer pool, which is updated incrementally. Then, the environment state, target state, and reward of the target state are randomly sampled from the experience buffer pool, and a conditional diffusion model is constructed to generate high-reward target states using the reward as a generation condition. Next, the global environment state at different time steps in different trajectories needs to be sampled from the experience buffer pool to learn the temporal structure mapping, mapping temporally similar states to geometrically similar hidden states. After obtaining the temporal structure mapping, the L2 distance d1 between the current environment state and the target state in the latent space is calculated, as well as the L2 distance d2 between the next time step and the target state in the latent space. The final reward function is d1 - d2. To address the credit allocation problem caused by sharing the same reward function in multi-agent systems, it is necessary to obtain the global environment state of two adjacent time steps and control the observation changes of each agent in the next time step, thereby decomposing the overall exploration reward into the sum of the influence of each agent on the target exploration. In the reward design process of the behavior exploration phase, random noise needs to be added to the environment state of the next time step, and the average distance between the current time step state and these noisy state sets in the latent space is calculated as the behavior exploration reward. Finally, a dual-headed Q-network is used to decouple the exploration strategy into a target exploration strategy and a behavior exploration strategy, and the reward functions of the two different stages are applied to the decision network of the agents in their respective stages.

[0088] The multi-agent cooperative method described in this invention aims to solve the problem of low cooperative efficiency of multi-agent systems in complex task environments, and includes the following steps:

[0089] Step S1: Collect trajectory data generated by the interaction between the agent and the environment, including the current environmental state, current reward, environmental reward, actions, etc., and save it to the experience buffer pool. Update the experience pool using an incremental update method.

[0090] This step includes three sub-steps: defining the trajectory data format, initializing the experience buffer pool, and incrementally updating the experience pool. The specific order of the steps is as follows:

[0091] Step S11, define the trajectory data format. Define the trajectory data format for a single trajectory data τ. t Represented as a quintuple: τ t = t ​,a t ,r t ,s t+1 ,R t >, where s t ∈S, representing the environmental state at time t, where S is the state space; a t ∈A, representing the action performed by the agent at time t, where A is the action space; Indicates immediate rewards for environmental feedback. For reward space; s t+1 ∈S, indicating execution of a t The state at the next moment after; R t For s t The discount return is calculated using the following formula: Where r t+k Let γ be the immediate reward of the environment at time t+k, and let γ be the discount factor, where γ∈[0,1].

[0092] Step S12, initialize the experience buffer pool. Define the experience buffer pool D as a fixed-capacity first-in-first-out queue: D = {τ1, τ2, ..., τ...} N},|D|=N max Where N max This is the maximum capacity of the experience pool; the default value is 1000.

[0093] Step S13: Incrementally update the experience pool.

[0094] For each new trajectory data τ1, perform the following operation: D=D∪{τ t},if|D| <N max ;D=D\{τ oldest}∪{τ t}, if |D|=N max , where τ oldest This is the earliest trajectory data stored in the experience pool.

[0095] Step S2: Randomly sample the environment state, target state, and reward of the target state from the experience buffer pool, and construct a conditional diffusion model to generate a high-reward target state using the reward as the generation condition.

[0096] In this step, such as Figure 2 As shown, the process includes four sub-steps: defining data sampling and preprocessing, the forward diffusion process of the diffusion model, the backward diffusion process of the diffusion model, and latent space decoding. The specific order of the steps is as follows:

[0097] Step S21, Data Sampling and Preprocessing. First, triplet data are sampled from the empirical buffer pool D. t ,s g ,R g ​>, where s t , s g belong to the same trajectory, and 0 < t < g < T, where T is the termination time step in a trajectory, R g is the discounted cumulative return of s g . Only when the cumulative return R t corresponding to a certain state s t is greater than ω, can it be used as the high-return state s t of the trajectory where the state s g is located, that is, the target state, where ω is a manually set high-return threshold. During the processing, the state pair (s t , s g ) needs to be mapped to a hidden vector through the encoder ε: z0 = ε(s t , s g ), where d is the dimension of the hidden space. ε is a trainable encoding network, adopting a two-stream encoding architecture: First, the visual features f t and f g of s t and s g are respectively extracted through a convolutional backbone network with shared weights, and then the two are concatenated along the feature dimension and input into a multi-layer perceptron (MLP) for non-linear fusion, and finally the hidden vector z0 is output through a linear projection layer. During the training process, the encoding network parameters are jointly optimized by minimizing the weighted sum of the reconstruction loss and the return prediction loss to ensure that the hidden space can both retain the spatio-temporal features of the original state pair and encode the semantic information related to the cumulative return. The reconstruction loss and return prediction loss functions are as follows:

[0098]

[0099] Step S22, in the conditional diffusion process, first, a forward diffusion process needs to be carried out: Gaussian noise is gradually added to the hidden vector z0 for a total of M steps:

[0100]

[0101] where β t is the noise scheduling coefficient, default set to 0.02; z t is the noise-added sample hidden vector when the forward diffusion time step is t, and where corresponds to the hidden variable related to the current state s t , corresponds to the hidden variable related to s g .

[0102] Step S23: When the forward diffusion time step reaches M, the reverse diffusion process needs to begin, with R... g As a condition, through Stepwise denoising to generate the target latent vector:

[0103]

[0104] in The input is the noise vector z t diffusion steps t and cumulative return R g During the inverse diffusion process at time step t∈{M,…,0}, the denoising network… Receive noise latent vector z t Embedded representation of diffusion time step t and target reward R g As a joint input. To achieve effective conditional control, a cross-modal attention mechanism is employed to integrate R... g Encoded as a 128-dimensional conditional vector c g By using channel splicing and noise latent vector z t Fusion. Specifically, a state injection operation needs to be performed before inputting the noise vector z into the network: the original noise vector z... t Replace with

[0105] An improved 3D U-Net architecture is adopted, and in the conditional injection module, an adaptive layer regression (AdaGN) is added after each residual block to integrate the time step t and the reward condition c. g Embedded vector injection normalization layer:

[0106]

[0107] Where h is an intermediate feature. and Prediction is achieved through temporal-conditional joint embedding. A cross-attention layer is inserted into the downsampling path to enable low-dimensional features to interact with s. t The network enhances spatial awareness by leveraging local feature interactions. Ultimately, the network outputs a mean μ. θ and diagonal covariance ∑ θ :

[0108]

[0109] Where α t For noise scheduling coefficients, ∈ θ For the predicted noise residual, v θ These are the interpolation weights for network prediction.

[0110] Step S24: When the reverse diffusion process reaches its final step, i.e., t=0, it needs to be decoded. z0 ′ Decoded into state pairs in the original state space:

[0111] Step S3: Sample the global environment state at different time steps in different trajectories from the experience buffer pool, learn the temporal structure mapping, and map temporally similar states to geometrically similar hidden states.

[0112] In this step, the principle is as follows: Figure 3 As shown, in the original state space, the optimal trajectory of the multi-agent system reaches the target state in the shortest time step. The trajectory in the latent space approximately reaches the target state in a straight line. This indicates that the latent space captures the temporal distance between states at different time steps under the optimal policy in the original state space. Learning the temporal structure mapping mainly includes three sub-steps: defining the temporal structure mapping function, defining the latent space distance metric, and training the temporal structure mapping. The specific steps are as follows:

[0113] Step S31, define the time structure mapping function: Map the states at different time steps in the original state space to the latent space, h = φ(s). t ),h∈H, where d represents the embedding dimension.

[0114] Step S32, define the distance between different states in the latent space as the state-target value estimate: V(s) t ,s t+k )=‖φ(s t )-φ(s t+k )‖2.

[0115] Step S33 requires training the temporal structure mapping using a reinforcement learning paradigm. The goal of reinforcement learning is to optimize the Bellman equation error:

[0116] in It is the target value of the objective value function, usually calculated through an objective network, with fixed parameters to reduce fluctuations; V(s) t ,s g ) is the value function to be optimized, predicting the current state s. t To the target state s g Expected return; -1(s t ≠s g The significance of this is that if the current state is not the target state, the reward value is -1, otherwise it defaults to 0, thereby encouraging the agent to quickly approach the target state. Let the truncated squared loss function be used to suppress the influence of outliers, and it is defined as follows:

[0117]

[0118] Where τ is the threshold, which controls the sensitivity to large errors.

[0119] Step S4: Calculate the L2 distance d1 between the current environment state and the target state in the latent space, the L2 distance d2 between the next time step and the target state in the latent space, and finally the reward function is d1-d2.

[0120] In this step, the final reward function d1-d2 is the target exploration reward function: r g =dis cur -disn ext Among them, dis cur The current state s refers to t With the target state The distance in the latent space is calculated using the following formula: It is worth noting It is determined by the initial state s0 of the current trajectory; dis next This refers to the state s at the next time step. t+1 With the target state The distance in the latent space is calculated using the following formula:

[0121]

[0122] Step S5: Obtain the global environment state of two adjacent time steps and control the observation changes of each agent in the next time step, thereby decomposing the overall exploration reward into the sum of the influence of each agent on the target exploration.

[0123] This step includes three sub-steps: defining the global state and individual observations, constructing a virtual local state, and calculating the individual contribution reward. The specific order of the steps is as follows:

[0124] Step S51, define the global state and individual observations. Each agent's local observations collectively constitute the global state. Specifically: Global State in Let N represent the local observation of the i-th agent at time step t, where N is the total number of agents and determines the cooperative complexity of the system.

[0125] Step S52: Construct a virtual local state. To quantify the contribution of a single agent, it is necessary to isolate the impact of its behavior on the global state. To generate the virtual state, the observations of other agents need to be fixed, and only the action result of the i-th agent needs to be updated: in This represents a new observation after the i-th agent performs an action. Simulate the global state after only the i-th agent takes action.

[0126] Step S53: Calculate individual contribution rewards. Individual contribution rewards. The contribution reward formula needs to balance the value of local actions with the fairness of overall collaboration:

[0127]

[0128] in Represents the local contribution reward of the i-th agent; where, The physical meaning is that the action of agent i causes the global state to change from s. t Become This difference quantifies the independent impact of agent i's behavior on the target value. If this value is greater than zero, it indicates that agent i's behavior brings the system closer to the target; otherwise, it indicates that agent i's behavior causes a deviation from the target and requires punishment. Furthermore, it is assumed that...

[0129]

[0130] Its design motivation is that if simply using May lead to This violates the fundamental constraints of multi-agent credit allocation, and since Residual, as a residual term, will affect the global reward... The difference between the total contribution and the sum of the direct contributions of all agents is amortized to ensure conservation. From a mathematical perspective, when the sum of individual contributions... Underestimating global reward If the residual is positive, it is allocated to supplement the distribution; otherwise, the distribution is reduced to avoid excessive rewards.

[0131] Step S6: Add random noise to the environmental state of the next time step, and calculate the average distance between the current time step state and these noisy state sets in the latent space as a reward for behavioral exploration.

[0132] In this step, the reward for behavioral exploration is:

[0133]

[0134] Where, δ i For the next state s t+1 Explicit random perturbations are used to simulate environmental dynamics or the uncertainty of agent actions, and M is the number of noise terms in the random sampling; This reflects the current situation. tIn the hidden space, the degree of exploration of the surrounding state, if The larger the value, the greater the current value of s. t The higher the uncertainty, the lower the uncertainty. The goal of S6 is to design an agent exploration reward model to encourage agents to actively explore those unknown states.

[0135] Step S7, using a dual-head Q network, such as Figure 4 As shown, the exploration strategy is decoupled into a goal exploration strategy and a behavior exploration strategy, and the reward functions of the two different stages are applied to the agent's decision network at their respective stages.

[0136] This step includes four sub-steps: defining and constructing the dual-head Q-network architecture, designing the switching and transfer conditions during the design phase, collaborative training based on the central hybrid network, and establishing a dynamic gradient isolation mechanism. The specific order of the steps is as follows:

[0137] Step S71: Construct a dual-head Q-network architecture. First, a target exploration Q-network needs to be configured for each agent i. Q-Network for Behavioral Exploration The target exploration Q-network receives local observations. i The Q-value of the target-guided action is output to quickly approach the global target; the behavior exploration Q-network receives the same local observations. i It outputs the Q-value of microscopic behaviors for fine-grained operation. This is achieved through independent parameters. and Achieve strategy decoupling to avoid coupling problems caused by the intertwining of exploration rewards.

[0138] Step S72: Design phase division and switching conditions. Define the target exploration phase (t∈[0,g]) and the behavior exploration phase (t∈[g,T]), where g is the time step of first reaching the vicinity of the target state. Generate a binary mask signal mask through global state detection. g If the group of agents does not reach the vicinity of the target area, mask g =0, enable target exploration Q network If the group of agents reaches the vicinity of the target area, mask g =1, switch to the Q-network for behavioral exploration

[0139] Step S73, based on the central hybrid network Q tot Collaborative training of various agents. During the goal exploration phase, the various agents... Input Hybrid Network Output joint Q value TD error optimization and During the behavior exploration phase, the various intelligent agents Input Hybrid Network Output joint Q value TD error optimization and

[0140] Step S74: Design a dynamic gradient isolation mechanism. During the target exploration phase, freeze the behavioral exploration Q-network. parameters Update only Freeze the target exploration Q network during the behavior exploration phase. parameters Update only

[0141] Example 3: As Figure 1 , Figure 2 , Figure 3 and Figure 4 As shown, a two-stage multi-agent cooperative method based on exploration reward shaping includes the following steps:

[0142] Step S1: Collect trajectory data generated by the interaction between the agent and the environment, including the current environmental state, current reward, environmental reward and action, and save it to the experience buffer pool. Update the experience pool using an incremental update method.

[0143] Step S2: Randomly sample the environment state, target state, and reward of the target state from the experience buffer pool, and construct a conditional diffusion model to generate a high-reward target state using the reward as the generation condition.

[0144] Step S3: Sample the global environment state at different time steps in different trajectories from the experience buffer pool, learn the temporal structure mapping, and map temporally similar states to geometrically similar hidden states.

[0145] Step S4: Calculate the L2 distance d1 between the current environment state and the target state in the latent space, the L2 distance d2 between the next time step and the target state in the latent space, and finally the reward function is d1-d2.

[0146] Step S5: Obtain the global environment state of two adjacent time steps and control the observation changes of each agent in the next time step, thereby decomposing the overall exploration reward into the sum of the influence of each agent on the target exploration.

[0147] Step S6: Add random noise to the environmental state of the next time step, and calculate the average distance between the current time step state and these noisy state sets in the latent space as a reward for behavioral exploration.

[0148] Step S7: Using a dual-head Q-network, the exploration strategy is decoupled into a goal exploration strategy and a behavior exploration strategy, and the reward functions of the two different stages are applied to the agent's decision network at their respective stages.

[0149] Step S1 updates the experience pool using an incremental update method, including:

[0150] Step S11, define the trajectory data format, define the trajectory data format for a single trajectory data τ t Represented as a quintuple: τ t = t ,a t ,r t ,s t+1 ,R t >, where s t ∈S, representing the environmental state at time t, where S is the state space; a t ∈A, representing the action performed by the agent at time t, where A is the action space; Indicates immediate rewards for environmental feedback. For reward space; s t+1 ∈S, indicating execution of a t The state at the next moment after; R t For s t The discount return is calculated using the following formula: Where r t+k Let γ be the immediate reward of the environment at time t+k, and let γ be the discount factor, where γ∈[0,1].

[0151] Step S12, initialize the experience buffer pool, and define the experience buffer pool D as a fixed-capacity first-in-first-out queue: D = {τ1, τ2, ..., τ N},|D|=N max , where N max This is the maximum capacity of the experience pool; the default value is 1000.

[0152] Step S13, incrementally update the experience pool. For each new trajectory data τ1, perform the following operation: D=D∪{τ t},if|D| <N max ;D=D\{τ oldest}∪{τ t}, if |D|=N max , where τ oldest This is the earliest trajectory data stored in the experience pool.

[0153] ​In step S2, the conditional diffusion model is constructed. The diffusion model learns the optimal state distribution from the experience pool and generates diverse and high-quality latent targets, avoiding overfitting caused by simply relying on limited samples. At the same time, conditional generation enables the model to adjust the quality of the generated states according to the return value, forming a more refined exploration gradient. In a multi-agent environment, this mechanism promotes experience sharing, making the diffusion model a refiner of collective wisdom, dynamically adapting to environmental changes while maintaining policy diversity and preventing premature convergence to local optima. This self-reinforcing cycle can continuously improve the exploration efficiency and accelerate policy optimization. The detailed steps are as follows:

[0154] Step S21, data sampling and preprocessing. First, sample triple data <s t , s g , R g > from the experience buffer D, where s t and s g belong to the same trajectory, and 0 < t < g < T, where T is the termination time step in a trajectory, and R g is the discounted cumulative return of s g . Only when the cumulative return R t corresponding to a certain state s t is greater than ω, can it be used as the high-return state s t of the trajectory where the state s g is located, that is, the target state, where ω is a manually set high-return threshold. During the preprocessing, the state pair (s t , s g ) needs to be mapped to a latent vector through the encoder ε: z0 = ε(s t , s g ), where d is the dimension of the latent space, and ε is a trainable encoding network, adopting a two-stream encoding architecture: First, extract the visual features f t of s g and f t of s g respectively through the convolutional backbone network with shared weights, and then concatenate the two along the feature dimension and input them into a multi-layer perceptron (MLP) for non-linear fusion, and finally output the latent vector z0 through a linear projection layer. During the training process, the encoding network parameters are jointly optimized by minimizing the weighted sum of the reconstruction loss and the return prediction loss to ensure that the latent space can not only retain the spatio-temporal features of the original state pair but also encode the semantic information related to the cumulative return. The reconstruction loss and return prediction loss functions are as follows:

[0155]

[0156] Step S22, the conditional diffusion process first requires a forward diffusion process: Gaussian noise is gradually added to the latent vector z0, for a total of M steps:

[0157]

[0158] Where β t This is the noise scheduling factor, set to 0.02 by default; z t It is the noisy sample latent vector at a forward diffusion time step t, and in Corresponding to the current state s t Related latent variables, Corresponding to s g Related latent variables.

[0159] Step S23: When the forward diffusion time step reaches M, the reverse diffusion process needs to begin, with R... g As a condition, through Stepwise denoising to generate the target latent vector:

[0160]

[0161] in The input is the noise vector z t diffusion steps t and cumulative return R g During the inverse diffusion process at time step t∈{M,…,0}, the denoising network Receive noise latent vector z t Embedded representation of diffusion time step t and target reward R g As a joint input, to achieve effective conditional control, a cross-modal attention mechanism is employed to integrate R... g Encoded as a 128-dimensional conditional vector c g By using channel splicing and noise latent vector z t Fusion, specifically, requires performing a state injection operation before inputting the noise vector z into the network: the original noise vector z is then injected into the network. t Replace with An improved 3D U-Net architecture is adopted, and in the conditional injection module, an adaptive layer regression (AdaGN) is added after each residual block to integrate the time step t and the reward condition c. g Embedded vector injection normalization layer:

[0162]

[0163] Where h is an intermediate feature. and Predicted by temporal-conditional joint embedding, a cross-attention layer is inserted into the downsampling path, enabling low-dimensional features to be associated with s. tBy leveraging local feature interactions, the spatial perception capability is enhanced. Ultimately, the network outputs a mean μ. θ and diagonal covariance ∑ θ :

[0164]

[0165] Where α t For noise scheduling coefficients, ∈ θ For the predicted noise residual, v θ These are the interpolation weights for network prediction.

[0166] Step S24: When the reverse diffusion process reaches its final step, i.e., t=0, it needs to be decoded. z0 ′ Decoded into state pairs in the original state space:

[0167] Step S3 involves learning the temporal structure mapping. This mapping enables the agent to understand environmental dynamics at an abstract level. The distances in the latent space reflect the temporal relationships between states, allowing the agent to reason about causality and continuity more naturally. This representation filters out redundant information from the original states, making policy learning more robust. It also helps the agent generalize based on latent space locations when encountering new states. Detailed steps include:

[0168] Step S31, define the time structure mapping function: Map the states at different time steps in the original state space to the latent space, h = φ(s). t ),h∈H, where d represents the embedding dimension.

[0169] Step S32, define the distance between different states in the latent space as the state-target value estimate: V(s) t ,s t+k )=‖φ(s t )-φ(s t+k )‖2.

[0170] Step S33 requires training the temporal structure mapping using a reinforcement learning paradigm. The goal of reinforcement learning is to optimize the Bellman equation error. in It is the target value of the objective value function, usually calculated through an objective network, with fixed parameters to reduce fluctuations; V(s) t ,s g ) is the value function to be optimized, predicting the current state s. t To the target state s g Expected return; -1(s t ≠s gThe significance of this is that if the current state is not the target state, the reward value is -1, otherwise it defaults to 0, thereby encouraging the agent to quickly approach the target state. Let the truncated squared loss function be used to suppress the influence of outliers, and it is defined as follows:

[0171]

[0172] Where τ is the threshold, which controls the sensitivity to large errors.

[0173] Step S4, the final reward function design, is based on latent space distance. This design deeply couples the agent's learning objective with the temporal structure of the environment. Latent space distance naturally reflects the dynamic reachability between states. Therefore, this reward method can more accurately guide the agent to take effective actions, avoiding misleading feedback due to irrelevant features in the original high-dimensional space. Furthermore, because the distance metric is continuous, the reward signal has good smoothness, providing a clear learning direction without affecting exploration efficiency due to reward sparsity. Detailed steps include:

[0174] The final reward letter is: r g =dis cur -dis next dis cur The current state s refers to t With the target state The distance in the latent space is calculated using the following formula:

[0175]

[0176] It is worth noting It is determined by the initial state s0 of the current trajectory; dis next This refers to the state s at the next time step. t+1 With the target state The distance in the latent space is calculated using the following formula:

[0177]

[0178] Step S5, the overall goal exploration reward, includes: evaluation and decomposition of steps based on the agent's local contribution. Reward decomposition and credit allocation accurately identify the contribution of each individual in a multi-agent system, avoiding inefficient policy learning caused by the ambiguity of team rewards. By decomposing the latent space distance reward into components related to each agent's actions, the merits of each strategy can be evaluated more fairly, preventing the emergence of lazy agents or individual strategies being submerged in collective rewards. This fine-grained feedback allows agents to adjust their behavior in a targeted manner, promoting individual strategy optimization while ensuring the efficient achievement of team goals, ultimately improving overall exploration efficiency and collaboration quality. Detailed steps include:

[0179] Step S51, define the global state and individual observations. The local observations of each agent together constitute the global state. Specifically: global state

[0180] in Let N represent the local observation of the i-th agent at time step t, where N is the total number of agents and determines the cooperative complexity of the system.

[0181] Step S52: Construct a virtual local state. To quantify the contribution of a single agent, it is necessary to isolate the influence of its behavior on the global state. To generate the virtual state, it is necessary to fix the observations of other agents and only update the action result of the i-th agent. in This represents a new observation after the i-th agent performs an action. Simulate the global state after only the i-th agent takes action.

[0182] Step S53, calculate individual contribution reward. The contribution reward formula needs to balance the value of local actions with the fairness of overall collaboration:

[0183] in Represents the local contribution reward of the i-th agent; where, The physical meaning is that the action of agent i causes the global state to change from s. t Become This difference quantifies the independent impact of agent i's behavior on the target value. If this value is greater than zero, it indicates that agent i's behavior brings the system closer to the target; otherwise, it indicates that agent i's behavior causes a deviation from the target and requires punishment. Additionally, it is assumed that:

[0184]

[0185] Its design motivation is that if simply using May lead to This violates the fundamental constraints of multi-agent credit allocation, and since Residual, as a residual term, will affect the global reward... The difference between the total contribution and the sum of the direct contributions of all agents is amortized to ensure conservation. From a mathematical perspective, when the sum of individual contributions... Underestimating global reward If the residual is positive, it is allocated to supplement the distribution; otherwise, the distribution is reduced to avoid excessive rewards.

[0186] Behavioral exploration rewards in step S6 for:

[0187]

[0188] Where, δ i For the next state s t+1 Explicit random perturbations are used to simulate environmental dynamics or the uncertainty of agent actions, and M is the number of noise terms in the random sampling; This reflects the current situation. t In the hidden space, the degree of exploration of the surrounding state, if The larger the value, the greater the current value of s. t The higher the uncertainty, the lower the uncertainty. The goal of S6 is to design an agent exploration reward model that encourages agents to actively explore unknown states. By integrating micro-behavioral strategy incentives into macro-goal-oriented exploration, agents can accumulate rich underlying action patterns while pursuing long-term goals, forming a more flexible and robust strategy library. This two-level exploration mechanism maintains directionality and avoids behavioral rigidity caused by over-targeting, enabling agents to steadily approach goals in complex environments while discovering better paths through fine-grained behavioral innovation.

[0189] Step S7 involves the decoupling of the exploration strategy in the dual-head Q-network. This phased dual-Q-network design achieves a dynamic balance between exploration and development within the QMIX framework. In the early stage, the goal-oriented exploration Q-network guides the agent to quickly locate high-value areas and establish a global policy direction. In the later stage, the behavior-oriented exploration Q-network focuses on local policy optimization, improving policy quality through fine-grained action exploration. Detailed steps include:

[0190] Step S71: Construct a dual-head Q-network architecture. First, a target exploration Q-network needs to be configured for each agent i. Q-Network for Behavioral Exploration

[0191] Among them, the target exploration Q-network accepts local observations. i The Q-value of the target-oriented action is output to quickly approach the global target; the behavior exploration Q-network receives the same local observations. i It outputs the Q-value of microscopic behaviors for fine-grained operation, through independent parameters. and Achieve strategy decoupling to avoid coupling problems caused by the intertwining of exploration rewards.

[0192] Step S72: Set phase division and switching conditions, define the target exploration phase (t∈[0,g]) and the behavior exploration phase (t∈[g,T]), where g is the time step of first reaching the vicinity of the target state, and generate a binary mask signal mask through global state detection. g If the group of agents does not reach the vicinity of the target area, mask g =0, enable target exploration Q network If the group of agents reaches the vicinity of the target area, mask g =1, switch to the Q-network for behavioral exploration

[0193] Step S73, based on the central hybrid network Q tot Collaborative training, during the goal exploration phase, integrates the various intelligent agents... Input Hybrid Network Output joint Q value TD error optimization and During the behavior exploration phase, the various intelligent agents Input Hybrid Network Output joint Q value TD error optimization and

[0194] Step S74, a dynamic gradient isolation mechanism, freezes the behavioral exploration Q-network during the target exploration phase. parameters Update only Freeze the target exploration Q network during the behavior exploration phase. parameters Update only

[0195] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A two-stage multi-agent cooperative method based on exploration reward shaping, characterized in that, Includes the following steps: Step S1: Collect trajectory data generated by the interaction between the agent and the environment, including the current environmental state, current reward, environmental reward and action, and save it to the experience buffer pool. Update the experience pool using an incremental update method. Step S2: Randomly sample the environment state, target state, and reward of the target state from the experience buffer pool, and construct a conditional diffusion model to generate a high-reward target state using the reward as the generation condition. Step S3: Sample the global environment state at different time steps in different trajectories from the experience buffer pool, learn the temporal structure mapping, and map temporally similar states to geometrically similar hidden states. Step S4: Calculate the L2 distance d1 between the current environment state and the target state in the latent space, the L2 distance d2 between the next time step and the target state in the latent space, and finally the reward function is d1-d2. Step S5: Obtain the global environment state of two adjacent time steps and control the observation changes of each agent in the next time step, thereby decomposing the overall exploration reward into the sum of the influence of each agent on the target exploration; Step S6: Add random noise to the environmental state of the next time step, and calculate the average distance between the current time step state and these noisy state sets in the latent space as a reward for behavioral exploration. Step S7: Using a dual-head Q-network, the exploration strategy is decoupled into a goal exploration strategy and a behavior exploration strategy, and the reward functions of the two different stages are applied to the agent's decision network at their respective stages.

2. The trajectory data generated by the interaction between the intelligent agent and the environment according to claim 1, characterized in that, Step S1 updates the experience pool using an incremental update method, including: Step S11, define the trajectory data format, define the trajectory data format for a single trajectory data τ t Represented as a quintuple: τ t = t ,a t ,r t ,s t+1 ,R t >, where s t ∈S, representing the environmental state at time t, where S is the state space; a t ∈A, representing the action performed by the agent at time t, where A is the action space; Indicates immediate rewards for environmental feedback. For reward space; s t+1 ∈S, indicating execution of a t The state at the next moment after; R t For s t The discount return is calculated using the following formula: Where r t+k Let γ be the immediate reward provided by the environment at time t+k, and let γ be the discount factor, where γ∈[0,1].​ Step S12, initialize the experience buffer pool, and define the experience buffer pool D as a fixed-capacity first-in-first-out queue: D = {τ1, τ2, ..., τ N },|D|=N max , where N max This represents the maximum capacity of the experience pool; the default value is 1000. Step S13, incrementally update the experience pool. For each new trajectory data τ1, perform the following operation: D=D∪{τ t },if|D| <N max ;D=D\{τ oldest }∪{τ t }, if |D|=N max , where τ oldest This is the earliest trajectory data stored in the experience pool.

3. The two-stage multi-agent cooperative method based on exploration reward shaping according to claim 1, characterized in that, Step S2 involves constructing a conditional diffusion model. This model learns the optimal state distribution from the experience pool and generates diverse and high-quality potential targets, avoiding overfitting caused by relying solely on a limited sample size. Simultaneously, conditional generation allows the model to adjust the quality of generated states based on reward values, forming a more refined exploration gradient. In a multi-agent environment, this mechanism promotes experience sharing, making the diffusion model a refiner of collective wisdom, dynamically adapting to environmental changes while maintaining policy diversity and preventing premature convergence to local optima. This self-reinforcing cycle continuously improves exploration efficiency and accelerates policy optimization. Detailed steps include: Step S21, data sampling and preprocessing. First, sample triple data <s t , s g , R g > from the experience buffer D, where s t and s g belong to the same trajectory, and 0 < t < g < T, where T is the termination time step in a trajectory, and R g is the discounted cumulative return of s g . Only when the cumulative return R t corresponding to a certain state s t is greater than ω, can it be used as the high-return state s t of the trajectory where the state s g is located, that is, the target state, where ω is a manually set high-return threshold. During the preprocessing, the state pair (s t , s g ) needs to be mapped to a hidden vector through the encoder ε: z0 = ε(s t , s g ), where d is the dimension of the hidden space, and ε is a trainable encoding network, adopting a two-stream encoding architecture: First, extract the visual features f t and f g of s t and s g respectively through a convolutional backbone network with shared weights. Subsequently, concatenate the two along the feature dimension and input them into a multi-layer perceptron (MLP) for non-linear fusion, and finally output the hidden vector z0 through a linear projection layer. During the training process, jointly optimize the encoding network parameters by minimizing the weighted sum of the reconstruction loss and the return prediction loss to ensure that the hidden space can not only retain the spatio-temporal features of the original state pair but also encode the semantic information related to the cumulative return. The reconstruction loss and the return prediction loss functions are as follows: Step S22, the conditional diffusion process first requires a forward diffusion process: Gaussian noise is gradually added to the latent vector z0, for a total of M steps: Where β t This is the noise scheduling factor, set to 0.02 by default; z t It is the noisy sample latent vector at a forward diffusion time step t, and in Corresponding to the current state s t Related latent variables, Corresponding to s g Related latent variables, Step S23: When the forward diffusion time step reaches M, the reverse diffusion process needs to begin, with R... g As a condition, through Stepwise denoising to generate the target latent vector: in The input is the noise vector z t diffusion steps t and cumulative return R g During the inverse diffusion process at time step t∈{M,…,0}, the denoising network Receive noise latent vector z t Embedded representation of diffusion time step t and target reward R g As a joint input, to achieve effective conditional control, a cross-modal attention mechanism is employed to integrate R... g Encoded as a 128-dimensional conditional vector c g By using channel splicing and noise latent vector z t Fusion, specifically, requires performing a state injection operation before inputting the noise vector z into the network: the original noise vector z is then injected into the network. t Replace with An improved 3D U-Net architecture is adopted, and in the conditional injection module, an adaptive layer regression (AdaGN) is added after each residual block to integrate the time step t and the reward condition c. g Embedded vector injection normalization layer: Where h is an intermediate feature. and Predicted by temporal-conditional joint embedding, a cross-attention layer is inserted into the downsampling path, enabling low-dimensional features to be associated with s. t By leveraging local feature interactions, the spatial perception capability is enhanced. Ultimately, the network outputs a mean μ. θ and diagonal covariance ∑ θ : Where α t For noise scheduling coefficients, ∈ θ For the predicted noise residual, v θ For the interpolation weights of the network prediction, Step S24: When the reverse diffusion process reaches its final step, i.e., t=0, it needs to be decoded. z0 ′ Decoded into state pairs in the original state space:

4. The two-stage multi-agent cooperative method based on exploration reward shaping according to claim 1, characterized in that, Step S3 involves learning the temporal structure mapping. This mapping enables the agent to understand environmental dynamics at an abstract level. The distances in the latent space reflect the temporal relationships between states, allowing the agent to reason about causality and continuity more naturally. This representation filters out redundant information from the original states, making policy learning more robust. It also helps the agent generalize based on latent space locations when encountering new states. Detailed steps include: Step S31, define the time structure mapping function: Map the states at different time steps in the original state space to the latent space, h = φ(s). t ),h∈H, where d is the embedding dimension. Step S32, define the distance between different states in the latent space as the state-target value estimate: V(s) t ,s t+k )=‖φ(s t )-φ(s t+k )‖2, Step S33 requires training the temporal structure mapping using a reinforcement learning paradigm. The goal of reinforcement learning is to optimize the Bellman equation error. in It is the target value of the objective value function, usually calculated through an objective network, with fixed parameters to reduce fluctuations; V(s) t ,s g ) is the value function to be optimized, predicting the current state s. t To the target state s g Expected return; -1(s t ≠s g The significance of this is that if the current state is not the target state, the reward value is -1, otherwise it defaults to 0, thereby encouraging the agent to quickly approach the target state. Let the truncated squared loss function be used to suppress the influence of outliers, and it is defined as follows: Where τ is the threshold, which controls the sensitivity to large errors.

5. The two-stage multi-agent cooperative method based on exploration reward shaping according to claim 1, characterized in that, Step S4, the final reward function design, is based on latent space distance. This design deeply couples the agent's learning objective with the temporal structure of the environment. Latent space distance naturally reflects the dynamic reachability between states. Therefore, this reward method can more accurately guide the agent to take effective actions, avoiding misleading feedback due to irrelevant features in the original high-dimensional space. Furthermore, because the distance metric is continuous, the reward signal has good smoothness, providing a clear learning direction without affecting exploration efficiency due to reward sparsity. Detailed steps include: The final reward letter is: r g =dis cur -dis next dis cur The current state s refers to t With the target state The distance in the latent space is calculated using the following formula: It is worth noting It is determined by the initial state s0 of the current trajectory; dis next This refers to the state s at the next time step. t+1 With the target state The distance in the latent space is calculated using the following formula:

6. The two-stage multi-agent cooperative method based on exploration reward shaping according to claim 1, characterized in that, Step S5, the overall goal exploration reward, includes: evaluation and decomposition of steps based on the agent's local contribution. Reward decomposition and credit allocation accurately identify the contribution of each individual in a multi-agent system, avoiding inefficient policy learning caused by the ambiguity of team rewards. By decomposing the latent space distance reward into components related to each agent's actions, the merits of each strategy can be evaluated more fairly, preventing the emergence of lazy agents or individual strategies being submerged in collective rewards. This fine-grained feedback allows agents to adjust their behavior in a targeted manner, promoting individual strategy optimization while ensuring the efficient achievement of team goals, ultimately improving overall exploration efficiency and collaboration quality. Detailed steps include: Step S51, define the global state and individual observations. The local observations of each agent together constitute the global state. Specifically: global state in Let N represent the local observation of the i-th agent at time step t, where N is the total number of agents and determines the cooperative complexity of the system. Step S52: Construct a virtual local state. To quantify the contribution of a single agent, it is necessary to isolate the influence of its behavior on the global state. To generate the virtual state, it is necessary to fix the observations of other agents and only update the action result of the i-th agent. in This represents a new observation after the i-th agent performs an action. Simulate the global state after only the i-th agent takes action. Step S53, calculate individual contribution reward. The contribution reward formula needs to balance the value of local actions with the fairness of overall collaboration: Represents the local contribution reward of the i-th agent; where, The physical meaning is that the action of agent i causes the global state to change from s. t Become This difference quantifies the independent impact of agent i's behavior on the target value. If this value is greater than zero, it indicates that agent i's behavior brings the system closer to the target; otherwise, it indicates that agent i's behavior causes a deviation from the target and requires punishment. Furthermore, it is assumed that... Its design motivation is that if simply using May lead to This violates the fundamental constraints of multi-agent credit allocation, and since Residual, as a residual term, will affect the global reward... The difference between the total contribution and the sum of the direct contributions of all agents is amortized to ensure conservation. From a mathematical perspective, when the sum of individual contributions... Underestimating global reward If the residual is positive, it is allocated to supplement the distribution; otherwise, the distribution is reduced to avoid excessive rewards.

7. The two-stage multi-agent cooperative method based on exploration reward shaping according to claim 1, characterized in that, Behavioral exploration rewards in step S6 for: Where, δ i For the next state s t+1 Explicit random perturbations are used to simulate environmental dynamics or the uncertainty of agent actions, and M is the number of noise terms in the random sampling; This reflects the current situation. t In the hidden space, the degree of exploration of the surrounding state, if The larger the value, the greater the current value of s. t The higher the uncertainty, the lower the uncertainty. The goal of S6 is to design an agent exploration reward model that encourages agents to actively explore unknown states. By integrating micro-behavioral strategy incentives into macro-goal-oriented exploration, agents can accumulate rich underlying action patterns while pursuing long-term goals, forming a more flexible and robust strategy library. This two-level exploration mechanism maintains directionality and avoids behavioral rigidity caused by over-targeting, enabling agents to steadily approach goals in complex environments while discovering better paths through fine-grained behavioral innovation.

8. The two-stage multi-agent cooperative method based on exploration reward shaping according to claim 1, characterized in that, In step S7, the exploration strategy decoupling step of the dual-head Q network is achieved. This phased dual-Q network design realizes a dynamic balance between exploration and development under the QMIX framework. In the early stage, the target exploration Q network guides the agent to quickly locate high-value areas and establish a global strategy direction. The Q-network, which focuses on later-stage behavior exploration, concentrates on local policy optimization, improving policy quality through fine-grained action exploration. Detailed steps include: Step S71: Construct a dual-head Q-network architecture. First, a target exploration Q-network needs to be configured for each agent i. Q-Network for Behavioral Exploration Among them, the target exploration Q-network accepts local observations. i The Q-value of the target-oriented action is output to quickly approach the global target; the behavior exploration Q-network receives the same local observations. i It outputs the Q-value of microscopic behaviors for fine-grained operation, through independent parameters. and To achieve strategy decoupling and avoid coupling problems caused by the intertwining of exploration rewards, Step S72: Set phase division and switching conditions, define the target exploration phase (t∈[0,g]) and the behavior exploration phase (t∈[g,T]), where g is the time step of first reaching the vicinity of the target state, and generate a binary mask signal mask through global state detection. g If the group of agents does not reach the vicinity of the target area, mask g =0, enable target exploration Q network If the group of agents reaches the vicinity of the target area, mask g =1, switch to the Q-network for behavioral exploration Step S73, based on the central hybrid network Q tot Collaborative training, during the goal exploration phase, integrates the various intelligent agents... Input Hybrid Network Output joint Q value TD error optimization and During the behavior exploration phase, the various intelligent agents Input Hybrid Network Output joint Q value TD error optimization and Step S74, a dynamic gradient isolation mechanism, freezes the behavioral exploration Q-network during the target exploration phase. parameters Update only Freeze the target exploration Q network during the behavior exploration phase. parameters Update only

Images