An asynchronous multi-agent collaborative decision-making method and system based on probability decomposition

Abstract

The present application relates to the field of artificial intelligence, automatic control and robot technology, and particularly relates to an asynchronous multi-agent collaborative decision-making method and system based on probability decomposition. The present application uses a filling mechanism to align the team decision-making timeline through a generalized value function approximation (G-VFA module), and after obtaining the value of the agent team, the individual value function is obtained through probability value function decomposition; the strategy network of each agent is updated by using an asynchronous policy gradient update (A-PG module) method; at the same time, the training process integrates off-policy learning, soft update mechanism and entropy regularization enhancement technology. The present application overcomes the idle waiting problem under the synchronous framework, can more efficiently utilize the time budget; has strong robustness, can adapt to different task requirements and environmental uncertainty; at the same time, has good training ability, through the integration of enhancement technology, the stability and efficiency of the training are ensured. The present application can effectively solve the problem of efficient search of asynchronous multi-agent under time constraints.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, automatic control and robotics, and in particular to an asynchronous multi-agent collaborative decision-making method and system based on probabilistic decomposition. Background Technology

[0002] Most existing efficient multi-robot search solutions, including algorithms based on multi-agent reinforcement learning, are typically built on a framework where all agents make decisions and execute actions synchronously at discrete global time steps. Forcing a synchronous execution scheme means that agents that prioritize completing their actions must wait for other agents to finish before entering the next decision cycle. This results in significant idle waiting time, severely reducing overall time utilization efficiency and task allocation effectiveness.

[0003] Current methods in the field include asynchronous multi-agent reinforcement learning, such as asynchronous QMIX and asynchronous value decomposition networks. Their main characteristic is that when an agent is still executing a time-consuming action (macro-action), the algorithm uses the state-action value (Q-value) at the start of the macro-action to approximate its current value, thereby achieving a combination of value functions across asynchronous decision-making timelines. However, these existing methods have inherent limitations when applied to time-constrained asynchronous multi-robot efficient search problems with probability maximization as the core objective. Specifically, methods like asynchronous QMIX introduce additional network structures (such as hybrid networks) to fit more complex value function relationships, increasing model complexity and computational overhead. Furthermore, they heavily rely on specific hyperparameter settings, resulting in poor generalization ability and robustness under different scenarios or the influence of scenario uncertainties. Furthermore, methods such as asynchronous value decomposition networks typically decompose the team value function based on the additive assumption, meaning the team value is approximately equal to the sum of individual values. However, this fundamentally conflicts with the non-additive optimization objective of "the probability of successfully capturing a moving target within a finite time." The probability maximization objective does not conform to a simple additive relationship, making it impossible for such methods to directly and accurately model and decompose the team objective. Another category is asynchronous multi-agent executor-critic methods, but these methods are usually designed to maximize cumulative rewards, which is incompatible with the objective of maximizing probability within a finite time. Moreover, without incorporating value decomposition, they may lead to "lazy" agents.

[0004] In summary, existing technologies have failed to effectively solve the problem of efficient search for asynchronous multi-robots under time constraints. The core issue is the lack of a method that can effectively decompose non-additive, probabilistic team goals into intelligent individuals with asynchronous decision-making timelines and drive them to make collaborative decisions. Summary of the Invention

[0005] The purpose of this invention is to overcome the problems of insufficient search efficiency of asynchronous multi-agent systems under time constraints and the inability to effectively decompose non-additive, probabilistic team goals into intelligent individuals with asynchronous decision-making timelines in the prior art, and to provide an asynchronous multi-agent collaborative decision-making method and system based on probabilistic decomposition.

[0006] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0007] An asynchronous multi-agent cooperative decision-making method based on probabilistic decomposition, the main steps of which are as follows: S1: Construct a global environment model based on the decision task to be decomposed and initialize basic parameters; the basic parameters include the basic parameters of the agent and the basic parameters of the controller; S2: Simulate each agent in the global environment model based on the basic parameters, and align the team decision timeline through a filling mechanism; S3: Under the aligned team decision timeline, define and evaluate the team value function; update the team value function using the time-constrained Bellman operator, and decompose the team value function to obtain the individual value function of each agent; S4: Based on the individual value function and actual decision steps of each agent, the policy network of the corresponding agent is updated using asynchronous policy gradient; S5: Each agent performs simulation training in the global environment model according to the corresponding policy network; S6: Repeat S2 to S5 for training until the policy network becomes stable and is considered to have converged, or until the preset number of training rounds is reached. Output the current policy network to the corresponding agent to complete the multi-agent collaborative decision-making task.

[0008] This method mainly consists of three parts: a generalized value function approximation (G-VFA) module, an asynchronous policy gradient update (A-PG) module, and an integration of enhancement techniques.

[0009] The Generalized Value Function Approximation (G-VFA) module employs a padding mechanism to align the asynchronous timelines of each agent to a unified team decision timeline. This mechanism maintains time consistency, reduces computational complexity, and eliminates the need for additional states. It also provides a synchronized and consistent time baseline for subsequent value function evaluation. After unifying the decision timeline, the team value function is evaluated and updated using a time-constrained Bellman operator. The team value function is decomposed into individual value functions through the principle of "asynchronous individual-global maximization". This principle aligns with probabilistic objectives and ensures that, under asynchronous conditions, the team value function can also be optimized when each agent independently optimizes its individual value function.

[0010] The Asynchronous Policy Gradient Update (A-PG) module first updates the agent... In team decision-making steps The individual value function at a given time maps back to its actual decision-making steps. Since the virtual state-action pairs introduced by the filling mechanism do not contain new environmental transition information and are only used to maintain the timeline format, filling or removing these virtual pairs does not change the approximate mathematical expectation of the value function. This ensures that the inverse mapping from the team timeline to the individual asynchronous timeline is feasible and correct. Based on the mapped individual value function, the "asynchronous policy gradient theorem" can be used to update the policy network. parameters This allows each agent to update its own policy network independently and asynchronously.

[0011] The integration of enhancement techniques mainly includes off-policy learning, soft update mechanisms, and entropy regularization. Off-policy learning utilizes a trajectory replay buffer. Historical trajectory experience improves training efficiency; the soft update mechanism improves training stability and ensures the continuity of the motion trajectory by controlling the update step size and balancing the ratio of exploration and utilization; entropy regularization adds a policy entropy weighting term to the policy optimization objective, enabling the agent to explore more in the early stages of training and avoid premature convergence to a suboptimal policy.

[0012] As a preferred embodiment of the present invention, the basic parameters of the intelligent agent and the controller in S1 respectively include: The basic parameters of an agent include: the agent's initial position, the agent's velocity attribute, and the individual policy network. and individual value network The controller's basic parameters include: a pre-trained or randomly initialized joint policy network. Value Network Target network and trajectory playback buffer .

[0013] As a preferred embodiment of the present invention, this method employs a bidirectional node graph environment modeling method to model the indoor environment as a weighted graph, the expression of which is:

[0014] in, A node represents a gate; Let be the edges, representing the set of edges that allow passage between doors; Let be the edge weight, and let represent the time travel. Gather the rooms.

[0015] Each door is modeled as a bidirectional node to distinguish the robot's orientation (e.g., B). C and C (B represents different directions). The weights inside the room represent the time required for the robot to turn around, while the weights outside the room represent the travel time between nodes.

[0016] Once the global environment model is built and all basic parameters are initialized, the simulation phase begins. In the Generalized Value Function Approximation (G-VFA) module, the team decision-making timeline is first aligned using a filling mechanism.

[0017] As a preferred embodiment of the present invention, the filling mechanism rule is as follows: When the intelligent agent team enters a new decision-making step When an agent has completed an action, its current completed state-action pair is recorded; when an agent is still performing an action, its most recent state-action pair is copied as a virtual state-action pair and recorded.

[0018] The filling mechanism overcomes the waiting idle problem in the synchronous framework and can make efficient use of the time budget.

[0019] As a preferred embodiment of the present invention, step S3 includes the following steps: S31: Based on the joint state Joint actions and the policy network Evaluation Team Value Function ; S32: Based on the system's smallest time granularity And the current decision-making time determines the team's decision-making steps. Then, based on the team decision-making process... The joint state generated at time and joint actions The team value function is updated using the time-constrained Bellman operator; the expression for the team decision step is:

[0020] in, Steps for team decision-making; For the current decision-making time; This represents the smallest time granularity of the system. In a preferred embodiment of the present invention, the calculation of the time-constrained Bellman operator in step S32 follows a recursive relation, which includes the following steps: S321: In the team decision-making process Next, each intelligent agent, according to its own actions... and individual strategy network Move to a new location, joint state Transition to the next union state And obtain the probability of capturing the target during the transfer process. ; S322: According to the joint state and joint state Corresponding joint actions Calculate the current team value function of the intelligent agent team. ; S323: Based on the current team value function of the agent team and the probability of capturing the target and the state transition probability during the transition process, the team value function is obtained through the time-constrained Bellman operator, and the expression is as follows:

[0021] in, For time-constrained Bellman operators; The optimized team value function; The state transition probability; To carry out joint operations Later from the joint state To the joint state The probability of capturing the target during the transfer process; For joint strategy; For the joint state The team value function at that time; In a noncyclic state space, the time-constrained Bellman operator is a shrinking map, thus ensuring that the sequence of value functions updated by this operator converges to a unique optimal value function.

[0022] S33: Obtain the individual value function by inverse solving the team value function. The two satisfy the decomposition form of the "asynchronous individual-global maximization" principle, and its expression is:

[0023] in, For team value function; The number of agents; For intelligent agents; It is an individual value function.

[0024] As a preferred embodiment of the present invention, the "asynchronous individual-global maximization" principle means that for an agent that is performing an action but has not yet completed the action, its available action space is limited to a single option (i.e., continue to perform the current action); while for an agent that has completed the action, independently maximizing its individual value function is equivalent to maximizing the team value function.

[0025] This formula expresses the team's success probability as "1 minus the product of the individual failure probabilities of all agents," matching the probabilistic objective; and ensures that, under asynchronous conditions, the team's overall objective can also be optimized when each agent independently optimizes its individual value function.

[0026] As a preferred embodiment of the present invention, step S4 includes the following steps: S41: Remove the virtual state-action pairs copied in the filling mechanism; S42: [The sentence is incomplete and likely refers to an intelligent agent.] In synchronized team decision-making steps Individual value function at time Mapping back to its previous asynchronous decision step The asynchronous decision step is obtained Individual value function at time ; S43: Based on the individual value function and gradient update formula for individual policy networks parameters Perform gradient updates; the gradient update formula is expressed as:

[0027] in, For intelligent agents According to strategy From its initial state The generated motion trajectory; The length of the trajectory; Because the virtual state-action pairs recorded by the filling mechanism do not contain new environmental transition information, removing the virtual state-action pair does not change the approximate mathematical expectation of the value function. This ensures that the inverse mapping from the team timeline to the individual asynchronous timeline is correct and feasible. Therefore, each agent can update its individual policy network using its own value function on its respective decision timeline.

[0028] As a preferred embodiment of the present invention, when performing asynchronous policy gradient updates, off-policy learning is introduced, which includes the following steps: a1: A single intelligent agent The trajectory integration of a complete training session is expressed as:

[0029] in, For intelligent agents A complete set of training trajectories. The length of the trajectory; a2: Integrates the trajectories of each agent's complete training run, and its expression is:

[0030] in, The set of trajectories of all agents during a complete training session; The number of agents; a3: Integrate the trajectories from each complete training session of all agents to obtain the trajectory replay buffer. Its expression is:

[0031] a4: During each training session, if the agent satisfies the following:

[0032] in This serves as a buffer for trajectory playback. This is the minimum threshold for the trajectory playback buffer. Then, using the trajectory importance sampling ratio, from the trajectory playback buffer... Sampling is performed in the process, and the samples are incorporated into the gradient update formula to update the policy network; The gradient update formula combining the trajectory importance sampling ratio is:

[0033] in, For intelligent agents According to behavioral strategies From its initial state The generated motion trajectory; The behavioral strategy represents the historical strategy in the trajectory playback buffer; The length of the trajectory; For individual value functions; The trajectory importance sampling ratio is the value of the agent's current individual policy. Behavioral strategies corresponding to historical trajectories stored in the trajectory playback buffer The ratio, its expression is:

[0034] in, For the behavioral strategy, its parameters are: ; The length of the trajectory; Since this value represents the agent's current individual policy. Behavioral strategies corresponding to historical trajectories stored in the trajectory playback buffer The ratio may result in a large variance; this method uses a trajectory playback buffer with a large capacity but a relatively small size; the trajectory playback buffer can store multiple trajectories and support multiple sampling strategies, thereby balancing data efficiency and computational stability.

[0035] As a preferred embodiment of the present invention, a soft update mechanism is introduced when performing asynchronous policy gradient updates; the soft update mechanism is divided into a commentator network end and an executor network end.

[0036] The update rules for the commentator's online platform are as follows: Target Network parameters Tracking online individual value networks at a slow pace parameters Its expression is:

[0037] in, This is the soft update coefficient.

[0038] This soft update enables the target network to gradually track changes in the main evaluation network, ensuring a stable learning process.

[0039] The update rule for the executor is as follows: A strategy gradient pruning mechanism is adopted to limit the ratio of change between old and new strategies; The gradient update formula with gradient clipping mechanism is:

[0040] in, For intelligent agents According to strategy From its initial state The generated motion trajectory; The length of the trajectory; This is the clipping function; For trajectory playback buffer Historical policy network for storage Parameters; Set the pruning threshold for the strategy; It is an individual value function.

[0041] By introducing a policy gradient pruning mechanism, excessive policy parameter updates are prevented. This technique limits the magnitude of policy changes by setting a threshold, thereby effectively pruning gradient values ​​that exceed this threshold.

[0042] As a preferred embodiment of the present invention, when performing asynchronous policy gradient updates, entropy regularization is introduced, wherein the entropy regularization is: Adding a weighted term of policy entropy to the gradient update formula, its expression is:

[0043] in, The regularization coefficient is used. For intelligent agents According to strategy From its initial state The generated motion trajectory; The length of the trajectory; It is an individual value function.

[0044] Regularization coefficient This is used to control the balance between exploration and exploitation; during training, Typically decays over time, allowing the algorithm to gradually shift its focus from exploration to exploitation as training progresses; by maximizing the entropy regularization objective function, this strategy is encouraged to avoid premature convergence to a particular strategy, and instead promote the exploration of diverse and potentially optimal strategies.

[0045] As a preferred embodiment of the present invention, it is optional, but preferably, to combine the three enhancement modules to construct a complete enhancement technique, resulting in a complete gradient update formula, the expression of which is:

[0046] in, For intelligent agents According to behavioral strategies From its initial state The generated motion trajectory; The behavioral strategy represents the historical strategy in the trajectory playback buffer; The length of the trajectory; For individual value functions; The entropy regularization value function; The importance sampling ratio of the trajectory; This is the strategy pruning function.

[0047] By introducing three optimized enhancement modules, the problems of low sample efficiency, high learning variance, and easy getting trapped in local optima are overcome, thus ensuring the training effect.

[0048] As a preferred embodiment of the present invention, an iterative training method is adopted, repeating steps S2 to S5, and performing simulation training in the global environment model until the policy network tends to stabilize and is considered to have converged, or until a preset number of training rounds is reached; the current policy network is output to the corresponding agent to complete the multi-agent collaborative decision-making task.

[0049] An asynchronous multi-agent cooperative decision-making system based on probabilistic decomposition includes at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to execute any of the above-described asynchronous multi-agent cooperative decision-making methods based on probabilistic decomposition.

[0050] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention overcomes the waiting and idle problem in synchronous frameworks through a filling mechanism, enabling efficient use of time budget. By employing the time-constrained Bellman operator and the probabilistic value function decomposition formula, it directly aligns asynchronous multi-agent reinforcement learning with the time-constrained probability maximization objective, allowing the algorithm to directly optimize the team value function metric (capture probability) rather than indirectly accumulating rewards. Simultaneously, each agent updates its individual policy network using an asynchronous gradient update method, ensuring that the updates to the individual agent's policy network are unaffected by team decisions. This effectively decomposes non-additive, probabilistic team objectives into individual agents with asynchronous decision-making timelines, driving them to make collaborative decisions. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the overall process of an asynchronous multi-agent collaborative decision-making method based on probabilistic decomposition as described in Embodiment 1 of the present invention.

[0052] Figure 2 This is a schematic diagram illustrating the effect of the filling mechanism in the asynchronous multi-agent collaborative decision-making method based on probabilistic decomposition described in Embodiment 2 of the present invention.

[0053] Figure 3 This is a performance comparison chart between the asynchronous multi-agent cooperative decision-making method (MA-APD) based on probabilistic decomposition described in Embodiment 2 of the present invention and existing baseline methods.

[0054] Figure 4 This is a simplified use case diagram of an asynchronous multi-agent cooperative decision-making method based on probabilistic decomposition as described in Embodiment 4 of the present invention.

[0055] Figure 5 This is a comparison of the learning curves of the asynchronous multi-agent collaborative decision-making method based on probabilistic decomposition described in Embodiment 4 of the present invention with existing baseline methods.

[0056] Figure 6 This is a schematic diagram of the structure of an asynchronous multi-agent cooperative decision-making system based on probability decomposition, which utilizes any of the aforementioned asynchronous multi-agent cooperative decision-making methods based on probability decomposition, as described in Embodiment 5 of the present invention. Detailed Implementation

[0057] The present invention will be further described in detail below with reference to experimental examples and specific embodiments. However, this should not be construed as limiting the scope of the above-mentioned subject matter of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0058] Example 1 like Figure 1 As shown, an asynchronous multi-agent cooperative decision-making method based on probabilistic decomposition includes the following steps: S1: Construct a global environment model based on the decision task to be decomposed and initialize basic parameters; the basic parameters include the basic parameters of the agent and the basic parameters of the controller.

[0059] S2: Simulate each agent in the global environment model based on the basic parameters, and align the team decision timeline through a filling mechanism.

[0060] S3: Under the aligned team decision timeline, define and evaluate the team value function; update the team value function using the time-constrained Bellman operator, and decompose the team value function to obtain the individual value function of each agent.

[0061] S4: Based on the individual value function and actual decision steps of each agent, the policy network of the corresponding agent is updated using asynchronous policy gradient.

[0062] S5: Each agent performs simulation training in the global environment model according to the corresponding policy network.

[0063] S6: Repeat S2 to S5 for training until the policy network becomes stable and is considered to have converged, or until the preset number of training rounds is reached. Output the current policy network to the corresponding agent to complete the multi-agent collaborative decision-making task.

[0064] Example 2 This embodiment is a specific implementation of the asynchronous multi-agent cooperative decision-making method based on probabilistic decomposition described in Embodiment 1. It is implemented in three different agent group sizes (2, 3, and 4 agents) and in two classic environments: an office and a library. The steps include: S1: Construct office and library environment models and initialize basic parameters according to the decision tasks to be decomposed; the basic parameters include the basic parameters of the agent and the basic parameters of the controller.

[0065] The global environment model in S1 is modeled as a weighted graph using a bidirectional node graph environment modeling method. Its expression is:

[0066] in, A node represents a gate; Let be the edges, representing the set of edges that allow passage between doors; Let be the edge weight, and let represent the time travel. Gather the rooms.

[0067] The basic parameters of the agent and controller in S1 include: Basic parameters of the agent: initial position of the random agent, agent velocity attribute, and random individual policy network. and random individual value network .

[0068] Controller basic parameters: Set total time budget Pre-trained or randomly initialized joint policy networks Value Network Target network and trajectory playback buffer ( ).

[0069] Once the global environment model is built and all basic parameters are initialized, the module for generalized value function approximation (G-VFA) is entered.

[0070] S2: Simulate each agent in the global environment model based on the basic parameters, and align the team decision timeline through a filling mechanism.

[0071] Because the execution time of each agent's actions is different, their decision steps (Corresponding decision time) The timeline is asynchronous. To enable centralized value assessment, this invention introduces a filling mechanism to align the asynchronous timeline to a unified team decision-making timeline. This process does not introduce additional states and does not interrupt the execution of the agent's actions.

[0072] like Figure 2 As shown, the filling mechanism rule is as follows: When the intelligent agent team enters a new decision-making step When an agent has completed an action, its current completed state-action pair is recorded; when an agent is still performing an action, its most recent state-action pair is copied as a virtual state-action pair and recorded.

[0073] The filling mechanism overcomes the waiting idle problem in the synchronous framework and can make efficient use of the time budget.

[0074] S3: Under the aligned team decision timeline, define and evaluate the team value function; update the team value function using the time-constrained Bellman operator, and decompose the team value function to obtain the individual value function of each agent.

[0075] S31: Based on the joint state Joint actions and the policy network Evaluation Team Value Function .

[0076] S32: Based on the system's smallest time granularity And the current decision-making time determines the team's decision-making steps. Then, based on the team decision-making process... The joint state generated at time and joint actions The team value function is updated using the time-constrained Bellman operator; the expression for the team decision step is as follows.

[0077]

[0078] in, Steps for team decision-making; For the current decision-making time; This represents the minimum time granularity of the system.

[0079] The time-constrained Bellman operator satisfies the following recursive steps: S321: In the team decision-making process Next, each intelligent agent, according to its own actions... and individual strategy network Move to a new location, joint state Transition to the next union state And obtain the probability of capturing the target during the transfer process. .

[0080] S322: According to the joint state and joint state Corresponding joint actions Calculate the current team value function of the intelligent agent team. .

[0081] S323: Based on the current team value function of the agent team and the probability of capturing the target and the state transition probability during the transition process, the team value function is obtained through the time-constrained Bellman operator, and the expression is as follows:

[0082] in, For time-constrained Bellman operators; The optimized team value function represents the expected probability that the team will successfully complete the task (such as capturing the target) within the remaining time. The state transition probability; To carry out joint operations Later from the joint state To the joint state The probability of capturing the target during the transfer process; For joint strategy; For the joint state The team value function at that time.

[0083] In a noncyclic state space, the time-constrained Bellman operator is a shrinking map, thus ensuring that the sequence of value functions updated by this operator converges to a unique optimal value function.

[0084] S33: Obtain the individual value function by inverse solving the team value function. The two satisfy the decomposition form of the asynchronous individual-global maximization principle, and its expression is:

[0085] in, For team value function; The number of agents; For intelligent agents; It is an individual value function.

[0086] The asynchronous individual-global maximization principle is as follows: For an agent that is performing an action but has not yet completed it, its available action space is limited to a single option (i.e., continue performing the current action); while for an agent that has completed its action, maximizing its individual value function independently is equivalent to maximizing the team value function.

[0087] This formula expresses the team's success probability as "1 minus the product of the individual failure probabilities of all agents," matching the probabilistic objective; and ensures that, under asynchronous conditions, the team's overall objective can also be optimized when each agent independently optimizes its individual value function.

[0088] Next, we enter the Asynchronous Policy Gradient Update (A-PG) module, which is responsible for updating the policies of each agent using the decomposed individual value function.

[0089] S4: Based on the individual value function and actual decision steps of each agent, the policy network of the corresponding agent is updated using asynchronous policy gradient.

[0090] S41: Remove the virtual state-action pairs copied in the filling mechanism.

[0091] S42: [The sentence is incomplete and likely refers to an intelligent agent.] In synchronized team decision-making steps Individual value function at time Mapping back to its previous asynchronous decision step The asynchronous decision step is obtained Individual value function at time .

[0092] Because the virtual state-action pairs recorded by the filling mechanism do not contain new environmental transition information, removing the virtual state-action pair does not change the approximate mathematical expectation of the value function. This ensures that the inverse mapping from the team timeline to the individual asynchronous timeline is correct and feasible. Therefore, each agent can update its individual policy network using its own value function on its respective decision timeline.

[0093] S43: Based on the individual value function and gradient update formula for individual policy networks parameters Perform gradient updates; the gradient update formula is expressed as follows.

[0094]

[0095] in, For intelligent agents According to strategy From its initial state The generated motion trajectory; The length of the trajectory.

[0096] This step completes the basic policy network update; the next step is to integrate enhancement technologies.

[0097] Since basic policy gradient updates have certain limitations, we will integrate three enhancement modules—policy learning, soft update mechanism, and entropy regularization—to obtain a complete gradient update formula:

[0098] in, For intelligent agents According to behavioral strategies From its initial state The generated motion trajectory; The behavioral strategy represents the historical strategy in the trajectory playback buffer; The length of the trajectory; For individual value functions; The entropy regularization value function; The importance sampling ratio of the trajectory; This is the strategy pruning function.

[0099] Meanwhile, in the commentator network of the soft update mechanism, its target network... parameters Tracking online individual value networks at a slow pace parameters Its expression is:

[0100] in, This is the soft update coefficient.

[0101] This step overcomes the problems of low sample efficiency, high learning variance, and susceptibility to local optima by introducing three optimized enhancement modules: off-policy learning, soft update mechanism, and entropy regularization. Off-policy learning then proceeds once the replay threshold is reached. At that time, through the trajectory playback buffer Historical trajectory experience is used to improve training efficiency; the soft update mechanism controls the update step size, i.e., sets a smaller soft update coefficient. Balancing the ratio of exploration and exploitation enhances training stability and ensures the continuity of movement trajectories; entropy regularization incorporates a weighted term of policy entropy into the policy optimization objective, adjusting the regularization coefficient accordingly. Gradually reduce the value to 0 to allow the agent to explore more in the early stages of training, thus avoiding premature convergence to a suboptimal strategy.

[0102] S5: Each agent performs simulation training in the global environment model according to the corresponding policy network.

[0103] S6: Repeat S2 to S5 for training until the policy network becomes stable and is considered to have converged, or until the preset number of training rounds is reached. Output the current policy network to the corresponding agent to complete the multi-agent collaborative decision-making task.

[0104] After completing tasks with different agent group sizes, nine existing baseline methods were introduced, mainly including classic multi-robot search algorithms and synchronous / asynchronous multi-agent reinforcement learning algorithms. These were trained in the same simulated office and library environments and with the same agent group sizes (2, 3, and 4). Figure 3 As shown, a performance comparison chart of training is obtained. MA-APD is the method of this invention, which has a significantly higher capture probability than other baseline methods. At the same time, it achieves the best performance while having high training efficiency.

[0105] Example 3 This embodiment uses a specific decision-making task to be decomposed as an example to demonstrate and illustrate the asynchronous multi-agent cooperative decision-making method based on probability decomposition described in Embodiment 2; the decision-making task to be decomposed in this embodiment is: Consider a warehouse environment whose floor plan can be abstracted as a structure containing 10 nodes ( to Weighted undirected graph Each node Represents a room or goods area, side The weight of an edge represents a corridor or passageway between rooms. This represents the average time (in seconds) required for the robot to traverse the path. The target (e.g., a misplaced box of goods) moves on the graph according to a random walk model, meaning that at any given moment, the target moves to any adjacent node of its current node or remains stationary with equal probability. Four robots of varying speeds are deployed. The search tasks involve different maximum movement speeds, resulting in varying times required to cover the same distance. The total task time budget is T = 10 seconds.

[0106] At this point, the corresponding asynchronous multi-agent cooperative decision-making method based on probability decomposition includes the following steps: S1: System initialization.

[0107] S11: Environment Modeling: Constructing a Weighted Graph For example, nodes and , The edges are connected, with edge weights of 1.2 seconds and 0.8 seconds respectively. All robots and the target move on this graph.

[0108] S12: Robot State Initialization: Determine the initial position (node) and velocity attributes for each robot. For example, R1 is initially located at... The speed is "slow" (edge ​​weight time coefficient is 1.5); Initially located The speed is "fast" (coefficient is 0.7); Initially located , Initially located Their speeds vary.

[0109] S13: Target Belief Initialization: Since the target location is unknown, its possible locations are described using a probability distribution (belief state) b across all nodes. At the start of the task, if there is no prior information, it can be initialized as a uniform distribution, i.e. ,for .

[0110] S14: Controller Initialization: The controller loads the pre-trained or randomly initialized policy network. and value network (i=1,2,3,4), its structure is detailed in S4. Simultaneously, the trajectory playback buffer is initialized. Target network parameters (Set to be with) (Same), and training hyperparameters.

[0111] S2: Asynchronous decision-making and filling.

[0112] Each robot operates independently. When it reaches a node (decision point) or a fixed time interval, it sends a decision request to the central controller, including its current node number. and local time The controller maintains a virtual, discrete "team time step". The time interval Based on the system's smallest time granularity setting, for example Second.

[0113] With robots and For example, in the initial period of a mission: exist Within seconds, all robots started up simultaneously. The speed is relatively fast, firstly in Arrive at the node in seconds And issue a decision request. At this point, the team's time step... Starting from 0, it has progressed to (Corresponding to 0.8 seconds / 0.1 seconds). The controller received the request.

[0114] The controller executes a "filling mechanism": it checks the team's time step. arrive Are there other robots in between (such as...) A decision has been made. Since this is the first decision, there are no historical decisions. To construct an aligned sequence of team states for evaluation. The controller needs to perform "status backfilling".

[0115] for to The controller infers (or "fills in") the robots' known initial positions and motion models at each virtual time step. The "predicted location". For example, in hour, May still be from arrive On the way, the status is "on the side". "Up, 0.5 * edge weight has been moved."

[0116] At the same time, regarding goals and beliefs The controller utilizes a random walk transition model, from Start by progressively forwarding the computation to... .

[0117] Ultimately, for exist Second( In response to this decision, the controller constructed a team state sequence of length 9. Each of them It includes the populated state and current goal belief of all four robots. .

[0118] After that, exist Arrive at the node in seconds And issue a decision request, corresponding to the team's time step. 15. The controller runs the filling mechanism again. This time, it finds that R2 is already... A decision was made at that time (to go to) 5), therefore, in order to 9 to When in state 15, The status will be based on its Decision at the time (to go) 5) and its speed of motion are calculated. 1. In itself to The state is based on its origin arrive The movement fills the void. The goal and belief continue to spread.

[0119] Through this mechanism, the controller can generate a timeline up to the current team time step for each decision. A complete team state sequence This provides a synchronous and consistent time baseline for subsequent value assessment.

[0120] S3: Evaluation and decomposition of generalized value function.

[0121] The central controller receives the robot In team time step The decision request is received, and an aligned state sequence is obtained. Then, it is necessary to evaluate each possible action in the current state. The long-term value of (i.e., traveling from the current node to its neighboring nodes).

[0122] S31: Constructing Input: For Each candidate action The controller then inputs the action along with the state sequence into a "time-constrained Bellman operator" for evaluation. The core of this operator is to recursively calculate the time-constrained Bellman operation from the current step. Until the mission ends (correspond Second, The team's expected returns.

[0123] S32: Recursively calculate the team's Q-value Defined in team time step Team status is ,robot Take action Other robots ( It will follow its current strategy. Under the conditions of action, the team from arrive The capture probability is Its calculation follows the following recursive relationship: Real-time capture check: in status Based on the position and target beliefs of all robots The computational robotics team is currently at the step The probability of immediately capturing the target For example, if a robot and a target are on the same node with a certain probability, then capture is possible.

[0124] State transitions and future rewards: If not captured immediately, consider state transitions. The objective transitions to the next belief based on the random walk model. Each robot, according to its actions... (for ) and strategies (For other robots) Move to a new location, forming The prediction.

[0125] Recursive future value: calculated in Below, the team's future capture probability ; yes exist According to its strategy Expectations of choosing an action value.

[0126] Bellman equation: Combining the above two steps, based on the asynchronous individual-global maximization principle, The value at this time is equivalent to This yields the modified Bellman equation:

[0127] This reflects the superposition of probabilities between "immediate capture" and "future capture".

[0128] S33: Core Probability Decomposition: In order to maximize team value Assigning tasks to individual robots to guide their independent learning, this invention introduces a key decomposition formula. Defined under the same conditions, the robot... itself (from) arrive The probability of capturing the target is Note that a team capture event is "at least one robot captures a target". Based on probability theory, and assuming that each robot capture event is conditionally independent under a given state (which is a reasonable approximation of the decomposition), we have:

[0129] Therefore, the team With individuals The relationship between them is expressed as follows:

[0130] in .

[0131] In particular, in the assessment action At that time, other robots of It is based on their current strategy The estimated expected value.

[0132] Using this formula, when the value is known (or has been calculated) Then, the current decision-making robot can be deduced. Individual value This serves as the standard for evaluating the quality of its actions. In actual calculations, This is obtained from the recursive process in step S32. This can be obtained by solving the above decomposition formula.

[0133] S4: Asynchronous strategy update.

[0134] S41: Value Mapping and Action Selection: Obtaining the corresponding value through S3 Each candidate action individual value. Policy network The current state sequence (or its compact representation, such as...) The controller takes local observation history and a summary of target beliefs as input and outputs an action probability distribution. The controller can be based on... The value is obtained by sampling the network using ε-greedy or directly based on the policy. Choose an action * (e.g., "go to node") "), and issue the action command to implement.

[0135] S42: Policy Gradient Update Decision data, including state sequences and selected actions. * Individuals obtained through calculation The values, along with the new states and rewards entered after the actions are performed, are organized into an experience trajectory and stored in the central trajectory replay buffer. During the training phase, regularly from Data from the middle sampling batch.

[0136] For the sampled The policy network based on the data from a particular decision. parameters Updates are performed using the policy gradient method. The gradient direction aims to maximize the expected return, which can be based on the advantage function. (in It is calculated using the state-value function. The gradient update formula is:

[0137] The key here is the method used to compute the policy gradient. The value is obtained through centralized evaluation and decomposition of S3, ensuring that the gradient signal contains information about team collaboration, despite the policy network. It operates independently.

[0138] S43: Network structure example: Policy Network : Input layer: Receiving robot Local observations (such as the current node code, neighbor node information), historical actions in the most recent L steps, and goal beliefs. A summary of some kind (such as a feature vector compressed by a small neural network). An example of the total input dimension is... .

[0139] GRU layer: A Gated Recurrent Unit (GRU) network with a hidden layer size of 128, used to process historical sequence information and extract temporal features.

[0140] Fully connected layer: The output of the last time step of the GRU is fed into a fully connected layer with 64 neurons and using the ReLU activation function.

[0141] Output layer: connected to a layer with... neurons ( yes The action space on each node is such that there are 5 adjacent nodes plus a pause. A fully connected layer is then connected to a Softmax activation function to output the probability of each action.

[0142] Value Network (used for baseline calculation) Or used directly for evaluation): The structure is similar to a policy network, but the final output layer consists of a single neuron (without an activation function or using linear activation), outputting the state value. The estimation. Its input also contains state sequence information.

[0143] S5: Training and Optimization.

[0144] The training of the entire system is offline and centralized, but the learned policies are distributed.

[0145] S51: Experience Replay: The central controller continuously collects experience data generated from all online interactions of the robots. Stored in the trajectory playback buffer During training, from Uniformly sample small batches of data.

[0146] S52: Target Network and Soft Update: For stable training, the value network for each robot Maintain a target network Parameters of the target network Tracking online network parameters at a slow speed The update rules are as follows:

[0147] in This is the soft update factor, for example, set to 0.01.

[0148] S53: Loss Function and Entropy Regularization: Value Networks The policy network is learned by minimizing the mean squared loss of the temporal difference error (TD-error). The loss function includes a policy gradient term and an entropy regularization term. The entropy regularization term encourages exploration, and its weights... In the early stages of training, a larger value can be set (e.g.) (), and then gradually decreases.

[0149] S54: Training loop: Repeat the process of data collection, sampling, loss calculation, backpropagation to update online network parameters, and soft update target network parameters until the policy performance converges (tends to stabilize).

[0150] Through the above steps, this invention successfully achieves the learning of an efficient distributed cooperative search strategy through centralized training in an asynchronous, multi-robot system with varying speeds, which can maximize the capture probability of dynamic targets within a limited time budget.

[0151] Example 4 This embodiment demonstrates the training results of the asynchronous multi-agent cooperative decision-making method based on probabilistic decomposition described in Embodiment 2 using a simple scenario. It also introduces four other existing baseline methods for comparison, aiming to illustrate that the present invention captures a higher probability than other baseline methods within a limited time budget. The specific steps are as follows: like Figure 4 As shown, two robots are defined ( , ), From the starting point Set off, From the starting point Departure, setting a time budget If the time is 3.5 seconds, then the mission objective is to cover as many endpoints as possible within the time budget, that is, to maximize the capture probability within 3.5 seconds. .

[0152] In actual implementation, the robot's specific path, total time, and endpoint coverage are obtained: (from Set off):

[0153] (from Set off):

[0154] Key constraint: Time budget The seconds are very tight. It takes 3.2 seconds to get to D (just within budget). It takes exactly 3.5 seconds to reach D (on the line). This means that any extra time wasted (such as waiting) will result in the inability to complete the overlay.

[0155] Key success factors: exist Seconds are not equal ,predict In the future, D can be efficiently covered (B→D takes only 1.5 seconds), so D will be left to... 1. Switch to C automatically. This is the benefit of division of labor brought about by "asynchronous + predictive". If exist You must wait for seconds arrive If the synchronization decision is made again in seconds (i.e., the synchronization strategy is implemented), then... With only 1.5 seconds remaining, it's too late to go to either C or D.

[0156] For the other four baseline methods, under the same time budget and the same simple model, the target acquisition results are as follows: Figure 5 The learning curve comparison chart is shown. As can be seen from the comparison chart, in a simple scenario, this method performs better in terms of time budget. It achieved a capture probability of approximately 89.65% per second, far exceeding other asynchronous multi-agent reinforcement learning baseline methods.

[0157] Example 5 like Figure 6 As shown, an asynchronous multi-agent cooperative decision-making system based on probabilistic decomposition includes at least one processor, a memory communicatively connected to the at least one processor, and at least one input / output interface communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed, enables the at least one processor to perform the asynchronous multi-agent cooperative decision-making method based on probabilistic decomposition described in the foregoing embodiments. The input / output interface may include a display, keyboard, mouse, and USB interface for inputting and outputting data.

[0158] Furthermore, this asynchronous multi-agent collaborative decision-making system based on probabilistic decomposition can be a desktop computer, mobile phone, tablet computer, wearable asynchronous multi-agent collaborative decision-making system based on probabilistic decomposition, etc., capable of deep information recognition.

[0159] Furthermore, the processor may include one or more processing cores. The processor connects various parts of the asynchronous multi-agent cooperative decision-making system based on probabilistic decomposition using various interfaces and lines. It executes various functions and processes data within the system by running or executing instructions, programs, code sets, or instruction sets stored in memory, and by calling data stored in memory. Optionally, the processor may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also be implemented separately as a communication chip, without being integrated into the processor.

[0160] The memory may include random access memory (RAM) or read-only memory (ROM). The memory can be used to store instructions, programs, code, code sets, or instruction sets, such as instructions or code sets used to implement the asynchronous multi-agent cooperative decision-making method based on probabilistic decomposition provided in this application. The memory may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described above, etc. The data storage area may also store data created during the use of the asynchronous multi-agent cooperative decision-making system based on probabilistic decomposition (such as a mapping table of modulation sequences and depths, image data, spectrogram data, etc.).

[0161] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory, magnetic disks, or optical disks.

[0162] When the integrated units of the present invention are implemented as software functional units and sold or used as independent products, they can also be stored in a computer-readable storage medium. The computer-readable storage medium stores program code, which can be called by a processor to execute the methods described in the above method embodiments. Based on this understanding, the technical solution of the embodiments of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer system (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium includes a non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program code that executes any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code can be compressed, for example, in an appropriate form.

[0163] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An asynchronous multi-agent cooperative decision-making method based on probabilistic decomposition, characterized in that, Includes the following steps: S1: Construct a global environment model based on the decision task to be decomposed and initialize basic parameters; the basic parameters include the basic parameters of the agent and the basic parameters of the controller; S2: Simulate each agent in the global environment model based on the basic parameters, and align the team decision timeline through a filling mechanism; S3: Under the aligned team decision timeline, define and evaluate the team value function; update the team value function using the time-constrained Bellman operator, and decompose the team value function to obtain the individual value function of each agent; S4: Based on the individual value function and actual decision steps of each agent, the policy network of the corresponding agent is updated using asynchronous policy gradient. S5: Each agent performs simulation training in the global environment model according to the corresponding policy network; S6: Repeat S2 to S5 for training until the policy network becomes stable and is considered to have converged, or until the preset number of training rounds is reached. Output the current policy network to the corresponding agent to complete the multi-agent collaborative decision-making task.

2. The asynchronous multi-agent cooperative decision-making method based on probabilistic decomposition according to claim 1, characterized in that, The basic parameters of the agent and controller in S1 include: Basic parameters of an agent: agent's initial position, agent's velocity attribute, individual policy network, and individual value network; Controller basic parameters: pre-trained or randomly initialized joint policy network, value network, target network, and trajectory playback buffer.

3. The asynchronous multi-agent cooperative decision-making method based on probabilistic decomposition according to claim 1, characterized in that, The filling mechanism in S2 is as follows: When the agent team enters a new team decision-making step For an agent that has completed an action, record its current state-action pair; For agents that are still performing actions, their most recent state-action pair is copied as a virtual state-action pair and recorded.

4. The asynchronous multi-agent cooperative decision-making method based on probabilistic decomposition according to claim 1, characterized in that, S3 includes the following steps: S31: Evaluate the team value function based on the joint state, joint actions, and the policy network; S32: Determine the team decision step based on the system's minimum time granularity and the current decision time; then, update the team value function using the time-constrained Bellman operator based on the joint state and joint actions generated at the team decision step; the expression for the team decision step is: in, Steps for team decision-making; For the current decision-making time; This represents the smallest time granularity of the system. S33: The individual value function is obtained by inverse solving the team value function. The two satisfy the asynchronous individual-global maximization principle, and their expression is: in, For team value function; The number of agents; For intelligent agents; It is an individual value function.

5. The asynchronous multi-agent cooperative decision-making method based on probabilistic decomposition according to claim 4, characterized in that, The calculation of the time-constrained Bellman operator in S32 follows a recursive relation, which includes the following steps: S321: In the team decision-making process Next, each intelligent agent, according to its own actions... and individual strategy network Move to a new location, joint state Transition to the next union state And obtain the probability of capturing the target during the transfer process. ; S322: According to the joint state and joint state Corresponding joint actions Calculate the current team value function of the intelligent agent team. ; S323: Based on the current team value function of the agent team and the probability of capturing the target and the state transition probability during the transition process, the team value function is obtained through the time-constrained Bellman operator, and the expression is as follows: in, For time-constrained Bellman operators; The optimized team value function; The state transition probability; To carry out joint operations Later from the joint state To the joint state The probability of capturing the target during the transfer process; For joint strategy; For the joint state The team value function at that time.

6. The asynchronous multi-agent cooperative decision-making method based on probabilistic decomposition according to claim 1, characterized in that, S4 includes the following steps: S41: Remove the virtual state-action pairs copied in the filling mechanism; S42: [The sentence is incomplete and likely refers to an intelligent agent.] In synchronized team decision-making steps Individual value function at time Mapping back to its previous asynchronous decision step The asynchronous decision step is obtained Individual value function at time ; S43: Based on the individual value function and gradient update formula for individual policy networks parameters Perform gradient updates; the gradient update formula is expressed as: in, For intelligent agents According to strategy From its initial state The generated motion trajectory; The length of the trajectory.

7. The asynchronous multi-agent cooperative decision-making method based on probabilistic decomposition according to claim 6, characterized in that, The gradient update described in S43 introduces off-policy learning; The off-policy learning includes the following steps: a1: A single intelligent agent The trajectory integration of a complete training session is expressed as follows: in, For intelligent agents A complete set of training trajectories. The length of the trajectory; a2: Integrates the trajectories of each agent's complete training run, and its expression is: in, The set of trajectories of all agents during a complete training session; The number of agents; a3: Integrate the trajectories from each complete training session of all agents to obtain the trajectory replay buffer. Its expression is: a4: During each training session, if the agent satisfies the following: in This serves as a buffer for trajectory playback. This is the minimum threshold for the trajectory playback buffer. Then, using the trajectory importance sampling ratio, from the trajectory playback buffer... Sampling is performed in the process, and the samples are incorporated into the gradient update formula to update the policy network; The gradient update formula combining the trajectory importance sampling ratio is: in, For intelligent agents According to strategy From its initial state The generated motion trajectory; For behavioral strategies; The length of the trajectory; For individual value functions; The trajectory importance sampling ratio is expressed as follows: in, For the behavioral strategy, its parameters are: ; The length of the trajectory.

8. The asynchronous multi-agent cooperative decision-making method based on probabilistic decomposition according to claim 6, characterized in that, The gradient update described in S43 introduces a soft update mechanism. The soft update mechanism is divided into a commentator network and an implementer network: The update rules for the commentator's online platform are as follows: Target Network parameters Tracking online individual value networks at a slow pace parameters Its expression is: in, This is a soft update coefficient; The update rule for the executor is as follows: A strategy gradient pruning mechanism is adopted to limit the ratio of change between old and new strategies; The gradient update formula with gradient clipping mechanism is: in, For intelligent agents According to strategy From its initial state The generated motion trajectory; The length of the trajectory; This is the clipping function; For trajectory playback buffer Historical policy network for storage Parameters; Set the pruning threshold for the strategy; It is an individual value function.

9. The asynchronous multi-agent cooperative decision-making method based on probabilistic decomposition according to claim 6, characterized in that, Entropy regularization is introduced during gradient update as described in S43. The entropy regularization is as follows: Adding a weighted term of policy entropy to the gradient update formula, its expression is: in, The regularization coefficient is used. For intelligent agents According to strategy From its initial state The generated motion trajectory; The length of the trajectory; It is an individual value function.

10. An asynchronous multi-agent cooperative decision-making system based on probabilistic decomposition, characterized in that, The method includes at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 9.

Images