Multi-agent control methods, devices, equipment and media for causal experience replay

Abstract

This invention primarily relates to the field of intelligent robot technology. It discloses a multi-agent control method, apparatus, device, and medium for causal experience replay. The method includes: acquiring an experience dataset collected when multiple agents perform a cluster formation task, the experience dataset including a subset of action vectors and a subset of reward values; generating a causal graph based on the subset of action vectors and the subset of reward values; determining the weight value of each action vector based on the adjustment result of each action vector; updating the subset of action vectors according to each weight value, and training a preset control strategy model using the updated experience dataset; and controlling multiple agents to perform a cluster formation task based on a control strategy generated from the trained control strategy model. This application can enhance the deep learning capabilities of intelligent agents and improve the interpretability of agent decisions.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot technology, specifically to a multi-agent control method, device, equipment, and medium for causal experience playback. Background Technology

[0002] A multi-agent system (MAS) is a system composed of multiple intelligent agents. These agents can be physical entities, such as drones, autonomous vehicles, or robots, or virtual entities, such as software agents in a network. The agents are capable of autonomous action and decision-making, and possess abilities such as perception, communication, and learning. Multi-agent swarm control (also known as collective swarm control or cooperative swarm control) allows multiple agents to cooperate and coordinate to complete swarm control tasks on large or complex objects. This technology has broad application prospects in logistics, warehousing, disaster relief, agriculture, and industrial automation. Traditional multi-agent swarm control methods require a large number of samples to learn effective strategies; the utilization rate of sample data is low, and it is difficult to interpret its decision-making process and results, which limits the application of multi-agent swarm control methods. Summary of the Invention

[0003] This invention provides a multi-agent control method, apparatus, device, and medium for causal experience replay, which can enhance the deep learning ability of agents and improve the interpretability of agent decisions.

[0004] This invention provides a multi-agent control method for causal experience replay, the method comprising:

[0005] Acquire an experience dataset collected when multiple intelligent agents perform cluster formation tasks, wherein the experience dataset includes a subset of action vectors and a subset of reward values;

[0006] Based on the action vector subset and the reward value subset, a causal graph is generated, the causal graph including the causal relationship between action vectors in the action vector subset and reward values ​​in the reward value subset;

[0007] A weight value for each action vector is determined based on the result of adjusting each action vector, the weight value being used to represent the degree to which the action vector affects the causal relationship;

[0008] The action vector subset is updated according to each weight value, and the preset control strategy model is trained using the updated experience dataset of the action vector subset.

[0009] The control strategy generated based on the trained control strategy model controls multiple agents to perform cluster formation tasks.

[0010] Optionally, the step of generating a causal graph based on the subset of action vectors and the subset of reward values, wherein the causal graph includes the causal relationship between action vectors in the subset of action vectors and reward values ​​in the subset of reward values, including:

[0011] An initial causal graph is constructed and multiple nodes are generated in the initial causal graph. Each node includes the action vector and the reward value corresponding to the action vector.

[0012] An edge is generated between every two nodes, and the edge serves as a causal relationship between the two nodes.

[0013] Based on the results of detecting the causal relationship corresponding to each edge, each edge is filtered and the type and direction of the filtered edges are determined;

[0014] The weight value of each edge is determined based on its type and direction;

[0015] Based on the comparison of the preset threshold with the weight value of each edge, the preferred edge is determined;

[0016] The initial causal graph is updated based on each of the preferred edges to obtain the updated causal graph.

[0017] Optionally, determining the weight value of each action vector based on the result of adjusting each action vector, the weight value representing the degree of influence of the action vector on the causal relationship, includes:

[0018] Adjust the action vector of the target node among the plurality of nodes;

[0019] Based on the action vectors before and after adjustment, an average treatment effect value is determined, wherein the average treatment effect value is used to represent the degree to which the action vector of the target node affects the reward value of the target node;

[0020] Based on the average treatment effect value, the weight value of the action vector of the target node is determined.

[0021] Optionally, updating the subset of action vectors according to each weight value and training a preset control strategy model using the updated empirical dataset of the action vector subsets includes:

[0022] Based on the weight value of the action vector and the error value of the reward value, the priority of the experience data group is determined, wherein the experience data group consists of experience data including the action vector and the reward value collected in a cluster formation task;

[0023] The control strategy model is trained using each of the empirical data sets, and the sampling probability of each empirical data set being sampled when training the control strategy model is determined based on the priority of each of the empirical data sets.

[0024] When training the control strategy model, the importance sampling weight is determined based on the functional relationship between the amount of cache space storing the experience dataset and the sampling probability, and is used to adjust the sampling probability corresponding to the target experience data group among the multiple experience data groups.

[0025] Optionally, the control policy generated based on the trained control policy model controls multiple agents to perform cluster formation tasks, including:

[0026] When the multiple agents perform cluster formation tasks, multiple action strategies are generated through the local control strategy model corresponding to each agent, wherein each agent executes the action corresponding to the target action strategy among the multiple action strategies;

[0027] Update the agent state and agent action corresponding to each action strategy;

[0028] Generate the current state vector based on each updated agent state, and generate the current action vector based on each updated agent action;

[0029] The current state vector and the current action vector are used as inputs to the trained control strategy model, and the first output value is output through the trained control strategy model.

[0030] The sample state vector and sample action vector in the empirical dataset are used as inputs to the trained control policy model, and the trained control policy model outputs a second output value.

[0031] Based on the loss function relationship between the first output value and the second output value, the parameters of the trained control strategy model are updated;

[0032] A control strategy is generated using the updated control strategy model, and multiple agents are controlled to perform cluster formation tasks according to the control strategy.

[0033] Optionally, the multi-agent control method for causal experience replay further includes the step of generating the subset of action vectors;

[0034] Generating the subset of action vectors includes:

[0035] A time series is generated based on the behavioral data of the agent in the aforementioned experience dataset;

[0036] The time series is divided into multiple subsequences;

[0037] Multiple clusters are generated and each of the subsequences is merged into the target cluster of the multiple clusters to obtain the subsequences after clustering.

[0038] Generate the action vector subset, and use each clustered subsequence as the action vector within the action vector subset.

[0039] Optionally, the behavior vector corresponding to each time point in the time series includes multiple variables;

[0040] The step of dividing the time series into multiple subsequences includes:

[0041] Based on the multiple variables, a precision matrix is ​​generated, where each element in the precision matrix represents the conditional independence relationship between each of the variables.

[0042] The time series is divided into multiple time segments, and multiple cluster centers are generated;

[0043] The similarity relationship between time segments and cluster centers is determined based on the current accuracy matrix;

[0044] Based on the similarity relationship between the time segments and the cluster centers, each time segment is assigned to the cluster corresponding to the target cluster center;

[0045] Each cluster center is used as a subsequence, and the multiple subsequences are used as the result of the clustering process.

[0046] Based on the results of the clustering process, the current precision matrix is ​​updated;

[0047] The updated precision matrix is ​​used as the current precision matrix, and the step of determining the similarity relationship between time segments and cluster centers based on the current precision matrix is ​​returned until the clustering process reaches the preset termination condition, and multiple subsequences are output.

[0048] The present invention also provides a multi-agent control device for causal experience replay, the device comprising:

[0049] The acquisition module is used to acquire the experience dataset collected when multiple intelligent agents perform cluster formation tasks. The experience dataset includes a subset of action vectors and a subset of reward values.

[0050] A generation module is used to generate a causal graph based on the subset of action vectors and the subset of reward values, wherein the causal graph includes the causal relationship between action vectors in the subset of action vectors and reward values ​​in the subset of reward values;

[0051] The calculation module is used to determine the weight value of each action vector based on the result of adjusting each action vector, the weight value being used to represent the degree to which the action vector affects the causal relationship;

[0052] The training module is used to update the subset of action vectors according to each weight value, and to train a preset control strategy model using the updated empirical dataset of the subset of action vectors.

[0053] The control module is used to control multiple agents to perform cluster formation tasks based on the control strategy generated by the trained control strategy model.

[0054] The present invention also provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the multi-agent control method for causal experience replay as described in any of the preceding claims.

[0055] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-agent control method for causal experience replay as described in any of the preceding claims.

[0056] The present invention has at least the following beneficial effects:

[0057] In this technical solution, firstly, action vectors and reward values ​​are extracted from an experience dataset to construct a causal graph to identify the causal relationship between actions and rewards, which serves as the basis for decision-making and enhances the interpretability of the results. Secondly, by assigning weight values ​​to each action vector, the influence of each action on the causal relationship is quantified. This weight allocation allows the model to focus more on actions that are more critical to task success, thereby improving the efficiency of sample data utilization. Then, by updating a subset of action vectors and training the control policy model with the updated dataset, priority is given to training on experiences with significant causal impact, helping the model quickly identify corresponding causal relationships. Finally, the updated subset of action vectors is used to train the control policy model, generating a more effective control policy, thereby optimizing the execution of multi-agent swarm formation tasks and helping to improve the application scope and effectiveness of multi-agent swarm formation control methods. Attached Figure Description

[0058] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0059] Figure 1This is a flowchart of the steps of the multi-agent control method for causal experience replay provided in this embodiment;

[0060] Figure 2 This is a flowchart of step S102 in the multi-agent control method for causal experience replay provided in this embodiment;

[0061] Figure 3 This is a flowchart of step S103 in the multi-agent control method for causal experience replay provided in this embodiment;

[0062] Figure 4 This is a flowchart of step S104 in the multi-agent control method for causal experience replay provided in this embodiment;

[0063] Figure 5 This is a flowchart of step S105 in the multi-agent control method for causal experience replay provided in this embodiment;

[0064] Figure 6 This is another step flowchart of the multi-agent control method for causal experience replay provided in this embodiment;

[0065] Figure 7 This is a flowchart of step S602 in the multi-agent control method for causal experience replay provided in this embodiment;

[0066] Figure 8 This is a flowchart of the steps involved in a multi-agent control method for implementing causal experience replay in an application scenario.

[0067] Figure 9 This is a flowchart illustrating the causal experience replay process of a multi-agent control method for implementing causal experience replay in an application scenario.

[0068] Figure 10 This is a schematic diagram illustrating the first effect of a multi-agent control method for realizing causal experience replay in an application scenario.

[0069] Figure 11 This is a schematic diagram illustrating the second effect of a multi-agent control method for realizing causal experience replay in an application scenario.

[0070] Figure 12 This is a schematic diagram illustrating the third effect of a multi-agent control method for realizing causal experience replay in an application scenario;

[0071] Figure 13 This is a schematic diagram illustrating the fourth effect of a multi-agent control method for realizing causal experience replay in an application scenario;

[0072] Figure 14 This is a schematic diagram of the structure of the multi-agent control device for causal experience replay provided in this embodiment;

[0073] Figure 15 This is a schematic diagram of the structure of the electronic device provided in this embodiment. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0075] Before describing the embodiments of the technical solutions of this application, the technical terms involved in the technical solutions of this application will be explained.

[0076] A multi-agent system (MAS) is a system composed of multiple intelligent agents capable of autonomous action and decision-making, and possessing abilities such as perception, communication, and learning. Cluster formation control (also known as collective formation control or cooperative formation control) refers to the task of controlling large or complex objects through mutual cooperation and collaboration among multiple intelligent agents. In cluster formation control tasks, each intelligent agent is responsible for different formation control positions and operations, and together they complete the formation control task through cooperation and information sharing.

[0077] In related technical fields, an architecture called Deep Implicit Coordination Graphs (DICG) has been proposed to address the coordination problem in Multi-agent Reinforcement Learning (MARL). By implicitly learning the coordination relationships between agents, it improves the efficiency and effectiveness of multi-agent tasks. However, due to the high model complexity and implicit coordination nature of DICG, the method has poor interpretability, making it difficult to intuitively understand how the model makes specific decisions. DICG does not distinguish between inputs and learns both non-causal and causal factors in the input, which can affect the overall model performance and may lead to insufficient generalization ability in different scenarios. In addition, since the policies of each agent are constantly changing, DICG may require more training steps to achieve stable results, which may be a disadvantage in resource-constrained scenarios.

[0078] The researchers in this application found that DICG learns the coordination relationships between agents implicitly, rather than through explicit rules or constraints. While this implicit learning method can capture complex interactions between agents, it also makes the model's internal workings more difficult to interpret. Furthermore, DICG considers both causal and non-causal factors at input, without reconstructing an invariant causal mechanism. This unprocessed data may negatively impact the control model for swarm formation control, leading to performance degradation. Additionally, DICG introduces an implicit coordination graph, which allows the model to dynamically adjust coordination strategies between agents. However, this also increases the algorithm's complexity, especially when dynamically adjusting multi-agent interactions. This complexity increases the uncertainty of the training process, making it difficult for the model to converge stably within a reasonable timeframe.

[0079] To address the shortcomings of existing technical solutions, this application proposes a multi-agent control method, apparatus, device, and medium for causal experience replay. First, by extracting action vectors and reward values ​​from the experience dataset, a causal graph is constructed to identify the causal relationship between actions and rewards, serving as the basis for decision-making and enhancing the interpretability of the results. Second, by assigning weight values ​​to each action vector, the influence of each action on the causal relationship is quantified. This weight allocation allows the model to focus more on actions that are more critical to task success, thereby improving the efficiency of sample data utilization. Then, by updating a subset of action vectors and training the control policy model with the updated dataset, training prioritizes experiences with significant causal impact, helping the model quickly identify corresponding causal relationships. Finally, the updated subset of action vectors is used to train the control policy model, generating a more effective control policy, thereby optimizing the execution of multi-agent swarm formation tasks and improving the application scope and effectiveness of multi-agent swarm formation control methods. Embodiments provided by this application are shown below.

[0080] Please refer to Figure 1 , Figure 1 This is a flowchart of the steps in a multi-agent control method that uses causal experience replay.

[0081] This embodiment provides a multi-agent control method for causal experience replay, the method comprising:

[0082] S101. Obtain the experience dataset collected when multiple agents perform cluster formation tasks. The experience dataset includes a subset of action vectors and a subset of reward values.

[0083] S102. Based on the action vector subset and the reward value subset, generate a causal graph. The causal graph includes the causal relationship between the action vectors in the action vector subset and the reward values ​​in the reward value subset.

[0084] S103. Determine the weight value of each action vector based on the adjustment results of each action vector. The weight value is used to represent the degree to which the action vector affects the causal relationship.

[0085] S104. Update the action vector subset according to each weight value, and use the updated empirical dataset of the action vector subset to train the preset control strategy model.

[0086] S105. A control strategy generated based on the trained control strategy model controls multiple agents to perform cluster formation tasks.

[0087] In step S102 of some embodiments, the causal graph is constructed using the PC algorithm (Pearl's Causal Inference), or the Partial Ancestor Graph (PAG) is generated using the RFCI (Really Fast Causal Inference) algorithm.

[0088] In some embodiments, an experience dataset is stored in an experience cache, which can be used to store the states of all agents at different times. Specifically, for agents... The experience cache stores all the agent's experience quintuples:

[0089]

[0090] in, For intelligent agents in The corresponding status, action, and reward at any given moment. for The state corresponding to each moment. for The status termination signal at a given moment indicates whether the formation has reached its target point.

[0091] During training, each agent relies on experience. and The loss is calculated by training with a batch of experiences from the experience buffer through priority experience replay, which includes data before and after the intervention.

[0092] Please refer to Figure 2 , Figure 2 This is a flowchart of step S102 in the multi-agent control method of causal experience replay.

[0093] In some embodiments, step S102 includes:

[0094] S201. Construct an initial causal graph and generate multiple nodes in the initial causal graph. Each node includes an action vector and the reward value corresponding to the action vector.

[0095] S202. Generate an edge between every two nodes, with the edge serving as the causal relationship between the two nodes.

[0096] S203. Based on the results of detecting the causal relationship corresponding to each edge, each edge is filtered and the type and direction of the filtered edges are determined.

[0097] S204. Determine the weight value of each edge based on its type and direction.

[0098] S205. Based on the results of comparing the preset threshold with the weight value of each edge, the preferred edge is determined.

[0099] S206. Update the initial causal graph according to each selected edge to obtain the updated causal graph.

[0100] Understandably, in this embodiment, an observation dataset containing all representative subsequences is first constructed, denoted as . in It is the first Each action vector This is the reward value corresponding to the action vector. Starting with all variables in the dataset, construct a completely undirected graph, where each node represents a variable in the observed dataset D.

[0101] The conditional independence test is used to check whether a direct causal relationship exists between two nodes. If in the condition set... The next two variables and If they are independent, then there is no direct causal relationship between them. The functional expression is as follows:

[0102]

[0103] if and Conditions independent of Remove and The edges between them.

[0104] After completing all necessary conditional independence tests and removing irrelevant edges, RFCI determines the direction of the remaining edges using v-structure detection: if nodes exist... ,satisfy and and Conditions independent of This forms a v-structure, representing yes and Common descendants:

[0105]

[0106] In the v-structure, The tip, referred to as the V-structure, represents the simultaneous influence from... and The impact.

[0107] Finally, a partial ancestor graph can be generated, with the edge types and directions as follows:

[0108] : To establish a causal relationship, indicating yes The direct cause.

[0109] : This indicates the presence of unobserved confounding factors, leading to and There is a connection between them.

[0110] The direction of causality is uncertain, indicating that... may be The reason is unknown, but it cannot be fully determined yet.

[0111] If there is uncertainty regarding the direction of causality, for example If the sign is uncertain, then the indeterminate sign is retained. Further causal inferences may be made based on more data to determine the final causal direction.

[0112] The pruning phase is performed after the initial partial ancestor graph is generated. Its purpose is to reduce the number of edges in the graph so that the final causal graph contains the most significant causal relationships, rather than all possible associations.

[0113] The importance of an edge is evaluated by its average treatment effect (ATE). Edges with larger ATEs indicate that they have a significant impact on the reward and should be retained.

[0114]

[0115] For each edge in the causal graph, according to The value of is used to classify edges into different categories, such as: high importance edges (significant causality), medium importance edges, and low importance edges (weak causality).

[0116] Define a threshold This determines whether to retain an edge. The decision is based on experience, data size, and model requirements. If the edge weight If so, then delete that edge:

[0117]

[0118] Importance below the threshold The edges are usually caused by noise or weakly correlated factors and do not make a significant contribution to causality.

[0119] To prevent overly aggressive pruning of the causal graph at different stages, a dynamically adjusted threshold can be used during the pruning process. For example, a lower threshold can be set in the early stages of training. This is to preserve more potential causal edges; as training progresses, the threshold is gradually increased to... This makes the cause-effect diagram simpler and more focused on the most important relationships.

[0120] Please refer to Figure 3 , Figure 3 This is a flowchart of step S103 in the multi-agent control method of causal experience replay.

[0121] In some embodiments, step S103 includes:

[0122] S301, Adjust the action vector of the target node among multiple nodes.

[0123] S302. Based on the action vectors before and after adjustment, determine the average treatment effect value, whereby the average treatment effect value is used to represent the degree to which the action vector of the target node affects the reward value of the target node.

[0124] S303. Based on the average treatment effect value, determine the weight value of the action vector of the target node.

[0125] It is understood that this embodiment employs a causal intervention mechanism to measure the degree of influence of causal variables in the causal graph on rewards, directly controlling a specific causal variable rather than passively observing it. In this embodiment, a specific causal variable in the causal graph... Exert control and change its value To evaluate the direct causal effect of this variable, the specific expression is as follows:

[0126]

[0127] Then, the impact of the intervention on the causal variable of reward was assessed using the average treatment effect size (ATE). Its ATE is as follows,

[0128]

[0129] in, and These are the rewards before and after the intervention. The larger the value, the more significant the impact of the intervention on the reward, indicating that the action vector has an important causal effect on the policy.

[0130] Please refer to Figure 4 , Figure 4 This is a flowchart of step S104 in the multi-agent control method of causal experience replay.

[0131] In some embodiments, step S104 includes:

[0132] S401. Based on the error values ​​of the weight values ​​and reward values ​​of the action vectors, determine the priority of the experience data group, wherein the experience data group consists of experience data including action vectors and reward values ​​collected in a cluster formation task.

[0133] S402. Train the control strategy model using each set of empirical data, and determine the sampling probability of each set of empirical data being sampled when training the control strategy model based on the priority of each set of empirical data.

[0134] S403. When training the control strategy model, the importance sampling weight is determined based on the functional relationship between the amount of cache space for storing the experience dataset and the sampling probability, and is used to adjust the sampling probability corresponding to the target experience dataset in multiple experience datasets.

[0135] Understandably, in order to prioritize learning experiences with strong causal influence during the training of the control policy model, this embodiment employs a mechanism of prioritizing experience replay. First, the priority of the experience data set is determined, wherein... Greater experience sets higher priorities, while also taking into account... Error, for experience Its priority for:

[0136]

[0137] in, For experience of Error and average treatment effect It is a hyperparameter used to balance the effects of TD error and individual treatment effects. It is a small positive number to avoid a priority of zero. This determines the sampling probability of each empirical data set being sampled when training the control policy model, calculated as follows:

[0138]

[0139] in, The impact of control priority on sampling probability.

[0140] Because priority experience playback involves non-uniform sampling, some experiences may have a higher sampling probability. To correct the bias caused by this non-uniform sampling, an importance sampling weight is introduced. The following formula is used to determine the importance sampling weight:

[0141]

[0142] in, It is the size of the experience buffer. It is experience The sampling probability, It is a hyperparameter that controls the degree of importance sampling. It is usually gradually increased from a small value to 1 to ensure that the correction of bias becomes more and more sufficient as the training process progresses.

[0143] It is understandable that a causal intervention mechanism is adopted, and based on the results of the causal intervention, experience fragments with strong causal influence are given higher weights to improve the model's learning efficiency for important causal factors. Then, a priority experience replay mechanism is adopted to improve the training efficiency of deep reinforcement learning by prioritizing the replay of experiences with strong causal relationships with reward results.

[0144] Please refer to Figure 5 , Figure 5 This is a flowchart of step S105 in the multi-agent control method of causal experience replay.

[0145] In some embodiments, step S105 includes:

[0146] S501. When multiple agents perform cluster formation tasks, multiple action policies are generated through the local control policy model corresponding to each agent. Each agent executes the action corresponding to the target action policy among the multiple action policies.

[0147] S502. Update the agent state and agent action corresponding to each action strategy.

[0148] S503. Generate the current state vector based on each updated agent state, and generate the current action vector based on each updated agent action.

[0149] S504. Use the current state vector and the current action vector as inputs to the trained control policy model, and output the first output value through the trained control policy model.

[0150] S505. Use the sample state vector and sample action vector in the experience dataset as input to the trained control policy model, and output the second output value through the trained control policy model.

[0151] S506. Update the parameters of the trained control strategy model based on the loss function relationship between the first output value and the second output value.

[0152] S507. Generate a control strategy using the updated control strategy model and control multiple agents to perform cluster formation tasks according to the control strategy.

[0153] In some embodiments, a MADDPG-based control policy model is used to control a multi-agent formation. In this MADDPG-based control policy model, Critic and Actor networks are constructed. For the Critic network, each agent has its own online evaluation network. and target evaluation network For each actor, there is their own online policy network. and target policy network By employing a centralized training method, when calculating the Critic's forward propagation, the states of all agents are concatenated into a state vector. Concatenate the actions of all agents into action vectors. ;Will As input to an online evaluation network, the output is one-dimensional. Value, that is By using global information about agents in the environment to "centralize" the training of its own evaluation network, and then using samples from the experience buffer, it can obtain... Next, we construct a loss function and approximate the optimal Q value using temporal difference (TD). Its expression is as follows:

[0154]

[0155] in, Here, a target evaluation network is used to calculate the action to be taken by the agent in the next state. It should be noted that the input of the target evaluation network for each agent only contains the local state information of the agent itself.

[0156] Meanwhile, to correct the bias caused by non-uniform sampling, importance sampling weights are introduced. The total loss function is obtained as follows:

[0157]

[0158] Update parameters using gradient descent. :

[0159]

[0160] Each agent generates its own action policy using a local control policy model based on the DDPG algorithm. It constructs its own Critic and Actor networks, with the input being the global state information obtained by the entire formation system from the environment. The output of the Critic is the Q-value of the state-action pair, and the output of the Actor is the deterministic action of the agent.

[0161] When computing the forward propagation of its own Actor, each agent only uses its own local observation vector. As input to the online policy network, it outputs a deterministic action. Consistent with Critic, importance sampling weights are added. The loss function is solved and its gradient with respect to the parameters is calculated. Then, the parameters are updated using gradient descent. The loss function and gradient are as follows:

[0162]

[0163]

[0164] It is understood that this embodiment implements a centralized training and distributed execution approach, and shares global information during training, enabling the agent to acquire more data from other agents during training, thereby making better decisions in an environment where other agents exist, and effectively reducing instability caused by mutual influence between agents.

[0165] Please refer to Figure 6 , Figure 6 This is another step in the multi-agent control method of causal experience replay.

[0166] In some embodiments, the specific implementation of generating a subset of action vectors is achieved through the following steps:

[0167] S601. Generate time series data based on the behavioral data of agents in the empirical dataset.

[0168] S602. Divide the time series into multiple subsequences.

[0169] S603. Generate multiple clusters and merge each subsequence into the target cluster to obtain the clustered subsequence.

[0170] S604. Generate a subset of action vectors, and use each clustered subsequence as the action vector within the subset of action vectors.

[0171] In some embodiments, the behavior vector corresponding to each time point in the time series includes multiple variables.

[0172] In one specific embodiment, the behavioral data within the experience dataset is first constructed into a multidimensional time series, denoted as . ,in Indicates the first The time series is divided into behavior vectors at each time step. Then, the time series is divided into subsequences with conditionally independent structures, and the K-means clustering algorithm is used to further integrate the subsequences initially divided by TICC in order to more effectively represent and utilize the information of these subsequences in causal experience replay.

[0173] Understandably, after the initial segmentation, a large number of subsequences may be obtained. To simplify the representation of these fragments, the K-means algorithm is further used to cluster these subsequences into a few representative fragments, which will serve as nodes in the causal graph.

[0174] Methods for integrating subsequences using the K-means clustering algorithm include:

[0175] First select Initial cluster centers Then, for each subsequence segment Find the nearest cluster center ,

[0176]

[0177] After all subsequences have been assigned, the mean of each cluster is calculated based on the currently assigned clusters and used as the new cluster centers.

[0178]

[0179] The clustering process stops when the change in cluster centers is less than the threshold or when the maximum number of iterations is reached.

[0180] Finally, the K-means clustering algorithm further clusters the initially segmented subsequences into a few representative sequences. These sequences will serve as core elements in causal inference, helping the model to more efficiently capture and analyze causal relationships in time series during causal inference.

[0181] Please refer to Figure 7 , Figure 7 This is a flowchart of step S102 in the multi-agent control method of causal experience replay.

[0182] In some embodiments, step S602 includes:

[0183] S701. Based on multiple variables, generate a precision matrix, where each element in the precision matrix represents the conditional independence relationship between each variable.

[0184] S702. Divide the time series into multiple time segments and generate multiple cluster centers.

[0185] S703. Determine the similarity relationship between time segments and cluster centers based on the current precision matrix.

[0186] S704. Based on the similarity relationship between time segments and cluster centers, assign each time segment to the cluster corresponding to the target cluster center.

[0187] S705. Take the cluster corresponding to each cluster center as a subsequence, and take multiple subsequences as the result of the clustering process.

[0188] S706. Update the current precision matrix based on the results of the clustering process.

[0189] S707. Use the updated precision matrix as the current precision matrix, return to execute the step of determining the similarity relationship between time segments and cluster centers based on the current precision matrix, until the clustering process reaches the preset termination condition, and output multiple subsequences.

[0190] In some embodiments, steps S701-S707 above are implemented using TICC. Specifically, a precision matrix is ​​used to represent the conditional independence relationships in time series segments, thereby discovering different behavioral patterns. The elements inside are:

[0191]

[0192] in, The first precision matrix represents the... Element. If , representing variables and They are independent given other variables.

[0193] Meanwhile, TICC uses the Expectation-Maximization (EM) algorithm to segment the time series, with the specific steps as follows:

[0194] E-step: The current value of the fixed-precision matrix, assigning each time segment to the nearest cluster based on similarity.

[0195] M-step: With the clustering assignments fixed, update the precision matrix to maximize the log-likelihood, while using regularization to ensure the sparsity of the precision matrix:

[0196]

[0197] in, It is the covariance matrix. This is a regularization parameter used to control sparsity. After several E-steps and M-steps, TICC divides the time series into multiple subsequences, each subsequence corresponding to a cluster.

[0198] Understandably, the above embodiments combine action sequences within the experience dataset into multivariate time series, segment them, and construct causal graphs to explore the causal relationship between different actions of the agent and rewards, so that the model ignores actions unrelated to rewards and enhances the interpretability of the model results.

[0199] Please see Figure 8 , Figure 8 This is a flowchart illustrating the steps of a multi-agent control method for implementing causal experience replay in an application scenario.

[0200] In this application scenario, the control strategy model for the multi-agent control method implementing causal experience replay is a MADDPG-based control strategy model. First, the agents, evaluation system, and policy network are initialized. The agent is the entity that performs actions; the evaluation system assesses the agent's performance; and the policy network generates the agent's actions. Therefore, the state space, action space, and policy function of each agent need to be defined for constructing the Critic and Actor networks in MADDPG. For the agents... Its state space It includes its own state information and the perception information of the local environment. The specific expression is as follows.

[0201]

[0202] in,( () represents the agent's current position in the environment. The current velocity of the agent. The direction of movement of the intelligent agent. The information is obtained by the intelligent agent based on the environmental information perceived by the camera, through a feature extraction network.

[0203] Action space represents the set of actions that an intelligent agent can perform, and its specific expression is as follows.

[0204]

[0205] in, These represent adjustments to speed and direction, respectively.

[0206] The policy function is used to determine the current state. Choose the appropriate action It optimizes through training to maximize long-term rewards, and the specific expression is as follows.

[0207]

[0208] in For intelligent agents The strategy parameters are optimized independently for each agent.

[0209] During the training of multi-agent formation control, a reward function needs to be defined to satisfy the following factors: obstacle avoidance, approaching the target point, maintaining a safe distance from other agents, maintaining formation control, and maintaining speed consistency. Actions that meet these factors will be rewarded; actions that do not will be penalized.

[0210] Obstacle avoidance rewards:

[0211] This portion of the reward is designed to ensure that agents can avoid obstacles and maintain a safe distance from other agents.

[0212]

[0213] in, It is the importance weight of obstacle avoidance. It is an intelligent agent Distance to obstacles and other intelligent agents, It is the safe distance threshold, the minimum distance that an agent must maintain from obstacles and other agents. When a collision occurs, the agent will be given an instantaneous penalty.

[0214] Rewards for getting close to the target location:

[0215] During formation control, it is necessary to calculate the distance to the target point to ensure that the entire cluster can complete the formation control task.

[0216]

[0217] in, It is a weighting coefficient based on proximity to the target location. The center position of the formation, The target location for the formation control mission. An instant reward will be given upon reaching the target location.

[0218] Formation maintenance reward:

[0219] intelligent agent The distance between the agent and other agents should be close to the preset ideal spacing. To ensure the stability of the formation control configuration:

[0220]

[0221] in, Maintaining formation control is of paramount importance. It is an intelligent agent and The actual distance between them It is an intelligent agent and The ideal spacing in the formation.

[0222] Speed ​​Consistency Reward:

[0223] To avoid breaking the formation, the agents need to maintain consistent speed. This reward is used to ensure that the agents... The speed matches that of other intelligent agents:

[0224]

[0225] in, It is the importance weight of speed consistency. and They are intelligent agents and The speed.

[0226] By combining all the rewards, an intelligent agent can be obtained. reward function for

[0227]

[0228] Then, an experience cache is set up to store the states of all agents at different times, and the experience dataset is saved. The experience dataset is used for training the MADDPG network.

[0229] In practice, the agent generates and executes actions based on the current policy network. The process includes: updating the state and termination signal d for the next time step: updating the environment state based on the agent's actions and generating a termination signal d to determine if the termination condition has been met. If d satisfies the termination condition: checking if the termination signal d satisfies the preset termination condition. If it does, the process ends; otherwise, execution continues. Retrieving and segmenting the time series data from the experience buffer: retrieving time series data from the buffer storing the agent's experience and segmenting it as needed for subsequent analysis. Generating a causal graph based on the segmented time series: constructing a causal graph using the segmented time series data to identify causal relationships between variables. Intervening in the causal variables within the causal graph and calculating the average treatment effect (ATE): intervening in the identified causal variables within the causal graph and calculating the average treatment effect (ATE) before and after the intervention to assess the variable's impact on the outcome. Prioritizing experience replay sampling based on TD error and ATE, and updating the network: using temporal difference (TD) error and ATE to guide experience replay, prioritizing the replay of experiences most helpful for learning to update the policy network. End: the end of the process, indicating the completion of the agent's learning process.

[0230] Please see Figure 9 , Figure 9 This is a flowchart of the causal experience replay process for a multi-agent control method that implements causal experience replay in an application scenario.

[0231] like Figure 9 As shown, the training process of the causal reinforcement control policy model includes: defining the environmental states of the agent, which the agent can perceive; the agent performs actions and interacts with the environment, which leads to changes in state; the agent determines its actions according to the policy network; the policy network is a learning model that outputs the optimal action based on the current state; storing the agent's experience of interacting with the environment, including state, action, reward, and next state; extracting data from the experience buffer to generate a causal graph to analyze the causal relationships between different states and actions; based on the causal graph, intervening in certain variables to evaluate their impact on the outcome; an evaluation network is used to evaluate the quality of the agent's actions in specific states, helping the agent learn better policies; and prioritizing the most helpful experiences for learning based on TD error and ATE (average treatment effect) to replay in order to update the policy network.

[0232] Please see Figures 10 to 13 In this embodiment, the effect of multiple agents performing formation control tasks is as follows: Figures 10 to 13 As shown.

[0233] Please refer to Figure 14 , Figure 14This is a schematic diagram of the structure of a multi-agent control device that replays causal experience.

[0234] This embodiment also provides a multi-agent control device for causal experience playback, the device comprising:

[0235] The acquisition module 801 is used to acquire the experience dataset collected when multiple agents perform cluster formation tasks. The experience dataset includes a subset of action vectors and a subset of reward values.

[0236] The generation module 802 is used to generate a causal graph based on a subset of action vectors and a subset of reward values. The causal graph includes the causal relationship between action vectors in the subset of action vectors and reward values ​​in the subset of reward values.

[0237] The calculation module 803 is used to determine the weight value of each action vector based on the result of adjusting each action vector. The weight value is used to represent the degree to which the action vector affects the causal relationship.

[0238] The training module 804 is used to update the action vector subset according to each weight value, and to train the preset control strategy model using the updated empirical dataset of the action vector subset.

[0239] The control module 805 is used to control multiple agents to perform cluster formation tasks based on the control strategy generated by the trained control strategy model.

[0240] It will be understood by those skilled in the art that all or some of the steps and apparatuses in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. As is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0241] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0242] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement any of the above-mentioned causal experience playback multi-agent control methods.

[0243] refer to Figure 15 , Figure 15 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0244] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0245] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store operating devices and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is invoked by the processor 901 to execute the multi-agent control method for causal experience playback of the embodiments of this application.

[0246] The input / output interface 903 is used to implement information input and output;

[0247] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0248] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0249] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0250] It is understood that the content of the above method embodiments is applicable to the embodiments of this electronic device. The specific functions implemented by the embodiments of this electronic device are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0251] This application also provides a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to implement the multi-agent control method for causal experience replay as described in any of the above specific embodiments.

[0252] This application also provides a computer program product, including a computer program or computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium and executes the computer program or computer instructions, causing the computer device to perform the multi-agent control method of causal experience replay as described in any of the preceding embodiments.

[0253] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0254] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices. It should be understood that in this application, “at least one” means one or more, and “more than one” means two or more.

[0255] In the several embodiments provided in this application, it should be understood that the disclosed apparatus, devices, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0256] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0257] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0258] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0259] Although the description of this application has been quite detailed and particularly focused on several of the described embodiments, it is not intended to limit itself to any of these details or embodiments or any particular embodiment. Rather, it should be considered as effectively covering the intended scope of this application by referring to the appended claims and taking into account the prior art, which provides for a broad possible interpretation of these claims. Furthermore, the foregoing description of this application with respect to embodiments foreseeable by the inventors is intended to provide a useful description, and non-substantial modifications to this application that have not yet been foreseen may still represent equivalent modifications.

Claims

1. A multi-agent control method for causal experience replay, characterized in that, The method includes: Acquire an experience dataset collected when multiple intelligent agents perform cluster formation tasks, wherein the experience dataset includes a subset of action vectors and a subset of reward values; Based on the action vector subset and the reward value subset, a causal graph is generated, the causal graph including the causal relationship between action vectors in the action vector subset and reward values ​​in the reward value subset; The action vectors of target nodes in multiple nodes of the causal graph are adjusted, wherein each node includes the action vector and the reward value corresponding to the action vector; Based on the action vectors before and after adjustment, an average treatment effect value is determined, wherein the average treatment effect value is used to represent the degree to which the action vector of the target node affects the reward value of the target node; Based on the average treatment effect value, the weight value of the action vector of the target node is determined; Based on the weight value of the action vector and the error value of the reward value, the priority of the experience data group is determined, wherein the experience data group consists of experience data including the action vector and the reward value collected in a cluster formation task; The preset control strategy model is trained using each of the empirical data sets, and the sampling probability of each empirical data set being sampled when training the control strategy model is determined based on the priority of each empirical data set. When training the control strategy model, the importance sampling weight is determined based on the functional relationship between the amount of cache space storing the experience dataset and the sampling probability, and is used to adjust the sampling probability corresponding to the target experience data group among multiple experience data groups. The control strategy generated based on the trained control strategy model controls multiple agents to perform cluster formation tasks.

2. The multi-agent control method for causal experience replay according to claim 1, characterized in that, The step of generating a causal graph based on the action vector subset and the reward value subset, wherein the causal graph includes the causal relationship between action vectors in the action vector subset and reward values ​​in the reward value subset, including: Construct an initial causal graph and generate multiple nodes in the initial causal graph; An edge is generated between every two nodes, and the edge serves as a causal relationship between the two nodes. Based on the results of detecting the causal relationship corresponding to each edge, each edge is filtered and the type and direction of the filtered edges are determined; The weight value of each edge is determined based on its type and direction; Based on the comparison of the preset threshold with the weight value of each edge, the preferred edge is determined; The initial causal graph is updated based on each of the preferred edges to obtain the updated causal graph.

3. The multi-agent control method for causal experience replay according to claim 1, characterized in that, The control policy generated based on the trained control policy model controls multiple agents to perform cluster formation tasks, including: When the multiple agents perform cluster formation tasks, multiple action strategies are generated through the local control strategy model corresponding to each agent, wherein each agent executes the action corresponding to the target action strategy among the multiple action strategies; Update the agent state and agent action corresponding to each action strategy; Generate the current state vector based on each updated agent state, and generate the current action vector based on each updated agent action; The current state vector and the current action vector are used as inputs to the trained control strategy model, and the first output value is output through the trained control strategy model. The sample state vector and sample action vector in the empirical dataset are used as inputs to the trained control policy model, and the trained control policy model outputs a second output value. Based on the loss function relationship between the first output value and the second output value, the parameters of the trained control strategy model are updated; A control strategy is generated using the updated control strategy model, and multiple agents are controlled to perform cluster formation tasks according to the control strategy.

4. The multi-agent control method for causal experience replay according to claim 1, characterized in that, The method further includes the step of generating the subset of action vectors; Generating the subset of action vectors includes: A time series is generated based on the behavioral data of the agent in the aforementioned experience dataset; The time series is divided into multiple subsequences; Multiple clusters are generated and each of the subsequences is merged into the target cluster of the multiple clusters to obtain the subsequences after clustering. Generate the action vector subset, and use each clustered subsequence as the action vector within the action vector subset.

5. The multi-agent control method for causal experience replay according to claim 4, characterized in that, The behavior vector corresponding to each time point in the time series includes multiple variables; The step of dividing the time series into multiple subsequences includes: Based on the multiple variables, a precision matrix is ​​generated, where each element in the precision matrix represents the conditional independence relationship between each of the variables. The time series is divided into multiple time segments, and multiple cluster centers are generated; The similarity relationship between time segments and cluster centers is determined based on the current accuracy matrix; Based on the similarity relationship between the time segments and the cluster centers, each time segment is assigned to the cluster corresponding to the target cluster center; Each cluster center is used as a subsequence, and the multiple subsequences are used as the result of the clustering process. Based on the results of the clustering process, the current precision matrix is ​​updated; The updated precision matrix is ​​used as the current precision matrix, and the step of determining the similarity relationship between time segments and cluster centers based on the current precision matrix is ​​returned until the clustering process reaches the preset termination condition, and multiple subsequences are output.

6. A multi-agent control device for causal experience replay, characterized in that, The device includes: The acquisition module is used to acquire the experience dataset collected when multiple intelligent agents perform cluster formation tasks. The experience dataset includes a subset of action vectors and a subset of reward values. A generation module is used to generate a causal graph based on the subset of action vectors and the subset of reward values, wherein the causal graph includes the causal relationship between action vectors in the subset of action vectors and reward values ​​in the subset of reward values; The calculation module is used to adjust the action vector of the target node among multiple nodes in the causal graph, wherein each node includes the action vector and the reward value corresponding to the action vector; Based on the action vectors before and after adjustment, an average treatment effect value is determined, wherein the average treatment effect value is used to represent the degree to which the action vector of the target node affects the reward value of the target node; Based on the average treatment effect value, the weight value of the action vector of the target node is determined; The training module is used to determine the priority of the experience data set based on the error value of the weight value of the action vector and the reward value, wherein the experience data set consists of experience data including the action vector and the reward value collected in a cluster formation task; The preset control strategy model is trained using each of the empirical data sets, and the sampling probability of each empirical data set being sampled when training the control strategy model is determined based on the priority of each empirical data set. When training the control strategy model, the importance sampling weight is determined based on the functional relationship between the amount of cache space storing the experience dataset and the sampling probability, and is used to adjust the sampling probability corresponding to the target experience data group among multiple experience data groups. The control module is used to control multiple agents to perform cluster formation tasks based on the control strategy generated by the trained control strategy model.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the multi-agent control method for causal experience playback as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the multi-agent control method for causal experience replay as described in any one of claims 1 to 5.

Images