Reinforcement learning method based on causal reasoning and hierarchical attention mechanism

Abstract

The invention relates to the field of multi-agent reinforcement learning, and discloses a reinforcement learning method based on causal reasoning and a hierarchical attention mechanism, which makes up the defects of inaccurate role allocation and low cooperation efficiency in a traditional multi-agent system and improves the overall cooperation performance. According to the method, a causal perception multi-agent cooperation model is constructed, firstly, a dynamic causal graph is constructed by using a structural causal model, and a causal influence vector between agents is calculated by using an optimized variational distribution estimator; then generating a causal-guided hierarchical attention weight based on a causal influence vector; then cooperative information transmission is realized through a cross-agent attention sharing mechanism, and enhanced feature representation is generated by using a cross-layer fusion mechanism; a causal perception role selector is further designed based on the global causal contribution degree and the local causal contribution degree, and dynamic allocation of agent roles is achieved; and finally, performing model training optimization through a causal influence internal reward mechanism.

Description

Technical Field

[0001] This invention relates to the field of collaborative multi-agent reinforcement learning technology, specifically to a collaborative multi-agent reinforcement learning method based on causal reasoning and hierarchical attention mechanisms. Background Technology

[0002] Collaborative multi-agent reinforcement learning is a machine learning method in which multiple agents learn through interaction in a shared environment to complete complex tasks. It has been widely applied in intelligent fields such as autonomous driving, drone coordination, and game AI. In collaborative multi-agent systems, agents need to maximize the overall benefits of the system through role division and coordination. This requires different agents to assume different functional roles to achieve effective teamwork.

[0003] Existing collaborative multi-agent reinforcement learning methods are predominantly based on a centralized training distributed execution (CTDE) architecture. These methods utilize global information for policy learning during the training phase, while each agent makes independent decisions based on local observations during the execution phase. When agents can form clear role divisions, specialized division of labor can significantly improve collaborative efficiency and task completion quality. However, when agents lack an effective role allocation mechanism, functional role degradation can easily occur, meaning all agents converge to similar policy spaces, losing their diversity advantage and severely limiting the system's collaborative effectiveness.

[0004] Existing methods for solving the role allocation problem can be broadly categorized into three types:

[0005] 1. Static assignment method based on predefined roles: Role types are predefined through manual design or heuristic rules, and then agents are assigned to fixed roles. This method is simple to implement, but lacks flexibility and cannot adapt to dynamic changes in the environment and the complexity of tasks.

[0006] 2. Dynamic Role Discovery Method Based on Behavior Clustering: This method dynamically discovers and assigns roles by analyzing the historical behavioral patterns of agents. While this approach offers some adaptability, it neglects the causal relationship between agent behavior and environmental states, making it difficult to accurately assess the effectiveness of role assignment.

[0007] 3. Hierarchical Role Coordination Approach: This approach employs a hierarchical decision-making architecture, assigning roles through high-level strategies and executing specific actions through low-level strategies. While capable of handling complex collaborative tasks, this method lacks explicit modeling of causal dependencies between agents and an effective intrinsic incentive mechanism to promote role exploration.

[0008] However, existing methods have the following key problems: (1) Lack of causal relationship modeling: There is a lack of explicit modeling of the causal dependency between agent behavior and system state, which makes it impossible to accurately quantify the real causal contribution of each agent to the global goal, resulting in a lack of theoretical basis for role allocation; (2) Limitations of static allocation mechanism: The role allocation strategy cannot be adaptively adjusted according to the dynamic changes of the environment and the real-time changes of causal influence, which lacks flexibility; (3) Insufficient incentive for role exploration: There is a lack of an intrinsic incentive mechanism based on causal reasoning to promote agents to actively explore diverse roles, which leads to agents passively accepting role allocation, easily converging to a policy space with similar functions, resulting in functional role degradation, and ultimately limiting the collaborative efficiency of multi-agent systems. Summary of the Invention

[0009] The technical problem to be solved by this invention is to propose a multi-agent role allocation method based on causal reasoning, which makes up for the lack of explicit modeling of the causal relationship between agent behavior and system state in existing methods, solves the problems of functional role degradation and insufficient role exploration incentives, and enables multi-agent systems to achieve more effective collaborative learning.

[0010] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0011] A reinforcement learning method based on causal reasoning and hierarchical attention mechanisms includes the following steps:

[0012] First, a multi-agent collaborative model for causal perception is constructed;

[0013] Step 1: Based on the interaction data observed by the multi-agent system, construct a dynamic causal graph through a structural causal model and an optimized variational distribution estimator, calculate the causal influence vector between agents, and perform dynamic causal graph reconstruction to adjust the strength of causal relationships;

[0014] Step 2: Decompose the agent's observation space into three levels: local, global, and temporal. Extract feature representations for each level and generate causal-guided hierarchical attention weights based on causal influence vectors. Weighted fusion of the agent's own attention with that of other agents is performed, followed by feature fusion within and between levels to obtain enhanced observation information.

[0015] Step 3: Calculate the global causal contribution of the system and the local causal contribution of the agent based on the reward for the action performed;

[0016] Step 4: Based on the observation space and dynamic causal graph, the behavior embedding is obtained by fusing them. Then, the enhanced observation information, behavior embedding, and causal influence vector are fused to obtain the role embedding.

[0017] Step 5: Dynamically assign roles based on global and local causal contributions, and adjust the allocation strategy through a threshold adaptive mechanism;

[0018] Step 6: Design a causal incentive mechanism to train the model;

[0019] In actual task execution, the multi-agent collaborative model dynamically adjusts the allocation of agent roles based on the current environmental state.

[0020] Furthermore, step 1 is detailed as follows:

[0021] Step 11: Construct a dynamic cause-effect graph; the dynamic cause-effect graph G of the multi-agent system at time step t. t Specifically, it includes:

[0022]

[0023] Where n represents the number of agents. Let t be the state observed by the nth agent at time step t. This refers to the action of the nth agent at time step t.

[0024] Step 12: Calculate the causal influence strength; for time t, calculate the causal influence strength of agent i on agent j:

[0025]

[0026] in, The causal influence of agent i on agent j at time step t; For the action of agent i at time step t; This represents the state observed by agent j at time step t+1. Let I be the state observed by agent i at time step t; I(·;·∣·) is the conditional mutual information;

[0027] Step 13: Obtain a lower bound estimate of causal effects based on optimized variational distribution;

[0028]

[0029] Where, q φ (∣) denotes the variational distribution with parameter φ, and p(∣) denotes the posterior distribution; The action taken by agent i at time t; Let i be the state of agent i at time t; Let j be the state of agent j at time t+1;

[0030] To obtain a tight lower bound, we need to optimize the variational distribution q. φ ():

[0031]

[0032] Where φ * This represents the optimal set of parameters for the optimized variational distribution parameter φ. This indicates that the parameter φ is being searched to maximize the following expression.

[0033] Step 14: Reconstruct a dynamic causal graph based on historical causal influences; first, calculate the average causal influence:

[0034]

[0035] in This represents the average causal influence. This indicates calculating the average; if Less than the threshold τ prune From G t Remove edge (i,j) from G, then update G. t The causal influence vector is then recalculated.

[0036] Furthermore, step 2 is detailed as follows:

[0037] Step 21: Decompose the observation of the i-th agent into three scales:

[0038]

[0039] in, Let be the observation space of the i-th agent at time t. For local environmental information, This is global task information. This is time-series information;

[0040] Step 22: Extract features from information at different levels:

[0041]

[0042]

[0043] Among them, CNN is a convolutional neural network with a set of trainable parameters θ1; MLP is a multilayer perceptron with a set of trainable parameters θ2; LSTM is a long short-term memory network with a set of trainable parameters θ3; using This represents the feature vectors at each level, including local environmental information feature vectors. Global task information feature vector Temporal information feature vector

[0044] Step 23: Causally Guided Hierarchical Attention Generation; Based on Average Causal Influence The agent's attention at different observation levels is dynamically adjusted, and a linear transformation and Softmax mechanism are used to normalize the attention weights.

[0045]

[0046] in, Let i be the hierarchical attention weights for agent i. Linear weights used for attention generation;

[0047] Step 24: Calculate the attention interaction weights between agents i and j:

[0048]

[0049] in, W represents the attentional interaction weights between agents i and j at the same level. share Weights are mapped to the shared attention level;

[0050] Step 25: Next, the attention of the user and other agents is weighted and fused, and the weighted attention weights are applied to the features to obtain weighted features:

[0051]

[0052] in, To integrate weights, The weighted features are then used as weighted features; finally, the weighted features obtained within and between levels are fused to obtain enhanced observation information.

[0053]

[0054] in, These are the weighted feature representations for the corresponding levels; The fusion weights for the corresponding levels; W cross This is the weight matrix for cross-level feature fusion.

[0055] Furthermore, the global causal contribution and local causal contribution are obtained in the following way:

[0056] Global causal contribution:

[0057]

[0058] in, The global causal contribution of agent i at time t; The expected reward for the optimal action; Let i be the reward that agent i expects to receive at the next time t+1 after performing a random action at time t. This represents the system's total reward at time t+1. To perform the optimal action on agent i Causal intervention; The counterfactual condition for setting the state of agent j to an empty state; Let be the causal interaction weights between agents i and j at time t;

[0059]

[0060] in, The local causal contribution of agent i at time t; Let be the reward obtained by agent i at time t+1. To force agent i to perform its optimal action Then, the reward that agent i expects to receive at time t+1. To determine the expected reward that agent i will receive at time t+1 after being forced to perform a random action; π random The strategy is to distribute the components evenly.

[0061] Furthermore, step 4 is detailed as follows:

[0062] First, using a gated recurrent unit, the behavioral embedding of agent i is generated:

[0063]

[0064] Among them, f φ This is a gated loop unit with parameter φ. Embedding behavior at time step t;

[0065] Then, the multidimensional input is fused using a Transformer encoder:

[0066]

[0067] Where g ψ For Transformer encoders, Embed for the final character.

[0068] Furthermore, step 5 is detailed as follows:

[0069] Role determination using dynamic thresholds:

[0070]

[0071] in, Assigning roles to agent i at time t, categorizing them into high-level roles. Level and low-level characters Level , The difference in role positioning for agent i at time t. This refers to a dynamic threshold, which enables dynamic role migration through the following adaptive mechanism:

[0072]

[0073] in, The mean of the difference in character positioning. The standard deviation of the role positioning difference is α, the mean weight coefficient, β, the standard deviation weight coefficient, and γ, the historical smoothing coefficient. α + β + γ = 1.

[0074] Finally, the agents are categorized into the corresponding role subsets:

[0075]

[0076] in, Let be the set of agents assigned to higher-level roles at time t. The set of agents assigned to low-level roles at time t.

[0077] Furthermore, the causal influence intrinsic reward mechanism specifically includes exploration rewards, collaboration rewards, attentional consistency rewards, and cross-layer balance rewards;

[0078] The formula for calculating the exploration reward is as follows:

[0079]

[0080] Where, λ explore The weighting factor for exploration rewards is used to adjust the importance of exploration rewards in the total intrinsic rewards. Let be the change in causal influence of agent i at time t compared to time t-1. For diversity indicator function, Embedding the role of agent i, Let be the strength of the causal influence of agent i's behavior at time t on agent j's state or behavior;

[0081] The formula for calculating the collaboration reward is as follows:

[0082]

[0083] in, As a reward for collaboration, Let λ represent the strength of the causal influence of agent j's behavior on agent i's state or behavior. coopThe weighting coefficient for collaboration rewards;

[0084] The formula for calculating the attention consistency reward is as follows:

[0085]

[0086] in, For attentional consistency rewards, λ attn The weighting coefficient for attentional consistency rewards;

[0087] The formula for calculating the cross-level balancing reward is as follows:

[0088]

[0089] in, To balance rewards across layers, λ balance The weighting coefficient for cross-level balancing rewards;

[0090] The total intrinsic reward is:

[0091]

[0092] The total reward is:

[0093]

[0094] in, The external reward is then used; the model parameters are then optimized using gradient descent.

[0095] The beneficial effects of this invention are as follows:

[0096] 1) A dynamic causal graph construction mechanism was established. The causal relationship between the actions and states of agents in a multi-agent system was captured in real time through the structural causal model (SCM) and the optimized variational distribution estimator. The strength of the causal relationship was adaptively adjusted through the dynamic causal graph reconstruction (DCGR) mechanism, so that the system can accurately identify and utilize the real causal influence patterns between agents.

[0097] 2) It integrates hierarchical attention mechanism and causal reasoning. Through the generation of hierarchical attention weights guided by causality, the agent can dynamically adjust the attention allocation at different observation levels (local, global, and temporal) based on the causal influence vector. It also achieves effective transmission of collaborative information through the cross-agent attention sharing (CACAS) mechanism, which significantly improves the collaborative efficiency of multi-agent systems.

[0098] 3) A causal awareness role selector (CRS) was constructed. Based on the dynamic threshold comparison of global causal contribution (GCC) and local causal contribution (LCC), the explicit definition and dynamic allocation of agent roles are realized. This overcomes the implicit and static problems of role allocation in traditional methods and improves the accuracy and adaptability of role allocation.

[0099] 4) A causal influence intrinsic reward mechanism was designed, which includes a comprehensive intrinsic reward function that includes exploration rewards, collaboration rewards, attention consistency rewards and cross-layer balance rewards to incentivize agents to learn behavioral patterns that are beneficial to overall collaboration. Attached Figure Description

[0100] Figure 1 This is a flowchart illustrating the overall architecture of the hierarchical attention causal role embedder in this invention.

[0101] Figure 2 This is a schematic diagram illustrating the construction of the dynamic cause-effect graph of the present invention;

[0102] Figure 3 This is a schematic diagram illustrating the principle of the multi-scale feature extraction and hierarchical attention mechanism of this invention.

[0103] Figure 4 This is a schematic diagram of the structure generated by the causal role embedding of the present invention;

[0104] Figure 5 This is a flowchart of the determination process of the Causal Perception Role Selector (CRS) of the present invention. Detailed Implementation

[0105] This invention aims to propose a reinforcement learning method (HACA) based on causal reasoning and hierarchical attention mechanisms to address the problem of behavioral homogenization in the dynamic allocation of roles in multi-agent systems. By introducing Global Causal Contribution (GCC) and Local Causal Contribution (LCC) mechanisms to replace traditional role representations, and combining a threshold adaptive adjustment strategy and an intrinsic reward mechanism based on causal influence, this method enables multi-agent systems to achieve better policy learning and collaborative performance in complex environments. The overall implementation process is as follows: Figure 1 As shown:

[0106] S1. Preprocess the environmental state and historical action trajectory of the multi-agent system, construct a causal graph and initialize the hierarchical attention weights;

[0107] S2. Calculate the global causal contribution value (GCC) of each agent through the global causal reasoning module to identify key decision nodes;

[0108] S3. Extract local interaction features between agents using the local attention mechanism and calculate the local causal contribution value (LCC).

[0109] S4. Dynamically assign roles based on GCC and LCC values, and adjust the assignment strategy through a threshold adaptive mechanism.

[0110] S5. Introduce a causal influence-based intrinsic reward mechanism and train a complete HACA model using gradient descent.

[0111] S6. During actual task execution, dynamically adjust the allocation of agent roles based on the current environmental state to achieve the optimal cooperation strategy.

[0112] Example:

[0113] The following example uses a multi-agent reinforcement learning environment with n agents and m role types to illustrate the solution of this invention in detail. The steps of a reinforcement learning method based on causal reasoning and hierarchical attention mechanisms are as follows: Figure 1 As shown, the specific implementation process is as follows:

[0114] Step 1: Causal Influence Calculation and Feature Preprocessing

[0115] Step 11, as follows Figure 2 As shown, by utilizing historical interaction data of a multi-agent system, a dynamic causal graph is constructed through a structural causal model (SCM) to capture the causal dependencies between agent actions and environmental states; at the same time, the observation space is decomposed into three levels: local environmental information, global task information, and temporal information.

[0116] make Let G be the set of actions of the nth agent at time step t, and the dynamic causal graph G of the multi-agent system at a certain time step t. t Specifically, it includes:

[0117]

[0118] Where n represents the number of agents. Let t be the state observed by the nth agent at time step t. This refers to the action of the nth agent at time step t.

[0119] Step 12: Calculate the causal influence strength; calculate the causal influence matrix to quantify the causal contribution of each agent's behavior to the system state; for time t, calculate the causal influence strength of agent i on agent j:

[0120]

[0121] in, The causal influence of agent i on agent j at time step t; For the action of agent i at time step t; This represents the state observed by agent j at time step t+1. Let I represent the state observed by agent i at time step t; I(·;·∣·) represents conditional mutual information.

[0122] Step 13: Obtain a lower bound estimate of causal effects based on optimized variational distribution.

[0123] For calculating CI, to avoid directly estimating complex probability distributions, we use an optimized variational distribution to obtain a lower bound estimate of causal effects, which is more computationally feasible and stable.

[0124]

[0125] in, For conditional mutual information, The action taken by agent i at time t; Let i be the state of agent i at time t; Let p(∣) be the state of agent j at time t+1, and let p(∣) represent the posterior distribution. It expresses expectation.

[0126] To approximate the mutual information, variational distributions are introduced. Using Jensen's inequality and the concavity of the logarithmic function, we can obtain the variational lower bound of conditional mutual information:

[0127]

[0128] in, Given a variational distribution, this is a learnable neural network with parameters φ, which attempts to adjust based on the next state of agent j. and the current state of agent i To "infer" the action taken by agent i at time t Expanding further, the expectation can be expressed as a summation (for the discrete case) or an integral (for the continuous case):

[0129]

[0130] To obtain a tight lower bound, we need to optimize the variational distribution q. φ ():

[0131]

[0132] Where φ * This represents the optimal set of parameters for the optimized variational distribution φ. This indicates that the parameter φ is being searched to maximize the following expression.

[0133] Step 14: Reconstruct the dynamic causal graph (DCGR) based on historical causal influences and update the causal graph structure in real time; after quantifying the causal influences, calculate the average value of the causal influences as the basis for dynamic adjustment of the causal graph:

[0134]

[0135] in This represents the average causal influence. This indicates calculating the average. If... Less than the threshold τ prune From G t Remove edge (i,j) from G, then update G. t The causal influence vector is then recalculated.

[0136] Step 2: Multi-scale feature extraction and hierarchical attention generation; such as Figure 3 As shown, features are extracted from different observation levels through a hierarchical attention mechanism, and enhanced observation, behavioral embedding, and causal features are fused to generate the initial agent role embedding.

[0137] Step 21: First, decompose the observation space into multi-scale representations and extract features at different levels of abstraction. Decompose the observation of the i-th agent into three scales:

[0138]

[0139] in, This refers to local environmental information, such as the states of neighboring agents. This includes global task information, such as resource distribution; This refers to time-series information, such as historical trajectories.

[0140] Step 22: Extract features from different levels of features:

[0141]

[0142] in, Represented as feature vectors at each level, including local environmental information feature vectors. Global task information feature vector Temporal information feature vector CNN stands for Convolutional Neural Network, and its set of trainable parameters is θ1; MLP stands for Multilayer Perceptron, also known as a fully connected neural network, and its set of trainable parameters is θ2; LSTM stands for Long Short-Term Memory Network, which is a special type of recurrent neural network, and its set of trainable parameters is θ3.

[0143] Step 23: Causal-guided hierarchical attention generation, based on the average causal influence. The agent's attention at different observation levels is dynamically adjusted, and a linear transformation plus a Softmax mechanism is used to normalize the attention weights.

[0144]

[0145] in, Let i be the hierarchical attention weights for agent i. Linear weights used for attention generation.

[0146] Step 24: To enhance information sharing among agents at the same level, calculate the attention interaction weights between agents i and j:

[0147]

[0148] in, W represents the attentional interaction weights between agents i and j at the same level. share Weights are shared attention mappings.

[0149] Step 25: Next, the attention of the user and other agents is weighted and fused, and the weighted attention weights are applied to the features to obtain weighted features:

[0150]

[0151] in, To integrate weights, These are weighted features. Finally, the weighted features obtained within and between levels are fused to obtain enhanced observation information.

[0152]

[0153] in, These are weighted feature representations for different levels. The fusion weights for the corresponding levels; W cross This is the weight matrix for cross-level feature fusion.

[0154] Step 3: Calculate the Global Causal Contribution (GCC) and Local Causal Contribution (LCC) based on the causal influence matrix:

[0155]

[0156] in, The global causal contribution of agent i at time t; The local causal contribution of agent i at time t; The expected reward for the optimal action; Let i be the reward that agent i expects to receive at the next time t+1 after performing a random (no policy) action at time t. This represents the system's total reward at time t+1. To perform the optimal action on agent i Causal intervention; The counterfactual condition for setting the state of agent j to an empty state; Let be the causal interaction weights between agents i and j at time t. Let be the reward obtained by agent i at time t+1. To force agent i to perform its optimal action Then, the reward that agent i expects to receive at time t+1 is... To determine the expected reward that agent i will receive at time t+1 after being forced to perform a random action; π random The strategy is to distribute the components evenly.

[0157] Step 4, Causal perception role selection and threshold adaptation, such as Figure 5 As shown, it includes the following sub-steps:

[0158] Step 41: Causal Role Embedding and Dynamic Role Assignment

[0159] like Figure 4 As shown, using a Transformer encoder, enhanced observation, behavior embeddings, and causal features are fused to generate the final behavior embeddings and role embeddings:

[0160]

[0161] Where f φ This is a gated recurrent unit (GRU) model with parameter φ; Embed the behavior at time step t.

[0162]

[0163] Where g ψ For Transformer encoders, multi-dimensional input is fused. Embed for the final character.

[0164] Step 42: Determine roles using dynamic thresholds, and locate and assign roles in the causal graph based on the calculated GCC and LCC:

[0165]

[0166] in, Assigning roles to agent i at time t, categorizing them into high-level roles.Level and low-level characters Level , Indicates a dynamic threshold. The difference in role positioning of agent i at time t.

[0167] To dynamically adapt to the environment and enable dynamic role migration, an adaptive dynamic threshold is introduced:

[0168]

[0169] in, The mean, Let α be the standard deviation, β be the mean weighting coefficient, γ be the standard deviation weighting coefficient, and α + β + γ = 1.

[0170] To guide agents in forming clear divisions of responsibilities in complex tasks, agents are divided into high-level and low-level roles. High-level roles focus on global planning and guidance, while low-level roles are responsible for local execution and detailed responses. This approach better addresses the challenges of multi-agent collaboration in non-stationary environments. Agents are typically categorized into corresponding role subsets:

[0171]

[0172] in, Let be the set of agents assigned to higher-level roles at time t. The set of agents assigned to low-level roles at time t.

[0173] Step 5: Model training and real-time execution;

[0174] Step 51: Design a multi-dimensional intrinsic reward mechanism based on causal influence, including exploration rewards, collaboration rewards, attentional consistency rewards, and cross-layer balance rewards.

[0175] Exploration Rewards:

[0176]

[0177] Where, λ explore The weighting factor for exploration rewards is used to adjust the importance of exploration rewards in the total intrinsic rewards. Let be the change in causal influence of agent i at time t compared to time t-1. For diversity indicator function, Let i represent the role of the intelligent agent. Let be the strength of the causal influence of agent i's behavior at time t on agent j's state or behavior.

[0178] Collaboration Rewards:

[0179]

[0180] in, As a reward for collaboration, λ represents the strength of the causal influence of agent j's behavior on agent i's state or behavior. coop This is the weighting coefficient for collaborative rewards, used to adjust the incentive intensity of collaborative behavior.

[0181] Attentional consistency reward:

[0182]

[0183] in, For attentional consistency rewards, λ attn The weighting coefficient for attention consistency reward. This represents the attention weight distribution of agent i in a specific layer.

[0184] Cross-tier balanced rewards:

[0185]

[0186] in, To balance rewards across layers, λ balance β is the weighting coefficient for cross-level balancing rewards. i Let be the activation intensity vector of agent i at different levels.

[0187] Total intrinsic reward:

[0188]

[0189] Total Rewards:

[0190]

[0191] in, For external rewards.

[0192] Step 52, Parameter Optimization. The model is trained using the Adam optimizer with a learning rate of 0.0001 and a batch size of 512. The overall model parameters are optimized using gradient descent, combining role assignment loss and intrinsic reward.

[0193] Step 53: Execute the role assignment task based on the trained model. During the execution phase, the system dynamically adjusts the agent role assignment according to the real-time calculated causal influence and role embedding, ensuring efficient collaboration and task completion of the multi-agent system in complex environments.

[0194] Although the present invention has been described herein with reference to embodiments thereof, the above embodiments are merely preferred embodiments of the present invention, and the implementation of the present invention is not limited to the above embodiments. It should be understood that those skilled in the art can design many other modifications and implementations, which will fall within the scope and spirit of the principles disclosed in this application.

Claims

1. A reinforcement learning method based on causal reasoning and hierarchical attention mechanisms, characterized in that, Includes the following steps: First, a multi-agent collaborative model for causal perception is constructed; Step 1: Based on the interaction data observed by the multi-agent system, construct a dynamic causal graph through a structural causal model and an optimized variational distribution estimator, calculate the causal influence vector between agents, and perform dynamic causal graph reconstruction to adjust the strength of causal relationships; Step 2: Decompose the agent's observation space into three levels: local, global, and temporal. Extract feature representations for each level and generate causal-guided hierarchical attention weights based on the causal influence vector. By weighted fusion of the attention of itself and other intelligent agents, and then feature fusion within and between levels, enhanced observation information is obtained. Step 3: Calculate the global causal contribution of the system and the local causal contribution of the agent based on the reward for the action performed; Step 4: Based on the observation space and dynamic causal graph, the behavior embedding is obtained by fusing them. Then, the enhanced observation information, behavior embedding, and causal influence vector are fused to obtain the role embedding. Step 5: Dynamically assign roles based on global and local causal contributions, and adjust the allocation strategy through a threshold adaptive mechanism; Step 6: Design a causal incentive mechanism to train the model; In actual task execution, the multi-agent collaborative model dynamically adjusts the allocation of agent roles based on the current environmental state.

2. The reinforcement learning method based on causal reasoning and hierarchical attention mechanism according to claim 1, characterized in that, Step 1 is described in detail as follows: Step 11: Construct a dynamic cause-effect graph; the dynamic cause-effect graph G of the multi-agent system at time step t. t Specifically, it includes: Where n represents the number of agents. Let t be the state observed by the nth agent at time step t. This refers to the action of the nth agent at time step t. Step 12: Calculate the causal influence strength; for time t, calculate the causal influence strength of agent i on agent j: in, The causal influence of agent i on agent j at time step t; For the action of agent i at time step t; This represents the state observed by agent j at time step t+1. Let I be the state observed by agent i at time step t; I(·;·∣·) is the conditional mutual information; Step 13: Obtain a lower bound estimate of causal effects based on optimized variational distribution; Where, q φ (∣) denotes the variational distribution with parameter φ, and p(∣) denotes the posterior distribution; The action taken by agent i at time t; Let i be the state of agent i at time t; Let j be the state of agent j at time t+1; To obtain a tight lower bound, we need to optimize the variational distribution q. φ (): Where φ * This represents the optimal set of parameters for the optimized variational distribution parameter φ. This indicates that the parameter φ is being searched to maximize the following expression. Step 14: Reconstruct a dynamic causal graph based on historical causal influences; first, calculate the average causal influence: in This represents the average causal influence. This indicates calculating the average; if Less than the threshold τ prune From G t Remove edge (i,j) from G, then update G. t The causal influence vector is then recalculated.

3. The reinforcement learning method based on causal reasoning and hierarchical attention mechanism according to claim 2, characterized in that, Step 2 is described in detail below: Step 21: Decompose the observation of the i-th agent into three scales: in, Let be the observation space of the i-th agent at time t. For local environmental information, This is global task information. This is time-series information; Step 22: Extract features from information at different levels: Among them, CNN is a convolutional neural network with a set of trainable parameters θ1; MLP is a multilayer perceptron with a set of trainable parameters θ2; LSTM is a long short-term memory network with a set of trainable parameters θ3; using This represents the feature vectors at each level, including local environmental information feature vectors. Global task information feature vector Temporal information feature vector Step 23: Causally Guided Hierarchical Attention Generation; Based on Average Causal Influence The agent's attention at different observation levels is dynamically adjusted, and a linear transformation and Softmax mechanism are used to normalize the attention weights. in, Let i be the hierarchical attention weights of agent i. Linear weights used for attention generation; Step 24: Calculate the attention interaction weights between agents i and j: in, W represents the attentional interaction weights between agents i and j at the same level. share Weights are mapped to the shared attention level; Step 25: Next, the attention of the user and other agents is weighted and fused, and the weighted attention weights are applied to the features to obtain weighted features: in, To integrate weights, The weighted features are then used as weighted features; finally, the weighted features obtained within and between levels are fused to obtain enhanced observation information. in, These are the weighted feature representations for the corresponding levels; The fusion weights for the corresponding levels; W cross This is the weight matrix for cross-level feature fusion.

4. The reinforcement learning method based on causal reasoning and hierarchical attention mechanism according to claim 3, characterized in that, The global causal contribution and local causal contribution are obtained in the following way: Global causal contribution: in, The global causal contribution of agent i at time t; The expected reward for the optimal action; Let i be the reward that agent i expects to receive at the next time t+1 after performing a random action at time t. This represents the system's total reward at time t+1. To perform the optimal action on agent i Causal intervention; The counterfactual condition for setting the state of agent j to an empty state; Let be the causal interaction weights between agents i and j at time t; in, The local causal contribution of agent i at time t; Let be the reward obtained by agent i at time t+1. To force agent i to perform its optimal action Then, the reward that agent i expects to receive at time t+1. To determine the expected reward that agent i will receive at time t+1 after being forced to perform a random action; π random The strategy is to distribute the components evenly.

5. The reinforcement learning method based on causal reasoning and hierarchical attention mechanism according to claim 4, characterized in that, Step 4 is described in detail below: First, using a gated recurrent unit, the behavioral embedding of agent i is generated: Among them, f φ This is a gated loop unit with parameter φ. Embedding behavior at time step t; Then, the multidimensional input is fused using a Transformer encoder: Where g ψ For Transformer encoders, Embed for the final character.

6. The reinforcement learning method based on causal reasoning and hierarchical attention mechanism according to claim 5, characterized in that, Step 5 is described in detail below: Role determination using dynamic thresholds: in, Assigning roles to agent i at time t, categorizing them into high-level roles. Level and low-level characters Level , The difference in role positioning for agent i at time t. This refers to a dynamic threshold, which enables dynamic role migration through the following adaptive mechanism: in, The mean of the difference in character positioning. The standard deviation of the role positioning difference is α, the mean weight coefficient, β, the standard deviation weight coefficient, and γ, the historical smoothing coefficient. α + β + γ = 1. Finally, the agents are categorized into the corresponding role subsets: in, Let be the set of agents assigned to higher-level roles at time t. The set of agents assigned to low-level roles at time t.

7. The reinforcement learning method based on causal reasoning and hierarchical attention mechanism according to claim 6, characterized in that, The causal influence of the intrinsic reward mechanism specifically includes exploration rewards, collaboration rewards, attentional consistency rewards, and cross-layer balance rewards; The formula for calculating the exploration reward is as follows: Where, λ explore The weighting factor for exploration rewards is used to adjust the importance of exploration rewards in the total intrinsic rewards. Let be the change in causal influence of agent i at time t compared to time t-1. For diversity indicator function, Embedding the role of agent i, Let be the strength of the causal influence of agent i's behavior at time t on agent j's state or behavior; The formula for calculating the collaboration reward is as follows: in, As a reward for collaboration, Let λ represent the strength of the causal influence of agent j's behavior on agent i's state or behavior. coop The weighting coefficient for collaboration rewards; The formula for calculating the attention consistency reward is as follows: in, For attentional consistency rewards, λ attn The weighting coefficient for attention consistency reward; the formula for calculating the cross-layer balancing reward is as follows: in, To balance rewards across layers, λ balance The weighting coefficients for cross-layer balancing rewards; the total intrinsic reward is: The total reward is: in, The external reward is then used; the model parameters are then optimized using gradient descent.

Images