Target condition reinforcement learning method and system based on causal inference
By introducing causal inference and causal graphs into target condition reinforcement learning, the problem of insufficient generalization ability in the external distribution environment is solved, and a more efficient and robust reinforcement learning model is realized, which is suitable for multi-tasking and complex environments.
Patent Information
- Application Number
- CN202510018280.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-07
- Publication Date
- 2025-04-18
AI Technical Summary
The existing target condition reinforcement learning methods show poor generalization ability when facing an external distribution environment, mainly due to the influence of unobserved confounders, insufficient robustness of abstract states, and limited interpretability and availability of causal models.
The target condition reinforcement learning method based on causal inference is adopted, and the global state is processed through abstract functions, abstract state is generated, unobserved confounding factors are cut off, and a new abstract path is constructed using the causal graph, and the Monte Carlo sampling method is optimized to learn the weighted adjacency matrix of the causal graph.
It significantly improves the generalization ability, learning efficiency, control accuracy and robustness of the reinforcement learning model, enhances the flexibility and scalability of multi-task learning, reduces resources and energy consumption, and provides explanatory power for strategy learning.
Smart Images
Figure CN119849589A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of reinforcement learning, and in particular to a target conditional reinforcement learning method and system based on causal inference. Background Art
[0002] Goal-conditional reinforcement learning is a technique designed to train intelligent agents to complete multiple tasks and achieve multiple goals. In recent years, it has been widely used in fields such as robotics and recommendation systems. However, existing GCRL methods often show poor generalization ability when facing out-of-distribution environments. This performance degradation is mainly due to unobserved confounding factors in the environment, which play a key role in the distribution difference between the training and test environments.
[0003] Most of the current technical means deal with the OOD generalization problem through the following methods: state abstraction, causal modeling, and generalization based on policy optimization. However, there are still the following problems and shortcomings:
[0004] 1. The impact of unobserved confounding factors:
[0005] Existing methods find it difficult to explicitly consider unobserved confounding factors, which, as common causal factors between the global state and the target state, introduce causal bias and lead to a decrease in generalization ability.
[0006] 2. The abstract state is not robust enough:
[0007] Traditional state abstraction methods mostly rely on prior knowledge of specific tasks and lack the ability to adapt to changes in causal relationships between different environments. When the test environment is significantly different from the training environment, the abstract state may fail.
[0008] 3. Limited interpretability and usability of causal models:
[0009] Most current causal modeling methods rely on explicit causal assumptions, which are difficult to satisfy in real-world applications. In addition, causal graph learning in high-dimensional environments faces problems with computational complexity and accuracy.
[0010] Therefore, we proposed a target-conditional reinforcement learning method and system based on causal inference to solve the problems raised above. Summary of the invention
[0011] The purpose of the present invention is to provide a target conditional reinforcement learning method and system based on causal inference to solve one of the problems raised by the above background technology, namely, the influence of unobserved confounding factors in the current market, the insufficient robustness of abstract states, and the limited interpretability and usability of causal models.
[0012] To achieve the above object, the present invention provides the following technical solutions:
[0013] The target-conditional reinforcement learning method based on causal inference includes the following steps:
[0014] Step 1: Process the global state s through the abstract function F(s; G) to generate an abstract state and remove features that are not related to the task;
[0015] Step 2: In the presence of unobserved confounding factors z, use the intervention probability P(a|do(s)) to cut off the backdoor path;
[0016] Step 3: By introducing the causal graph G, a new abstract path is constructed and the intervention probability of the final strategy output is represented;
[0017] Step 4: Estimate the intervention probability through Monte Carlo sampling method, and optimize the goal to maximize the cumulative expected reward of the strategy;
[0018] Step 5: Learn the weighted adjacency matrix W(θ) of the causal graph by optimizing the objective function L(θ) and constrain the causal graph to be an acyclic graph.
[0019] As a further optimization scheme of the present invention, completing the abstract function F(s; G) includes the following steps:
[0020] The global state s is input into the feature extraction network to obtain a feature vector, the causal graph G is used as input to obtain a causal feature vector, the feature vector is concatenated with the causal feature vector to obtain a concatenated vector, and the concatenated vector is input into the abstract state generation network to obtain an abstract state S ab .
[0021] As a further optimization scheme of the present invention, in step 2, the formula used for the intervention probability is defined as follows:
[0022]
[0023] Where G represents the set of all possible values g, P(a|s,g) represents the probability of a occurring given the global state s and the confounding factor g, and P(g) is the marginal probability of the confounding factor g.
[0024] As a further optimization solution of the present invention, in step 3, the formula used based on the causal graph G is defined as follows:
[0025]
[0026] Where G represents the set of all possible causal graphs G′, P(s ab |s,G′) represents the abstract state s given the global state s and a specific causal graph G′ ab is the conditional probability of , and P(G′) is the marginal probability of the causal graph G′.
[0027] The intervention probability of the final strategy output is expressed as:
[0028]
[0029] Among them, P(a|do(s ab )) indicates that the intervention do(s) is being performed ab ) after the event a The probability of occurrence, G is the set of all possible causal graphs G′, P(a|s ab ,G′,g) represents the given abstract state s ab , a specific causal graph G′ and a confounding factor g, the event a The conditional probability of occurrence, P(G′) is the marginal probability of the causal graph G′.
[0030] As a further optimization scheme of the present invention, in step 4, for estimating P(s ab |do(s)), the Monte Carlo sampling method can be expressed as:
[0031]
[0032] Among them, K is the number of sampling, G k is the causal graph obtained in the kth sampling, P(s ab |s,G k ) is in the kth sampling, given the global state s and the causal graph G k When the abstract state s ab The conditional probability of .
[0033] For the estimation of P(a|do(s ab ), the Monte Carlo sampling method can be expressed as:
[0034]
[0035] Among them, M is the number of sampling, G m and g m In the m The causal graph and confounding factors obtained from the subsampling, π(s ab ,G m ,g m ) is in m In the sampling, given the abstract state s ab 、Causal graph G m and confounding factor g m The probability estimate of event a when .
[0036] The target conditional reinforcement learning system based on causal inference includes a causal graph learning module, an abstract network module and a policy network module;
[0037] Among them, the causal graph learning module is used to generate and update the causal graph G;
[0038] Abstract network module: used to filter and generate abstract states S using causal graphs ab ;
[0039] Policy network module: used to generate the agent's strategy a=π(s ab ,G,g).
[0040] Compared with the prior art, the present invention has the following beneficial effects:
[0041] By introducing causal inference and causal graphs, the present invention effectively improves the generalization ability, learning efficiency, control accuracy and robustness of the reinforcement learning model, while enhancing its flexibility and scalability in multi-task learning, significantly saving resources and energy consumption. By cutting off mixed causal paths, abstracting state space, and optimizing data usage efficiency, the intelligent agent can still accurately perform tasks in changing environments and complex tasks, reducing training costs, and providing interpretability for strategy learning, providing strong technical support for the efficient, accurate and low-cost operation of the intelligent agent in a variety of scenarios.
[0042] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 It is the evaluation environment diagram under different OOD settings of the present invention;
[0044] Figure 2 A comparison diagram of COGITATION and the baseline method under the chemical environment of the present invention;
[0045] Figure 3 It is the average success rate diagram of three OOD settings and three downstream tasks of the present invention;
[0046] Figure 4 It is a comparison diagram between the causal graph learned in the present invention and the real causal graph. DETAILED DESCRIPTION
[0047] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0048] Embodiment 1
[0049] The target condition reinforcement learning method based on causal inference is characterized by comprising the following steps:
[0050] Step 1: Process the global state s through the abstract function F(s; G) to generate an abstract state and remove features that are not related to the task;
[0051] Specifically, completing the abstract function F(s; G) includes the following steps:
[0052] The global state s is input into the feature extraction network to obtain a feature vector, the causal graph G is used as input to obtain a causal feature vector, the feature vector is concatenated with the causal feature vector to obtain a concatenated vector, and the concatenated vector is input into the abstract state generation network to obtain an abstract state S ab .
[0053] Generate abstract state S ab The formula definition is as follows:
[0054] S ab =F(s;G)
[0055] Step 2: In the presence of unobserved confounding factors z, use the intervention probability P(a|do(s)) to cut off the backdoor path;
[0056] The formula used for the probability of intervention is defined as follows:
[0057]
[0058] Where G represents the set of all possible values g, P(a|s,g) represents the probability of a occurring given the global state s and the confounding factor g, and P(g) is the marginal probability of the confounding factor g.
[0059] Step 3: By introducing the causal graph G, a new abstract path is constructed and the intervention probability of the final strategy output is represented;
[0060] The formula used based on the causal graph G is defined as follows:
[0061]
[0062] Where G represents the set of all possible causal graphs G′, P(s ab |s,G′) represents the abstract state s given the global state s and a specific causal graph G′ ab is the conditional probability of , and P(G′) is the marginal probability of the causal graph G′.
[0063] The intervention probability of the final strategy output is expressed as:
[0064]
[0065] Among them, P(a|do(s ab )) indicates that the intervention do(s) is being performed ab ) after the event a The probability of occurrence, G is the set of all possible causal graphs G′, P(a|s ab ,G′,g) represents the given abstract state s ab , the conditional probability of event a under the conditions of a specific causal graph G′ and confounding factor g, P(G′) is the marginal probability of the causal graph G′.
[0066] Step 4: Estimate the intervention probability through Monte Carlo sampling method, and optimize the goal to maximize the cumulative expected reward of the strategy;
[0067] Specifically, for estimating P(s ab |do(s)), the Monte Carlo sampling method can be expressed as:
[0068]
[0069] Among them, K is the number of sampling, G k is the causal graph obtained in the kth sampling, P(s ab |s,G k ) is in the kth sampling, given the global state s and the causal graph G k When the abstract state s ab The conditional probability of .
[0070] For the estimation of P(a|do(s ab ), the Monte Carlo sampling method can be expressed as:
[0071]
[0072] Among them, M is the number of sampling, G m and g m In the m The causal graph and confounding factor obtained from the subsampling, π(s ab ,G m ,g m ) is in m In the sampling, given the abstract state s ab 、Causal graph G m and confounding factor g m The probability estimate of event a when .
[0073] Step 5: Learn the weighted adjacency matrix W(θ) of the causal graph by optimizing the objective function L(θ) and constrain the causal graph to be an acyclic graph.
[0074] Specifically, the formula used to optimize the target is defined as follows:
[0075]
[0076] Where J(π) is the value or performance indicator of strategy π, E represents the expected value, Indicates status s Following the state distribution p under policy π π , represents the action a selected according to the strategy π, γ is the discount factor used to balance the immediate reward and future reward, r(st,at,gt) is the immediate reward under the state st, action at and confounding factor gt at time step t, is the discounted cumulative reward from time step t=0 to infinity.
[0077] The objective function L(θ) is used to learn the weighted adjacency matrix W(θ) of the causal graph using the following formula:
[0078]
[0079] Among them, L(θ) is the optimization objective function used to learn the weighted adjacency matrix W(θ), is the normalization factor, where n is the number of data points, ||X-XW(θ)|| 2 is the reconstruction error term, which measures the difference between the original data matrix X and the data reconstructed by the weighted adjacency matrix W(θ), λ||W(θ)||1 is the L1 regularization term, which is used to promote the sparsity of W(θ), α is the regularization parameter, which is used to balance the weight of the reconstruction error term and the DAG regularization term, and h(W(θ) is the DAG regularization term, which is used to ensure that the graph represented by W(θ) is a directed acyclic graph (DAG) without cycles.
[0080] The state space S is defined as follows:
[0081] S={s1,s2,....,s n}
[0082] Among them, S is a set containing n different state variables, s1, s2, ...., s n Represent the 1st to nth state variables in the state space respectively.
[0083] The action space A is defined as follows:
[0084] A={a1,a2,....,a m}
[0085] Among them, A is a set containing m different actions, a1, a2, ..., a m Represent the 1st to 2nd actions in the action space respectively. m action.
[0086] The definition of the target space G can be rearranged as follows:
[0087] G={g1,g2,....,g k}
[0088] Among them, G is a set containing k different targets, g1, g2, ..., g k They represent the 1st to kth targets in the target space respectively.
[0089] Embodiment 2
[0090] A target-conditional reinforcement learning system based on causal inference, including a causal graph learning module, an abstract network module, and a policy network module;
[0091] Among them, the causal graph learning module is used to generate and update the causal graph G;
[0092] Abstract network module: used to filter and generate abstract states S using causal graphs ab ;
[0093] Policy network module: used to generate the agent's strategy a=π(s ab ,G,g).
[0094] In summary, the present invention cuts off the mixed causal paths in the environment through causal inference, effectively eliminating the deviation between the training distribution and the test distribution of the target conditional reinforcement learning model, thereby significantly improving the generalization ability of the model in the out-of-distribution environment. When the target state changes, the method of the present invention can still accurately perform the task.
[0095] The present invention uses causal graphs to abstract the state, eliminates state variables that are not related to the task, and reduces the dimension of the state space. This state abstraction significantly reduces the computational complexity of policy learning and improves the training efficiency of the model. At the same time, the decongestion adjustment method of Monte Carlo sampling further optimizes the efficiency of training data use.
[0096] Through decongestion adjustment, the agent can effectively avoid decision bias caused by unobserved confounding factors during strategy execution, making the agent more accurate in tracking task targets. For example, in a manipulation task, the target object can be more accurately located and manipulated even when there are interference factors in the environment.
[0097] The present invention has stronger robustness to complex environments through dynamic learning and adjustment of causal graphs. Even in high-noise or high-complexity environments, the agent can still maintain good performance. And the learnability of causal graphs enables the method of the present invention to adapt to different environments and task requirements. This method is not only applicable to traditional tasks in reinforcement learning, but can also be extended to multiple scenarios such as multi-objective optimization and robot control, and has a wide range of applicability.
[0098] By improving learning efficiency and reducing training data requirements, the present invention reduces the computing resources and energy consumption required to train reinforcement learning models. At the same time, de-confusion adjustment enables the intelligent agent to complete tasks more efficiently, reducing the operating cost in practical applications.
[0099] The state abstraction and causal inference method introduced in this invention provides a general framework for multi-task learning of intelligent agents. This method can migrate strategies between multiple different task environments through dynamic adaptation of causal graphs, further improving multi-task processing capabilities.
[0100] The introduction of causal graphs makes the policy learning process more interpretable. By analyzing the causal relationship graph, we can clarify the impact path of different state variables and target variables in the agent's decision-making, providing a basis for problem diagnosis and policy optimization in complex task environments.
[0101] Experimental comparison
[0102] Purpose:
[0103] 1. Evaluate the performance of COGITATION under ID (in-distribution) and OOD (out-of-distribution) generalization settings.
[0104] 2. Verify the ability of COGITATION to identify true causal graphs and compare it with methods based on conditional independence testing (CIT).
[0105] 3. Evaluate the sampling efficiency improvement of the state abstraction obtained by COGITATION when generalizing downstream tasks (H3).
[0106] 4. Study the impact of backdoor adjustment achieved through Monte Carlo sampling on generalization ability (H4).
[0107] 5. Evaluate the scalability and robustness of COGITATION (H5).
[0108] Baseline method:
[0109] SAC: Soft Actor-Critic, which increases the diversity of actions through entropy, is a well-known reinforcement learning baseline method.
[0110] GNN: Graph Neural Network, and considers dense dependencies of each state variable.
[0111] ICIN: Provides causal reasoning capabilities to agents, assuming access to true causal graphs to accomplish goal-oriented tasks.
[0112] ICIL: Proposes an invariant feature learning structure to capture the implicit causal relationships of multiple tasks and facilitates the generalization of different tasks.
[0113] DBC: Learning abstract states via dual analog metrics,considering effective downstream control.
[0114] GRADER: Formulating the goal-conditioned reinforcement learning problem as variational likelihood maximization with causal graphs as latent variables to promote generalization.
[0115] CDL: Model-based methods help improve generalization ability by learning causal dynamic models and strictly eliminating the dependencies between state variables and actions.
[0116] As attached Figure 3 The figure shows the average success rate (10 random seeds) for three OOD settings (color, shape, friction) and three downstream tasks (Reach, Pick, Match) under the operation environment. The best and second best results are indicated by (\textbf{bold}) and (\underline{underline}), respectively.
[0117] As attached Figure 1 The evaluation environment under different OOD settings is shown in Figure 1. The environment settings include ID and OOD. The ID setting uses the same target distribution for training and testing, while the OOD setting changes the target distribution by introducing a confusion factor to test the generalization ability under different target distributions.
[0118] As attached Figure 2 As shown, COGITATION is compared with multiple baseline methods in the chemical environment, for chain graphs, collision graphs, jungle graphs, and complete graphs in ID and OOD settings. All results are the average test rewards calculated based on 10 random seeds, showing that COGITATION performs best in all environment settings, especially in the OOD setting.
[0119] We test the generalization ability of RL agents by modifying common reinforcement learning benchmark environments. We design two custom settings: in-distribution (ID) and out-of-distribution (OOD), corresponding to different target distributions. During training and testing, we use $p_{\text{train}}(g)$ and $p_{\text{test}}(g)$ to represent the target distribution. In the ID setting, the same target distribution is used for training and testing, while the OOD setting introduces distribution differences due to the confusion factor.
[0120] Chemical environment: Two fixed-color nodes are added as interference items, and the color change mechanism of the remaining five nodes is controlled by the underlying causal graph. The agent can only change the color of one node at each step, and the goal is to match the specified color. In the OOD setting, a pseudo relationship is introduced by changing the target color.
[0121] MuJoCo Operation Environment: A MuJoCo-based operation environment is used to verify the performance of COGITATION in complex rigid body dynamics. There are movable objects (blue balls and red boxes) and immovable objects (black balls and black boxes) in the environment. The agent's goal is to stack the balls on the boxes.
[0122] In the OOD setting, we made the following modifications: (a) "OOD-color": modify the color of the ball and the box in the test environment; (b) "OOD-shape": change the shape of the box in the test environment; (c) "OOD-friction": reduce the friction of the table in the test environment.
[0123] Experimental results:
[0124] For overall performance (H1): Figure 1 and attached Figure 3 As shown, COGITATION outperforms all baseline methods in the chemical environment (ID and OOD settings), and also performs best in the OOD-shape and OOD-friction settings of the operational environment (and second best in the OOD-color setting).
[0125] In the OOD setting of the chemical environment, the gap with the baseline is significantly larger than that of the ID setting, showing that the method of the present invention still performs well in non-trivial generalization tasks. SAC performs poorly in all chemical and operational environment tasks. GNN and DBC fail in most OOD generalization tasks. The ability to reveal the causal relationships behind the tasks makes ICIN, ICIL and GRADER outperform SAC, GNN and DBC. Although CDL's effective state abstraction brings strong performance, it is affected by the pseudo-correlation introduced by the confusion factor in the OOD setting, resulting in its performance being inferior to COGITATION.
[0126] Causal discovery analysis (H2): To verify the effectiveness of causal graph learning in COGITATION, the causal discovery effect of the conditional independence test (CIT) method used in GRADER and CDL is evaluated. The performance of COGITATION in a complex causal graph environment is verified by indicators such as accuracy, recall, precision, F1 score, and ROC AUC, showing its advantages in capturing nonlinear relationships. Figure 4 The comparison between the causal graph learned in the \textit{Collider} environment and the real causal graph is shown, indicating that COGITATION can accurately reveal the causal relationships in the environment.
[0127] Advantages of state abstraction (H3}): To verify the effectiveness of state abstraction, COGITATION is compared with a version without state abstraction ("COGITATION w / o SA") and also compared with baseline methods such as DBC and CDL.
[0128] The results are attached Figure 2 and attached Figure 3 As shown in the figure, COGITATION outperforms other methods in both ID and OOD settings. DBC performs worse than COGITATION and CDL due to the lack of causal integration. Although CDL performs well in some cases due to effective state abstraction, it is affected by pseudo-correlation in OOD settings, resulting in insufficient generalization ability.
[0129] Backgate adjustment ablation study (H4}): Backgate adjustment is implemented via Monte Carlo sampling. Ablation experiments are performed on backgate adjustment to study the impact of different sampling times $K$ on the results. When $K=1$, effective backgate adjustment cannot be performed and the agent performs poorly in the OOD generalization task. As $K$ increases to 5 and 10, the performance improves, and the best performance is achieved when $K=10$, proving the effectiveness of backgate adjustment.
[0130] Scalability of COGITATION (H5): The scalability of COGITATION was evaluated in chemical environments with increasing numbers of entity nodes (7, 12, 17). The results showed that as the number of nodes increases, the complexity of the causal graph also increases, and the task becomes more difficult. Despite this, COGITATION still shows the best performance when dealing with increasingly complex environments and different downstream tasks, demonstrating the robustness and scalability of our algorithm.
[0131] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0132] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0133] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0134] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.
Claims
1. A target-conditional reinforcement learning method based on causal inference, characterized in that: The following steps are involved: Step 1: Process the global state s through the abstract function F(s; G) to generate an abstract state and remove features that are not related to the task; Step 2: In the presence of unobserved confounding factors z, use the intervention probability P(a|do(s)) to cut off the backdoor path; Step 3: By introducing the causal graph G, a new abstract path is constructed and the intervention probability of the final strategy output is represented; Step 4: Estimate the intervention probability through Monte Carlo sampling method, and optimize the goal to maximize the cumulative expected reward of the strategy; Step 5: Learn the weighted adjacency matrix W(θ) of the causal graph by optimizing the objective function L(θ) and constrain the causal graph to be an acyclic graph.
2. The target conditional reinforcement learning method based on causal inference according to claim 1, characterized in that: In step 1, the abstract function F(s; G) comprises the following steps: The global state s is input into the feature extraction network to obtain a feature vector, the causal graph G is used as input to obtain a causal feature vector, the feature vector is concatenated with the causal feature vector to obtain a concatenated vector, and the concatenated vector is input into the abstract state generation network to obtain an abstract state S ab .
3. The target conditional reinforcement learning method based on causal inference according to claim 1, characterized in that: In step 2, the formula used for the probability of intervention is defined as follows: Where G represents the set of all possible values g, P(a|s,g) represents the probability of a occurring given the global state s and the confounding factor g, and P(g) is the marginal probability of the confounding factor g.
4. The target conditional reinforcement learning method based on causal inference according to claim 1, characterized in that: In step 3, the formula used based on the causal graph G is defined as follows: Where G represents the set of all possible causal graphs G′, P(s ab |s,G′) represents the abstract state s given the global state s and a specific causal graph G′ ab The conditional probability of , and P(G′) is the marginal probability of the causal graph G′; The intervention probability of the final strategy output is expressed as: Among them, P(a|do(s ab )) indicates that the intervention do(s) is being performed ab ), the probability of event a occurring after s, G is the set of all possible causal graphs G′, P(a|s ab ,G′,g) represents the given abstract state s ab , the conditional probability of event a under the conditions of a specific causal graph G′ and confounding factor g, P(G′) is the marginal probability of the causal graph G′.
5. The target conditional reinforcement learning method based on causal inference according to claim 1, characterized in that: In step 4, for the estimated P(s ab |do(s)), the Monte Carlo sampling method can be expressed as: Among them, K is the number of sampling, G k is the causal graph obtained in the kth sampling, P(s ab |s,G k ) is in the kth sampling, given the global state s and the causal graph G k When the abstract state s ab The conditional probability of For the estimation of P(a|do(s ab ), the Monte Carlo sampling method can be expressed as: Among them, M is the number of sampling, G m and g m In the m The causal graph and confounding factor obtained from the subsampling, π(s ab ,G m ,g m ) is in the mth sampling, given the abstract state s ab 、Causal graph G m and confounding factor g m The probability estimate of event a when .
6. A target-conditional reinforcement learning system based on causal inference, characterized in that: The target conditional reinforcement learning method based on causal inference as described in any one of claims 1 to 5 comprises a causal graph learning module, an abstract network module and a policy network module; Among them, the causal graph learning module is used to generate and update the causal graph G; Abstract network module: used to filter and generate abstract states S using causal graphs ab ; Policy network module: used to generate the agent's strategy a=π(s ab ,G,g).