Complex system-oriented causal structure identification and causal effect estimation method

Abstract

The invention discloses a causal structure identification and causal effect estimation method for a complex system, and relates to the technical field of data processing, and the method comprises the following steps: S1, building a causal structure identification and causal effect estimation system for the complex system; s2, importing an analysis data file by a causal data analysis module; according to the complex system-oriented causal structure identification and causal effect estimation method, various business data contents such as cross-linking rules, key indexes and the like are imported and analyzed by connecting experimental data; the invention provides a relatively complete causal structure identification and causal effect estimation method aiming at the conditions that the analysis and utilization of data generated in the current complex system simulation process are insufficient, and the practice of causal structure identification and causal effect estimation under a complex causal graph is lacked. Therefore, the problem that causal structure identification and causal effect estimation of a complex system are difficult is solved, and important data support is provided for improving system efficiency and improving optimization.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a causal structure identification and causal effect estimation method for complex systems. Background Art

[0002] Today's complex systems generate a large amount of experimental data during the simulation process, but the utilization rate of experimental data is still insufficient. How to make good use of the existing experimental data and accurately mine and analyze the causal relationship between data has become an important issue. In order to effectively mine and verify the causal relationship between many variables (such as parameters, factors, simulation targets, etc.) in typical simulation scenarios, a causal structure identification and causal effect estimation method for complex systems based on complex system experimental data is proposed. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides a method for causal structure identification and causal effect estimation for complex systems, which solves the problems raised in the above background technology.

[0004] To achieve the above objectives, the present invention is implemented through the following technical solutions: a method for causal structure identification and causal effect estimation for complex systems, comprising the following steps: S1: Build a causal structure identification and causal effect estimation system for complex systems; The causal relationship identification system for complex systems constructed includes: a causal data analysis module, a causal structure identification module and a causal effect estimation module; The causal data analysis module is used to import and analyze training experiment data, provide common functions for data cleaning, and automatically construct a causal graph through the PC algorithm and causal entropy algorithm; The causal structure identification module is used to design a causal structure identification algorithm based on d-separation and do-calculus according to the rules and conditions of the backdoor criteria, the frontdoor criteria and the instrumental variables; The causal effect estimation module is used to design the matching logic of the causal effect algorithm according to the causal structure identification results, and provide a verification function for the causal effect estimation results; S2: Causal data parsing module imports parsed data files: Import and analyze system simulation experiment data in the form of database and file, and provide outlier removal, normalization, data interpolation and related data cleaning functions; automatically build causal graphs based on the analyzed data through PC algorithm and causal entropy algorithm, and import causal graph data into causal structure recognition module; S3: The causal structure recognition module takes the causal graph data as input and can perform specified causal variable structure recognition and batch causal structure recognition; S4: The causal effect estimation module takes experimental data, causal structure identification results and a set of adjustment variables as input, calculates the estimated results of the causal effects between specified causal variables, and estimates the causal effects in batches, and verifies the credibility of the causal effect evaluation results through data subset verification.

[0005] Optionally, the S3: causal structure identification module takes the causal graph data as input, and can perform the steps of specifying the causal variable structure identification and batch causal structure identification. Before identifying the causal structure, it is first necessary to determine whether the causal graph satisfies the DAG completeness. If not, subsequent identification work cannot be performed. In order to realize the identification of the causal structure and batch causal structure belonging to any causal variable in the complex causal graph, based on the backdoor criterion, frontdoor criterion and instrumental variable rules, the causal structure is divided into: causal structure that satisfies the backdoor criterion, causal structure that satisfies the frontdoor criterion and causal structure that satisfies the instrumental variable.

[0006] Optionally, when using the rules based on the backdoor criteria, front-door criteria and instrumental variables to identify the causal structure, logical rules are designed to identify the causal structure of the causal graph based on the Bayesian network characteristics of the causal graph, concepts such as d-separation and do-calculus, and obtain the identification result of the causal structure, as well as the corresponding set of adjustment variables: the set of adjustment variables that meet the backdoor criteria, the set of adjustment variables that meet the front-door criteria and the set of adjustment variables that meet the instrumental variables, for the causal effect estimation module to evaluate the causal effect; causal variables can be selected for causal structure identification, and each node in the causal graph can be traversed to identify the causal structure between all causal variable nodes.

[0007] Optionally, when identifying the causal structure between the causal variables, no identification priority is set, and the causal structure identification module saves all structure identification results that meet the conditions.

[0008] Optionally, the S4: causal effect estimation module takes experimental data, causal structure identification results and a set of adjustment variables as input, calculates the estimated results of the causal effects between specified causal variables, and batch estimates the causal effects, and verifies the credibility of the causal effect evaluation results through data subset verification. According to the causal structure identification results obtained by the causal structure identification module and the corresponding set of adjustment variables, applicable rules of the evaluation algorithm are designed. For the causal structure that meets the backdoor criterion, causal effect evaluation algorithms such as univariate linear regression, multivariate linear regression, propensity score stratification, and propensity score matching are provided, and suitable evaluation algorithms are further allocated based on the set of adjustment variables.

[0009] Optionally, for the causal structure that meets the front-door criterion, a two-layer linear regression algorithm based on the front-door criterion is provided; for the causal structure that meets the instrumental variables, a two-layer linear regression algorithm based on the instrumental variables and a binary instrument / Wald estimation algorithm are provided, and a suitable causal effect evaluation algorithm is further assigned according to the dimension and distribution information of the adjustment variable set and the dependent variable; for the case where the causal structure identification result is not unique, including the causal structure that meets both the back-door criterion and the instrumental variables, the priority of causal structure matching is designed, and the user can design the priority order of the causal structure identification results as needed, and the system will assign a suitable causal effect evaluation algorithm to the causal variable according to the causal structure with the highest priority.

[0010] Optionally, it supports causal effect estimation of the specified causal variables, and the user can independently select an appropriate causal effect evaluation algorithm based on the causal structure identification result; it supports batch causal effect estimation, and calculates the causal effect for all nodes with causal relationships in the causal graph; for the calculated causal effect evaluation results, the credibility of the evaluation results is tested by the data subset verification method, and the evaluation results that pass the test will be saved, and the evaluation results that fail the test will be discarded; the causal effect evaluation results can be statistically classified, and the causal effect evaluation results of the statistical classification can be discussed and analyzed by experts.

[0011] The present invention provides a method for causal structure identification and causal effect estimation for complex systems, which has the following beneficial effects: This causal structure identification and causal effect estimation method for complex systems imports and analyzes various business data contents such as cross-linking rules and key indicators by connecting experimental data. In response to the current problems of insufficient utilization of data analysis generated in the simulation process of complex systems and lack of practice in causal structure identification and causal effect estimation under complex causal diagrams, a relatively complete causal structure identification and causal effect estimation method is proposed, thus solving the problem of difficulty in causal structure identification and causal effect estimation of complex systems, and providing important data support for improving system efficiency and improving optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flowchart of the complex system causal structure identification and causal effect estimation of the present invention; Figure 2 It is a flowchart of the evaluation scenario design provided by S1 of the present invention; Figure 3 It is a flowchart of the generation of the evaluation data set provided by S2 of the present invention; Figure 4 It is a flowchart of the causal relationship discovery and identification provided by S3 of the present invention; Figure 5 It is a flowchart of the causal effect estimation provided by S4 of the present invention. DETAILED DESCRIPTION

[0013] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0014] See also Figures 1 to 5 The present invention provides a technical solution: a method for causal structure identification and causal effect estimation for complex systems, comprising the following steps: S1: Build a causal structure identification and causal effect estimation system for complex systems; The causal relationship identification system for complex systems is built, which includes: causal data analysis module, causal structure identification module and causal effect estimation module; Causal data analysis module: used to import and analyze training and experiment data, provide common functions for data cleaning, and automatically construct causal graphs through PC algorithm and causal entropy algorithm; Causal structure identification module: used to design a causal structure identification algorithm based on d-separation and do-calculus according to the rules and conditions of backdoor criteria, frontdoor criteria and instrumental variables; Causal effect estimation module: used to identify results based on causal structure, design matching logic of causal effect algorithm, and provide verification function of causal effect estimation results; S2: Causal data parsing module imports parsed data files: Import and analyze system simulation experiment data in the form of database and file, and provide outlier removal, normalization, data interpolation and related data cleaning functions; automatically build causal graphs based on the analyzed data through PC algorithm and causal entropy algorithm, and import causal graph data into causal structure recognition module; The causal graph is automatically constructed through the PC algorithm and the causal entropy algorithm. The causal graph data format definition mainly includes two parts: nodes and edges. The node information includes the node ID information (used to identify the node) and the label information (the name information of the data represented), which is convenient for drawing and displaying the causal graph nodes; the edge information includes the starting node (dependent variable node) and the target node (result variable node) of the edge, which is convenient for determining the direction of each edge in the causal graph; S21: Based on the variables in the observed sample data, select any two variables x and y to conduct automatic causal relationship discovery. You can choose the PC algorithm and the causal entropy algorithm at will. Select the PC algorithm and enter S22; select the causal entropy algorithm and enter S23; S22: When the PC algorithm is selected, the algorithm first generates a fully connected undirected graph. For any two variables x and y, the variable Z other than x and y is used as a condition to perform an independence test on x and y. If x and y are independent, the edge relationship is removed; if they are related, the relationship is retained. After traversing all x and y, the V-shaped structure in the causal graph, that is, the collision, is identified. After determining the collision structure, the direction of some causal pairs can be determined. Finally, for the causal pairs for which the causal relationship cannot be determined, conditional entropy is used to determine the direction.

[0015] S23: When the causal entropy algorithm is selected, the algorithm first defines the causal entropy, finds the set of all causal pairs pointing to y through the aggregation algorithm, and then deletes the non-causal pairs through the progressive removal algorithm.

[0016] S24: Generate causal pairs with directions according to the algorithm and automatically draw the causal graph; S3: The causal structure recognition module recognizes the causal structure based on the causal graph data: Supports causal structure identification and batch causal structure identification of specified causal variables. The structural causal model and potential outcome framework define three types of criteria for causal variables: backdoor criteria, frontdoor criteria, and instrumental variables, as shown in Table 1: Table 1 Causal Principle Rule Definition Adjust variable set definition Backdoor Guidelines 1. All the backdoor paths between the dependent variable and the result variable are blocked, with all the variables in the adjustment variable set as conditions; 2. The adjustment variable set does not contain any child nodes of the dependent variable The set of variables that satisfy the backdoor criteria and participate in the causal effect estimation Front door guidelines 1. When all variables in the adjustment variable set are used as conditions, all directed paths from the dependent variable to the effect variable will be blocked; 2. There is no backdoor path for the causal relationship: dependent variable → adjustment variable set; 3. Using the dependent variable as a condition will block all backdoor paths for the adjustment variable set → effect variable. Variables that meet the front-door criteria and participate in causal effect estimation Instrumental variables 1. It is an exogenous variable; 2. It is directly related to the dependent variable; 3. It is independent of the confounding variables; 4. It has no direct causal relationship with the dependent variable A collection of instrumental variables that participate in causal effect estimation Different causal variables have different criteria, and the applicable causal effect estimation algorithms are also different. Therefore, it is necessary to first identify the structure to which each causal variable belongs, and thus define three causal variable structures: causal structures that meet the backdoor criterion, causal structures that meet the frontdoor criterion, and causal structures that meet the instrumental variables, combining d-separation, intervention calculus, and Bayesian network structure characteristics; The steps for identifying the causal structure of a design are as follows: S31: Check the completeness of the causal graph DAG: It is required that all edges in the causal graph are directed edges, and there is no closed loop path for any pair of nodes; S32: To test the causal structure to which the causal variable belongs, it is necessary to judge all three causal structures and save all causal structure identification results: S321: Identification of causal structures that meet the backdoor criteria; Using the relevant definition of the backdoor criteria, the specific identification steps are as follows: S3211: Check whether there is a directed path from the dependent variable to the result variable. If not, it means there is no causal relationship between the two nodes and no subsequent operations are performed; S3212: Find whether there is a backdoor adjustment variable set. This step uses the do calculation function to delete all outward-pointing edges of the dependent variable in the causal graph, that is, to control the dependent variable and obtain a new causal graph; S323: Use the causal graph obtained in S3212 to check whether the empty set is a backdoor adjustment variable set (the empty set can be used as a backdoor adjustment variable set, that is, there is no confounding variable between the causal variables). This process is implemented through the d-separation function. If the empty set can be used as a backdoor adjustment variable set, no subsequent operations are performed, otherwise proceed to the next step; S324: Use the original causal graph to check all other variable sets, remove the dependent variable node and its descendant nodes and the effect variable node, and only the observable variables meet the conditions of the backdoor adjustment variable set. If the empty set can separate the node in the adjustment variable set at this time from the cause / effect variable node d-, then remove this node from the adjustment variable set; S325: using the original causal graph, for the set of adjustment variables obtained in S324:, judging whether the nodes therein can separate the dependent variable and the result variable d-, and the set finally obtained is the set of adjustment variables that meets the backdoor criterion; S322: Identification of causal structures that meet the front-door criteria: Using the relevant definition of the front door criteria, the specific identification steps are as follows: S3221: Check whether there is a directed path from the dependent variable to the result variable. If not, it means there is no causal relationship between the two nodes and no subsequent operations are performed; S3222: Through the do calculation function, all the incoming edges of the dependent variable (the edges from all parent nodes pointing to the dependent variable) are deleted, and a new causal graph is obtained; S3223: Use the original causal graph to preliminarily screen the front-door adjustment variable set. The screening rule is: all descendant node sets of the dependent variable - the result variable set - all descendant node sets of the result variable, and take the intersection of the screened front-door adjustment variable set and all observable node sets; S3224: For the set of front-door adjustment variables screened by S3223, use the causal graph obtained by S3222 to determine whether each node therein can d-separate the dependent variable and the effect variable: a variable that meets the front-door criteria must be on a directed path between the dependent variable and the effect variable, that is, the variable will d-separate the dependent variable and the effect variable; S3225: For the set of front-door adjustment variables screened by S3224, the original causal graph is used, requiring that there is no back-door path between the set of front-door adjustment variables and the dependent variable (judged by the empty set); S3226: For the set of front-door adjustment variables screened by S3225, using the causal graph obtained by S3222, with the dependent variable as the condition (using the do calculation function to delete all the outgoing edges of the dependent variable), all back-door paths between the set of front-door adjustment variables and the result variable will be blocked, and finally the front-door adjustment variables will be obtained; S323: Identification of causal structures consistent with instrumental variables: Using the relevant definition of instrumental variables, the specific identification steps are as follows: S3231: Check whether there is a directed path from the dependent variable to the result variable. If not, it means there is no causal relationship between the two nodes and no subsequent operations are performed; S3232: Delete all the incoming edges of the dependent variable (edges from all parent nodes pointing to the dependent variable) through the do calculation function, and the resulting causal graph replaces the previous causal graph; S3233: remove the nodes in the parent node set of the dependent variable that exist in the parent node of the result variable as the candidate set of instrumental variables; S3234: remove the nodes in the descendant node sets of all parent nodes of the result variable from the candidate set obtained in S3233, which is the instrumental variable set; S4: The causal effect estimation module takes the parsed experimental data, causal structure identification results and adjustment variable set as input to calculate and test the causal effect; Match appropriate causal effect estimation algorithms for different causal structures and verify the accuracy of causal effect estimation results; support causal effect estimation of specified causal variables and batch causal effect estimation; cope with multiple causal structure identification results, support causal structure priority setting; The matching rules of the designed causal effect algorithm are as follows: S41: Calculation of causal effect estimates: According to the three definitions of causal structure identification results, the causal effect estimation calculation can also be divided into three cases: causal effect estimation based on backdoor criteria, causal effect estimation based on frontdoor criteria and causal effect estimation based on instrumental variables; S411: Causal effect estimation based on the backdoor criterion: When using the causal effect estimation algorithm based on the backdoor criterion, there are two prerequisites: the causal structure identification result is unique and is the backdoor criterion, and the causal structure identification result is not unique and the priority is the backdoor criterion; four causal effect estimation algorithms based on the backdoor criterion are provided: univariate linear regression, multivariate linear regression, propensity score stratification, and propensity score matching. The corresponding usage rules of the four algorithms are designed as follows: S4111: Univariate linear regression: The backdoor adjustment variable set is an empty set, and the outcome variable is set unchanged. The only dependent variable set that satisfies the backdoor adjustment variable set is the current dependent variable. S4112: Multiple linear regression: The backdoor adjustment variable set is an empty set, and the outcome variable is set unchanged. In addition to the current dependent variable, all dependent variable sets that satisfy the empty set of the backdoor adjustment variable set also contain other nodes; S4113: Propensity score stratification: The backdoor adjustment variable set is not an empty set and the number does not exceed 5. The dependent variable is one-dimensional and follows a binary distribution. The number of propensity stratification levels is 5 by default. Users can change the number of stratification levels uniformly as needed. S4114: Propensity score matching: The set of backdoor adjustment variables is not empty and the number is more than 5, and the dependent variable is one-dimensional and follows a bivariate distribution; S4112: Estimation of causal effects based on the front-door criterion: When using the causal effect estimation algorithm based on the front-door criterion, there are two prerequisites: the causal structure identification result is unique and is the front-door criterion, and the causal structure identification result is not unique and has the priority of the front-door criterion; the causal effect evaluation algorithm based on the front-door criterion supports the two-layer linear regression algorithm based on the front-door criterion; S4113: Estimation of causal effects based on instrumental variables: When using the causal effect estimation algorithm based on instrumental variables, there are two prerequisites: the causal structure identification result is unique and is an instrumental variable, and the causal structure identification result is not unique and has a priority of an instrumental variable. The causal effect evaluation algorithm based on instrumental variables supports a two-layer linear regression algorithm and a binary instrument / Wald estimation algorithm based on instrumental variables. The matching rules corresponding to the algorithm are designed as follows: S41131: Bivariate Instruments / Wald Estimation: Both the instrumental variable set and the dependent variable are one-dimensional. If the instrumental variable follows a bivariate distribution, the Wald estimation expression is used to calculate the causal effect; if the instrumental variable does not follow a bivariate distribution, the variance expression is used to calculate the causal effect. S41132: Two-level linear regression: The dimension of the instrumental variable or the dependent variable is greater than one, and the dimension of the instrumental variable set is greater than or equal to the dimension of the dependent variable; S42: Causal effect assessment results test: The causal effect estimation results are recalculated using a refutation method based on data subset verification. The significance test (p-value) is used to determine whether to accept the refutation calculation results. A threshold is set for the two calculation results. If the threshold is exceeded, it is considered that the deviation between the causal effect calculation result and the refutation calculation result is unacceptable, that is, it fails the test; According to the above steps, the causal structure identification and causal effect estimation method for complex systems is completed. The technical solution proposed in the present invention accurately extracts and identifies causal relationships through data analysis, providing technical support for improving system efficiency and improving optimization.

[0017] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for causal structure identification and causal effect estimation for complex systems, characterized by: The following steps are involved: S1: Build a causal structure identification and causal effect estimation system for complex systems; The causal relationship identification system for complex systems constructed includes: a causal data analysis module, a causal structure identification module and a causal effect estimation module; The causal data analysis module is used to import and analyze training experiment data, provide common functions for data cleaning, and automatically construct a causal graph through the PC algorithm and causal entropy algorithm; The causal structure identification module is used to design a causal structure identification algorithm based on d-separation and do-calculus according to the rules and conditions of the backdoor criteria, the frontdoor criteria and the instrumental variables; The causal effect estimation module is used to design the matching logic of the causal effect algorithm according to the causal structure identification results, and provide a verification function for the causal effect estimation results; S2: Causal data parsing module imports parsed data files: Import and analyze system simulation experiment data in the form of database and file, and provide outlier removal, normalization, data interpolation and related data cleaning functions; automatically build causal graphs based on the analyzed data through PC algorithm and causal entropy algorithm, and import causal graph data into causal structure recognition module; S3: The causal structure recognition module takes the causal graph data as input and can perform specified causal variable structure recognition and batch causal structure recognition; S4: The causal effect estimation module takes experimental data, causal structure identification results and a set of adjustment variables as input, calculates the estimated results of the causal effects between specified causal variables, and estimates the causal effects in batches, and verifies the credibility of the causal effect evaluation results through data subset verification.

2. The method for causal structure identification and causal effect estimation for complex systems according to claim 1, characterized in that: The S3: causal structure identification module takes the causal graph data as input, and can perform the steps of specifying the causal variable structure identification and batch causal structure identification. Before identifying the causal structure, it is first necessary to determine whether the causal graph satisfies the DAG completeness. If not, subsequent identification work cannot be performed. In order to realize the identification of the causal structure and batch causal structure belonging to any causal variable in the complex causal graph, based on the backdoor criterion, frontdoor criterion and instrumental variable rules, the causal structure is divided into: the causal structure that satisfies the backdoor criterion, the causal structure that satisfies the frontdoor criterion and the causal structure that satisfies the instrumental variable.

3. The method for causal structure identification and causal effect estimation for complex systems according to claim 1, characterized in that: When using the rules based on the backdoor criteria, front-door criteria and instrumental variables to identify the causal structure, logical rules are designed to identify the causal structure of the causal graph based on the Bayesian network characteristics of the causal graph, concepts such as d-separation and do-calculus, and the identification result of the causal structure and the corresponding adjustment variable sets are obtained: the adjustment variable set that meets the backdoor criteria, the adjustment variable set that meets the front-door criteria and the adjustment variable set that meets the instrumental variables, for the causal effect estimation module to evaluate the causal effect; causal variables can be selected for causal structure identification, and each node in the causal graph can be traversed to identify the causal structure between all causal variable nodes.

4. The method for causal structure identification and causal effect estimation for complex systems according to claim 1, characterized in that: When identifying the causal structure between the causal variables, no identification priority is set, and the causal structure identification module saves all structure identification results that meet the conditions.

5. The method for causal structure identification and causal effect estimation for complex systems according to claim 1, characterized in that: The S4: causal effect estimation module takes experimental data, causal structure identification results and adjustment variable sets as inputs, calculates the estimated results of the causal effects between specified causal variables, and estimates the causal effects in batches. In the step of verifying the credibility of the causal effect evaluation results through data subset verification, the applicable rules of the evaluation algorithm are designed according to the causal structure identification results obtained by the causal structure identification module and the corresponding adjustment variable set. For the causal structure that meets the backdoor criterion, causal effect evaluation algorithms such as univariate linear regression, multivariate linear regression, propensity score stratification, propensity score matching, etc. are provided, and a suitable evaluation algorithm is further allocated based on the adjustment variable set.

6. The method for causal structure identification and causal effect estimation for complex systems according to claim 1, characterized in that: For causal structures that meet the front-door criteria, a two-layer linear regression algorithm based on the front-door criteria is provided; for causal structures that meet instrumental variables, a two-layer linear regression algorithm based on instrumental variables and a binary instrument / Wald estimation algorithm are provided, and a suitable causal effect evaluation algorithm is further assigned according to the dimension and distribution information of the adjustment variable set and the dependent variable; for situations where the causal structure identification results are not unique, including causal structures that meet both the back-door criteria and instrumental variables, a priority for causal structure matching is designed, and users can design the priority order of causal structure identification results as needed. The system will assign a suitable causal effect evaluation algorithm to the causal variable according to the causal structure with the highest priority.

7. The method for causal structure identification and causal effect estimation for complex systems according to claim 1, characterized in that: Supports causal effect estimation of specified causal variables. Users can select appropriate causal effect evaluation algorithms based on causal structure identification results. Supports batch causal effect estimation, calculating causal effects for all nodes with causal relationships in the causal graph. The credibility of the calculated causal effect evaluation results will be tested through the data subset verification method. The evaluation results that pass the test will be saved, and the evaluation results that fail the test will be discarded. The causal effect evaluation results can be statistically classified and the statistically classified causal effect evaluation results can be discussed and analyzed by experts.

Images