Hidden variable identification method, device and equipment based on causal analysis and storage medium

Abstract

The invention discloses a hidden variable identification method and device based on causal analysis, equipment and a storage medium, and relates to the technical field of artificial intelligence and causal inference, in the whole process, information gain is used for dynamically quantifying a variable causal effect to realize hidden variable identification, so that a hidden variable judgment result can reflect the real existence of the hidden variable instead of data noise, and the hidden variable identification accuracy is improved. And the judgment accuracy is obviously improved. The method comprises the steps that iteration judgment is carried out on observation variables in an observation variable data set to obtain causal information gains of target observation variables obtained through iteration judgment, the observation variable data set comprises a plurality of observation variables, and the target observation variables are observation variables meeting shielding conditions in each iteration judgment process; the shielding condition is that no direct causal relationship exists between the observation variable and the target variable; calculating an average information gain and a final non-fitting error according to the causal information gain of the target observation variable; and if the average information gain is smaller than the final non-fitting error, determining that the target system has the hidden variable.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and causal inference, and in particular to a hidden variable identification method and device based on causal analysis, equipment and a storage medium. BACKGROUND

[0002] A hidden variable refers to a key implicit factor that has a direct or indirect influence on the correlation between system output or observed variables in a specific system, but due to measurement technology limitations, observation cost constraints, or conceptual abstraction, etc., its specific value or state cannot be directly obtained by existing detection means. Traditional methods such as structural equation models or latent class analysis can partially estimate hidden variables, but usually rely on model settings or distribution assumptions, and have poor adaptability to high-dimensional nonlinear systems.

[0003] In the related technical field, there are hidden variable identification schemes based on causal analysis, but traditional causal analysis methods usually imply the full observation variable assumption, i.e., it is assumed that all key factors that have an impact on the causal relationship are observable, or it relies on the assumption of Gaussian noise independence. However, in complex systems (such as biological regulatory networks, social behavior analysis, economic system prediction, etc.), there are often variables that cannot be directly observed or measured, leading to incomplete modeling of causal structures and inability to restore the true causal relationship.

[0004] In recent years, information theory methods (such as mutual information and KL divergence) have been introduced into the field of causal analysis to measure the information dependence or distribution difference between variables, but the core feature of a hidden variable is not directly observed, and its existence is only indirectly reflected by regulating the correlation between observed variables. The essence of mutual information and KL divergence is to characterize the global statistical correlation or distribution difference between observed variables, and only the existence possibility of a hidden variable can be inferred from the indirect statistical characteristics of observed data, which cannot achieve accurate identification and quantitative characterization of a hidden variable, and cannot meet the high-precision requirements of complex systems for hidden variable analysis. SUMMARY

[0005] Therefore, the present application provides a hidden variable identification method and device based on causal analysis, equipment and a storage medium, which mainly aims to solve the problem that the existing causal analysis method can only infer the existence possibility of a hidden variable from the indirect statistical characteristics of observed data, and cannot achieve accurate identification and quantitative characterization of a hidden variable, which cannot meet the high-precision requirements of complex systems for hidden variable analysis.

[0006] In a first aspect, a hidden variable identification method based on causal analysis is provided, which comprises: iteratively determining each observation variable in the observation variable dataset to obtain a causal information gain of a target observation variable obtained by the iterative determination, the observation variable dataset comprising a plurality of observation variables, the target observation variable being an observation variable satisfying a shielding condition in each iteration determination process, the shielding condition being that the observation variable has no direct causal relationship with the target variable; calculating an average information gain and a final un-fitting error according to the causal information gain of the target observation variable; if the average information gain is less than the final un-fitting error, determining that there is a hidden variable in a target system, the target system being a system represented by the observation variable dataset after shielding the target observation variable.

[0007] Further, the iteratively determining each observation variable in the observation variable dataset to obtain a causal information gain of a target observation variable obtained by the iterative determination comprises: obtaining a correlation between a target variable and an observation variable in the observation variable dataset; iteratively determining each observation variable in the observation variable dataset according to the correlation to obtain a causal information gain of a target observation variable obtained by the iterative determination.

[0008] Further, the obtaining a correlation between a target variable and an observation variable in the observation variable dataset comprises: setting a target variable and an observation variable in the observation variable dataset; calculating a correlation coefficient of the target variable and the observation variable to obtain the correlation between the target variable and the observation variable; Correspondingly, the iteratively determining each observation variable in the observation variable dataset according to the correlation to obtain a causal information gain of a target observation variable obtained by the iterative determination comprises: screening a candidate observation variable related to the target variable according to the correlation to iteratively determine each observation variable in the candidate observation variable to obtain a causal information gain of a target observation variable output by the iterative determination.

[0009] Further, the iteratively determining each observation variable in the observation variable dataset according to the correlation to obtain a causal information gain of a target observation variable obtained by the iterative determination comprises: in the process of iteratively determining each observation variable in the observation variable dataset, pre-shielding the observation variables in order of correlation between the target variable from low to high, and calculating a causal information gain of a current observation variable; If the causal information gain of the current observed variable is greater than zero, then the current observed variable is deleted from the set of observed variables, the current observed variable is determined as the target observed variable that meets the masking condition, the causal information gain of the target observed variable is obtained, the current iteration decision is skipped, and the pre-masking of observed variables in order of increasing correlation with the target variable is repeated. If the causal information gain of the current observed variable is less than or equal to zero, then the current observed variable is retained in the set of observed variables, and the next observed variable in the set of observed variables is iteratively determined until the current observed variable meets the masking condition.

[0010] Furthermore, the calculation of the causal information gain of the currently observed variable includes: Different causal hypothesis models are constructed using the observed variable dataset. The causal hypothesis models are used to express the causal relationship between the target variable and the current observed variable. The different causal hypothesis models include a first causal hypothesis model and a second causal hypothesis model. The first causal hypothesis model is used to assume that there is no direct causal relationship between the current observed variable and the target variable. The second causal hypothesis model is used to assume that there is a direct causal relationship between the current observed variable and the target variable. Based on pre-constructed causal hypothesis models, the observed variable dataset is used to fit the models, and the causal information gain of the current observed variable is calculated according to the prediction error of the different causal hypothesis models.

[0011] Furthermore, the step of using the observed variable dataset to fit the model based on pre-constructed causal hypothesis models, and calculating the causal information gain of the current observed variable according to the prediction error of the different causal hypothesis models, includes: Based on the pre-constructed first causal hypothesis model and second causal hypothesis model, the observed variable data are divided into training set and test set; The different causal hypothesis models are fitted using the partitioned training set to obtain the model fitting results. The model fitting results are evaluated using the partitioned test set to obtain the prediction error of the first causal hypothesis model and the prediction error of the second causal hypothesis model. The causal information gain of the current observed variable is calculated based on the prediction errors of the first causal hypothesis model and the second causal hypothesis model.

[0012] Further, after calculating the causal information gain of the current observed variable based on the prediction error of the first causal hypothesis model and the prediction error of the second causal hypothesis model, the method further includes: If the causal information gain of the current observed variable is greater than a set value, then it is determined that there is a direct causal relationship between the current observed variable and the target variable in the observed variable dataset; If the causal information gain of the current observed variable is less than or equal to a set value, then it is determined that there is no direct causal relationship between the current observed variable and the target variable in the observed variable dataset.

[0013] Secondly, a latent variable identification device based on causal analysis is provided, the device comprising: The acquisition unit is used to iteratively determine each observed variable in the observed variable dataset to obtain the causal information gain of the target observed variable obtained by the iterative determination. The observed variable dataset includes multiple observed variables, and the target observed variable is the observed variable that satisfies the masking condition in each iterative determination process. The masking condition is that there is no direct causal relationship between the observed variable and the target variable. The calculation unit is used to calculate the average information gain and the final unfit error based on the causal information gain of the target observed variable; The determining unit is used to determine that the target system has latent variables if the average information gain is less than the final unfit error, wherein the target system is a system represented by the dataset of observed variables after masking the target observed variables.

[0014] Further, the acquisition unit includes: The first acquisition module is used to acquire the correlation between the target variable and the observed variable in the observed variable dataset; The second acquisition module is used to iteratively determine each observed variable in the observed variable dataset according to the correlation, so as to obtain the causal information gain of the target observed variable obtained by the iterative determination.

[0015] Furthermore, the first acquisition module is specifically used for: Define the target variable and the observed variables in the observed variable dataset; Calculate the correlation coefficient between the target variable and the observed variable to obtain the correlation between the target variable and the observed variable; The second acquisition module is specifically used for: Candidate observed variables related to the target variable are selected based on the correlation, and each observed variable among the candidate observed variables is iteratively determined to obtain the causal information gain of the target observed variable output by the iterative determination.

[0016] Furthermore, the second acquisition module is specifically used for: During the iterative determination of each observed variable in the observed variable dataset, the observed variables are pre-masked in order of increasing correlation with the target variable, and the causal information gain of the current observed variable is calculated. If the causal information gain of the current observed variable is greater than zero, then the current observed variable is deleted from the set of observed variables, the current observed variable is determined as the target observed variable that meets the masking condition, the causal information gain of the target observed variable is obtained, the current iteration decision is skipped, and the pre-masking of observed variables in order of increasing correlation with the target variable is repeated. If the causal information gain of the current observed variable is less than or equal to zero, then the current observed variable is retained in the set of observed variables, and the next observed variable in the set of observed variables is iteratively determined until the current observed variable meets the masking condition.

[0017] Furthermore, the second acquisition module is specifically used for: Different causal hypothesis models are constructed using the observed variable dataset. The causal hypothesis models are used to express the causal relationship between the target variable and the current observed variable. The different causal hypothesis models include a first causal hypothesis model and a second causal hypothesis model. The first causal hypothesis model is used to assume that there is no direct causal relationship between the current observed variable and the target variable. The second causal hypothesis model is used to assume that there is a direct causal relationship between the current observed variable and the target variable. Based on pre-constructed causal hypothesis models, the observed variable dataset is used to fit the models, and the causal information gain of the current observed variable is calculated according to the prediction error of the different causal hypothesis models.

[0018] Furthermore, the second acquisition module is specifically used for: Based on the pre-constructed first causal hypothesis model and second causal hypothesis model, the observed variable data are divided into training set and test set; The different causal hypothesis models are fitted using the partitioned training set to obtain the model fitting results. The model fitting results are evaluated using the partitioned test set to obtain the prediction error of the first causal hypothesis model and the prediction error of the second causal hypothesis model. The causal information gain of the current observed variable is calculated based on the prediction errors of the first causal hypothesis model and the second causal hypothesis model.

[0019] Furthermore, the second acquisition module is specifically used for: After calculating the causal information gain of the current observed variable based on the prediction error of the first causal hypothesis model and the prediction error of the second causal hypothesis model, if the causal information gain of the current observed variable is greater than a set value, then it is determined that there is a direct causal relationship between the current observed variable and the target variable in the observed variable dataset. If the causal information gain of the current observed variable is less than or equal to a set value, then it is determined that there is no direct causal relationship between the current observed variable and the target variable in the observed variable dataset.

[0020] Thirdly, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the aforementioned latent variable identification method based on causal analysis.

[0021] Fourthly, a latent variable identification device based on causal analysis is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor. When the processor executes the program, it implements the aforementioned latent variable identification method based on causal analysis.

[0022] By employing the above technical solution, this application provides a method, apparatus, device, and storage medium for identifying latent variables based on causal analysis. Compared with current methods that use information theory to identify latent variables, this application iteratively determines each observed variable in the observed variable dataset to obtain the causal information gain of the target observed variable obtained through iterative determination. The observed variable dataset includes multiple observed variables, and the target observed variable is the observed variable that satisfies the masking condition in each iterative determination process. The masking condition is that there is no direct causal relationship between the observed variable and the target variable. Based on the causal information gain of the target observed variable, the average information gain and the final unfit error are calculated. If the average information gain is less than the final unfit error, it is determined that the target system has latent variables, and the target system is the system represented by the observed variable dataset after masking the target observed variable. The entire process uses information gain to dynamically quantify the causal effect of each observed variable in the dataset on the target variable, thereby shielding target observed variables that do not have a direct causal contribution to the target variable and focusing on the set of observed variables that make a true causal contribution. This reduces the interference of spurious associations on the determination of latent variables from the source. At the same time, the average information gain and the final misfit error are introduced for quantitative comparison. If the cumulative causal information of the target observed variable is insufficient to fully explain the change of the target variable, it is determined that there is a latent variable in the target system. This allows the determination result of latent variables to reflect their true existence rather than data noise, which significantly improves the accuracy of the determination.

[0023] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart illustrating a latent variable identification method based on causal analysis in one embodiment of this application; Figure 2 yes Figure 1 A flowchart illustrating a specific implementation method of step 101; Figure 3 yes Figure 2 A flowchart illustrating a specific implementation method for step 202; Figure 4 yes Figure 3 A flowchart illustrating a specific implementation method of step 301; Figure 5 This is a flowchart of the latent variable identification process based on causal analysis in one embodiment of this application; Figure 6 This is a flowchart of the causal information gain calculation process in one embodiment of this application; Figure 7 This is a schematic diagram of the latent variable identification device based on causal analysis in one embodiment of this application; Figure 8 This is a schematic diagram of the device structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0025] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0026] In related technologies, information theory methods (such as mutual information and KL divergence) have been introduced into the field of causal analysis to measure information dependence or distributional differences between variables. However, the core characteristic of latent variables is that they are not directly observed, but their existence is indirectly reflected by regulating the relationship between observed variables. The essence of mutual information and KL divergence is to characterize the global statistical association or distributional differences between observed variables. They can only infer the possibility of the existence of latent variables through the indirect statistical characteristics of observed data, and cannot achieve accurate identification and quantitative characterization of latent variables, making it difficult to meet the high-precision requirements of latent variable analysis for complex systems.

[0027] To address this problem, this embodiment provides a latent variable identification method based on causal analysis, such as... Figure 1 As shown, it includes the following steps: 101. Perform iterative judgment on each observed variable in the observed variable dataset to obtain the causal information gain of the target observed variable obtained by the iterative judgment.

[0028] In this embodiment, the observed variable dataset includes multiple observed variables. The target observed variable is the observed variable that satisfies the masking condition in each iteration of the determination process. The masking condition is that there is no direct causal relationship between the observed variable and the target variable.

[0029] Specifically, the observed variable dataset refers to the collection of data used to analyze all original observable variables, such as gene expression data, social science survey data, meteorological data, or mechanical system monitoring data. The original observable variables here are data carriers directly obtained from the target system through detection, measurement, and other means. The core of this dataset is that it is directly observable, numerically quantifiable, and includes various types such as effective observed variables that have a causal contribution to the target variable and target observed variables that do not have a causal contribution.

[0030] In order to gradually remove target observed variables that do not contribute causally from the set of observed variables, it is necessary to iteratively judge each observed variable in the set of observed variables. Specifically, the iterative judgment process refers to screening the observed variables in the set of observed variables round by round according to the set rules, so as to avoid missing key effective observed variables or misjudging the target observed variable in a single screening, and to determine that the observed variables finally retained only include effective observed variables that have a direct causal relationship with the target variable.

[0031] Specifically, in each iteration, all observed variables in the observed variable data are treated as variables to be determined. Based on the masking condition, it is sequentially determined whether there is a direct causal relationship between the variables to be determined and the target variable. If there is no direct causal relationship between the variables to be determined and the target variable, the variables to be determined are identified as the target observed variables and removed from the observed variable data. The current iteration is then terminated. After removing the target observed variable, the variables to be determined are retrieved again, and the above iteration process is repeated. Based on the masking condition, it is sequentially determined whether there is a direct causal relationship between the variables to be determined and the target variable, until there are no observed variables in the variables to be determined that satisfy the masking condition. It can be understood that each iteration ends when the target observed variable is detected in the observed variable data. After the target observed variable is detected, the causal information gain of the target observed variable is calculated. Here, the causal information gain quantifies the direct causal contribution of a single observed variable to the target variable. Its essence is the incremental causal information provided by the target observed variable to explain the uncertainty of the target variable, excluding indirect associations, spurious associations, and confounding interference. Generally speaking, the causal information gain of the target observed variable is less than or equal to a set threshold, which is close to 0.

[0032] 102. Calculate the average information gain and the final unfit error based on the causal information gain of the target observed variable.

[0033] Understandably, after each iteration of the decision process selects the target observed variable, the causal information gain of the target observed variable is recorded. Correspondingly, multiple iterations of the decision process will record the causal information gain of multiple target observed variables. The average information gain is obtained by averaging the cumulative causal information of all target observed variables. For example, after iterative decision-making, the dataset of observed variables yields the causal information gains of 6 target observed variables, and the corresponding average information gain is obtained by averaging the causal information gains of the 6 target observed variables.

[0034] In this embodiment, the final misfit error is the uncertainty remaining in the target variable after all target observed variables have been removed. It characterizes the gap in the target variable uncertainty that the total direct causal contribution of the observed variable dataset after removing the target observed variable still cannot cover. Specifically, it can be calculated based on the conditional entropy quantification in information theory, using the observed variable dataset after removing the target observed variable as the basis. The corresponding conditional entropy is the final misfit error, which is mainly used to measure the uncertainty that still exists in the target variable under the premise of the effective observed variable values ​​after removing the target observed variable.

[0035] 103. If the average information gain is less than the final unfit error, then it is determined that the target system has latent variables.

[0036] In this embodiment, the average information gain is compared with the final misfit error. Essentially, this is done by quantitatively comparing the average contribution strength of the causal variables that mask the target variable in the observed variable dataset with the unexplained uncertainty gap, in order to determine whether the causal information of the observed variable dataset after masking the target variable is complete, and thus infer whether the latent variable exists.

[0037] If the average information gain is less than the final misfit error, it indicates that the existing causal information is insufficient to cover the changes in the target variable. This gap needs to be filled by unobserved latent variables, further indicating that there are latent variables in the target system.

[0038] If the average information gain is greater than or equal to the final unfit error, it indicates that the average causal contribution of the target variable has been able to cover the remaining uncertainty gap. In this case, the unexplained uncertainty may be caused only by data noise or small modeling errors, rather than latent variables, further indicating that there are no latent variables in the target system.

[0039] Here, the system is the real physical or logical system to be analyzed, such as a biological gene regulation system or an industrial equipment operating system. During the screening process, after masking the target observed variable in the observed variable dataset, only valid observed variables with a direct causal relationship to the target variable are retained. At this point, the target system is the system represented by the observed variable dataset after masking the target observed variable. This target system focuses on valid observed variables that can reflect the true causal relationship of the target system. By analyzing the completeness of the causal information in the target system, it is determined whether latent variables exist in the target system.

[0040] The latent variable identification method based on causal analysis provided in this application differs from current methods that use information theory to identify latent variables. This application iteratively determines the causal information gain of the target observed variable by performing iterative judgments on each observed variable in the observed variable dataset. The observed variable dataset includes multiple observed variables, and the target observed variable is the observed variable that satisfies the masking condition in each iterative judgment process. The masking condition is that there is no direct causal relationship between the observed variable and the target variable. Based on the causal information gain of the target observed variable, the average information gain and the final unfit error are calculated. If the average information gain is less than the final unfit error, it is determined that the target system has latent variables, and the target system is the system represented by the observed variable dataset after masking the target observed variable. The entire process uses information gain to dynamically quantify the causal effect of each observed variable in the dataset on the target variable, thereby shielding target observed variables that do not have a direct causal contribution to the target variable and focusing on the set of observed variables that make a true causal contribution. This reduces the interference of spurious associations on the determination of latent variables from the source. At the same time, the average information gain and the final misfit error are introduced for quantitative comparison. If the cumulative causal information of the target observed variable is insufficient to fully explain the change of the target variable, it is determined that there is a latent variable in the target system. This allows the determination result of latent variables to reflect their true existence rather than data noise, which significantly improves the accuracy of the determination.

[0041] In practical applications, causality inevitably leads to correlation, and strong correlations are more likely to stem from statistical regularities governing the core role of causality. For example, in industrial systems, the correlation coefficient between equipment operating temperature and the occurrence of failures is much higher than the correlation coefficient between ambient humidity and the probability of failure. In other words, the stronger the correlation between the observed variable and the target variable, the higher the probability that a causal relationship exists between them. Specifically, such as... Figure 2 As shown, step 101 includes the following steps: 201. Obtain the correlation between the target variable and the observed variables in the observed variable dataset.

[0042] 202. Based on the correlation, perform iterative determination on each observed variable in the observed variable dataset to obtain the causal information gain of the target observed variable obtained by the iterative determination.

[0043] In this embodiment, the correlation coefficient between the target variable and the observed variable is obtained by calculating the correlation coefficient using the Pearson correlation coefficient or partial correlation coefficient. Specifically, a target variable and observed variables are defined in the observed variable dataset; the correlation coefficient between the target variable and the observed variables is calculated to obtain the correlation between them. This process can be performed using the following formula:

[0044] in, For covariance, For the observation vector, For the target variable, The standard deviation is denoted as .

[0045] Accordingly, candidate observed variables related to the target variable are screened based on their correlation, and each observed variable among the candidate observed variables is iteratively judged to obtain the causal information gain of the target observed variable output by the iterative judgment.

[0046] Understandably, the correlation coefficient is a statistic that quantifies the degree of association between an observed variable and a target variable. A correlation coefficient with an absolute value close to 1 indicates that the changes in the expressions of the two variables are more synchronized, while a correlation coefficient with an absolute value close to 0 indicates a weaker association. Here, a threshold coefficient can be set. In other words, correlation can reflect, to a certain extent, whether there is a direct causal relationship between the observed variable and the target variable. If an iterative judgment process is directly applied to all observed variables, it will lead to a large number of invalid variables consuming resources. Correlation screening can quickly eliminate observed variables that are completely unrelated to the target variable. These observed variables do not need to be included in the iterative judgment process, significantly reducing the complexity of iterative judgment and causal information gain calculation, and improving the efficiency of the technical process. Correspondingly, observed variables with a correlation coefficient greater than the set threshold can be added to the candidate observed variables as observed variables that are associated with the target variable.

[0047] In practical applications, the process of iteratively determining the correlation between observed variables in a dataset essentially transforms the priority of potential causal contributions into a variable determination order. This ensures that low-probability observed variables are analyzed first, followed by high-probability observed variables. This allows for the priority screening of less correlated observed variables when they meet the screening criteria. Specifically, for example... Figure 3 As shown, step 202 includes the following steps: 301. During the iterative determination of each observed variable in the observed variable dataset, the observed variables are pre-masked in order of increasing correlation with the target variable, and the causal information gain of the current observed variable is calculated.

[0048] 302. If the causal information gain of the current observed variable is greater than zero, then delete the current observed variable from the set of observed variables, determine the current observed variable as the target observed variable that meets the masking condition, obtain the causal information gain of the target observed variable, jump out of the current iteration decision, and repeat the pre-masking of observed variables in order of increasing correlation with the target variable.

[0049] 303. If the causal information gain of the current observed variable is less than or equal to zero, the current observed variable is retained in the set of observed variables, and the next observed variable in the set of observed variables is iteratively determined until the current observed variable meets the masking condition.

[0050] In this embodiment, the order of correlation with the target variable from low to high is to first mask low-probability observed variables and retain high-probability observed variables, which significantly improves the efficiency of iterative judgment.

[0051] It should be noted that each iteration of the determination process requires pre-masking the observed variables in a predetermined order, which is the order of correlation between the observed variables and the target variable from low to high. Specifically, in the current round of determination, based on the observed variable dataset, the observed variables are pre-masked in the predetermined order. The causal information gain is calculated for the current observed variable. If the causal information gain of the current observed variable is greater than zero, it means that the current observed variable is the target observed variable that meets the masking condition. Accordingly, the target observed variable is deleted from the observed variable dataset, the determination process of the current round ends, the causal information gain of the target observed variable is recorded, and the determination process proceeds to the next round. Similarly, based on the observed variable dataset after the target variable is masked, the observed variables are pre-masked in the predetermined order, and the determination process of the previous round is repeated. If the causal information gain of the current observed variable is less than or equal to zero, it means that the current observed variable does not meet the masking condition, and the causal information gain is calculated for the next observed variable.

[0052] For example, the observed variable dataset includes five observed variables A1, A2, A3, A4, and A5, sorted from low to high relevance to obtain A2, A3, A1, A4, and A5. In the first iteration, after pre-masking observed variable A1, the causal information gain of observed variable A1 is calculated to be greater than zero, indicating that observed variable A1 is a target observed variable that meets the masking condition. The causal information gain of the target observed variable A1 is obtained, and the target observed variable A1 is deleted from the observed variable set accordingly. The second iteration proceeds, after pre-masking observed variable A2, the causal information gain of observed variable A2 is calculated to be less than zero, indicating that observed variable A2 does not meet the masking condition. The next observed variable, A3, is then pre-masked. After pre-masking observed variable A3, the causal information gain of observed variable A3 is calculated. If the information gain is greater than zero, it indicates that the observed variable A3 is a target observed variable that meets the masking condition. The causal information gain of the target observed variable A3 is obtained, and A3 is removed from the observed variable set accordingly. The process then proceeds to the third iteration. After pre-masking observed variable A2, if the causal information gain of A2 is less than zero, it indicates that A2 does not meet the masking condition. The process continues to pre-mask the next observed variable A4. If the causal information gain of A4 is less than zero, it indicates that A4 does not meet the masking condition. The process continues to pre-mask the next observed variable A5. If the causal information gain of A5 is less than zero, it indicates that A5 does not meet the masking condition. At this point, no observed variable in the current observed variable dataset meets the masking condition, and the iteration process ends.

[0053] In practical applications, observed variable datasets often contain interference, such as confounding variables (variables that simultaneously affect both the observed and target variables) and data noise. Combining different causal hypothesis models can more efficiently eliminate interference. Specifically, for example... Figure 4 As shown, step 301 includes the following steps: 401. Construct different causal hypothesis models using the observed variable dataset.

[0054] 402. Based on the pre-constructed causal hypothesis models, the observed variable dataset is used to fit the model, so as to calculate the causal information gain of the current observed variable according to the prediction error of the different causal hypothesis models.

[0055] In this embodiment, the causal hypothesis model is used to express the causal relationship between the target variable and the currently observed variable. Different causal hypothesis models include a first causal hypothesis model and a second causal hypothesis model. The first causal hypothesis model is used to assume that there is no direct causal relationship between the currently observed variable and the target variable, while the second causal hypothesis model is used to assume that there is a direct causal relationship between the currently observed variable and the target variable.

[0056] The above first causal hypothesis model The following formula can be used to represent it:

[0057] in, To exclude observed variables With target variable Other variables besides This represents the prediction error of the first causal hypothesis model.

[0058] The above second causal hypothesis model The following formula can be used to represent it:

[0059] in, To exclude observed variables With target variable Other variables besides This represents the prediction error of the second causal hypothesis model.

[0060] Specifically, in calculating the causal information gain of the current observed variable based on the prediction errors of different causal hypothesis models, the observed variable data can be divided into training and test sets after pre-constructing a first causal hypothesis model and a second causal hypothesis model. The training set is used to fit the different causal hypothesis models, yielding the model fitting results. The test set is used to evaluate the model fitting results, obtaining the prediction errors of the first and second causal hypothesis models. Based on the prediction errors of the first and second causal hypothesis models, the causal information gain of the current observed variable is calculated. The division of the training and test sets can be achieved using K-fold cross-validation or leave-one-out cross-validation to ensure the robustness of the model evaluation.

[0061] Specifically, in each partition, the first causal hypothesis model and the second causal hypothesis model are trained separately, and the prediction error of the causal hypothesis model is calculated using the mean squared error. It can be expressed using the following formula:

[0062] in, For the true value, For predicted values, This represents the number of samples.

[0063] Accordingly, the prediction error of the first causal hypothesis model is calculated using the above formulas. Prediction error of the second causal hypothesis model .

[0064] Furthermore, by comparing the first causal hypothesis model... Prediction error Second causal hypothesis model Prediction error Calculate causal information gain It can be expressed using the following formula: -

[0065] Correspondingly, if the causal information gain of the current observed variable is greater than a set value, then it is determined that there is a direct causal relationship between the current observed variable and the target variable in the observed variable dataset; that is, when When, it indicates that the currently observed variable is introduced. It can significantly reduce prediction error, and is considered right There is a direct causal relationship.

[0066] If the causal information gain of the current observed variable is less than or equal to a set value, then it is determined that there is no direct causal relationship between the current observed variable and the target variable in the observed variable dataset; that is, when When, it indicates that the currently observed variable is introduced. It does not significantly reduce prediction error, and is considered right There is no direct causal relationship.

[0067] Furthermore, each iterative decision-making process records the causal information gain of the target observed variable. The corresponding multiple iterations will record the causal information gain of multiple target observation variables. Calculate the average information gain It can be expressed using the following formula:

[0068] in, The number of observed variables for the target.

[0069] Correspondingly, if the average information gain Greater than the final unfit error This indicates that the target system has error increments that cannot be explained by existing observed variables, meaning that the target system has unobserved latent variables. If the average information gain... Less than or equal to the final unfit error This indicates that the observed variables of the target system can explain the error increment, meaning that there are no unobserved latent variables in the target system.

[0070] In practical applications, the specific process of identifying latent variables based on causal analysis can be found in [reference needed]. Figure 5 As shown, Figure 5The overall process of latent variable identification in this application is systematically demonstrated. Figure 5 The flowchart in section A illustrates the process of latent variable identification from a macro perspective. First, it receives the input of observed variables, calculates and sorts the correlation between each observed variable and the target variable, and then identifies the potential latent variables one by one. Figure 5 The B-block diagram serves as the core execution process for latent variable identification. Through multiple iterations, it calculates the causal information gain in each iteration fitting process and compares the mean of the causal information gain error obtained from multiple iterations with the final unfit error. Based on the comparison results, it determines whether the target variable has latent variables.

[0071] In practical applications, the specific process for calculating causal gain information can be found in [reference needed]. Figure 6 As shown, the difference in fitting loss is regarded as causal gain information. Cross-validation is used to compare the causal gain information of the masked observed variables. If the causal gain information is less than or equal to zero, it means that the observed variable has a causal contribution to the target variable and the observed variable is retained. Otherwise, the observed variable is deleted.

[0072] Furthermore, as a specific implementation of the above method, embodiments of this application provide a latent variable identification device based on causal analysis, such as... Figure 7 As shown, the device includes: an acquisition unit 51, a calculation unit 52, and a determination unit 53.

[0073] The acquisition unit 51 is used to iteratively determine each observed variable in the observed variable dataset to obtain the causal information gain of the target observed variable obtained by the iterative determination. The observed variable dataset includes multiple observed variables, and the target observed variable is the observed variable that satisfies the masking condition in each iterative determination process. The masking condition is that there is no direct causal relationship between the observed variable and the target variable. The calculation unit 52 is used to calculate the average information gain and the final unfit error based on the causal information gain of the target observed variable; The determining unit 53 is used to determine that the target system has latent variables if the average information gain is less than the final unfit error, wherein the target system is a system represented by the dataset of observed variables after masking the target observed variables.

[0074] The latent variable identification device based on causal analysis provided in this invention, compared with the current method of identifying latent variables using information theory, obtains the causal information gain of the target observed variable by iteratively judging each observed variable in the observed variable dataset. The observed variable dataset includes multiple observed variables, and the target observed variable is the observed variable that satisfies the masking condition in each iterative judgment process. The masking condition is that there is no direct causal relationship between the observed variable and the target variable. Based on the causal information gain of the target observed variable, the average information gain and the final unfit error are calculated. If the average information gain is less than the final unfit error, it is determined that there is a latent variable in the target system, and the target system is the system represented by the observed variable dataset after masking the target observed variable. The entire process uses information gain to dynamically quantify the causal effect of each observed variable in the dataset on the target variable, thereby shielding target observed variables that do not have a direct causal contribution to the target variable and focusing on the set of observed variables that make a true causal contribution. This reduces the interference of spurious associations on the determination of latent variables from the source. At the same time, the average information gain and the final misfit error are introduced for quantitative comparison. If the cumulative causal information of the target observed variable is insufficient to fully explain the change of the target variable, it is determined that there is a latent variable in the target system. This allows the determination result of latent variables to reflect their true existence rather than data noise, which significantly improves the accuracy of the determination.

[0075] In specific application scenarios, the acquisition unit includes: The first acquisition module is used to acquire the correlation between the target variable and the observed variable in the observed variable dataset; The second acquisition module is used to iteratively determine each observed variable in the observed variable dataset according to the correlation, so as to obtain the causal information gain of the target observed variable obtained by the iterative determination.

[0076] In specific application scenarios, the first acquisition module is specifically used for: Define the target variable and the observed variables in the observed variable dataset; Calculate the correlation coefficient between the target variable and the observed variable to obtain the correlation between the target variable and the observed variable; The second acquisition module is specifically used for: Candidate observed variables related to the target variable are selected based on the correlation, and each observed variable among the candidate observed variables is iteratively determined to obtain the causal information gain of the target observed variable output by the iterative determination.

[0077] In specific application scenarios, the second acquisition module is further used for: During the iterative determination of each observed variable in the observed variable dataset, the observed variables are pre-masked in order of increasing correlation with the target variable, and the causal information gain of the current observed variable is calculated. If the causal information gain of the current observed variable is greater than zero, then the current observed variable is deleted from the set of observed variables, the current observed variable is determined as the target observed variable that meets the masking condition, the causal information gain of the target observed variable is obtained, the current iteration decision is skipped, and the pre-masking of observed variables in order of increasing correlation with the target variable is repeated. If the causal information gain of the current observed variable is less than or equal to zero, then the current observed variable is retained in the set of observed variables, and the next observed variable in the set of observed variables is iteratively determined until the current observed variable meets the masking condition.

[0078] In specific application scenarios, the second acquisition module is further used for: Different causal hypothesis models are constructed using the observed variable dataset. The causal hypothesis models are used to express the causal relationship between the target variable and the current observed variable. The different causal hypothesis models include a first causal hypothesis model and a second causal hypothesis model. The first causal hypothesis model is used to assume that there is no direct causal relationship between the current observed variable and the target variable. The second causal hypothesis model is used to assume that there is a direct causal relationship between the current observed variable and the target variable. Based on pre-constructed causal hypothesis models, the observed variable dataset is used to fit the models, and the causal information gain of the current observed variable is calculated according to the prediction error of the different causal hypothesis models.

[0079] In specific application scenarios, the second acquisition module is further used for: Based on the pre-constructed first causal hypothesis model and second causal hypothesis model, the observed variable data are divided into training set and test set; The different causal hypothesis models are fitted using the partitioned training set to obtain the model fitting results. The model fitting results are evaluated using the partitioned test set to obtain the prediction error of the first causal hypothesis model and the prediction error of the second causal hypothesis model. The causal information gain of the current observed variable is calculated based on the prediction errors of the first causal hypothesis model and the second causal hypothesis model.

[0080] In specific application scenarios, the second acquisition module is further used for: After calculating the causal information gain of the current observed variable based on the prediction error of the first causal hypothesis model and the prediction error of the second causal hypothesis model, if the causal information gain of the current observed variable is greater than a set value, then it is determined that there is a direct causal relationship between the current observed variable and the target variable in the observed variable dataset. If the causal information gain of the current observed variable is less than or equal to a set value, then it is determined that there is no direct causal relationship between the current observed variable and the target variable in the observed variable dataset.

[0081] Based on the above, Figures 1-4 Accordingly, this application embodiment also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described method. Figures 1-4 The method for identifying latent variables based on causal analysis is shown.

[0082] Based on this understanding, the technical solution of this application can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive), and includes several instructions to cause a computer device (such as a personal computer, server, or network device) to execute the methods described in the various implementation scenarios of this application.

[0083] Based on the above, Figures 1-4 The method shown, and Figure 7 To achieve the above objectives, the virtual device embodiment shown in this application also provides a physical device for identifying latent variables based on causal analysis. Specifically, this physical device can be a computer, smartphone, tablet, smartwatch, server, or network device, etc. The physical device includes a storage medium and a processor; the storage medium stores a computer program; the processor executes the computer program to achieve the above-described... Figures 1-4 The method for identifying latent variables based on causal analysis is shown.

[0084] Optionally, the physical device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0085] In an exemplary embodiment, see Figure 8The aforementioned physical device includes a communication bus, a processor, a memory, and a communication interface. It may also include an input / output interface and a display device. The various functional units can communicate with each other via the bus. The memory stores a computer program, and the processor executes the program stored in the memory to perform the latent variable identification method based on causal analysis described in the above embodiments.

[0086] Those skilled in the art will understand that the entity device structure for latent variable identification based on causal analysis provided in this embodiment does not constitute a limitation on the entity device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0087] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the entity device identified based on causal analysis of latent variables, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing entity device.

[0088] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware platforms, or it can be implemented in hardware. By applying the technical solution of this application, compared with the existing methods, this application uses information gain to dynamically quantify the causal effect of each observed variable in the observed variable dataset on the target variable, so as to shield the target observed variables that have no direct causal contribution to the target variable, focus on the set of observed variables with real causal contribution, and reduce the interference of false associations on the determination of latent variables from the source; at the same time, the average information gain and the final misfit error are introduced for quantitative comparison. If the cumulative causal information of the target observed variable is insufficient to fully explain the change of the target variable, it is determined that there is a latent variable in the target system, so that the latent variable determination result can reflect its real existence rather than data noise, which significantly improves the determination accuracy.

[0089] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application. Those skilled in the art will understand that the modules in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or can be modified to be located in one or more apparatuses different from this embodiment. The modules of the above-described embodiment can be combined into one module, or further divided into multiple sub-modules.

[0090] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of any particular implementation scenario. The above disclosures are merely a few specific implementation scenarios of this application; however, this application is not limited thereto, and any variations conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A method for identifying latent variables based on causal analysis, characterized in that, include: Each observed variable in the observed variable dataset is iteratively determined to obtain the causal information gain of the target observed variable obtained by the iterative determination. The observed variable dataset includes multiple observed variables, and the target observed variable is the observed variable that satisfies the masking condition in each iterative determination process. The masking condition is that there is no direct causal relationship between the observed variable and the target variable. Calculate the average information gain and the final unfit error based on the causal information gain of the target observed variable; If the average information gain is less than the final unfit error, then it is determined that the target system has latent variables, and the target system is the system represented by the dataset of observed variables after masking the target observed variables.

2. The method according to claim 1, characterized in that, The iterative determination of each observed variable in the observed variable dataset to obtain the causal information gain of the target observed variable obtained by the iterative determination includes: Obtain the correlation between the target variable and the observed variables in the observed variable dataset; Based on the correlation, each observed variable in the observed variable dataset is iteratively determined to obtain the causal information gain of the target observed variable obtained by the iterative determination.

3. The method according to claim 2, characterized in that, The process of obtaining the correlation between the target variable and the observed variables in the observed variable dataset includes: Define the target variable and the observed variables in the observed variable dataset; Calculate the correlation coefficient between the target variable and the observed variable to obtain the correlation between the target variable and the observed variable; Accordingly, the iterative determination of each observed variable in the observed variable dataset based on the correlation to obtain the causal information gain of the target observed variable obtained by the iterative determination includes: Candidate observed variables related to the target variable are selected based on the correlation, and each observed variable among the candidate observed variables is iteratively determined to obtain the causal information gain of the target observed variable output by the iterative determination.

4. The method according to claim 2, characterized in that, The step of iteratively determining each observed variable in the observed variable dataset based on the correlation to obtain the causal information gain of the target observed variable obtained by the iterative determination includes: During the iterative determination of each observed variable in the observed variable dataset, the observed variables are pre-masked in order of increasing correlation with the target variable, and the causal information gain of the current observed variable is calculated. If the causal information gain of the current observed variable is greater than zero, then the current observed variable is deleted from the set of observed variables, the current observed variable is determined as the target observed variable that meets the masking condition, the causal information gain of the target observed variable is obtained, the current iteration decision is skipped, and the pre-masking of observed variables in order of increasing correlation with the target variable is repeated. If the causal information gain of the current observed variable is less than or equal to zero, then the current observed variable is retained in the set of observed variables, and the next observed variable in the set of observed variables is iteratively determined until the current observed variable meets the masking condition.

5. The method according to claim 4, characterized in that, The calculation of the causal information gain of the currently observed variable includes: Different causal hypothesis models are constructed using the observed variable dataset. The causal hypothesis models are used to express the causal relationship between the target variable and the current observed variable. The different causal hypothesis models include a first causal hypothesis model and a second causal hypothesis model. The first causal hypothesis model is used to assume that there is no direct causal relationship between the current observed variable and the target variable. The second causal hypothesis model is used to assume that there is a direct causal relationship between the current observed variable and the target variable. Based on pre-constructed causal hypothesis models, the observed variable dataset is used to fit the models, and the causal information gain of the current observed variable is calculated according to the prediction error of the different causal hypothesis models.

6. The method according to claim 5, characterized in that, The step of fitting the model to a dataset of observed variables based on pre-constructed causal hypothesis models, and calculating the causal information gain of the current observed variable according to the prediction errors of the different causal hypothesis models, includes: Based on the pre-constructed first causal hypothesis model and second causal hypothesis model, the observed variable data are divided into training set and test set; The different causal hypothesis models are fitted using the partitioned training set to obtain the model fitting results. The model fitting results are evaluated using the partitioned test set to obtain the prediction error of the first causal hypothesis model and the prediction error of the second causal hypothesis model. The causal information gain of the current observed variable is calculated based on the prediction errors of the first causal hypothesis model and the second causal hypothesis model.

7. The method according to claim 6, characterized in that, After calculating the causal information gain of the current observed variable based on the prediction error of the first causal hypothesis model and the prediction error of the second causal hypothesis model, the method further includes: If the causal information gain of the current observed variable is greater than a set value, then it is determined that there is a direct causal relationship between the current observed variable and the target variable in the observed variable dataset; If the causal information gain of the current observed variable is less than or equal to a set value, then it is determined that there is no direct causal relationship between the current observed variable and the target variable in the observed variable dataset.

8. A latent variable identification device based on causal analysis, characterized in that, include: The acquisition unit is used to iteratively determine each observed variable in the observed variable dataset to obtain the causal information gain of the target observed variable obtained by the iterative determination. The observed variable dataset includes multiple observed variables, and the target observed variable is the observed variable that satisfies the masking condition in each iterative determination process. The masking condition is that there is no direct causal relationship between the observed variable and the target variable. The calculation unit is used to calculate the average information gain and the final unfit error based on the causal information gain of the target observed variable; The determining unit is used to determine that the target system has latent variables if the average information gain is less than the final unfit error, wherein the target system is a system represented by the dataset of observed variables after masking the target observed variables.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images