Causal effect estimation method and system based on offline reinforcement learning

Abstract

The present invention relates to the technical field of causal effect estimation, and in particular to a causal effect estimation method and system based on offline reinforcement learning, the method comprising: obtaining an observation data matrix and a causal adjacency matrix, and extracting a first conditional average intervention effect estimate; performing a sensitivity analysis on the first conditional average intervention effect estimate, and extracting sensitivity analysis results; constructing a causal effect estimation model based on the sensitivity analysis results; and optimizing the causal effect estimation model with the goal of minimizing the sensitivity of unobserved confounding factors. The purpose is to solve the limitation of existing causal effect estimation methods that rely on the assumption that there are no unobserved confounding factors, and to achieve more accurate causal effect estimation when no confounding factors are observed.

Description

Technical Field

[0001] The present invention relates to the technical field of causal effect estimation, and in particular to a causal effect estimation method and system based on offline reinforcement learning. Background Art

[0002] Causal effect estimation aims to quantify the direct impact of an intervention or treatment on an outcome variable, and to distinguish changes caused by the intervention from natural changes through scientific methods. This process is crucial to understanding causal relationships in complex systems. In existing technologies, causal effect estimation mainly relies on several advanced statistical and machine learning methods, such as dual machine learning (DML), dual robust learning (DRL), and cross-learning-based methods, which aim to overcome the difficulties in implementing traditional randomized controlled trials and potential bias problems in observational data; DML uses the advantages of machine learning to accurately control confounding variables; DRL combines model prediction and inverse probability weighting strategies to enhance the stability and accuracy of estimates; and the cross-learning strategy improves the reliability of individual treatment effect estimates.

[0003] At present, with the increasing safety requirements in fields such as aerospace, healthcare, autonomous vehicles, and industrial control systems, there is a stronger demand for accuracy and reliability in the evaluation of intervention measures; however, existing methods all rely on the assumption of no unobserved confounding factors (conditional independence assumption), that is, it is assumed that all factors affecting the results have been fully considered; since this assumption of no confounding cannot be verified in practice, that is, we cannot clearly rule out the existence of unobserved confounding factors; unobserved confounders may bias the estimation of causal effects, resulting in false correlations and incorrect causal inferences. Summary of the invention

[0004] In order to solve the limitation of existing causal effect estimation methods that rely on the assumption of no unobserved confounding factors and achieve more accurate causal effect estimation when no confounding factors are observed, the present invention provides a causal effect estimation method and system based on offline reinforcement learning. The technical solutions adopted are as follows:

[0005] The technical solution of the first aspect of the present invention provides a causal effect estimation method based on offline reinforcement learning, the method comprising:

[0006] Obtain the observed data matrix and the causal adjacency matrix and extract the first conditional average intervention effect estimate;

[0007] A sensitivity analysis was performed on the estimated value of the mean intervention effect of the first condition, and the sensitivity analysis results were extracted;

[0008] Construct a causal effect estimation model based on the sensitivity analysis results;

[0009] The causal effect estimation model is optimized with the goal of minimizing the sensitivity to unobserved confounders.

[0010] Furthermore, a sensitivity analysis was performed on the estimated value of the average intervention effect of the first condition, and the sensitivity analysis results were extracted including:

[0011] Introducing randomly generated unobserved confounders;

[0012] Calculate the second conditional mean intervention effect estimate based on randomly generated unobserved confounders;

[0013] Sensitivity analysis results were extracted based on the second-condition mean intervention effect estimate and the first-condition mean intervention effect estimate.

[0014] Furthermore, the causal effect estimation model is constructed based on the sensitivity analysis results, including:

[0015] Construct actor modules based on the first-condition mean intervention effect estimates, including:

[0016] Using the convolutional layer of the policy neural network, the observation data submatrix of the current step is mapped according to the output channel and core number of the convolutional layer;

[0017] Use the fully connected layer of the policy neural network to map the observation data matrix into action prediction values;

[0018] Calculate the action-value function of the current step and update the policy parameters of the actor module using random search.

[0019] Furthermore, the objective function of updating the strategy parameters of the actor module using random search is:

[0020]

[0021] In the formula, Represents the policy function Depends on the policy neural network parameters The objective function of Indicates that the distribution The state of expectations; represents the set of all strategies, i.e., the experience pool; represents the Bellman residual; Represents the policy function Dependency policy neural network parameters Down, choose The probability of Indicates status Take action The action-value function of represents the temperature coefficient; Represents the policy function, which is used to define the probability distribution of selecting each action under different states; Indicates The state of the step, that is, the observation data sub-matrix; Indicates The state action of the step is the estimated value of the average intervention effect of the first condition; Indicates distribution in strategy Down state expectations.

[0022] Furthermore, constructing a causal effect estimation model based on the sensitivity analysis results also includes:

[0023] Build the critic module, including:

[0024] The next action prediction value of the policy neural network is used as input;

[0025] Based on the Q function neural network, calculate the predicted action value of the current step and the target value of the action value of the next step;

[0026] Calculate the action value target value of the current step based on the next step action value target value and sensitivity analysis results;

[0027] According to the root mean square error between the action value target value of the current step and the predicted action value, random search is used to update the policy parameters of the Q function neural network.

[0028] Furthermore, the expression for calculating the action value target value of the current step is:

[0029]

[0030] In the formula, Indicates that based on immediate rewards , next state And the termination status indicator The target value for the next step; Indicates the next step status, that is ; Indicates the next action, i.e. ; represents the discount factor; Indicated in strategy Next, next state Take the next step The action value of represents the temperature coefficient; Indicates the next step status Take the next step The logarithmic probability of Represents the minimum operator.

[0031] Furthermore, the objective function of updating the policy parameters of the Q-function neural network using random search is:

[0032]

[0033] In the formula, Represents the policy function Depends on the Q-function neural network parameters The objective function of Indicates that from the experience pool Medium state action pair expectations; Representation strategy Next, status Take action The action value function of .

[0034] Furthermore, constructing a causal effect estimation model based on the sensitivity analysis results also includes:

[0035] Construct a reward module and update the Q function neural network parameters of the critic module according to the sensitivity analysis results.

[0036] Furthermore, the expression for optimizing the causal effect estimation model with the goal of minimizing the sensitivity of unobserved confounding factors is:

[0037]

[0038] In the formula, represents the optimal strategy; Indicates about the status and actions Distribution of Indicates status Take action and transition to the state The immediate reward function of represents the temperature coefficient; Representation strategy state Entropy under Indicates that the distribution The state of and actions expectations.

[0039] The technical solution of the second aspect of the present invention provides a causal effect estimation system based on offline reinforcement learning, using the causal effect estimation method based on offline reinforcement learning described in the technical solution of the first aspect of the present invention, and the system includes:

[0040] a data acquisition module configured to acquire an observation data matrix and a causal adjacency matrix and extract a first conditional average intervention effect estimate;

[0041] a sensitivity analysis module, configured to perform a sensitivity analysis on the estimated value of the average intervention effect of the first condition and extract the sensitivity analysis results;

[0042] A causal effect estimation model building module, configured to build a causal effect estimation model based on sensitivity analysis results;

[0043] The causal effect estimation model optimization module is configured to optimize the causal effect estimation model with the goal of minimizing the sensitivity of unobserved confounders.

[0044] The present invention has the following beneficial effects:

[0045] The causal effect estimation method based on offline reinforcement learning provided by the present invention performs sensitivity analysis by introducing randomly generated unobserved confounding factors, extracts the sensitivity analysis results, constructs a causal effect estimation model with the conditional average intervention effect estimate as the action and the sensitivity analysis results as the reward, and optimizes the causal effect estimation model with the goal of minimizing the sensitivity of the unobserved confounding factors; by introducing sensitivity analysis feedback to continuously update the reinforcement learning strategy, causal effect estimation is allowed under the condition of relaxing the unconfounding assumption, effectively dealing with potential unobserved confounding factors, thereby significantly improving the robustness and reliability of causal effect estimation in complex real-world scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A method flow chart of a causal effect estimation method based on offline reinforcement learning provided by one embodiment of the present invention;

[0048] Figure 2 A schematic diagram of the structure of a causal effect estimation system based on offline reinforcement learning provided by an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of causal discovery results of 137 variables under real data provided by an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of estimating the causal effect of elevator control on each variable provided by one embodiment of the present invention;

[0051] Figure 5 A schematic diagram of estimating the causal effect of different aileron controls on each variable provided by an embodiment of the present invention;

[0052] Figure 6 A schematic diagram of estimating the causal effect of different rudder controls on each variable provided by an embodiment of the present invention;

[0053] Figure 7 A schematic diagram of causal discovery results after applying the method provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of a causal effect estimation method based on offline reinforcement learning proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0055] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0056] The specific scheme of the causal effect estimation method based on offline reinforcement learning provided by the present invention is described in detail below with reference to the accompanying drawings.

[0057] See also Figure 1 , which shows a method flow chart of a causal effect estimation method based on offline reinforcement learning provided by an embodiment of the present invention, the method comprising:

[0058] Step S100: Obtain the observation data matrix and the causal adjacency matrix, and extract the first conditional average intervention effect estimate; specifically, in the model preparation step, the historical data matrix is ​​recorded as , the causal adjacency matrix is Represents the causal relationship between variables in the historical data matrix; the training process involves multiple cycles, each cycle represents a complete training data set A complete iteration over multiple steps; at each step, Randomly divided into batches of the same size to reduce the correlation between samples, improve training efficiency, and provide sample diversity; specifically, in the Step, yes Execution times Operations: Each time from Random selection lines, forming a OK Matrix of columns ,generate A small matrix can be expressed as ,in , ; Randomly select a set of training data sets from the training data set for iterative update, and extract the first conditional average intervention effect estimate; Specifically, for each split batch , the corresponding causal effect estimate is calculated using traditional methods or causal machine learning methods (offline action estimation), which can be expressed as:

[0059]

[0060] In the formula, Indicated in The conditional mean intervention effect estimate for the step, is a 1×1 dimensional matrix, serving as the label for the training causal effect estimation model; The outcome variable represents the individual's experience with the intervention; The outcome variable represents the individual's outcome when he or she does not receive the intervention; The eigenvector representing an individual is ;

[0061] For example, in the scenario of pilot simulation flight, the observation data matrix may contain various operation data of the pilot in the simulated flight mission, the state data of the aircraft, and the physiological and psychological reaction data of the pilot; the causal adjacency matrix is ​​a matrix that describes the causal relationship between variables in the data set; specifically: the observation data matrix includes: the pilot's input to the flight control system, such as the operation of the joystick (to control lift), aileron (to control roll), and rudder (to control yaw); aircraft state indicators, such as altitude, speed, attitude (pitch angle, roll angle, yaw angle); pilot's physiological signals, such as electroencephalogram (EEG), heart rate, heart rate variability, etc.; pilot's facial expressions and emotional states can be obtained through facial recognition technology; the causal adjacency matrix contains the following information (expressed as 0-1 variables): the causal relationship between the pilot's operation and the change of the aircraft state; the causal relationship between the pilot's operation and his physiological and psychological state;

[0062] Step S200: Perform sensitivity analysis on the estimated value of the average intervention effect of the first condition and extract the sensitivity analysis results, including:

[0063] Step S210: introducing randomly generated unobserved confounders; specifically, since there is no baseline true value of the causal effect in the real world environment, it becomes challenging to determine the reliability of the estimated causal effect; as the assumption of unobserved confounders is relaxed, the sensitivity of the CATE estimate to these factors becomes increasingly important; in order to analyze the sensitivity of the estimate, this embodiment adopts a method of introducing randomly generated confounders, which is a placebo test; specifically, by introducing randomly generated unobserved confounders (that is, newly added backdoor variables and corresponding random data) to perform simulation analysis, the sensitivity of the conditional average intervention effect estimate (CATE) is examined; first, randomly generated unobserved confounders are introduced, and these factors are added to the estimator identified by the estimated CATE, and random data are generated for them, and an augmented data set is constructed. Subsequent calculations are all based on this modified data environment containing simulated unobserved confounders; when the results obtained using the modified estimator are close to the results obtained before the introduction of random confounders, it indicates that the sensitivity of the CATE estimate is low;

[0064] Step S220: Calculate the second conditional average intervention effect estimate based on the randomly generated unobserved confounders; specifically, based on the augmented data set to which the randomly generated unobserved confounders are added, use the corresponding causal effect estimation method to calculate the second conditional average intervention effect estimate; the calculation process is essentially similar to the usual way of calculating CATE, but the simulated unobserved confounders are incorporated into the input data;

[0065] Step S230: extract the sensitivity analysis results according to the estimated value of the average intervention effect of the second condition and the estimated value of the average intervention effect of the first condition; specifically, the absolute value of the difference or the relative change rate and other indicators can be calculated to measure the degree of deviation between the two; after obtaining the relevant indicators for measuring the degree of deviation, the sensitivity p-value can be determined by statistical test methods or based on pre-set rules; for example, assuming that the before and after CATE estimates are not substantially affected by unobserved confounding factors (that is, in an ideal, insensitive situation), the difference or relative change rate and other indicators should obey the normal distribution, and then according to the actually calculated deviation degree indicator value, the probability corresponding to the deviation degree is solved by the probability calculation function corresponding to the statistical distribution, and this probability value is the sensitivity p-value;

[0066] Step S300: constructing a causal effect estimation model based on the sensitivity analysis results;

[0067] Step S300 specifically includes:

[0068] Step S310: constructing an actor module based on the first conditional average intervention effect estimate; specifically, the actor module is based on the first conditional average intervention effect estimate As an action, take a submatrix of the observation data matrix as a state;

[0069] Step S310 includes:

[0070] Step S310: Use the convolution layer of the strategy neural network to map the observation data submatrix of the current step according to the output channel and core number of the convolution layer; specifically, use the one-dimensional convolution layer in the strategy neural network to map Matrix of columns Map to dimensional matrix, represents the output channel of the convolutional layer, Represents the number of convolutional layer cores;

[0071] Step S320: Map the observation data matrix to action prediction values ​​using the fully connected layer of the policy neural network; specifically, use two fully connected layers to map the matrix Mapping to 1D predicted action value , represents the estimated conditional average intervention effect estimate (CATE);

[0072] Step S330: Calculate the action value function of the current step and use random search to update the strategy parameters of the actor module; specifically, the action value function of the current step can be expressed as:

[0073]

[0074] In the formula, Indicated in strategy Next, status Take action The action-value function of represents the policy function; represents the immediate reward, that is, in the state Take action The immediate reward function of represents the discount factor; represents the temperature coefficient; Indicates the next step status; Indicates the next action; express expectations; Indicates in status Take action The logarithmic probability of

[0075] The objective function for updating the policy parameters of the actor module using random search is:

[0076]

[0077] In the formula, Represents the policy function Depends on the policy neural network parameters The objective function of Indicates that the distribution The state of expectations; represents the set of all strategies, i.e., the experience pool; represents the Bellman residual; Represents the policy function Dependency policy neural network parameters Down, choose The probability of Indicates status Take action The action-value function of represents the temperature coefficient; Represents the policy function, which is used to define the probability distribution of selecting each action under different states; Indicates The state of the step, that is, the observation data sub-matrix; Indicates The state action of the step is the estimated value of the average intervention effect of the first condition; Indicates distribution in strategy Down state expectation; minimizing the KL divergence makes the learned policy distribution as close as possible to the optimized target distribution; therefore, minimizing the KL divergence guides the learned policy toward a low-sensitivity behavior policy, effectively minimizing sensitivity;

[0078] Specifically, this embodiment provides a policy neural network consisting of a 1-dimensional convolutional layer and two fully connected layer neural networks for learning the Gaussian distribution mean and covariance of the action strategy; the difference from the Q-function neural network is that the input of the policy neural network is n-dimensional observations plus 1-dimensional actions; the purpose of the policy neural network is to select appropriate actions from the observations at each step to maximize future rewards and the entropy of the strategy, and the policy network samples actions through reparameterized skills.

[0079] Step S320: constructing a critic module; specifically, a Q-function neural network consisting of a 1-dimensional convolutional layer and two fully connected layer neural networks is used to learn the action-value relationship between observation and action;

[0080] Step S321: taking the next action prediction value of the strategy neural network as input;

[0081] Step S322: Based on the Q function neural network, calculate the predicted action value of the current step and the target action value of the next step;

[0082] Step S323: Based on the next action value target value and the sensitivity analysis result, the action value target value of the current step is calculated; specifically, the expression for calculating the action value target value of the current step is:

[0083]

[0084] In the formula, Indicates based on immediate rewards , next state And the termination status indicator The target value for the next step; Indicates the next step status, that is ; Indicates the next action, i.e. ; represents the discount factor; Indicated in strategy Next, next state Take the next step The action value of represents the temperature coefficient; Indicates the next step status Take the next step The logarithmic probability of represents the minimum operator; The use of operators is an integral part of the SAC algorithm, which aims to balance exploration and exploitation by minimizing a cost function that includes entropy regularization. This choice promotes a robust, adaptive strategy by prioritizing actions with the lowest expected cost, consistent with SAC's goal of achieving efficiency through entropy maximization;

[0085] Step S324: Based on the root mean square error between the action value target value of the current step and the predicted action value, the strategy parameters of the Q function neural network are updated using random search; specifically, the objective function of the strategy parameters of the Q function neural network updated using random search is:

[0086]

[0087] In the formula, Represents the policy function Depends on the Q-function neural network parameters The objective function of Indicates that from the experience pool Medium state action pair expectations; Representation strategy Next, status Take action The action-value function of

[0088] In summary, the critic module first transforms the predicted action value of the actor module’s policy neural network at step k+1 into as input; then, the critic module generates logarithmic probability and calculates entropy based on this value; the critic module calculates the target value by calculating the action value; then, using the predicted action value of the Q function neural network, taking the observation and actual action value of the current step as input, it minimizes the target action value and the predicted action value The root mean square error between them is used to obtain the optimal Q function, thereby evaluating the action value of the actor model.

[0089] Step S330: Construct a reward module to update the Q function neural network parameters of the critic module according to the sensitivity analysis results; specifically, the main purpose of the reward module is to use the traditional method or causal machine learning method (offline action estimation) to output the actual value of CATE A sensitivity analysis is performed, and the sensitivity p-value is fed back to the critic module to update the Q function parameters. This embodiment uses the feedback of the sensitivity analysis to adaptively optimize the causal effect estimation strategy to maintain robustness. Even in the case where there may be potential unobserved confounding factors, the sensitivity analysis method used quantifies the impact of these unobserved confounding factors on the causal effect estimation. By using a quantitative measure of the estimated reliability, the stability of the causal effect estimation can be ensured even when there may be unobserved confounding factors. The causal effect estimation method based on offline reinforcement learning uses this sensitivity analysis feedback to iteratively improve the causal effect estimation strategy, thereby improving the accuracy and reliability of the results.

[0090] Step S400: Optimizing the causal effect estimation model with the goal of minimizing the sensitivity of unobserved confounding factors.

[0091] Specifically, the expression for optimizing the causal effect estimation model with the goal of minimizing the sensitivity of unobserved confounding factors is:

[0092]

[0093] In the formula, represents the optimal strategy, which maximizes the expected cumulative reward and entropy; Indicates about the status and actions Distribution of Indicates status Take action and transition to the state The immediate reward function of represents the temperature coefficient; Representation strategy state Entropy under Indicates that the distribution The state of and actions The expectation of ; this formula can adaptively balance the improvement of the current causal effect estimate and the exploration of alternative estimation strategies that may be more robust to unobserved confounding; by iteratively updating the causal effect estimation strategy based on the feedback of the sensitivity analysis, the ECEE-RL architecture can continuously improve the estimation strategy and gradually converge to the optimal strategy ;

[0094] Specifically, the causal effect estimation module provided by the present invention consists of three modules: actor, critic and reward; the three modules together form a stateless Markov decision process (MDP) for strategy evaluation and optimization, and finally obtain the optimal strategy with the goal of maximizing the expected cumulative reward and entropy; in this stateless MDP, the "state" is implicitly represented as a combination of the current "action" (causal effect estimation) and "reward" (sensitivity p-value), which reflects the performance of the current causal effect estimation strategy; different "states" correspond to different estimation strategies, and by continuously adjusting the strategy to maximize the "reward", the architecture improves the overall performance of the causal effect estimation. Therefore, the "state" transition in this case essentially represents the performance transition of the causal effect estimation strategy; the proposed architecture frames the causal effect estimation problem as a stateless MDP, thereby solving the limitations of existing methods in dealing with complex data environments; first of all, the reason why MDP can effectively capture model sensitivity and perform causal effect estimation lies in its ability to simulate the uncertainty and environmental dynamics in the decision-making process. MDP provides a structured approach to understanding and modeling the behavior of complex systems by defining four basic elements: state, action, reward, and state transition probability; within this framework, each state implicitly contains a performance indicator of the current strategy, such as the error in parameter estimation; the action corresponds to the causal effect estimation strategy selected based on the current state; the reward reflects the result of taking a specific action, especially pointing to the sensitivity of the model to unobserved confounding factors; and the state transition probability describes how the system state (strategy performance) changes after taking an action. Importantly, the MDP framework allows the causal effect estimation process to be viewed as an optimization problem, whose goal is to maximize long-term rewards - that is, to minimize the sensitivity in causal estimation in the face of uncertainty and dynamic changes in data; in this way, MDP not only provides a mechanism for dynamically adjusting the causal effect estimation strategy, but also improves the adaptability and robustness of the estimation method by considering the expected return of the entire decision-making process. In addition, the wide applicability of causal effect estimation using MDP also stems from its solid foundation in mathematics and statistics. In addition, the flexibility of the MDP framework allows complex probabilistic models and machine learning algorithms to be incorporated into the model to finely simulate and predict the behavior of dynamic systems, thereby achieving highly accurate and reliable causal effect estimation in a wide range of fields and applications.

[0095] Compared with the prior art, the present invention has the following beneficial effects: the present invention effectively relaxes the assumption of no unobserved confounding factors by introducing an offline reinforcement learning (RL) method; in traditional methods, the assumption of no unobserved confounding factors is a key prerequisite for causal effect estimation, but in practical applications, this assumption is often difficult to meet; the present invention can alleviate this problem to a certain extent through the offline RL method, thereby improving the robustness of causal effect estimation; the present invention utilizes an offline actor-critic (SAC) method, regards causal effect estimation and sensitivity analysis as "actions" and "rewards", and realizes adaptive optimization of strategies by minimizing the sensitivity of confounding factors. Compared with the prior art, this method can dynamically adjust The present invention optimizes the causal effect estimation process by adjusting the estimation strategy, making the estimation result more accurate and reliable; the present invention adopts an offline RL method to perform causal effect estimation without interactive decision-making, thereby reducing the computing resources and time cost required for online learning; compared with the existing DRL and XLearner methods, the present invention has higher computing efficiency in large-scale data processing and high-dimensional feature space; at the same time, since the offline RL method can gradually optimize the causal effect estimation strategy through repeated iterative learning, the present invention shows stronger robustness in dealing with data noise and model uncertainty; compared with the traditional DML and XLearner methods, the present invention can better adapt to complex and dynamic data environments.

[0096] In summary, the causal effect estimation method based on offline reinforcement learning provided by the present application relaxes this assumption through sensitivity analysis feedback, compared with the traditional method and causal machine learning method that rely on the strong no-confounding assumption; compared with the method of using a fixed estimation strategy in the prior art, the present application can adaptively update the strategy based on sensitivity analysis feedback, focusing on estimating the average causal effect under dynamic treatment, rather than discovering, automatically deciding or testing the effect difference between two treatments; compared with the time-consuming determination of causal relationships and data collection in online reinforcement learning, it is difficult to complete the large number of interactions required for online reinforcement learning. The present application provides a more practical solution through offline reinforcement learning. When relaxing the assumption of no unobserved confounding factors, we do not introduce new assumptions; on the contrary, we strengthen the understanding and application of sensitivity analysis of existing data, which enables us to improve the reliability of causal effect estimation; by utilizing the feedback of sensitivity analysis, the causal effect estimation strategy is adaptively optimized to maintain robustness, even in the presence of potential unobserved confounding factors. The sensitivity analysis method adopted quantifies the impact of these unobserved confounding factors on our causal effect estimation. This quantitative measure of the reliability of the estimate enables us to ensure the robustness of our findings, even in the presence of unobserved confounding factors. This method uses this sensitivity analysis feedback to iteratively improve the causal effect estimation strategy, thereby increasing the accuracy and reliability of the results.

[0097] Taking the pilot simulation aircraft scenario as an example, the overall algorithm flow is as follows: first, the pre-training strategy is initialized using all causal effect estimation methods, and then the following operations are performed iteratively at each step of each cycle:

[0098] a. Randomly sample a batch of data from the data set;

[0099] b. Calculate the current action and predicted action;

[0100] c. Calculate the strategy value and q value;

[0101] d. Update the Q function and policy neural network parameters;

[0102] e. Update the target Q function parameters;

[0103] Specifically, input: data matrix , the causal adjacency matrix is , neural network parameters and , temperature parameters ; Initialize the Q function neural network parameters and policy neural network parameters; Randomly select e rows of data to form ; Calculate the current action and predict actions ; Calculate sensitivity p-value; Calculate target q-value and predicted q-value; Update Q function parameters to minimize ; Update the policy parameters ⌀ to minimize ; Update target network parameters; Output optimal strategy ; In the scenario of pilot simulation flight, the output CATE can be: Causal effect of pilot operation: estimate the causal impact of different pilot operations (such as control inputs of joystick, aileron, and rudder) on the change of aircraft state; for example, the impact of quickly pulling up or down the control stick on the aircraft attitude and speed; Causal effect of physiological signals: analyze the relationship between the pilot's physiological response (such as heart rate, heart rate variability, electroencephalogram (EEG) indicators) and flight operations or flight stress; Causal effect of emotional state: evaluate the causal relationship between the pilot's emotional changes (such as happiness, sadness, surprise, fear, etc. obtained through facial expression recognition technology) and specific flight events or operations; Causal effect of flight performance indicators: estimate the impact of flight operations on flight performance indicators (such as altitude, speed, engine parameters, fuel consumption, etc.); Causal effect of flight safety-related indicators: analyze the impact of pilots' operating habits, physiological state and emotional state on flight safety, for example, specific operations may lead to an increase in flight risks; Causal effect of human-computer interaction: study how the interaction between pilots and flight simulators affects the effectiveness of flight training and the performance of pilots. Results of strategy optimization: Outputs the optimal strategies optimized through the actor-critic framework. These strategies are mappings from a certain state of the aircraft to a series of operations by the pilot, which can minimize the impact of confounding factors and maximize the effectiveness, safety, and rationality of the pilot's operations. These output data can provide important feedback for pilot training, helping pilots understand how their operations affect flight results, so as to conduct targeted training and improvements. At the same time, it also provides a scientific basis for flight instructors and aviation psychology researchers to evaluate and improve training procedures.

[0104] See also Figures 3 to 7 As shown, an example of this method under real data is provided:

[0105] The real data set in this example includes human-computer interaction data collected from pilots during simulated flight on a flight simulator, including pilot electroencephalogram (EEG) data, facial recognition data, and simulator flight status and control data; the data collection process is automatically recorded at a frequency of 2-10 times per second. After removing the timestamp variable, there are 137 data items and indicators, represented as ; These data were collected on December 4, 2021 from a 22-year-old Asian male pilot, by wearing a head-mounted EEG sensor (Emotiv) and placing a facial feature and emotion identifier (Facereader) directly in front of the pilot. Using this data, we apply a causal effect estimation method based on offline reinforcement learning to analyze the causal effects of various pilot control behaviors (such as rapid upward and downward pilot elevator control) during the flight mission on certain features of interest to us (such as EEG, emotions); By analyzing the causal effects, similar to understanding the underlying dynamic mechanisms, we can perform higher-level analysis, such as "In a specific situation, when the pilot quickly controls the aircraft to dive, how does the pilot's EEG change?" These questions are crucial to studying the human-machine coupling mechanism between the pilot and the aircraft, and are of great significance to pilot safety management.

[0106] In order to analyze the causal effect, we used the pilot's elevator control, aileron control, and rudder control as different treatments; the pilot's EEG AF4, EEG T8, EEG PZ, happiness, sadness, surprise, fear and other indicators as the results, and used the variables Representation; The data recorded at each timestamp is regarded as a sample; Since it is impossible for us to collect all the variables related to the pilot's driving, hidden variables are likely to affect both the processing and the results. Therefore, the causal effect estimation method based on offline reinforcement learning is more suitable for this task;

[0107] Since there are no known causal relationships in real data, we use causal discovery methods to treat the discovered causal relationships as known conditions; similarly, we assume that the causal effect is identifiable and the collected real data follow the two basic assumptions of causal inference. First, we use the popular connection coordinate description and reparameterization (CCDr) causal discovery algorithm to analyze 137 variables; the results of causal discovery are shown in Figure 3 As shown, Figure 3 The size and color of the nodes and edges do not represent any meaning; and the diagram has been verified by experts in the field of flight control, and the key causal chain is highly consistent with professional knowledge;

[0108] Based on causality Figure 3 , we analyze the CATE of different results with different processing values. The relevant results are as follows: The observations of are continuous, , we use K-means to discretize the processing values, and finally Divided into two categories Our goal is to analyze For different results of , denoted as :

[0109]

[0110] Among them, the positive value of CATE indicates that the elevator control, aileron control and rudder control operation levels are equal to 1 relative to the causal effect value of the elevator control, aileron control and rudder control levels being equal to 0, and the negative value is the opposite;

[0111] See also Figure 4 As shown, Figure 4 To handle elevator control For each variable, the CATE diagram records The experimental results show that different levels of elevator control It has causal effects on the pilots' biosignals, emotions, and indicators related to the simulator's physical state during the experiment; in terms of the simulator's physical state, the three largest causal effects are depth position , Engine cylinder head temperature and the reflection of right aileron 2 The three factors that have the greatest impact on pilots’ bioelectric signals, facial features, and emotions are chin droop ,disgust and EEG AF3 ; This indicates that different elevator control levels have direct causal effects on pilots' chin drop, disgust, EEG AF3, depth position, engine cylinder head temperature, and right aileron reflex; elevator control is used to achieve pitch angle control, but depth position rather than vertical position is most affected, which may be related to the slowing of depth position changes during elevator control; among all variables, the one with the smallest causal effect is mass , lips open and extended , left squeeze dimple , only a weak causal effect;

[0112] See also Figure 5 As shown, Figure 5 To handle different aileron controls For each variable's CATE, The experimental results show that different aileron control The horizontal position has a causal effect on the human-related bioelectric signals of the pilot and the physical state of the simulator during the experiment. The largest causal effect is the horizontal position , Indicated Airspeed 、CQ.AF3 , among all the variables, the one with the smallest causal effect is the aircraft magnetic heading , Pleasure , Vertical Position ; Aileron control is used to achieve horizontal turning, and the horizontal position of the flight simulator The greatest impact, followed by indicated airspeed and CQ.AF3 signal;

[0113] See also Figure 6 As shown, Figure 6 To handle different rudder controls For each variable's CATE, The experimental results show that different rudder control It has a causal effect on the pilot's bioelectric signals, facial features and emotions related to people during the experiment, as well as the physical state of the simulator. Among the physical states of the simulator, the largest causal effect is the ground height. , right squeeze dimple ; Among bioelectric signals and facial features, lip extension , potency 、CQ.AF4 The rudder is an asymmetric load used for anti-yaw and adjustment. It is mainly used to offset the sideslip when turning on the ground, rather than directly used to control the steering direction. Therefore, the causal effect of horizontal position, depth position and vertical position is not obvious, but the causal effect of the horizon height controlled by the aircraft is large (negative value), which may be because the rudder is mainly used on the ground; among all variables, the pitch moment , Deflection of left aileron 1 , surprise have minimal causal effects;

[0114] See also Figure 7 As shown, Figure 7 The estimated CATE, Figure 7 The size of the node indicates the degree, and the color depth and size of the edge indicate the numerical value of CATE; the thicker and darker the line, the larger the estimated CATE value; Figure 7 It can be clearly shown in The causal effect relationship between each variable; in addition to some physical parameters of the simulator, EEG AF3, CQ.AF3, CQ.AF4 and various facial features and emotions are also worth paying attention to and exploring; among them, EEG AF3 is the electrode position named according to the standard method of EEG electrode placement; CQ.AF3 and CQ.AF4 represent the related indicators of the EEG signals from the AF3 and AF4 electrode positions after filtering, AF4 is located in the right area of ​​the frontal lobe, AF3 is located in the right area of ​​the frontal lobe, and CQ.AF4 is located in the right area of ​​the frontal lobe. The electrodes are mainly located in the left area of ​​the anterior frontal lobe; it can be seen that the causal effect estimation method based on offline reinforcement learning provided by this method is still applicable in the real situation where all relevant variables cannot be collected and there are hidden variables affecting processing and results. The discovered causal relationship can be used as a known condition for analysis with the help of causal discovery methods; this method can effectively analyze the conditional average intervention effect (CATE) of different pilot control behaviors (such as elevator control, aileron control, and rudder control) as processing on multiple results (including relevant indicators such as pilot bioelectric signals, facial features, emotions, and simulator physical status), and clearly present the size and relationship of the causal effects of different control behaviors on each variable, which helps to deeply understand the human-machine coupling mechanism between pilots and aircraft, and is of great significance to pilot safety management and other aspects.

[0115] In summary, the causal effect estimation method based on offline reinforcement learning effectively solves the limitation of existing causal effect estimation methods that rely on the assumption of no unobserved confounding factors through a series of innovative mechanisms and achieves more accurate estimation. This method first introduces randomly generated unobserved confounding factors for sensitivity analysis to simulate the impact of factors that may exist but are not observed in reality on the estimation, so as to measure the robustness of the estimation. For example, in the pilot flight simulator data experiment, although all relevant variables cannot be obtained, this simulation analysis can evaluate the reliability of the causal effect estimation of different control behaviors on various indicators. At the same time, using technical means such as Q function neural network, the actor module, critic module and reward module are constructed with the observation information in the data, and the complex relationship pattern between actions (interventions) and results under different states is learned from a large amount of offline data, rather than simply relying on the simple model construction under the assumption of no confounding factors. In the process of policy optimization, the policy parameters are dynamically adjusted according to the error between the action value function and the target value, so that in the complex environment where unobserved confounding factors exist, the real causal effect relationship can be continuously approached, so that when faced with unobserved confounding factors that are common in actual data, relatively more accurate and more reliable causal effect estimation results can still be given, which provides strong support for application scenarios such as human-computer coupling mechanism research and expands the application potential of causal effect estimation in complex actual situations. In the field of healthcare, causal effect estimation methods based on offline reinforcement learning can improve treatment effect estimation while considering unobserved patient characteristics; as can be seen from our experiments, the improved robustness of this method to unobserved confounding factors (reflected in lower sensitivity p-values) is particularly valuable in medical research, because all relevant patient information may not be captured. For economics and policy evaluation, causal effect estimation methods based on offline reinforcement learning provide a robust tool for estimating causal effects in the presence of unobserved confounding factors, which is common in observational studies. The ability of this method to adapt and improve its estimation over time may be particularly useful in capturing complex economic relationships. In the field of educational research, causal effect estimation methods based on offline reinforcement learning can enhance our understanding of intervention effects by taking into account unobserved student or school characteristics. Their improved accuracy and robustness can achieve more reliable estimates of educational program effects.

[0116] See also Figure 2 As shown, the technical solution of the second aspect of the present invention provides a causal effect estimation system based on offline reinforcement learning, using the causal effect estimation method based on offline reinforcement learning described in the technical solution of the first aspect of the present invention, and the system includes:

[0117] a data acquisition module configured to acquire an observation data matrix and a causal adjacency matrix and extract a first conditional average intervention effect estimate;

[0118] a sensitivity analysis module, configured to perform a sensitivity analysis on the estimated value of the average intervention effect of the first condition and extract the sensitivity analysis results;

[0119] A causal effect estimation model building module, configured to build a causal effect estimation model based on sensitivity analysis results;

[0120] The causal effect estimation model optimization module is configured to optimize the causal effect estimation model with the goal of minimizing the sensitivity of unobserved confounders.

[0121] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0122] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A causal effect estimation method based on offline reinforcement learning, characterized in that: The method comprises: Obtain the observation data matrix and causal adjacency matrix of the pilot's simulated flight scenario, and extract the first conditional average intervention effect estimate; the observation data matrix includes the pilot's electroencephalogram data, facial recognition data, and the simulator's flight status and control data; the causal adjacency matrix includes the causal relationship between the pilot's operation and the aircraft's state change, and the causal relationship between the pilot's operation and the physiological and psychological state; A sensitivity analysis was performed on the estimated value of the first conditional average intervention effect, and the sensitivity analysis results were extracted, including: Construct actor modules based on the first-condition mean intervention effect estimates, including: Using the convolutional layer of the policy neural network, the observation data submatrix of the current step is mapped according to the output channel and core number of the convolutional layer; Use the fully connected layer of the policy neural network to map the observation data matrix into action prediction values; Calculate the action value function of the current step and use random search to update the policy parameters of the actor module; Build the critic module, including: The next action prediction value of the policy neural network is used as input; Based on the Q function neural network, calculate the predicted action value of the current step and the target value of the action value of the next step; Based on the next step action value target value and sensitivity analysis results, calculate the current step action value target value; According to the root mean square error between the action value target value of the current step and the predicted action value, the policy parameters of the Q function neural network are updated using random search; Construct a reward module and update the Q function neural network parameters of the critic module based on the sensitivity analysis results; Construct a causal effect estimation model based on the sensitivity analysis results; The causal effect estimation model is optimized with the goal of minimizing the sensitivity to unobserved confounders.

2. The causal effect estimation method according to claim 1, characterized in that: A sensitivity analysis was performed on the estimated value of the average intervention effect of the first condition, and the sensitivity analysis results were extracted including: Introducing randomly generated unobserved confounders; Calculate the second conditional mean intervention effect estimate based on randomly generated unobserved confounders; Sensitivity analysis results were extracted based on the second-condition mean intervention effect estimate and the first-condition mean intervention effect estimate.

3. The causal effect estimation method according to claim 1, characterized in that: The objective function for updating the policy parameters of the actor module using random search is: In the formula, Represents the policy function Depends on the policy neural network parameters The objective function of Indicates that the distribution The state of expectations; represents the set of all strategies, i.e., the experience pool; represents the Bellman residual; Represents the policy function Dependency policy neural network parameters Down, choose probability; Indicates status Take action The action-value function of represents the temperature coefficient; Represents the policy function, which is used to define the probability distribution of selecting each action under different states; Indicates The state of the step, that is, the observation data sub-matrix; Indicates The state action of the step is the estimated value of the average intervention effect of the first condition; Indicates distribution in strategy Down state expectations.

4. The causal effect estimation method according to claim 1, characterized in that: The expression for calculating the action value target value of the current step is: In the formula, Indicates that based on immediate rewards , next state And the termination status indicator The target value for the next step; Indicates the next step status, that is ; Indicates the next action, i.e. ; represents the discount factor; Indicated in strategy Next, next state Take the next step The action value of represents the temperature coefficient; Indicates the next step status Take the next step The logarithmic probability of Represents the minimum operator.

5. The causal effect estimation method according to claim 1, characterized in that: The objective function of updating the policy parameters of the Q-function neural network using random search is: In the formula, Represents the policy function Depends on the Q-function neural network parameters The objective function of Indicates that from the experience pool Medium state action pair expectations; Representation strategy Next, status Take action The action value function of .

6. The causal effect estimation method according to claim 1, characterized in that: The expression for optimizing the causal effect estimation model with the goal of minimizing the sensitivity of unobserved confounding factors is: In the formula, represents the optimal strategy; Indicates about the status and actions Distribution of Indicates status Take action and transition to the state The immediate reward function of represents the temperature coefficient; Representation strategy state Entropy under Indicates that the distribution The state of and actions expectations.

7. A causal effect estimation system based on offline reinforcement learning, characterized in that: The causal effect estimation method based on offline reinforcement learning according to any one of claims 1 to 6 is adopted, and the system comprises: a data acquisition module configured to acquire an observation data matrix and a causal adjacency matrix and extract a first conditional average intervention effect estimate; a sensitivity analysis module, configured to perform a sensitivity analysis on the estimated value of the average intervention effect of the first condition and extract the sensitivity analysis results; A causal effect estimation model building module, configured to build a causal effect estimation model based on sensitivity analysis results; The causal effect estimation model optimization module is configured to optimize the causal effect estimation model with the goal of minimizing the sensitivity of unobserved confounders.

Images