A method for explaining agent behavior based on causal inference
Through causal inference and multi-layer perceptron neural network, the behavior causal structure diagram is constructed, and the interpretability problem of reinforcement learning agent behavior model is solved, real prediction and explanation of agent behavior is realized, and the agent training method is optimized.
Patent Information
- Application Number
- CN202111625582.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-28
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-12-28
AI Technical Summary
The behavioral model of reinforcement learning agents lacks interpretability and is difficult to fully utilize in actual combat, resulting in obstacles to human trusting agents' behavior. The existing interpretation methods rely on human writing logic or data relevance and cannot conform to human logic.
Using the agent behavior interpretation method based on causal inference, through the collection and causal analysis of reinforcement learning training data, a behavior causal structure diagram is constructed, combined with a multi-layer perceptron neural network, a reasonable behavior causal structure model is generated for prediction. The agent takes action next and explains its behavior.
It improves the interpretability of the agent's behavior, can truly predict the agent's next action, and concisely describe its behavioral characteristics through a causal relationship diagram, optimizing the agent's training method.
Smart Images

Figure CN114358247B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of auxiliary decision making and causal inference, and in particular relates to a method for explaining the behavior of a reinforcement learning intelligent agent. Background Art
[0002] In recent years, the application value of drones has been highlighted in several local wars and has attracted the attention of more and more researchers. The quality of drone air confrontation game decision-making directly reflects its intelligence level and affects its combat effectiveness when confronting manned aircraft or drones. At present, the mainstream method to solve the problem of drone air confrontation game decision-making is reinforcement learning, such as policy gradient, actor-critic and other methods. Reinforcement learning does not require a large training data set and sufficient prior knowledge. The agent can learn from scratch, continuously learn in the process of continuous interaction with the environment, adjust its own strategy, and achieve the selection of the optimal behavior. However, deep reinforcement learning faces the problem of lack of interpretability, making it difficult for the analysis results of the algorithm to be fully utilized in actual combat. The behavior of the agent obtained by reinforcement learning is a black box model and lacks interpretability. This has a certain obstacle to human trust in the behavior of the agent. Using the agent training data to analyze the agent's behavior model is an important means of explaining the agent. Humans can predict the behavior of the agent by modeling the agent's behavior, explain why the agent does a certain behavior, and the optimal solution that the agent can make under the current environmental state. Therefore, explaining the behavior of intelligent agents has important guiding significance for humans to trust intelligent agents and optimize the training methods of intelligent agents.
[0003] At present, in the explanation of the behavior of intelligent agents, people's explanation of the behavior of intelligent agents basically relies on the interpretability of the model itself and the correlation between the training data. For example, the behavior logic of the intelligent agent generated based on rules is completely written by humans. In a certain specific situation, it is very clear what decision the intelligent agent will take next and what kind of behavior it will make, and there will be no ambiguous choices. The intelligent agent model obtained in this way is highly interpretable, but it is too dependent on the intelligent agent behavior logic written by humans, which makes it difficult to reflect the intelligence of the machine and is ineffective when handling complex tasks. Another way to generate intelligent agents is to train them through a large amount of data and neural networks. In this way, the relationship between a certain behavior of the intelligent agent and certain quantities can be found by discovering the correlation between the data of training the intelligent agent to explain the basis for the generation of the intelligent agent's behavior. Although this method takes advantage of the superiority of the machine in processing large amounts of data and the generalization ability of the generated model, it has a very serious defect in explaining the intelligent agent. The explanation obtained by the correlation between the data cannot conform to human logic. Summary of the invention
[0004] The purpose of the present invention is to solve or improve the defect that the above-mentioned intelligent agent behavior model is difficult to explain. To this end, the present invention provides an intelligent agent behavior explanation method based on causal relationship inference. The causal behavior structure diagram obtained by this method can predict the next action of the intelligent agent and explain the basis for the intelligent agent to perform a certain behavior. It is of great significance in improving the interpretability of the intelligent agent behavior and subsequently optimizing the training method of the intelligent agent.
[0005] The technical solution of the present invention:
[0006] A method for explaining the behavior of an intelligent agent based on causal inference. This method first collects training data for an intelligent agent trained using reinforcement learning, mainly including the environmental state, actions taken, and reward information during the training process of the intelligent agent, and selects a certain proportion of the aircraft intelligent agent training data as a data set; then, based on the connection between the data in the data set and combined with prior knowledge, the causal relationship between the data is discovered; then, the training data is regressed and fitted to generate a reasonable behavior causal structure model; finally, the real-time observation data is input into the behavior causal structure model to predict the actions that the intelligent agent may take and explain the behavior of the intelligent agent. The process of this method includes:
[0007] a) Collect sample data offline during the reinforcement learning training process;
[0008] The sample data is obtained through the interaction between the agent and the environment during the reinforcement learning training process. The sample data mainly includes three parts, namely the state of the environment, the action of the agent and the reward obtained by the agent. The entire time series of a reinforcement learning training agent task exploration is regarded as a set of sample data. The state of the environment, the action of the agent and the reward obtained by the agent are closely related to the reinforcement learning training process. The collected data is shown in Table 1.
[0009] b) Combining the causality between data and human experience to obtain a behavioral causal structure diagram;
[0010] The process of constructing the behavior causal structure diagram is as follows:
[0011] We model the behavior explanation of the intelligent agent trained by reinforcement learning by constructing a behavior causal structure diagram. We can perform causal analysis on the data generated during the training of the intelligent agent and construct a reasonable behavior causal structure diagram based on human experience and knowledge, which is the basis for behavior explanation modeling. Our goal is to express the relationship between the environment state, action, and reward, and to perform causal discovery on the relationship between data, taking into account the temporal nature of the data generated by reinforcement learning.
[0012] (1) Discovering causality between data
[0013] There are two ways to discover causality between data:
[0014] One is based on the independence test method, through the sample correlation coefficient of two variables X and Y: Judgment independence, where X i With Y i Represents the value of the variable, and Represents the mean of the data. Based on independence, the Markov hypothesis is combined to verify causality.
[0015] Another method is to add noise to the model: Y = f(X, E), X ⊥ E; the model is divided into a linear model Y = a · X + E and a nonlinear model Y = f2(f1(x) + E). Among them, X and Y represent variables, a represents the weight parameter, x represents the value of the variable, f, f1, f2 represent function equations, and E represents the data noise model.
[0016] The above two methods were verified separately. The first solution is simple to implement, but its accuracy is not high when dealing with complex problems. The accuracy of the second method is affected by various hyperparameters and is complex to implement, but it is excellent for complex reasoning problems.
[0017] (2) Select a suitable data causal model based on the agent's aerial mission, and build a reasonable behavior causal structure diagram based on the discovery of causality between data and the human understanding and analysis of the agent's mission, i.e., prior knowledge. The behavior causal structure diagram consists of nodes and directed edges connecting these nodes. Nodes represent random variables, directed edges between nodes represent the relationship between nodes, and conditional probabilities represent the strength of the relationship between nodes. The random variables include the state of the environment, the agent's actions, and the rewards obtained by the agent.
[0018] c) Use behavioral causal structure diagram to build a causal structure diagram model
[0019] The collected sample data is input into the multi-layer perceptron neural network for offline training, and the node-to-node transfer matrix of the causal structure graph model is learned to obtain the relationship weights between nodes, which represent the strength of the relationship between nodes. The obtained weights are added to the behavior causal structure graph obtained in step b), and the behavior of the next action of the intelligent agent is predicted by comparing the current state of the intelligent agent with the behavior causal structure graph with parameters, and the behavior model of the intelligent agent is explained by comparing the predicted results with the actual results of the intelligent agent.
[0020] The behavioral causal structure diagram can make a qualitative analysis of the behavior of the intelligent agent, but it is far from enough to accurately predict the next action of the intelligent agent. A quantitative regression model between variables is trained based on the existing data.
[0021] The multilayer perceptron neural network uses the ReLU function as an activation function. The ReLU (rectified linear unit) function provides a very simple nonlinear transformation. Given an element x, the function is defined as: ReLU(x) = max(x, 0)
[0022] The multi-layer perceptron neural network introduces one or more hidden layers based on the single-layer neural network. The hidden layer is located between the input layer and the output layer. The multi-layer perceptron network model is:
[0023] O=(XW h +b h )W o +b o =XW h W o +b h W o +b o
[0024] O = output of the network ∈ R N×H
[0025] X = input |∈R N×D , D represents the number of features
[0026] W h = weight of the hth layer |∈R D×H , H represents the number of hidden units in the first layer
[0027] b h = Output of the h-th layer network ∈ R N×H
[0028] W o = weight of layer o|∈R D×H
[0029] b o = Output of the oth layer network∈R N×H
[0030] Figure 1 A neural network diagram of a multilayer perceptron is shown; in the multilayer perceptron shown in the model diagram, it can be seen from the model diagram that the neurons in the hidden layer of the multilayer perceptron neural network are fully connected to each input in the input layer, and the neurons in the output layer are also fully connected to each neuron in the hidden layer. Therefore, the hidden layer and output layer in the multilayer perceptron neural network are both fully connected layers.
[0031] Furthermore, 70% of the sample data is used as the training set and 30% as the test set. The training adopts the minibatch method and the batch size is selected as 64. After all the sample data are fully trained once, the training is stopped and the weight matrix and bias term in the final forward propagation process are output. The overall training process is as follows Figure 2 This is shown in the training section.
[0032] d) Use the causal structure graph model to reasonably explain the behavior of the intelligent agent when it performs tasks.
[0033] Intelligent agent action prediction and behavior interpretation: With a trained model, it can be used to interpret intelligent agent behavior. The environmental state at each moment in the reinforcement learning task is combined with the trained weight matrix and bias term and substituted into the forward propagation process. The output is a two-dimensional vector, which represents the probability of the action occurring at the current moment and the probability of not occurring. The largest of the two is whether the action is executed at the current moment. This is repeated once for all actions. Find the action that is most likely to occur.
[0034] In the above process, a), b) and c) are offline processes, the purpose of which is to obtain a trained prediction model, and d) is the online application of the model, the purpose of which is to obtain a reasonable explanation of the agent's behavior.
[0035] The present invention can analyze the behavior patterns of the intelligent agents by collecting data in the training environment of the aircraft intelligent agents, and complete the interpretation of the behavior of the intelligent agents. Offline training is performed using data from the intelligent agent reinforcement learning training process; the prediction model obtained through training is used for online prediction. By using the connection between the data and combining prior knowledge, it can be found whether there is causality between the behavior of the intelligent agent and the change in the state of the environment and the reward obtained by the intelligent agent. In combination with expert knowledge, the causal relationship between the data can be reasonably discovered. The next action of the intelligent agent can be predicted more realistically through data fitting based on the causal relationship graph. In line with this, the behavioral decisions of the intelligent agent can also be explained. The causal model of the intelligent agent behavior obtained through the intelligent agent training data can intuitively and concisely describe the behavioral characteristics of the intelligent agent, and become a means to optimize the intelligence of the intelligent agent. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 Flowchart of the present invention.
[0037] Figure 2 Neural network diagram of a multilayer perceptron.
[0038] Figure 3 Example picture.
[0039] Figure 4 Flowchart of the method of the present invention. DETAILED DESCRIPTION
[0040] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.
[0041] Causal structure learning is performed by retaining the data from the aircraft agent training process. The main data include the current time (time), relative distance (distance), closing rate (closure), the current heading angle (psi_V), altitude (h), speed (velocity), climb rate (h_dot), angle of attack (alpha), sideslip angle (beta), overload (n_load), blood volume (blood), and remaining oil volume (oil) displayed by the red and blue sides respectively.
[0042] Based on the independence test method, the independence is judged by calculating the sample correlation coefficient between two variables, and the causality is verified based on the independence combined with the Markov hypothesis. For more complex data such as the relative coordinates of the aircraft, the method of adding noise to the model is adopted to construct the causal structure diagram between the data, and the multi-layer perceptron is used to learn the parameters of the graph transition matrix, and the causal threshold is set to finally obtain the following Figure 4 Diagram of the aircraft confrontation causal structure.
[0043] According to the current state of the aircraft, such as attitude and health, Figure 3 Calculate the state of the aircraft at the next moment. For example, by predicting that the angle of attack of the aircraft will increase at the next moment, it is determined that the aircraft has performed a maneuver to increase the angle of attack. If the actual simulation output result is indeed that the angle of attack increases, then it can be shown that the causal structure model can well explain the behavior of the aircraft intelligent agent and play an explanatory role in the results of reinforcement learning training.
[0044] The process of the method of the present invention combining steps 1), 2), 3) and 4) is as follows Figure 4 As shown in Figure 2, data collection and network training are offline processes, while emission point prediction is an online process.
[0045] For each sequence sample of reinforcement learning, each time point has a true value of the selected action, that is, the label, and for each time point, the prediction model will also give a predicted action, and the difference between the two is used as a criterion for measuring the prediction effect. Here, accuracy, precision and recall are selected as evaluation indicators. For the binary classification problem, the samples can be divided into four situations according to the combination of their true category and predicted category. The specific classification is shown in Table 2.
[0046] The accuracy A, precision P and recall R are defined as
[0047]
[0048]
[0049]
[0050] The experimental results of the agent behavior interpretation method based on causal inference show that the actions predicted by this method are basically similar to the behavioral decisions of the agent trained by reinforcement learning.
[0051] In summary, the actions predicted by the agent behavior interpretation method based on causal inference are basically consistent with the agent's behavior, which means that the agent's behavior can be explained by the behavioral causal structure model; secondly, through this behavior interpretation method, some unintelligent characteristics of the agent can be discovered, and in turn the agent's training method can be adjusted.
[0052] Table 1 Agent training data samples
[0053] time T environment <![CDATA[S i <!-- 4 -->]]> action <![CDATA[A i ]]> award <![CDATA[r i ]]>
[0054] Table 2 Classification results confusion matrix
[0055]
Claims
1. A method for explaining agent behavior based on causal inference, characterized in that: The agent behavior interpretation method first collects training data of an agent trained by reinforcement learning, including the environment state, actions taken and reward information during the agent training process, and selects a certain proportion of the aircraft agent training data as a data set; Then, the causal relationship between the data is discovered based on the connection between the data in the data set and combined with prior knowledge; then, the training data is regressed and fitted to generate a reasonable behavioral causal structure model; finally, the real-time observation data is input into the behavioral causal structure model to predict the actions that the intelligent agent may take and explain the behavior of the intelligent agent; The specific process is as follows: a) Collect sample data offline during the reinforcement learning training process; The sample data is obtained during the interaction between the agent and the environment during the reinforcement learning training process. The sample data consists of three parts, namely the state of the environment, the action of the agent, and the reward obtained by the agent. The entire time series of a reinforcement learning training task exploration is regarded as a set of sample data. b) Combining the causality between data and human experience to obtain a behavior causal structure diagram; the process of constructing the behavior causal structure diagram is: (1) Discovering causality between data There are two ways to discover causality between data: One is based on the independence test method, through the sample correlation coefficient of two variables X and Y: Judgment independence, where X i With Y i Represents the value of the variable, and Represents the mean of the data; verifies causality based on independence and Markov assumption; Another method is to add noise to the model: Y = f(X, E), X ⊥ E; the model is in the linear model Y = aX + E and the nonlinear model Y = f2(f1(x) + E); where X and Y represent variables, a represents the weight parameter, x represents the value of the variable, f, f1, f2 represent function equations, and E represents the data noise model; (2) Selecting a suitable data causal model according to the agent's aerial mission, and constructing a reasonable behavior causal structure diagram based on the discovery of causality between data and combining human understanding and analysis of the agent's mission, i.e., prior knowledge; the behavior causal structure diagram consists of nodes and directed edges connecting these nodes; the nodes represent random variables, the directed edges between nodes represent the mutual relationship between nodes, and the conditional probability represents the strength of the relationship between nodes; the random variables include the state of the environment, the agent's actions, and the rewards obtained by the agent; c) Use behavioral causal structure diagram to build a causal structure diagram model Input the collected sample data into the multi-layer perceptron neural network for offline training, learn the node-to-node transfer matrix of the causal structure graph model, and obtain the relationship weights between nodes, where the weights represent the strength of the relationship between nodes; add the obtained weights to the behavior causal structure graph obtained in step b), predict the behavior of the next action of the intelligent agent by comparing the current state of the intelligent agent with the behavior causal structure graph with parameters, and interpret the behavior model of the intelligent agent by comparing the predicted results with the actual results of the intelligent agent; The multilayer perceptron neural network uses the ReLU function as an activation function. The ReLU function provides a very simple nonlinear transformation. Given an element x, the function is defined as: ReLU(x) = max(x, 0) The multi-layer perceptron neural network introduces more than one hidden layer based on the single-layer neural network. (hidden layer); the hidden layer is located between the input layer and the output layer. The multi-layer perceptron network model is: O=(XW h +b h )W o +b o =XW h W o +b h W o +b o O = output of the network ∈ R N×H X = input |∈R N×D , D represents the number of features W h = weight of the hth layer |∈R D×H , H represents the number of hidden units in the first layer b h = Output of the h-th layer network ∈ R N×H W o = weight of layer o|∈R D×H b o = Output of the oth layer network∈R N×H The neurons in the hidden layer of the multilayer perceptron neural network are fully connected to each input in the input layer, and the neurons in the output layer are also fully connected to each neuron in the hidden layer; therefore, the hidden layer and output layer in the multilayer perceptron neural network are both fully connected layers; After all sample data are fully trained once, stop training and output the weight matrix and bias term in the final forward propagation process; d) Use the causal structure graph model to reasonably explain the behavior of the intelligent agent when it performs tasks; The environmental state at each moment in the reinforcement learning task is combined with the trained weight matrix and bias term and substituted into the forward propagation process. The output is a two-dimensional vector, which represents the probability of the action occurring and not occurring at the current moment. The largest of the two determines whether the action is executed at the current moment. This is repeated once for all actions to find the action that is most likely to occur.
