Systems and methods for interpreting artificial intelligence behavior policies using causal reasoning

Abstract

Systems and methods for interpreting artificial intelligence behavior policies using causal reasoning are provided. In one embodiment, a method includes generating a data structure including states as determined by an AI behavior policy and actions to be performed in those states, determining, with an offline first computing system, state factors associated with the states and responsibility scores for the state factors, each responsibility score indicates a causal effect of each action associated with one of the states, generating a causal ML model using the first computing system based on the state factors and the responsibility scores, determining a state factor associated with a current state using an online second computing system based on the causal ML model, and identifying the one or more state factors as a causal cause of the action caused by the current state. Other example methods and systems for providing interpretation of AI behavior policies with causal reasoning are also disclosed.

Description

[0001] introduction

[0002] The information provided in this section is for the purpose of generally presenting the background of the present disclosure. The work of the presently named inventors is neither explicitly nor implicitly admitted to be prior art against the present disclosure to the extent it is described in this section nor to the extent that it may otherwise not qualify as prior art at the time of filing. Technical Field

[0003] The present disclosure relates to systems and methods for explaining artificial intelligence (AI) behavior strategies using causal reasoning. Background Art

[0004] AI can be used in fully autonomous or semi-autonomous vehicles to determine the actions to be taken by the vehicle. For example, AI algorithms can be applied to calculate and determine the driving behavior of such a vehicle. In such examples, graphical models such as probability trees are often used to automate decision making. Sometimes, information about the behavior of the automated vehicle can be provided by design and heuristic rules. Summary of the invention

[0005] A method for providing an explanation of an AI behavior strategy using causal reasoning, comprising: generating a data structure including states and actions to be performed in those states as determined by the AI ​​behavior strategy, determining state factors associated with the states and responsibility scores for the state factors using an offline first computing system, each responsibility score indicating a causal impact of each action associated with one of the states, generating a causal machine learning (ML) model using the first computing system based on the state factors and the responsibility scores, determining state factors associated with a current state based on the generated causal ML model using an online second computing system, and identifying one or more state factors as causal causes of an action caused by the current state.

[0006] In other features, the method further comprises reformulating the states and actions into a table represented by the index based on one or more criteria.

[0007] In other features, the AI ​​behavior policy is an AI behavior policy for an autonomous vehicle, and the one or more criteria include a defined number of segments, each segment representing a different area adjacent to the autonomous vehicle.

[0008] In other features, the method further comprises assigning a value to the index based on the defined discretized representation.

[0009] In other features, determining the status factor and the responsibility score using the offline first computing system includes determining the status factor and the responsibility score based on a value of the index.

[0010] Among other features, the AI ​​behavior policy is an AI behavior policy for an autonomous vehicle, and the state factor is associated with a semantic abstraction.

[0011] Among other features, the semantic abstraction includes one or more segments adjacent to the autonomous vehicle.

[0012] In other features, the method further comprises displaying a notification on a display in the autonomous vehicle regarding a causal reason for the behavior resulting from the current state.

[0013] In other features, the notification includes a graphical representation that highlights an area adjacent to the autonomous vehicle in which the at least one segment is located.

[0014] In other features, the notification includes a description of an area adjacent to the autonomous vehicle in which the at least one segment is located.

[0015] In other features, the size of the area adjacent to the autonomous vehicle is adjustable based on parameters of the autonomous vehicle and / or a density of traffic distribution near the autonomous vehicle.

[0016] In other features, a color of the highlighted area is adjustable based on a confidence value associated with at least one segment.

[0017] A method for providing explanations of AI behavior policies using causal reasoning includes receiving a causal ML model, determining state factors associated with a current state based on the causal ML model, identifying one or more state factors as causal causes of an action caused by the current state, and displaying a notification about the causal causes of the action caused by the current state.

[0018] Among other features, the AI ​​behavior policy is an AI behavior policy for an autonomous vehicle, and the state factor is associated with a semantic abstraction.

[0019] Among other features, the semantic abstraction includes one or more segments adjacent to the autonomous vehicle.

[0020] In other features, displaying the notification regarding the causal reason for the action resulting from the current state includes displaying the notification regarding the causal reason for the action resulting from the current state on a display in the autonomous vehicle.

[0021] In other features, the notification includes a graphical representation that highlights an area adjacent to the autonomous vehicle in which the at least one segment is located.

[0022] In other features, the notification includes a description of an area adjacent to the autonomous vehicle in which the at least one segment is located.

[0023] In other features, the size of the area adjacent to the autonomous vehicle is adjustable based on parameters of the autonomous vehicle and / or a density of traffic distribution near the autonomous vehicle.

[0024] In other features, a color of the highlighted area is adjustable based on a confidence value associated with at least one segment.

[0025] This disclosure also includes the following examples:

[0026] Example 1. A method for providing an explanation of an artificial intelligence (AI) behavior strategy using causal reasoning, the method comprising:

[0027] generating a data structure comprising states and actions to be performed in those states as determined by the AI ​​behavior policy;

[0028] determining, using the offline first computing system, state factors associated with the states and responsibility scores for the state factors, each responsibility score indicating a causal impact of each action associated with one of the states;

[0029] generating a causal machine learning (ML) model using a first computing system based on the state factors and the responsibility scores;

[0030] Determining, using an online second computing system, a state factor associated with a current state based on the generated causal ML model; and

[0031] Identifies one or more state factors as the causal cause of the action caused by the current state.

[0032] Example 2. The method of Example 1 further comprises reformulating the states and actions into a table represented by an index based on one or more criteria.

[0033] Example 3. The method of Example 2, wherein:

[0034] The AI ​​behavior strategy is an AI behavior strategy for an autonomous vehicle; and

[0035] The one or more criteria include defining a number of segments, each segment representing a different area adjacent to the autonomous vehicle.

[0036] Example 4. The method of Example 2 further comprises assigning a value to the index based on a defined discretized representation.

[0037] Example 5. The method of Example 4, wherein determining the status factor and the responsibility score using the offline first computing system includes determining the status factor and the responsibility score based on a value of an index.

[0038] Example 6. The method of Example 1, wherein:

[0039] The AI ​​behavior strategy is an AI behavior strategy for an autonomous vehicle; and

[0040] The state factors are associated with semantic abstractions.

[0041] Example 7. The method of Example 6, wherein the semantic abstraction includes one or more segments adjacent to the autonomous vehicle.

[0042] Example 8. The method of Example 7, further comprising displaying a notification on a display in the autonomous vehicle regarding a causal reason for an action resulting from a current state.

[0043] Example 9. The method of Example 8, wherein the notification includes a graphical representation that highlights an area adjacent to the autonomous vehicle in which the at least one segment is located.

[0044] Example 10. The method of Example 9, wherein the notification includes a description of an area adjacent to the autonomous vehicle in which the at least one segment is located.

[0045] Example 11. The method of Example 9, wherein the size of the area adjacent to the autonomous vehicle is adjustable based on parameters of the autonomous vehicle and / or a density of traffic distribution near the autonomous vehicle.

[0046] Example 12. The method of example 9, wherein a color of the highlighted area is adjustable based on a confidence value associated with the at least one segment.

[0047] Example 13. A method for providing an explanation of an artificial intelligence (AI) behavior strategy using causal reasoning, the method comprising:

[0048] Receive causal machine learning (ML) models;

[0049] Determine the state factors associated with the current state based on the causal ML model;

[0050] Identify one or more state factors as causal causes of the action resulting from the current state; and

[0051] Displays notifications about the causal reasons for an action resulting from the current state.

[0052] Example 14. The method of Example 13, wherein:

[0053] The AI ​​behavior strategy is an AI behavior strategy for an autonomous vehicle; and

[0054] The state factors are associated with semantic abstractions.

[0055] Example 15. The method of Example 14, wherein the semantic abstraction includes one or more segments adjacent to the autonomous vehicle.

[0056] Example 16. The method of Example 15, wherein displaying a notification regarding a causal reason for an action resulting from a current state comprises displaying a notification regarding a causal reason for an action resulting from a current state on a display in the autonomous vehicle.

[0057] Example 17. The method of Example 16, wherein the notification includes a graphical representation that highlights an area adjacent to the autonomous vehicle in which the at least one segment is located.

[0058] Example 18. The method of Example 17, wherein the notification includes a description of an area adjacent to the autonomous vehicle in which the at least one segment is located.

[0059] Example 19. The method of Example 17, wherein the size of the area adjacent to the autonomous vehicle is adjustable based on parameters of the autonomous vehicle and / or a density of traffic distribution near the autonomous vehicle.

[0060] Example 20. The method of example 17, wherein a color of the highlighted area is adjustable based on a confidence value associated with the at least one segment.

[0061] Further areas of applicability of the present disclosure will become apparent from the detailed description, claims and drawings.The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The present disclosure will become more fully understood from the detailed description and accompanying drawings, in which:

[0063] Figure 1 is a functional block diagram of an example system for generating a causal machine learning model summarizing states, resulting actions, and influential state factors according to the present disclosure;

[0064] Figure 2 is a functional block diagram of an example system according to the present disclosure, the example system being used to Figure 1 The causal machine learning model uses causal reasoning to provide explanations for AI behavior strategies;

[0065] Figure 3 According to the present disclosure, Figure 2 transportation that is part of the system;

[0066] Figure 4 Describes the present disclosure Figure 3 A graphical view of the means of transport, the Figure 3 A vehicle is centered within the defined perimeter and has an area surrounding the vehicle divided into eight sectors;

[0067] Figure 5 Describes the present disclosure Figure 3 A graphical view of the means of transport, the Figure 3 a vehicle centered within a defined perimeter and having an area surrounding the vehicle divided into eight sectors, wherein the vehicle is merging onto a highway;

[0068] Figure 6 is a flow chart of an example control process for generating a causal machine learning model for subsequent prediction of causes of states and actions according to the present disclosure;

[0069] Figure 7 is a flow chart of an example control process for training a causal machine learning model for subsequent prediction of causes of states and actions according to the present disclosure; and

[0070] Figure 8-10 is a flow chart of an example control process for providing explanations of AI behavior policies using causal reasoning according to the present disclosure.

[0071] In the drawings, reference numerals may be repeated to identify similar and / or identical elements. DETAILED DESCRIPTION

[0072] AI algorithms are applied to calculate and determine automated decisions, such as driving behaviors for semi-autonomous and fully autonomous vehicles. Although automated decision making is generally beneficial, especially in vehicle applications, conventional systems do not typically explain or fully explain the reasoning behind the automated behavior to the user (e.g., driver, passenger, etc.). For example, in complex automated behaviors caused by applying AI algorithms to real-life problems, it may be desirable to provide notifications that explain the behavior to the user. Such notifications can improve the user's trust in the system by explaining why the system takes a specific action. For example, automated driving can enhance the experience of its users by adding automatically calculated notifications, explaining vehicle actions (such as sudden braking, acceleration, changing lanes, etc.).

[0073] The system and method according to the present disclosure enables the calculation of the explanation of the automated behavior by utilizing the identification of the state factor with the highest responsibility score (e.g., causal influence) for the automated action selected for execution. As further explained herein, the system and method herein can utilize an offline computing solution to determine the responsibility score on the state factor and its subset and generate a causal model that summarizes the state, action and influential state factors, and utilize an online computing solution to provide a causal explanation in real time based on the causal model constructed offline. Using such a causal explanation, the content of the action taken (e.g., vehicle action) can be displayed or otherwise provided to the user to explain, thereby improving the user's situational awareness and increasing the coverage of the situation when the information may be found to be relevant to the user. For example, in a vehicle application, potential threats that affect the driving maneuvers of an automated driving vehicle can be marked in real time and displayed on a human-machine interface (HMI) display. Using this information, the user can become more aware and understand the automated behavior, and as a result, gain trust in the automated system.

[0074] Reference now Figure 1-2 , presents a block diagram of an example system 100, 200 for providing explanations of AI behavior policies using causal reasoning. Figure 1-2 As shown in , system 100 generally includes control module 102, database 104, and memory circuit 106, and system 200 generally includes control module 202, memory circuit 106, and display module 208. In various embodiments, and as further explained herein, system 100 can be used in an offline state or mode (e.g., not implemented for real-time use) to determine responsibility scores over state factors and generate a causal model summarizing states, actions, and influential state factors. Additionally, system 200 can be used in an online state or mode (e.g., implemented for real-time use) to provide causal explanations in real time based on a causal model constructed offline.

[0075] In various embodiments, the Figure 1-2Systems 100, 200. For example, system 100 can be employed in an offline scenario of a vehicle application to collect data to determine the causes and resulting actions of a given state and construct a summary (e.g., a model) that can be used online, as explained herein. System 200 can then be employed in an online scenario of a vehicle application to use the previously constructed or implemented model, as explained herein. In some examples, the constructed model can also be used offline (e.g., as a summary). In the vehicle application example, system 200 or a portion thereof can be implemented in any suitable autonomous (e.g., semi-autonomous or fully autonomous) vehicle, such as an electric vehicle (EV), an internal combustion engine vehicle, etc. For example, Figure 3 An autonomous vehicle 300 is depicted, comprising Figure 2 The control module 202 and the display module 208 of the autonomous vehicle 300, and one or more sensors 350 (e.g., one or more sensors and / or cameras) installed on the outside and / or inside of the vehicle 300. The sensors 350 generally obtain information about the surroundings or environment of the vehicle 300 and transmit such information to the control module 202 for calculating possible actions of the autonomous vehicle 300 based on the obtained information and for implementing one or more possible actions.

[0076] Although Figure 1-2 The systems 100, 200 are generally described herein with respect to automated driving of a vehicle (e.g., vehicle 300), but it should be appreciated that the systems 100, 200 can be implemented using suitable non-vehicle applications. For example, the systems and methods herein can be implemented in route selection (navigation) and congestion control, EV automated charging (e.g., actively deciding when to charge and when to supply energy back to the grid, etc.), thermal comfort control, resource management (e.g., fleet control, etc.), etc. In addition, the systems 100, 200 can be implemented in other vehicle applications, such as other modes of transportation (e.g., airplanes, trains, etc.), etc.

[0077] refer to Figure 1 In the system 100, the control module 102 collects data related to various states and actions resulting from the states according to the AI ​​behavior strategy. Once the data is collected, the control module 102 can generate a data structure (e.g., organize such data into a table form). In such an example, the data structure (e.g., a table) can have two columns, one for the state and the other for the corresponding action (e.g., a single action for the state). Each action can be calculated according to the AI ​​strategy as an action to be performed once the state is encountered. In various embodiments, the states and resulting actions can be indexed (e.g., 0-6 for seven different actions as referenced below).

[0078] In some examples, database 104 may store information, code, etc. related to data collection. For example, database 104 may store a description of what actions need to be taken in each state. In a model-based planning system, the AI ​​policy may be calculated offline by control module 102 and may be retrieved as a function of the state. However, in a different planning system, the AI ​​policy may be known as a result of running certain code, such as in a Monte Carlo tree search. In such an example, it may be necessary to collect pairs of states and corresponding actions for further causal interpretation using the policy.

[0079] In various embodiments, the collected states may relate to various parameters of the vehicle (e.g., ego), and the actions are actions taken by the vehicle. Parameters associated with the vehicle may include, for example, the position of the vehicle (e.g., x and y position), the heading of the vehicle, the speed of the vehicle (e.g., longitudinal speed and lateral speed), lane change status, distance to target, relative heading of other neighboring vehicles, relative position to other neighboring vehicles, relative speed to other neighboring vehicles, etc. The resulting actions may include, for example, different levels of vehicle acceleration / deceleration, directional lane changes, etc. For example, states may include "ego_pos_x", "ego_pos_y", "ego_heading", "ego_vel_lon", "ego_vel_lat", "ego_lane", "ego_lane_change_status", "ego_distance_to_target", "inRadius", "laneChangeStatus", "relHeading", "relPosLat", "relPosLon", "relVelLat", "relVelLon". In such an example, some parameters, such as relative parameters, may be repeated for each of a defined number (e.g., fifteen, twenty, twenty-five, etc.) of vehicles considered in traffic around the ego vehicle. Additionally, the actions may include seven different actions, such as, for example, "strong_acc," "Accelerate," "strong_dec," "decelerate," "Left_lane_change," "Right_lane_change," and "none."

[0080] Data collection can be done in different ways. For example, the collection can include clean target collection or dirty target collection. In clean target collection, the control module 102 receives the AI ​​policy and performs sample collection by applying the AI ​​policy on many evaluations to collect data about its performance. Dirty target collection is similar to clean target collection, but the control module 102 performs sample collection by applying the received AI policy on many evaluations, where the calculation can branch the evaluation at predetermined important points during the evaluation to obtain a stronger coverage of the state / action space. In other examples, the collection can include a clean bootstrap of a reinforcement learning (RL) training process or a dirty bootstrap of an RL training process. For example, during a clean bootstrap RL training process, each iteration starts with applying the current AI policy of the current iteration to the environment, and obtains samples (states and resulting actions) indicating the policy. In such an example, the collection can be refined by considering a sliding window over the collection (e.g., forgetting older samples obtained using older AI policies and retaining newer samples as a result of a more refined AI policy). Dirty bootstrap is similar to clean bootstrap, but also allows branching at predetermined important points during evaluation.

[0081] In various embodiments, the collected data may be reformulated based on defined semantic criteria. For example, the control module 102 may receive defined semantic criteria and then apply the semantic criteria to the collected data. In some examples, the semantic criteria may include a defined section around the vehicle. For example, Figure 4 Depicted is a portion of the image centered within the defined perimeter 400. Figure 3 300. As shown, the area surrounding the ego vehicle 300 is divided into eight sections 402, 404, 406, 408, 410, 412, 414, 416. In this example, each section 402, 404, 406, 408, 410, 412, 414, 416 represents a different area adjacent to the vehicle 300.

[0082] Then, the control module 102 can restate the state and action into a table represented by an index based on the criteria. For example, Table 1 below shows the vehicles (the closest vehicle, the second closest vehicle, and the third closest vehicle) of the traffic according to their relative Figure 4 24 indexes (listed as index 0 to index 23) of the semantic meanings of the eight segments 402, 404, 406, 408, 410, 412, 414, 416 of FIG. In such an example, traffic in each state is represented by 24 indexes.

[0083]

[0084] Next, the control module 102 can apply the defined discretization expression to reduce the dimension of the data domain. For example, the control module 102 can assign values ​​to the index based on the defined discretization expression. In such an example, each index 0 to index 23 can be assigned a coded value, which is, for example, taken from [0, 728]. For example, Table 2 below shows the features ("laneChangeStatus", "relHeading", "relPosLat", "relPosLon", "relVelLat", "relVelLon") of six different states and a condition matrix (positive, same, negative). In the following example, conditions can be set based on the range of values. For example, if the value is in the first value range, the condition can be set to "positive". Additionally, if the value is in the second value range, the condition can be set to "same". In addition, if the value is in the third value range, the condition can be set to "negative". Then, for each combination of setting conditions (positive, same or negative) in a row, a coded value between 0-728 can be assigned. For example, if the combination of setting conditions is all "positive", the coded value is assigned to 1, as shown in Table 2. In various embodiments, each value associated with the different states may be calculated based on a conventional expectation-maximization (E / M) algorithm to obtain a discretized model (or representation) of the vehicle.

[0085]

[0086] The control module 102 may then determine state factors associated with the states and responsibility scores for the state factors. In such examples, each responsibility score indicates a causal impact of one of the resulting actions associated with one of the states. For example, the control module 102 may determine that one or more neighboring vehicles (e.g., the state factors in this case) may be responsible for the resulting action of the state (e.g., strong acceleration). The control module 102 may then determine a responsibility score for each neighboring vehicle that participated in (e.g., was responsible for) the resulting vehicle action. In such examples, the control module 102 may determine the state factors and the associated responsibility scores based on the assigned values ​​of the indexes.

[0087] In various embodiments, the control module 102 may employ an algorithm to determine the state factors and their responsibility scores. For example, the control module 102 may implement one or more algorithms based on the algorithm provided in the article entitled “Causal Explanations for Sequential Decision Making Under Uncertainty” and published on January 10, 2023, which is incorporated herein in its entirety.

[0088] In such an example, the control module 102 may identify a vehicle within a defined distance from the ego vehicle (e.g., Figure 4 In various embodiments, the control module 102 may determine the proximity of neighboring vehicles and the self-vehicle (e.g., Figure 4 The Euclidean distance between the vehicles 300) is determined, and each vehicle within the defined distance is identified based on the Euclidean distance of the vehicle.

[0089] The control module 102 may then determine a responsibility score for each identified vehicle. In various embodiments, the control module 102 may determine the responsibility score based on equations (1)-(4) below. For example, with respect to a given set of states S, S′ (of equation (1) below) includes those states in which the features being tested as weak causes maintain their assignments as in the original state s in the state set S. However, variables in contingency states may have value assignments that are different from the values ​​in the original state s in the state set S. As such, if it is desirable to show that the feature being tested as a cause is indeed the cause, this means that, regardless of what assignments are applied to other variables in the state set S (of length determined by the β (beta) parameter), changing these values ​​should not change the action decision discovered by the AI ​​policy. In other words, S′ includes all states s in the state set S such that the action for that state does not change (and remains the same action selected for the corresponding state s in the state set S) when the value of a variable in the contingency set has changed. Equation (1) below is the set of all states S′. yes The set of states in which the actions are different when the assignment of the contingency set is changed.

[0090] In the following equation (3), σ(sigma) is the set (i.e., ) (where changing the assignment changes the original action) is the same size as the set (i.e., ) (where changing the assignment does not change the original action). In such an example, σ(sigma) is a function with a numerator of S′ and a denominator of The fractional definition of , as shown in equation (3).

[0091] In the following equation (4), ρ (rho) represents the responsibility score, and β (beta) represents the size of the contingency set. For example, the features in state s in the state set S can be x1, x2, x3, and x4. When testing whether x1 is the cause of action a selected for a certain assignment of the values ​​of this vector, x1 can be tested for single states taken from x2, x3, x4 or pairs or triplets. In this case, β (beta) can be 1, 2, or 3. But when calculating the cause of an action, it is possible to define beta_max as 2. In such an example, only single states and pairs are tested, not triplets of x2, x3, and x4.

[0092]

[0093]

[0094]

[0095]

[0096] For example, Figure 5 Depicted in Figure 4 Definition of perimeter 400 centered Figure 3 A graphical view of the self-vehicle 300, and Figure 4 The eight sections 402, 404, 406, 408, 410, 412, 414, 416 represent different areas adjacent to the vehicle 300. Figure 4 In the example of FIG. 4 , the ego vehicle 300 merges onto the highway, and there are multiple vehicles on the left side of the ego vehicle 300 in sections 402 , 404 , and 416 . Figure 5 In the example of FIG. 4 , the control module 102 may identify vehicles 21, 22, 23 (e.g., within the defined perimeter 400) as state factors responsible for causing the vehicle 300 to accelerate, and determine a responsibility score of 0.7 for such state factors. Additionally, the control module 102 may identify vehicles 0, 22, 23 (e.g., within the defined perimeter 400) as state factors responsible for causing the vehicle 300 to accelerate strongly, and determine a responsibility score of 0.45 for such state factors. In other examples, vehicles 0, 21, 22, 23 and / or other vehicles in segments 402, 404, 416 may cause other actions of the vehicle 300.

[0097] Continue to refer Figure 1, the control module 102 can then generate a causal machine learning (ML) model based on the state factors and the responsibility scores. For example, the control module 102 can utilize data with actual causes for each data point to generate a causal ML model, which can be subsequently used to predict causes of states and actions, as further explained below. In such an example, the data can include states, actions resulting from the states, state factors responsible for the actions, and associated responsibility scores.

[0098] The control module 102 may generate the causal ML model in any suitable manner. For example, the control module 102 may divide the data set associated with the state factors and the associated responsibility scores into a training subset and a test subset. The control module 102 may train the causal ML model for predicting the cause of a given state using the training subset. The control module 102 may then test the causal ML model with the test subset to determine whether the causal ML model is sufficiently trained. If not, the control module 102 may continue to train the causal ML model. If so, the control module 102 may store the trained causal ML model in the memory circuit 106.

[0099] exist Figure 1 In the example of , the causal ML model can be any suitable model. For example, the causal ML model can include an integration of trees (e.g., random forests, XGBoost, etc.), neural networks, heuristic rule-based models, etc.

[0100] Reference now Figure 2 In the system 200, the control module 202 can be utilized in an online state or mode (e.g., implemented for real-time use) to provide causal explanations in real time based on the causal ML model constructed offline. In such an example, the control module 202 can be located in the vehicle 300, such as in Figure 3 In such an example, the control module 202 can be considered a vehicle control module. In other examples, the control module 202 can be external to the vehicle 300, but communicate with a vehicle control module in the vehicle 300.

[0101] exist Figure 2 In the example of , the control module 202 may receive or otherwise access the causal ML model stored in the memory circuit 106. Then, when the system 200 is online, the control module 202 determines the state factors (e.g., semantic segments or any other abstract or semantic representations on top of the raw state features) and the responsibility scores of the state factors based on the generated causal ML model.

[0102] For example, and as in Figure 2 As shown in FIG. , the control module 202 can receive real-time traffic information (e.g., Figure 3The current state 210 of the vehicle 300) driven by the vehicle 300 and the resulting action 212 (e.g., a signal representing the state and the resulting action). In such an example, the control module 202 can generally be as described above with respect to Figure 1 The received data may be organized as explained by the control module 102. For example, the control module 202 may semantically organize the data (eg, based on defined semantic criteria) and discretize the semantic data (eg, based on defined discretization representations), as explained above.

[0103] The control module 202 can then apply the offline constructed causal ML model to the data and output the semantic causes predicted for the action in real time. For example, the control module 202 can determine state factors and responsibility scores for the state factors based on the causal ML model. In such an example, the state factors can include one or more semantic segments or regions adjacent to the vehicle 300 that have a causal impact on the received current state 210 and the resulting action 212.

[0104] The control module 202 may then identify one or more state factors as causal causes of the received action caused by the state based on the determined responsibility score. For example, the control module 202 may define one or more similarity metrics to compare the current state 210 with the state learned by the causal ML model. The similarity metric may be used because the data points encountered in real time may not be the same as the data points encountered in the dataset used to train the causal ML model. The control module 202 may then predict the state factors (e.g., one of the segments adjacent to the vehicle 300) that comprise the potential cause of the action based on the similarity metric (e.g., without specifying which vehicle in the real-time scenario is the actual cause). For example, the control module 202 may predict the identified segment or area (e.g., state factor) with the highest responsibility score as the causal cause of the action caused by the current state. As an example, the control module 202 may predict that a particular identified segment or area on the left side of the vehicle has the highest impact (e.g., responsibility score) on decelerating the vehicle 300. In other examples, the control module 202 may predict that the northeast segment (e.g., Figure 5 The identified vehicles in section 402 of are affecting the current driving behavior of the vehicle 300.

[0105] In various embodiments, the control module 202 can cause the display module 208 to display a notification regarding the causal reasons for the action caused by the current state 210. For example, the control module 202 can send a notification to a user located at Figure 3The display module 208 in the vehicle 300 transmits a signal. In such an example, the signal can include a specific identified segment (e.g., state factor) or multiple identified segments (e.g., state factors) with the highest responsibility score as the causal cause of the action 212 caused by the current state 210. The display module 208 (e.g., dynamic HMI display) can then display a notification for the occupant of the vehicle 300 (e.g., driver, passenger, etc.) identifying the specific segment(s) as the causal cause of the action caused by the current state 210.

[0106] The notification displayed by the display module 208 may include any suitable type and / or format of information for the occupant. For example, the notification may include a graphical representation that highlights the area adjacent to the vehicle 300 in which the identified segment(s) are located. For example, the display module 208 may display something like Figure 5 , but with segment 416 highlighted. In such an example, the highlighted segment may include the status factor with the highest responsibility score.

[0107] In other examples, the notification may include a description of the area adjacent to the vehicle 300 in which the identified segment(s) are located. For example, the notification may generally explain the pattern of the semantic area relative to the traffic pattern. In such an example, the notification may provide that a potential threat is to the left (relative to the vehicle 300) in a congested area of ​​traffic, that a potential threat is behind (relative to the vehicle 300) when we slow down, etc.

[0108] In some examples, the characteristics of the notification can be changed relative to parameters associated with the vehicle 300. For example, the notification can include one or more areas (e.g., segments) adjacent to the vehicle. In such an example, the size of the one or more areas can be adjusted based on the speed of the vehicle 300, the density of traffic distribution around the vehicle 300, etc. For example, when the speed of the vehicle 300 is high (e.g., exceeding a defined threshold such as 100 km / hour, 115 km / hour, etc.) and traffic is congested, the notification can include a larger segment. Alternatively, when traffic is sparse and / or the speed of the vehicle 300 is relatively slow, the notification can include a smaller segment.

[0109] In still other examples, the notification may include an indication of a confidence level, causal reason, etc. associated with the highlighted area. For example, the display module 108 may adjust the color of the highlighted area based on the confidence value associated with the (one or more) identified vehicles. In such an example, the color may be adjusted within a color range, where lighter colors express less confident values ​​and darker colors express more confident values. In other examples, the color intensity of the highlighted area may be varied, where a lower intensity (e.g., dim) expresses less confident values ​​and a high intensity expresses a high confidence value. In various embodiments, the confidence value may be determined based on (one or more) similarity metrics used to compare the current state 210 to the state learned by the causal ML model.

[0110] Figure 6-10 The diagram shows Figure 1 The system 100 and / or Figure 2 The example control processes 600, 700, 800, 900, 1000 that can be used by the system 200. More specifically, the control processes 600, 700 can be composed of Figure 1 The system 100 is implemented when it is in an offline state or mode (e.g., not implemented for real-time use), and the control processes 800, 900, 1000 can be implemented by Figure 2 200 is implemented when the system 200 is in an online state or mode (e.g., implemented for real-time use). Although the example control processes 600, 700, 800, 900, 1000 are relative to Figure 1 The system 100 and / or the control module 102, 202 Figure 2 Although described with reference to the system 200, any of the control processes 600, 700, 800, 900, 1000 may be adopted by another suitable system.

[0111] exist Figure 6 In the example of FIG. 6 , the control process 600 may be used to generate a causal ML model for subsequent prediction of states and causes of actions. As shown, the control process 600 begins at 602, wherein Figure 1 The control module 102 of the vehicle collects data related to various states and actions resulting from the states according to the AI ​​behavior strategy. As explained above, the AI ​​behavior strategy may be received by the control module 102 or generally accessible by the control module 102, and the states collected may relate to various parameters of the vehicle, and the actions are actions taken by the vehicle. Control then proceeds to 604 and 606.

[0112] At 604, the control module 102 reformulates or organizes the collected data into a table with indexes based on the defined criteria as explained above. For example, as explained above, the defined criteria may be a semantic criterion with a defined section around the vehicle. Then, at 606, the control module 102 assigns a value to each index based on the defined discretized representation to, for example, reduce the dimensionality of the data domain as explained above. Control then proceeds to 608.

[0113] At 608 , the control module 102 determines the state factors associated with the collected state and the responsibility scores of the state factors. For example, the control module 102 may employ an algorithm to determine the state factors and their responsibility scores based on equations (1)-(4) as explained above. Control then proceeds to 610 .

[0114] At 610, the control module 102 generates a causal ML model based on the determined state factors and responsibility scores. For example, and as explained herein, the control module 102 can train the causal ML model using the determined state factors and responsibility scores. Control then proceeds to 612, where the control module 102 stores the causal ML model in a memory circuit (e.g., Figure 1 Then, control can end, as in Figure 6 as shown in .

[0115] exist Figure 7 In the example of FIG. 7 , the control process 700 may be used to train a causal ML model for subsequent prediction of causes of states and actions. As shown, the control process 700 begins at 702, where the control module 102 accesses state factors associated with a state and responsibility scores for the state factors (e.g., which may be in Figure 6 Control then proceeds to 704, where the control module 102 divides the accessed data into a training subset and a test subset. Control then proceeds to 706.

[0116] At 706, the control module 102 trains the causal ML model based on the training subset of state factors and the associated responsibility scores. Control then proceeds to 708, where the control module 102 generates an output score using the causal ML model based on the testing subset of state factors and the associated responsibility scores. Control then proceeds to 710.

[0117] At 710, the control module 102 determines that the output score is greater than a defined threshold. For example, the defined threshold may be any suitable value that indicates whether the causal ML model is sufficiently trained. If at 710 the output score is not greater than the defined threshold, control returns to 706, where the control module 102 continues to train the causal ML model. Otherwise, if at 710 the output score is greater than the defined threshold, control proceeds to 712, where the control module 102 stores the causal ML model in a memory circuit (e.g., Figure 1 Then, control can end, as in Figure 7 as shown in .

[0118] exist Figure 8 In the example of FIG. 8 , a control process 800 may be used to provide an explanation of an AI behavior strategy using causal reasoning. As shown, the control process 800 begins at 802, wherein Figure 2 The control module 202 receives or otherwise accesses a signal for a vehicle (e.g., Figure 3 Control then proceeds to 804, where the control module 202 receives the current state of the behavior strategy (e.g., Figure 2 The current state 210) and the resulting action (e.g., Figure 2 The result of the operation is obtained in action 212). Control then proceeds to 806.

[0119] At 806, the control module 202 applies the trained causal ML model to the current state and the resulting action to identify the state factor and its associated responsibility score. For example, and as explained above, the control module 202 can identify areas around the vehicle and determine their associated responsibility scores. In such an example, the area can be a function of the state factor, and as explained above, the responsibility score can be calculated for the state factor according to the algorithm for finding the responsibility score. Control then proceeds to 808.

[0120] At 808, the control module 202 identifies the one or more state factors having the highest associated responsibility scores as causal causes of the current state and the resulting action. For example, and as explained above, if one or more areas around the vehicle have the highest responsibility scores, the control module 202 can predict that the area(s) include a potential threat. For example, the control module 202 can determine that the area to the left of the vehicle has the highest responsibility score, and then identify that area as likely to include a potential threat (e.g., the vehicle) as a reason for the resulting action. Control then proceeds to 810.

[0121] At 810, the control module 202 displays a notification with a causal reason for the action caused by the current state. For example, and as explained above, the control module 202 may send a notification to a user located at Figure 3 300 of the transportation Figure 2 The display module 208 transmits a signal to cause the display module 208 to display a notification providing the causal reason. As explained above, the displayed notification may include various types and / or formats of information for occupants (e.g., driver, passenger, etc.) in the vehicle 300. Control may then end, such as at Figure 8 as shown in .

[0122] exist Fig. 9 In the present invention, the control process 900 can be used to provide an explanation of the AI ​​behavior strategy using causal reasoning. As shown, the control process 900 begins with steps 802, 804, 806, and 808, as described above with respect to Figure 8 The control process 800 is explained. Then, Fig. 9 The control process 900 proceeds to 910.

[0123] At 910, the control module 202 determines a confidence value for the identified one or more state factors (e.g., the current state and the cause vehicle(s) of the resulting action). For example, and as explained above, the control module 202 may determine the confidence value based on one or more similarity metrics used to compare the current state with the state learned by the (offline constructed) causal ML model. Control then proceeds to 912.

[0124] At 912, the control module 202 determines whether the confidence value is greater than a defined threshold. If so, control proceeds to 914, where the control module 202 displays (e.g., causes) the first color (or first color range). Figure 1 The display module 108 displays a defined area (e.g., one or more sections) adjacent to the vehicle 300 with one or more status factors identified. In such an example, the first color may be a dark shade of a specific color, a dark color (e.g., red, etc.), etc. Then, control may end, such as at Fig. 9 If no at 912, control proceeds to 916, where the control module 202 displays the defined area adjacent to the vehicle 300 with the identified one or more status factors in a second color (or second color range), such as a light shade of a particular color (e.g., the same general color as the first color, but in a lighter shade), a light color (e.g., yellow). Control may then end, as in Fig. 9 as shown in .

[0125] exist Fig.10 In the present invention, the control process 1000 can be used to provide an explanation of the AI ​​behavior strategy using causal reasoning. As shown, the control process 1000 begins at steps 802, 804, 806, and 808, as described above with respect to Figure 8 The control process 800 is explained. Then, Fig.10 The control process 1000 proceeds to 1010, where the control module 202 displays a notification having a defined area adjacent to the vehicle 300 (e.g., an identified segment adjacent to the vehicle) having one or more state factors identified with the highest responsibility score (causal influence) for the current state and the resulting action, as explained above. Control then proceeds to 1012.

[0126] At 1012, the control module 202 determines one or more parameters associated with the vehicle 300. For example, and as explained above, the control module 202 may determine the speed of the vehicle 300 and / or the density of traffic distribution around the vehicle 300 based on information received from one or more vehicle sensors. Control then proceeds to 1014.

[0127] At 1014, the control module 202 determines whether the parameter(s) associated with the vehicle 300 is greater than one or more defined thresholds. For example, the control module 202 may compare the speed of the vehicle 300 to a defined speed value and the density of the traffic profile to a defined traffic density value. If no at 1014, control may end, such as at Fig.10 However, if at 1014 the parameter(s) exceed the defined threshold(s), control proceeds to 1016 where the control module 202 adjusts the size of the defined area of ​​the notification. For example, if the speed of the vehicle 300 is greater than the defined speed value and the density of the traffic distribution indicates heavy traffic based on the defined traffic density value, the control module 202 may increase the defined area of ​​the notification. Control may then end, as in Fig.10 as shown in .

[0128] The foregoing description is merely illustrative in nature and is by no means intended to limit the present disclosure, its application or use. The broad teachings of the present disclosure can be implemented in a variety of forms. Therefore, although the present disclosure includes specific examples, the true scope of the present disclosure should not be so limited, because other modifications will become apparent after studying the drawings, the specification and the following claims. It should be understood that, without changing the principles of the present disclosure, one or more steps in the method can be performed in different orders (or simultaneously). In addition, although each embodiment is described above as having certain features, any one or more of those features described with respect to any embodiment of the present disclosure can be implemented in any other embodiment and / or combined with the features of any other embodiment, even if the combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and the arrangement of one or more embodiments with each other is still within the scope of the present disclosure.

[0129] Spatial and functional relationships between elements (e.g., between modules, circuit elements, semiconductor layers, etc.) are described using various terms, including "connected," "engaged," "coupled," "adjacent," "immediately adjacent," "on," "above," "below," and "disposed." Unless explicitly described as "direct," when describing a relationship between a first and a second element in the above disclosure, the relationship may be a direct relationship in which no other intervening elements exist between the first and second elements, but may also be an indirect relationship in which one or more intervening elements (either spatially or functionally) exist between the first and second elements. As used herein, the phrase "at least one of A, B, and C" should be interpreted to mean a logical (A or B or C) using a non-exclusive logical "or," and should not be interpreted to mean "at least one of A, at least one of B, and at least one of C."

[0130] In the various figures, the direction of the arrows, as indicated by the arrow heads, generally indicates the flow of information (such as data or instructions) of interest to the illustration. For example, when element A and element B exchange various information, but the information transmitted from element A to element B is relevant to the illustration, the arrow may be directed from element A to element B. The unidirectional arrow does not imply that no other information is transmitted from element B to element A. In addition, for information sent from element A to element B, element B may send a request or receipt confirmation of the information to element A.

[0131] In this application, including the definitions below, the term "module" or the term "controller" may be replaced with the term "circuit". The term "module" may refer to, be part of, or include: an application specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or grouped) that executes code; a memory circuit (shared, dedicated, or grouped) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system on a chip.

[0132] The module may include one or more interface circuits. In some examples, the interface circuit may include a wired or wireless interface connected to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functionality of any given module of the present disclosure may be distributed among multiple modules connected via the interface circuit. For example, multiple modules may allow load balancing. In a further example, a server (also referred to as a remote or cloud) module may implement some functionality on behalf of a client module.

[0133] The term "code" as used above may include software, firmware, and / or microcode, and may refer to a program, a routine, a function, a class, a data structure, and / or an object. The term "shared processor circuit" covers a single processor circuit that executes some or all code from multiple modules. The term "group processor circuit" covers a processor circuit that executes some or all code from one or more modules in conjunction with an additional processor circuit. References to multiple processor circuits cover multiple processor circuits on discrete dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term "shared memory circuit" covers a single memory circuit that stores some or all code from multiple modules. The term "group memory circuit" covers a memory circuit that stores some or all code from one or more modules in conjunction with additional memory.

[0134] The term "memory circuit" is a subset of the term "computer-readable medium". The term "computer-readable medium" as used herein does not encompass transient electrical or electromagnetic signals propagated through a medium (such as on a carrier wave); therefore, the term "computer-readable medium" may be considered to be tangible and non-transitory. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital magnetic tape or hard drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0135] The apparatus and methods described in this application may be implemented in part or in whole by a special-purpose computer, which is created by configuring a general-purpose computer to perform one or more specific functions embodied in a computer program. The functional blocks, flow chart components and other elements described above serve as software specifications that can be transformed into computer programs by the routine work of skilled technicians or programmers.

[0136] The computer program includes processor executable instructions stored on at least one non-transitory tangible computer readable medium. The computer program may also include or rely on stored data. The computer program may include a basic input / output system (BIOS) that interacts with the hardware of the special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0137] A computer program may include: (i) descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation), (ii) assembly code, (iii) target code generated by a compiler from source code, (iv) source code for execution by an interpreter, (v) source code for compilation and execution by a just-in-time compiler, etc. By way of example only, source code may be written using syntax from languages ​​including: C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Fortran, Perl, Pascal, Curl, OCaml, HTML5 (Hypertext Markup Language Version 5), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Visual Lua, MATLAB, SIMULINK, and

Claims

1. A method for providing an explanation of an artificial intelligence (AI) behavior strategy using causal reasoning, the method comprising: generating a data structure comprising states and actions to be performed in those states as determined by the AI ​​behavior policy; determining, using the offline first computing system, state factors associated with the states and responsibility scores for the state factors, each responsibility score indicating a causal impact of each action associated with one of the states; generating a causal machine learning (ML) model using a first computing system based on the state factors and the responsibility scores; Determining, using an online second computing system, a state factor associated with a current state based on the generated causal ML model; as well as Identifies one or more state factors as the causal cause of the action caused by the current state. 2 . The method of claim 1 , further comprising reformulating the states and actions into a table represented by an index based on one or more criteria. The method of claim 2 , further comprising assigning a value to the index based on a defined discretized representation.

4. The method according to claim 3, wherein: Determining the status factor and the responsibility score using the offline first computing system includes determining the status factor and the responsibility score based on a value of the index.

5. The method according to claim 1, wherein: The AI ​​behavior strategy is an AI behavior strategy for an autonomous vehicle; and The state factors are associated with semantic abstractions.

6. The method according to claim 5, wherein: The semantic abstraction includes one or more segments adjacent to the autonomous vehicle. 7 . The method of claim 6 , further comprising displaying a notification on a display in the autonomous vehicle regarding a causal reason for an action resulting from a current state.

8. A method for providing an explanation of an artificial intelligence (AI) behavior strategy using causal reasoning, the method comprising: Receive causal machine learning (ML) models; Determine the state factors associated with the current state based on the causal ML model; Identify one or more state factors as causal causes of the action caused by the current state; as well as Displays notifications about the causal reasons for an action resulting from the current state.

9. The method according to claim 8, wherein: The AI ​​behavior strategy is an AI behavior strategy for an autonomous vehicle; and The state factors are associated with semantic abstractions.

10. The method according to claim 9, wherein: The semantic abstraction includes one or more segments adjacent to the autonomous vehicle.

Images