Computer system

Abstract

To effectively support human decision-making.SOLUTION: A storage device stores a model configured to output an action predicted based on a value in response to input data. An arithmetic device is configured to acquire data to be explained including values of a plurality of components to be explained in order to explain first prediction processing of the model that outputs a first predicted action in response to first input data. The arithmetic device determines contributions of each of the plurality of components to be explained to a value and an uncertainty of the value in the first prediction processing. The arithmetic device detects risk components in the first prediction processing, for each of the plurality of components to be explained, based on the contributions. The arithmetic device presents information on the risk components.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a computer system, for example, a computer system for assisting a user's decision-making.

Background Art

[0002] As the background art of the present disclosure, for example, International Publication No. 2022 / 024559 (Patent Document 1) is known. Patent Document 1 discloses a medical support system for assisting a medical act by a doctor. For example, "The medical support system includes a control unit, a recognition unit for recognizing an operative field environment, and a machine learning model for estimating an operation to be performed by the medical support system based on the recognition result of the recognition unit. The control unit outputs judgment basis information regarding the operation estimated by the machine learning model to an information presentation unit. The control unit further includes an arithmetic unit for calculating a reliability regarding the estimation result of the machine learning model, and outputs the reliability to the information presentation unit." is disclosed (see, for example, the abstract).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When using a planning model based on reinforcement learning or mathematical optimization for assisting human decision-making, interpretability can be a problem. For example, there are no clues regarding what situations the plan proposed by the model emphasizes and whether there is a better plan in a normal model. Therefore, a technique for enhancing the interpretability of the plan recommended by the model is desired so as to effectively assist human decision-making.

Means for Solving the Problems

[0005] A computer system according to one aspect of the present invention includes an arithmetic unit and a memory device, the memory device storing a model that outputs value-based predictive behavior for input data, the arithmetic unit acquiring explainable data including values ​​of a plurality of explainable elements for explaining a first prediction process by the model that outputs a first predictive behavior from first input data, determining the contribution values ​​of each of the plurality of explainable elements to the value and the uncertainty of the value in the first prediction process, detecting risk elements in the first prediction process based on the contribution values ​​among the plurality of explainable elements, and presenting information on the risk elements. [Effects of the Invention]

[0006] According to one aspect of this disclosure, it is possible to effectively support human decision-making. Other issues, configurations, and effects not mentioned above will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0007] [Figure 1] This figure shows an example of the hardware configuration of a computer system, including a decision support system according to one embodiment of this specification. [Figure 2] This figure shows an example of a computer system's software configuration. [Figure 3] This shows an example of the structure of information included in the status transmitted from the operational management system to the decision support system. [Figure 4] This shows an example of the information structure included in the predicted behavior by the behavior prediction unit. [Figure 5] This shows an example of the information structure included in the behavioral value and the uncertainty of the behavioral value, as predicted by the behavioral value prediction unit. [Figure 6] This shows an example of the information structure included in the contribution values ​​of elements to behavioral value and uncertainty, as performed by the XAI execution unit. [Figure 7] This shows an example of the information structure included in behavior improvement proposals, as developed by the Behavior Improvement Proposal Exploration Department. [Figure 8] This shows an example of the information structure included in the evaluation values ​​of the original behavior and improved behavior by the behavior evaluation unit. [Figure 9] This flowchart shows an embodiment of the processing procedure when a reinforcement learning model, specifically a Q-learning model, is applied to the operation and management of power. [Figure 10] A flowchart illustrating an example of a method for extracting high-risk factors in step S13 is shown. [Figure 11] A flowchart illustrating another example of a method for extracting high-risk factors in step S13 is shown. [Figure 12] An example of the XAI settings screen is shown. [Figure 13] An example of a screen displaying risk factors is shown. [Figure 14] An example of a behavior improvement suggestion screen generated by the behavior improvement suggestion suggestion unit is shown. [Figure 15] An example of the behavioral assessment results screen is shown. [Modes for carrying out the invention]

[0008] The embodiments of the present invention will be described in detail below with reference to the drawings. Where necessary for convenience, the description will be divided into multiple sections or embodiments. Unless otherwise specified, these are not unrelated, and one may be a modification, detail, or supplementary explanation of part or all of the other. Furthermore, when referring to the number of elements, etc. (including number, numerical value, quantity, range, etc.), unless otherwise specified or clearly limited to a specific number in principle, the number is not limited to that specific number and may be greater than or less than that number.

[0009] The system in one embodiment of this specification may be a physical computer system (one or more physical computers) or a system built on a cloud infrastructure or other computing resource group (multiple computing resources). The computer system or computing resource group may include one or more interface devices (e.g., including communication devices and input / output devices), one or more storage devices (e.g., including memory (main memory) and auxiliary storage devices), and one or more arithmetic units.

[0010] When a function is realized by executing a program containing a command code by an arithmetic unit, since the defined processing is appropriately performed while using a storage device and / or an interface device, etc., the function may be regarded as at least part of the arithmetic unit. The processing described with the function as the subject may be the processing performed by the arithmetic unit or a system having the arithmetic unit. The program may be installed from a program source.

[0011] The program source may be, for example, a program distribution computer or a computer-readable storage medium (e.g., a computer-readable non-transitory storage medium). The description of each function is an example, and a plurality of functions may be combined into one function, or one function may be divided into a plurality of functions.

[0012] FIG. 1 is a diagram showing an example of the hardware configuration of a computer system including a decision-making support system according to an embodiment of the present specification. The computer system shown in FIG.​ 1 includes an operation management system 110, a decision-making support system 100, and a user terminal 120. Each device is connected to each other via a network 140. The type of the network 140 is arbitrary, for example, a WAN (Wide Area Network) and a LAN (Local Area Network), etc. Also, the connection method of the network 140 may be either wired or wireless.

[0013] The operation management system 110 manages the operation of the user terminal target system. The operation management system 110 may be, for example, an operation management system for a power grid, an operation management system for railway operation, an operation management system for a supply chain, or an operation management system for a production line, etc. In the following, mainly, an example of an operation management system for a power grid will be described.

[0014] The hardware configuration of the operation management system 110 includes a CPU 101, a memory 102, an auxiliary storage device 103, and a network interface 104. The hardware components communicate via an internal bus. The CPU 101 executes programs stored in the memory 102. The memory 102 stores programs executed by the CPU 101 and information necessary for the programs. The memory 102 also includes a work area temporarily used by the programs.

[0015] The auxiliary storage device 103 permanently stores data. The auxiliary storage device 103 may be a storage medium such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive), or a non-volatile memory. The programs and information stored in the memory 102 may be stored in the auxiliary storage device 103. In this case, the CPU 101 reads the programs and information from the auxiliary storage device 103, loads the programs and information into the memory 102, and executes the programs loaded into the memory 102. The network interface 104 connects to other devices via a network.

[0016] The decision-making support system 100 generates information to support the decision-making of users who use the operation management system 110 and presents it to the users. Also, upon receiving an instruction from the user, it gives an instruction to the operation management system 110. The instruction to the operation management system 110 may be given directly from the user terminal 120 to the operation management system 110.

[0017] The hardware configuration of the decision-making support system 100 includes a CPU 111, a memory 112, an auxiliary storage device 113, and a network interface 114. The hardware components communicate via an internal bus or the like. The CPU 111, the memory 112, the auxiliary storage device 113, and the network interface 114 are hardware components similar to the CPU 101, the memory 102, the auxiliary storage device 103, and the network interface 104.

[0018] The user terminal 120 is a terminal used by the user. The user terminal 120 accepts user input to generate a descriptive text for the policy model and presents the user with an explanation of the basis for the policy model's estimations. The hardware configuration of the user terminal 120 includes a CPU 121, memory 122, auxiliary storage device 123, network interface 124, input device 125, and output device 126. The hardware components communicate via an internal bus.

[0019] The CPU 121, memory 122, auxiliary storage device 123, and network interface 124 are hardware components similar to the CPU 101, memory 102, auxiliary storage device 103, and network interface 104.

[0020] The input device 125 is a device for inputting data, and includes a keyboard, mouse, and touch panel. The output device 126 is a device for outputting data, and includes a display and touch panel.

[0021] In the above-described device, the CPU operates as a functional unit having a predetermined function by executing processing according to a program. In the following description, when the program is the subject of the processing, it means that the CPU or the device on which the CPU is implemented is executing the program that realizes the functional unit.

[0022] In the configuration example shown in Figure 1, different computers perform the tasks of operation management, decision support, and user interface, respectively. In other examples, all or some combinations of these tasks may be performed by a single computer. For example, the operation management system 110 and the decision support system 100 may be implemented as virtual computers running on a single computer.

[0023] As described above, a computer system can consist of one or more computers, each comprising one or more arithmetic units and one or more storage devices, including non-transient storage media. Memory, auxiliary storage devices, or combinations thereof are storage devices. A CPU is an example of an arithmetic unit. An arithmetic unit can consist of a single processing unit or multiple processing units, and may include one or more arithmetic units or multiple processing cores. An arithmetic unit can be implemented as one or more central processing units, microprocessors, microcomputers, microcontrollers, digital signal processors, state machines, logic circuits, graphics processing devices, chip-on systems, and / or any device that manipulates signals based on control instructions.

[0024] Figure 2 shows an example of the software configuration of a computer system, specifically illustrating the program modules of the operation management system 110 and the decision support system 100, and an overview of their processing.

[0025] The operation management system 110 includes a status acquisition unit 311 and an action input unit 312. The status acquisition unit 311 acquires information about the current state of the target system. The action input unit 312 acquires information about the next action of the target system from the decision support system 100.

[0026] The decision support system 100 includes an action and action value prediction unit 210, an action risk factor analysis unit 220, an action improvement proposal search and presentation unit 230, and an action improvement proposal evaluation unit 240. The action and action value prediction unit 210 includes an action prediction unit 211 and an action value prediction unit 212.

[0027] The behavior prediction unit 211 obtains information on the current state 215 of the target system from the state acquisition unit 311 of the operation management system 110, and predicts the next optimal action 216 of the target system based on the state 215. The behavior prediction unit 211 may include explanatory variables different from the state 215, such as environmental variables, and may also be subject to constraints.

[0028] The action value prediction unit 212 predicts the value of a predicted action and its uncertainty 213 based on the state 215 of the target system and the predicted action 216 of the action prediction unit 211. The action and action value prediction unit 210 can be configured using any algorithm that predicts actions and the value of those actions, such as reinforcement learning, imitation learning models, or decision trees. An example using reinforcement learning is described below.

[0029] The behavioral risk factor analysis unit 220 includes an information acquisition unit 221, an XAI (eXplainable Artificial Intelligence) execution unit 222, an XAI setting unit 223, a risk factor analysis unit 224, and a risk factor presentation unit 225.

[0030] The information acquisition unit 221 acquires information to be processed from the behavior and behavior value prediction unit 210. Specifically, it acquires information on the current state of the target system 215, the predicted behavior 216 by the behavior prediction unit 211, and the behavior value and uncertainty of the behavior value 213 by the behavior value prediction unit 212.

[0031] The XAI execution unit 222 calculates the contribution values ​​227 of the specified input elements to the action value and the uncertainty of the action value 213. The XAI execution unit 222 calculates the contribution values ​​227 of each element to the action value and the uncertainty of the action value, respectively. The XAI execution unit 222 decomposes the action value and the uncertainty of the action value, respectively, into the contribution values ​​of the elements.

[0032] The XAI execution unit 222 may calculate the contribution values ​​using any algorithm. For example, the XAI execution unit 222 can use SHAP (Shapley Additive Explanation), LIME (Local Interpretable Model-Agnostic Explanations), Integrated gradient, etc.

[0033] The XAI setting unit 223 obtains setting data for the XAI execution unit 222 from the user terminal 120 and sets it in the XAI execution unit 222. The XAI setting unit 223 displays a setting screen on the user terminal 120 and receives the settings entered on that setting screen from the user terminal 120. Details of the settings will be described later, but the type and value of the element to be explained for calculating the contribution value, the behavioral value prediction model to be explained, etc. are specified.

[0034] The risk factor analysis unit 224 analyzes the contribution values ​​227 of the elements to behavioral value and the uncertainty of behavioral value to determine the elements that are risk factors for behavioral value and the uncertainty of behavioral value. For example, the risk factor analysis unit 224 determines that elements that make a negative contribution to both behavioral value and uncertainty, that is, elements that lower behavioral value and increase the uncertainty of behavioral value, are risk factors.

[0035] The risk factor presentation unit 225 presents the analysis results of the risk factor analysis unit 224 to the user. In this example, the risk factor presentation unit 225 generates a screen showing the analysis results of the risk factor analysis unit 224 and displays it on the user terminal 120.

[0036] The behavior improvement suggestion search and presentation unit 230 includes a behavior improvement suggestion search unit 231 and a behavior improvement suggestion presentation unit 232. The behavior improvement suggestion search and presentation unit 230 obtains the analysis results of the risk factor analysis unit 224, along with the predicted behavior 216 from the behavior prediction unit 211, from the behavior risk factor analysis unit 220.

[0037] The behavior improvement suggestion search unit 231 analyzes the analysis results of the risk factor analysis unit 224 and searches for behavior improvement suggestions 235 for the optimal predicted behavior 216 of the behavior prediction unit 211. The behavior improvement suggestion presentation unit 232 presents the behavior improvement suggestions 235 proposed by the behavior improvement suggestion search unit 231 to the user. In this example, the behavior improvement suggestion presentation unit 232 generates a screen showing the behavior improvement suggestions 235 proposed by the behavior improvement suggestion search unit 231 and displays it on the user terminal 120.

[0038] The behavior improvement proposal evaluation unit 240 includes a behavior evaluation unit 241 and a behavior evaluation result display unit 242. The behavior improvement proposal evaluation unit 240 obtains one or more behavior improvement proposals 235 proposed by the behavior improvement proposal search unit 231 and predicted behaviors 216 by the behavior prediction unit 211 from the behavior improvement proposal search and presentation unit 230.

[0039] The behavior evaluation unit 241 takes the behavior improvement proposal 235 proposed by the behavior improvement proposal search unit 231 as input and outputs an evaluation value thereof. For example, the behavior evaluation unit 241 may perform a Monte Carlo simulation to calculate the expected value in a reinforcement learning environment that includes probabilistic behavior, or it may perform a time evolution simulation using the behavior prediction unit 211.

[0040] In this example, the behavior improvement proposal evaluation unit 240 calculates the evaluation value 245 of the original behavior of the improvement proposal, i.e., the predicted behavior 216 by the behavior prediction unit 211 and the behavior improvement proposal 235. The behavior evaluation result display unit 242 presents the evaluation results of the behavior evaluation unit 241 to the user. In this example, the behavior evaluation result display unit 242 generates a screen showing the evaluation results of the behavior evaluation unit 241 and displays it on the user terminal 120.

[0041] The user, using the user terminal 120, refers to the presented evaluation value and specifies an action 246 to instruct the operation management system 110 to perform. The specified action 246 may be a predicted action 216, an action improvement suggestion 235, or any of these actions updated by the user. The action input unit 312 of the operation management system 110 obtains the action 246 specified by the user from the action improvement suggestion evaluation unit 240 and commands the target system to perform that action 246.

[0042] The following describes an example of a reinforcement learning algorithm for power operation and management. The operation management system 110 operates and manages multiple generators and controls their power generation. The power generation of the generators is required to be set appropriately according to the generator failure probability and power demand. The decision support system 100 uses a reinforcement learning algorithm to predict the optimal action of the operation management system 110.

[0043] Examples of reinforcement learning algorithms include the Actor-Critic algorithm and the Q-learning algorithm. Examples of models include deep neural networks and table-based models, as well as any other type of model. Below, the action value will be referred to as the Q-value. The Q-value is the expected long-term reward, and in the example of power management, it would include the cost of power generation and the damage caused by a power outage.

[0044] The following describes an example of a data structure in power operation management. Figure 3 shows an example of the structure of information contained in state 215 transmitted from the operation management system 110 to the decision support system 100. State 215 shows the value of each element of the state. In the example of power operation management, the element is, for example, a generator, and the value of the state element shows the current power generation amount of the generator.

[0045] Figure 4 shows an example of the information structure included in the predicted behavior 216 by the behavior prediction unit 211. The predicted behavior 216 shows the value of each element of the behavior. In power operation management, the elements of the behavior are, for example, generators, similar to the elements of the state. The value of the behavior element indicates the amount of power generated by the generator in the next step.

[0046] Figure 5 shows an example of the structure of the information contained in the behavioral value and behavioral value uncertainty 213 by the behavioral value prediction unit 212. The behavioral value and behavioral value uncertainty 213 represent the Q value, epistemic uncertainty (EU), and eleatoric uncertainty (AU) of the predicted behavior 216. The Q value represents the behavioral value. The epistemic uncertainty and eleatoric uncertainty represent the uncertainty of the Q value, respectively.

[0047] Various methods have been proposed to quantify the uncertainty of machine learning inference results. The uncertainty of inference results can be decomposed into two types: epistemic uncertainty, which arises from insufficient training data, and eleatoric uncertainty, which arises from noise and other factors present in the data itself.

[0048] In regression problems, the variance between models when multiple models are trained on the same training dataset is defined as follows: Let μi be the predicted value for a given input value of model i (i=1...N), and let σi be the variance. In this example, the predicted value is the Q-value. The variance σi is decomposed into σe, which represents epistemic uncertainty, and σa, which represents aleatoric uncertainty.

[0049] The variance σe, which represents epistemic uncertainty, represents the variance of the difference between the estimated value μi of each model and the mean of the estimated values ​​of all models. If sufficient data is available for training, this value will decrease, thus representing uncertainty caused by insufficient data. On the other hand, σa, which represents aleatoric uncertainty, represents the mean of the variances of the estimated values ​​of each model, and represents uncertainty caused by the difficulty of estimating the data itself, which arises due to the randomness of the training parameters and data sampling, even if the training has converged.

[0050] Several methods are known for calculating epistemic and eleatoric uncertainty. For example, a reinforcement learning method that combines the concepts of distributed reinforcement learning and Bayesian estimation can calculate eleatoric and epistemic uncertainty separately, in addition to the Q-value for each state and action.

[0051] Figure 6 shows an example of the information structure included in the element contribution values ​​227 for behavioral value and uncertainty, as generated by the XAI execution unit 222. The element contribution values ​​227 for behavioral value and uncertainty represent the contribution values ​​272 for the Q value, 273 for epistemic uncertainty, and 274 for eleatoric uncertainty for each of the input elements 271.

[0052] Input element 271 represents the identifier of the behavioral element shown by the predicted behavior 216, i.e., the identifier of the generator. The contribution values ​​272, 273, and 274 for the Q value, epistemic uncertainty, and eleatoric uncertainty, respectively, represent the values ​​of the behavioral elements shown by the predicted behavior 216, i.e., the contribution values ​​based on the power output of each generator.

[0053] A positive or negative contribution value 272 to the Q-value indicates that the element's contribution increases or decreases the Q-value. A positive or negative contribution value 273 to epistemic uncertainty or 274 to aleatoric uncertainty indicates that uncertainty increases or decreases. Here, increasing the Q-value is a more appropriate action, and decreasing uncertainty is a more appropriate action.

[0054] Figure 7 shows an example of the structure of information contained in the action improvement plan 235, generated by the action improvement plan search unit 231. The action improvement plan 235 shows the value of the improvement plan for each element of the action. In this example, the value of the improvement plan for the element of the action indicates the amount of power generated by the generator in the next step.

[0055] Figure 8 shows an example of the information structure included in the evaluation values ​​245 of the original behavior and improved behavior by the behavior evaluation unit 241. Figure 8 shows, for example, an example of the behavior evaluation value for the original behavior or one proposed behavior improvement. The behavior evaluation unit 241 generates a table with the structure shown in Figure 8 for each of the original behavior and one or more proposed behavior improvement measures. Note that there may be cases where no proposed behavior improvement measures exist.

[0056] The behavior evaluation value 245 represents the sum of rewards in each episode of the simulation. In one embodiment of this specification, the behavior evaluation unit 241 evaluates the predicted behavior 216 by the behavior prediction unit 211 and the proposed behavior improvement 235 by the behavior improvement suggestion unit 231 based on the simulation.

[0057] Each episode can consist of multiple steps, from a step that satisfies predetermined start conditions to a step that satisfies predetermined end conditions. Each step represents a single action performed by the agent (operation management system 110) within the environment. In this example, the action represents the amount of power generated by the generator and is included in the state of the next step. For example, in this example, the initial state of each episode is fixed, and the subsequent state depends on the results of each action.

[0058] Each step represents the interaction between the environment and the action. For example, each step represents the state of the environment, the action, and the reward. The current state and the action taken in response to that state determine the next state. The reward is determined by the combination of the state of the environment and the action taken in response to that state. In the example shown in Figure 8, the sum of rewards is the sum of the rewards for all steps in each episode.

[0059] Next, an example of the process performed by the decision support system 100 will be described. The decision support system 100 presents the user with areas for improvement regarding the actions calculated by the model, thereby supporting their decision-making. The areas for improvement may include not only potential problems that need improvement, but also specific improvement proposals.

[0060] Figure 9 is a flowchart illustrating an embodiment of the processing procedure when a reinforcement learning model, specifically a Q-learning model, is applied to power operation management. In step S11, the XAI setting unit 223 performs settings for risk factor analysis using XAI. The XAI setting unit 223 presents a setting screen on the user terminal 120 and accepts input from the user. Setting items can include the subject to be explained, explanatory factors, and baseline. Some items are set automatically, and some items may be specified by the user.

[0061] Figure 12 shows an example of the XAI settings screen 400. The XAI settings screen 400 includes section 401 showing the behavioral value prediction model to be explained, section 402 showing the data to be explained (states and behaviors), section 403 showing the explanatory factors, and section 404 showing the baseline.

[0062] The behavioral value prediction model and the data (state and behavior) to be explained are automatically set by the XAI setting unit 223. For example, the selected data to be explained is the current state 215 or predicted behavior 216 obtained from the behavior and behavioral value prediction unit 200. The elements of the current state 215 or predicted behavior 216 are the elements to be explained.

[0063] Explanatory factors can be specified by the user, and either a state or a behavior can be selected. If a state is specified, its contribution value is calculated, and important state combinations that constitute risk factors are extracted. If a behavior is specified, its contribution value is calculated, and after presenting the risk factors, suggestions for behavioral improvement are made. The following describes an example where a behavior is selected as the explanatory factor.

[0064] The baseline is an option for algorithms that use a baseline to calculate contribution values, such as SHAP. For example, if behavior is specified as an explanatory factor, the XAI setting unit 223 displays section 404, which sets the value of the behavior element as the baseline. The default value is the current behavior. In the example of power operation management, the current power generation amount of each generator is shown as the default value. The user may change the default value. When the user has identified the state and behavior they want to focus on in the system, they set the data in the XAI setting unit 223 and select the evaluation start button to start execution. Note that the contribution value may be calculated using other environment variables.

[0065] Returning to Figure 9, in step S12, upon receiving an instruction from the user via the XAI setting unit 223 to perform an evaluation, the XAI execution unit 222 calculates the action value and the contribution value 227 to the uncertainty of the action value of the input elements. As explained with reference to Figure 12, the input elements for which the XAI execution unit 222 calculates the contribution value are the elements of the current state 215 or predicted action 216 specified by the user.

[0066] The calculation of contribution values ​​can be performed using, for example, SHAP. SHAP uses a baseline as the basis for calculating contribution values. The XAI execution unit 222 determines the contribution value of the input elements based on the relative values ​​of the input elements relative to the baseline values. In this example, the XAI execution unit 222 calculates the contribution value of each input element to the action value and the uncertainty of the action value.

[0067] For example, the XAI execution unit 222 uses an explanatory model that outputs contribution values. The explanatory model is generated based on the configuration of the action and action value prediction unit 200. The XAI execution unit 222 calculates relative values ​​from the values ​​of the input elements and the baseline values ​​of the input elements. The XAI execution unit 222 inputs the relative values ​​of each input element into the explanatory model and calculates the contribution values ​​of each input element to the action value and its uncertainty. Note that algorithms different from SHAP that do not require a baseline may be used. Also, various XAI algorithms are widely known, and detailed explanations are omitted here.

[0068] In step S13, the risk factor analysis unit 224 extracts high-risk elements based on the behavioral value and the contribution to uncertainty. In this example, the behavioral value is the Q value. Furthermore, in step S14, the risk factor presentation unit 225 presents these high-risk elements to the user on the user terminal 120.

[0069] Figure 10 shows a flowchart of an example of a method for extracting high-risk elements in step S13. In step S21, the risk factor analysis unit 224 obtains the value of epistemic uncertainty calculated by the XAI execution unit 222. Next, in step S22, the risk factor analysis unit 224 compares the epistemic uncertainty with a preset threshold.

[0070] If the epistemic uncertainty is greater than the threshold (S22:NO), the risk factor analysis unit 224 determines in step S23 that it will receive instructions from the user regarding the extraction of risk elements. Epistemic uncertainty is uncertainty caused by insufficient learning, and the acceptable level depends on the problem. Therefore, if the epistemic uncertainty is greater than the threshold, the unit receives instructions from the user regarding the next process to be performed, such as extracting risk elements as is or repeating the learning process. For example, the risk factor presentation unit 225 presents information about the epistemic uncertainty and accepts the user's specifications. Step S23 may be omitted.

[0071] If, in step S22, the epistemic uncertainty is below a threshold (S22: YES), then in step S24, the risk factor analysis unit 224 extracts generators whose contribution to the behavioral value is negative. Next, in step S25, the risk factor analysis unit 224 extracts generators from the extracted generators whose contribution to the aleatoric uncertainty is positive. This allows for the appropriate extraction of generators that represent risk factors.

[0072] Figure 11 shows a flowchart of another example of a method for extracting high-risk elements in step S13. In step S31, the risk factor analysis unit 224 obtains the contribution values ​​of each generator to the behavioral value and the contribution values ​​to eleatoric uncertainty, which are calculated by the XAI execution unit 222.

[0073] In step S32, the risk factor analysis unit 224 extracts generators with negative contribution values ​​to behavioral value and arranges them in ascending order of contribution value. Next, in step S33, the risk factor analysis unit 224 extracts a predetermined number of generators with positive contribution values ​​to eleatoric uncertainty from the generators with higher rankings. This allows for the appropriate extraction of generators that constitute risk elements. Note that the method of extracting risk factors described with reference to Figure 10 or 11 is just one example, and other methods may be used for extraction.

[0074] Returning to Figure 9, in step S14, the risk factor presentation unit 225 presents the high-risk elements extracted by the risk factor analysis unit 224. Figure 13 shows an example of the risk factor presentation screen 420. For example, the risk factor presentation unit 225 generates the risk factor presentation screen 420 and displays it on the user terminal 120.

[0075] In the example in Figure 13, the risk factor presentation screen 420 includes a table 421 showing the behavioral value and the contribution of each input element to the uncertainty of the behavioral value. Table 421 shows the identifier of the input element, the contribution of each input element to the behavioral value, the contribution of each input element to the epistemic uncertainty, and the contribution of each input element to the eleatoric uncertainty. In this example, the identifier of the input element is the same as the identifier of the generator.

[0076] The risk factor presentation screen 420 further includes a section 422 that highlights the high-risk elements extracted by the risk factor analysis unit 224. The method of highlighting the high-risk elements is optional. For example, in Table 421, high-risk elements may be highlighted to distinguish them from non-high-risk elements. In Figure 13, high-risk element 2 is highlighted in Table 421. The method of highlighting is optional.

[0077] The risk factor presentation screen 420 further includes a section 423 where the user specifies the behavioral elements for exploring improvement suggestions. In the example in Figure 13, two behavioral elements, specifically the power generation amounts of generators 13 and 15, are specified. Only elements not identified as high-risk elements may be specified, or any element may be specified. By accepting user specifications, improvement suggestions can be generated by changing the values ​​of the elements desired by the user.

[0078] When the "Start Search" button is selected, the system searches for suggested behavioral improvements, as described below. The search for suggested behavioral improvements involves changing the power output of these two generators from the values ​​of the predicted behavior 216 to find a behavioral improvement that improves the behavioral value and its uncertainty.

[0079] The risk factor presentation screen 420 displays the contribution values ​​to epistemic uncertainty in addition to the contribution values ​​to behavioral value and aleatoric uncertainty. Users can specify input elements of interest and conduct a search. Alternatively, instead of or in addition to searching for behavioral improvement suggestions that modify user-specified behavioral elements, a search for behavioral improvement suggestions using a variable number of some or all of the extracted high-risk elements may be automatically performed. By changing only the values ​​of the high-risk elements, more appropriate behavioral improvements can be explored. Some of the presentation information shown in Figure 13 may be omitted, and other information may be added.

[0080] Returning to Figure 9, in step S15, the behavior improvement suggestion search unit 231 creates a response surface for the values ​​of the extracted high-risk elements. The extracted high-risk elements may be all the high-risk elements extracted in step S13 or user-specified high-risk elements. The behavior improvement suggestion search unit 231 forms a response surface for each value to be referenced for determining the behavior improvement suggestion.

[0081] In the example described below, the behavior improvement suggestion search unit 231 searches for behavior improvement suggestions based on behavioral value and aleatoric uncertainty. Therefore, the behavior improvement suggestion search unit 231 creates response surfaces for both behavioral value and aleatoric uncertainty. The input parameters for the response surfaces are the values ​​of the extracted high-risk elements, i.e., the power output of the extracted generators. By considering the uncertainty of behavioral value in addition to the behavioral value itself, more appropriate behavior improvement suggestions can be found.

[0082] Next, in step S16, the behavior improvement suggestion search unit 231 searches for values ​​of high-risk elements that improve both behavioral value and uncertainty. For example, the behavior improvement suggestion search unit 231 searches for values ​​of high-risk elements that improve both behavioral value and aleatoric uncertainty, that is, values ​​that increase behavioral value and values ​​that decrease aleatoric uncertainty.

[0083] Behavioral value and aleatoric uncertainty can be in a trade-off relationship. Therefore, it may be acceptable to allow users to specify which one they prioritize. For example, the behavior improvement suggestion search unit 231 searches for a behavior improvement suggestion that maximizes (behavioral value - a * aleatoric uncertainty). The coefficient a is a positive constant that can be specified by the user.

[0084] The behavior improvement suggestion search unit 231 may further search based on epistemic uncertainty. For example, the behavior improvement suggestion search unit 231 searches for behavior suggestions where both the behavioral value and aleatoric uncertainty are improved among behavior suggestions where the epistemic uncertainty is below a threshold. This makes it possible to search for behaviors with low epistemic uncertainty, that is, behaviors that have been taken during learning (behaviors where safety is guaranteed). This threshold may be different from or the same as the threshold in the example explained with reference to Figure 10.

[0085] Furthermore, in the XAI setting screen 400 shown in Figure 12, if "State" is selected as an explanatory factor, alternative action options different from the predicted action may be explored. For example, action options in which both the action value and eleatoric uncertainty are within acceptable limits may be explored.

[0086] Next, in step S17, the behavior improvement suggestion search unit 231 determines whether a value for a high-risk element (behavior improvement suggestion) that improves both the behavioral value and the aleatoric uncertainty has been found. If a suggestion to improve the above two items is found (S17: YES), in step S18, the behavior improvement suggestion presentation unit 232 presents the behavior improvement suggestion to the user on the user terminal 120.

[0087] Figure 14 shows an example of the behavior improvement suggestion screen 450 generated by the behavior improvement suggestion suggestion unit 232. In the example shown in Figure 14, generators 13 and 15 are extracted as high-risk elements, and behavior improvement suggestions are explored and proposed that change their power generation amounts from the predicted behavior 216.

[0088] The behavior improvement suggestion screen 450 includes a response phase 451 for the Q value, which is the behavioral value, and a response phase 454 for Aleatoric uncertainty. In the two response phases 451 and 454, the horizontal axis shows the value (power generation amount) of the behavioral element of generator number 13, and the vertical axis shows the value (power generation amount) of the behavioral element of generator number 15. In this example, the power generation amount of the generator is calculated using a predetermined function, and the behavior improvement suggestion screen 450 displays the values ​​of predetermined parameters that determine the power generation capacity in that function as values ​​representing the power generation amount of the generator.

[0089] In the Q-value response phase 451, the black circle 452 indicates the coordinates of the power generation amounts of generators 13 and 15 in the original action shown by the predicted action 216. The cross mark 453 indicates the coordinates of the power generation amounts of generators 13 and 15 in the improved action plan. In the Aleatoric uncertainty response phase 454, the black circle 455 indicates the coordinates of the power generation amounts of generators 13 and 15 in the original action shown by the predicted action 216. The cross mark 456 indicates the coordinates of the power generation amounts of generators 13 and 15 in the improved action plan. Note that response phases 451 and 454 may also be indicated by a specific pattern, for example, to clearly show the portion where the epistemic uncertainty is below a threshold.

[0090] Section 457 shows the power output of generators 13 and 15 in the original behavior shown by predicted behavior 216, and the power output of generators 13 and 15 in the improved behavior. The power output of generator 13 changes from 1 to -0.25, and the power output of generator 15 changes from 1 to -0.25.

[0091] Table 458 shows the Q-value, epistemic uncertainty, and eleatoric uncertainty for both the original behavior and the proposed behavioral improvement, as indicated by the predicted behavior 216. The Q-value and eleatoric uncertainty of the proposed behavioral improvement are better than those of the original behavior.

[0092] If the user wants to see suggested behavioral improvements with other behavioral elements changed, they select button 459. In response to the selection of button 459, the decision support system 100 starts processing from step S14 shown in Figure 9 and displays the risk factor presentation screen 420 on the user terminal 120.

[0093] The user can obtain evaluations of both the original behavior and the proposed behavioral improvement by selecting the evaluation start button 460. The user specifies the number of simulation episodes to be executed for evaluation in box 461. When the evaluation start button 460 is selected, the behavioral improvement evaluation unit 240 executes the specified number of simulations for both the original behavior and the proposed behavioral improvement.

[0094] Returning to Figure 9, in step S19, the behavior evaluation unit 241 performs evaluations of both the original behavior and the proposed behavior improvement. The behavior evaluation unit 241 runs a specified number of simulations for both the original behavior and the proposed behavior improvement, and calculates evaluation values ​​245 for both the original behavior and the proposed behavior improvement. As explained with reference to Figure 8, the evaluation values ​​245 for the original behavior and the proposed behavior improvement represent the sum of rewards in each episode of the simulation for both the original behavior and the proposed behavior improvement. The sum of rewards represents the behavioral value in each episode.

[0095] Considering the balance of electricity supply and demand, it is necessary to change the behaviors that are dependent on the behavior being focused on. Since a rigorous evaluation requires settings that take into account the constraints woven into the environment, conducting a detailed evaluation through simulation can generate and present information that is useful for user decision-making.

[0096] If, in step S17, no action improvement plan is found that improves the action value and uncertainty (S17:NO), in step S20, the action evaluation unit 241 performs a predetermined number of simulations on the original action and calculates the evaluation value.

[0097] In step S21, the behavior evaluation result display unit 242 presents the evaluation results from the behavior evaluation unit 241 to the user. The behavior evaluation result display unit 242 generates an evaluation result screen and displays it on the user terminal 120. Figure 15 shows an example of the behavior evaluation result screen 480. The behavior evaluation result screen 480 shows both the evaluation result of the predicted behavior (original behavior) 216 predicted by the behavior prediction unit 211 and the evaluation result of the behavior improvement proposal 235 from the behavior improvement proposal search unit 231.

[0098] Table 481 shows the composition of the original behavior, specifically the identifiers and values ​​of each element of the behavior. In this example, the element of the behavior is a generator, and its value is the amount of power generated. Graph 482 shows the simulation results of the original behavior. Specifically, Graph 482 shows a histogram of the sum of rewards in the simulation of the original behavior. The horizontal axis represents the sum of rewards, and the vertical axis represents the frequency.

[0099] Table 483 shows the structure of the behavior improvement plan, specifically the identifiers and values ​​of each behavior element. In this example, the behavior element is a generator, and its value is the amount of power generated. Graph 484 shows the simulation results of the behavior improvement plan. Specifically, Graph 484 shows a histogram of the sum of rewards in the simulation of the behavior improvement plan. The horizontal axis represents the sum of rewards, and the vertical axis represents the frequency.

[0100] Section 485 presents comparative information between the simulation results of the original behavior and the simulation results of the proposed behavior improvement. Specifically, Section 485 shows statistical information on the sum of rewards for both the original behavior and the proposed behavior improvement. In the example in Figure 15, Section 485 shows the mean sum of rewards, the standard deviation of the sum of rewards, the worst-case scenario with the smallest sum of rewards, and the probability that the sum of rewards is less than -5. The source of the arrow indicates the value for the original behavior, and the tip of the arrow indicates the value for the proposed behavior improvement. By showing the comparative results of the two behaviors in this way, it is possible to effectively support the user's decision-making.

[0101] If no behavioral improvement plan that meets the specified conditions is found, only the evaluation result of the predicted behavior (original behavior) 216 predicted by the behavioral prediction unit 211 may be displayed.

[0102] The user specifies the original action using button 486 and the proposed action improvement using button 487. Returning to Figure 9, in step S22, when the action evaluation result display unit 242 receives the action specified by the user, it transmits the information of the specified action 246 to the operation management system 110.

[0103] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are described in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations.

[0104] Furthermore, each of the above-mentioned configurations, functions, and processing units may be implemented in hardware, either partially or entirely, by designing them as integrated circuits, for example. Alternatively, each of the above-mentioned configurations and functions may be implemented in software by having the processor interpret and execute programs that implement each function. Information such as programs, tables, and files that implement each function can be stored in memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card or SD card.

[0105] Furthermore, the control lines and information lines shown are those deemed necessary for explanatory purposes, and not all control lines and information lines are necessarily shown in the actual product. In practice, it is reasonable to assume that almost all components are interconnected. [Explanation of symbols]

[0106] 100 Decision support system, 101 CPU, 102 Memory, 103 Auxiliary storage device, 104 Network interface, 110 Operation management system, 112 Memory, 113 Auxiliary storage device, 114 Network interface, 120 User terminal, 122 Memory, 123 Auxiliary storage device, 124 Network interface, 125 Input device, 126 Output device, 140 Network, 210 Behavior and behavioral value prediction unit, 220 Behavioral risk factor analysis unit, 230 Behavioral improvement proposal search and presentation unit, 240 Behavioral improvement proposal evaluation unit

Claims

1. A computer system, The computing unit and Includes a storage device, The aforementioned storage device stores a model that outputs value-based predictive behavior for input data, The aforementioned computing device is To explain the first prediction process by the model that outputs a first predicted action from the first input data, we obtain explanatory data that includes the values ​​of multiple explanatory elements, Determine the contribution of each of the aforementioned multiple explanatory elements to the value and the uncertainty of the value in the first prediction process. Based on the contribution values, the risk elements in the first prediction process are detected in the plurality of explanatory elements. The information on the aforementioned risk factors is presented, The data to be explained above is the first predicted behavior, The computing device is a computer system that searches for and presents action improvement proposals for the first predicted action that improve the value and the uncertainty of the value by changing the values ​​of some elements in the first predicted action.

2. The computer system according to Claim 1, The computing device is a computer system that detects, as risk elements, explanatory elements that contribute to worsening both the value and the uncertainty of the value.

3. The computer system according to Claim 1, The aforementioned elements include the aforementioned risk element, the computer system.

4. The computer system according to Claim 1, A computer system in which some of the aforementioned elements are specified by the user.

5. The computer system according to Claim 1, The aforementioned computing device is The first predicted behavior and the proposed behavior improvement are evaluated by simulation. A computer system that presents the results of the aforementioned evaluation.

6. The computer system according to claim 1, The uncertainty of the aforementioned value is the Aleatoric uncertainty of the aforementioned value, The computing device is a computer system that searches for proposed improvements to the action in an action where the epithemic uncertainty of the value is below a threshold.

7. A method performed by the system, The system stores a model that outputs value-based predictive behavior for input data, The aforementioned method, The system acquires explanatory data, which includes the values ​​of multiple explanatory elements, for explaining the first prediction process by the model that outputs a first predicted action from the first input data. The system determines the contribution of each of the plurality of explanatory elements to the value and the uncertainty of the value in the first prediction process, The system detects risk elements in the first prediction process based on the contribution values ​​in the plurality of explanatory elements. The system presents information on the risk elements, The data to be explained above is the first predicted behavior, The method described above is a method in which the system searches for and presents action improvement proposals that improve the value and the uncertainty of the value for the first predicted action by changing the values ​​of some elements in the first predicted action.

Images