Information processor, machine learning method, and information processing method

Abstract

To provide a technique that allows a user to easily determine what kind of future scenario AI is assuming and outputting.SOLUTION: An information processor according to a preferred aspect of the invention includes: an agent for outputting a response on the basis of a state observed from environment with stochastic state transitions; an individual evaluation model for evaluating the response assuming that a part of the stochastic state transition occurs; and a plan explanation processing unit for outputting information based on the evaluation in association with information based on the response.SELECTED DRAWING: Figure 16

Description

Technical Field

[0001] The present invention relates to a technique for evaluating and presenting explanations for plans and actions output by a machine learning system.

Background Art

[0002] Reinforcement learning, which is one of the machine learning systems, is a mechanism for learning the parameters of a machine learning model (artificial intelligence: AI) so that actions leading to appropriate rewards are output in an environment (task) where rewards are given for actions. Due to its high performance, its scope of application has been expanded to tasks such as social infrastructure and medical sites. For example, in order to minimize damage caused by predicted natural disasters, etc., it is possible to formulate a prior countermeasure plan to appropriately allocate resources such as personnel in advance. However, in order to utilize a machine learning system in such mission-critical tasks, in addition to high usefulness, it is required to satisfy requirements regarding various properties such as transparency, fairness, and interpretability. Therefore, research on XAI (Explainable AI), which is a technique for explaining the basis of judgment of a machine learning system, has been rapidly progressing.

[0003] As an XAI technology for reinforcement learning, in Non-Patent Document 1, the parts of an image input to an AI model that the AI emphasizes are visualized with a heatmap. Especially in the framework of supervised learning, development of such explanation techniques for input data has been actively carried out. On the other hand, since the actions of AI in reinforcement learning are learned in anticipation of future rewards and events, attention has been focused on "future-oriented explanations" for future events intended by AI rather than "past-oriented explanations" using input data.

[0004] For example, in Non-Patent Document 2, a method is proposed that uses the scenario with the highest occurrence probability for a series of future events (state transitions) (hereinafter referred to as scenarios) that will occur after the action to be explained for the explanation.

[0005] Furthermore, Non-Patent Document 3 proposes a method for visualizing the intent behind the actions of a reinforcement learning AI by using a supervised learning AI model that outputs a table of all possible future state transitions and actions.

[0006] Furthermore, Patent Document 1 proposes a method for separating the AI, which evaluates a value called the Q-value representing the quality of behavior, according to each objective function. This makes it easier to learn behaviors that satisfy multiple objectives simultaneously and also provides insights into adjusting the weights of each objective function. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2019-159888 [Non-patent literature]

[0008] [Non-Patent Document 1] S. Greydanus, A. Koul, J. Dodge, and Alan Fern, “Visualizing and Understanding Atari Agents,” Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden, PMLR 80, 2018. [Non-Patent Document 2] JVD Waa, JV Diggelen, KVD Bosch, and M. Neerincx, “Contrastive Explanations for Reinforcement Learning in terms of Expected Consequences,” ArXiv, 1807.08706, 2018. [Non-Patent Document 3] H. Yau, C. Russell, and S. Hadfield, “What Did You Think Would Happen? Explaining Agent Behavior through Intended Outcomes,” Workshop on Extending Explainable AI Beyond Deep Models and Classifiers, Vienna, Austria, PMLR 119, 2020. [Overview of the Initiative] [Problems that the invention aims to solve]

[0009] The technology described in Non-Patent Document 2 is insufficient for interpreting the intentions of the AI. Reinforcement learning considers various scenarios and selects actions that are expected to be effective, including, for example, scenarios where the AI's actions are highly effective even with low probabilities, and risky scenarios where the reward remains low. Therefore, considering only the scenario with the highest probability of occurrence does not provide enough information to explain the AI's intentions. Rather than definitively selecting one scenario, a function is needed that selects scenarios according to the user's interests.

[0010] While the technology described in Non-Patent Document 3 allows for a comprehensive comparison of the AI's intended states, its practical application in real-world situations is difficult because a vast number of state transitions and actions are possible.

[0011] Although XAI is not considered in the technology described in Patent Document 1, even if we were to try to explain the AI's intentions using multiple objective functions, while it would be possible to extract the ones that the AI ​​prioritizes from the multiple objective functions, it would be difficult to determine the specific future scenarios that the AI ​​has envisioned.

[0012] Therefore, the objective of this invention is to provide a technology that allows users to easily determine what future scenarios the AI ​​is assuming when generating its output. [Means for solving the problem]

[0013] A preferred aspect of the present invention is an information processing device comprising: an agent that outputs a response based on a state observed from an environment involving probabilistic state transitions; an individual evaluation model that evaluates the response as if a part of the probabilistic state transitions were to occur; and a planning explanation processing unit that outputs information based on the evaluation in association with information based on the response.

[0014] More specifically, in the above, the agent and the individual evaluation model are machine learning models, the state is a feature obtained from the environment, and the individual evaluation model evaluates the response using the feature and the response as input.

[0015] More specifically, when training the agent, the individual evaluation model, and the expected value evaluation model that evaluates the Q value expectedly by looking at the entire set of probabilistic state transitions, the agent and the expected value evaluation model are trained using training data, and the individual evaluation model is trained using only a portion of the training data.

[0016] Another preferred aspect of the present invention is an information processing method performed by an information processing device comprising: a first learning model that inputs feature quantities based on an environment with probabilistic state transitions and outputs a response; and a second learning model that evaluates the response as if a part of the probabilistic state transitions were determined, the information processing method performing: a first step of inputting the feature quantities to the first learning model and causing it to output a response; a second step of inputting the feature quantities and the response to the second learning model and obtaining an evaluation value of the response; and a third step of outputting information based on the evaluation value in association with the response. [Effects of the Invention]

[0017] This technology allows users to easily determine what future scenarios the AI ​​is anticipating when generating its output. Other challenges, configurations, and effects will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0018] [Figure 1] It is a block diagram showing an example of the system configuration (hardware) of a machine learning system evaluation device. [Figure 2] It is a table diagram showing an example of the data structure of environmental data. [Figure 3] It is a table diagram showing an example of the data structure of feature data. [Figure 4] It is a table diagram showing an example of the data structure of a plan. [Figure 5] It is a table diagram showing an example of the data structure of environmental transition conditions. [Figure 6] It is a flowchart showing an example of a work flow for learning a plan generation agent, an expected value, and an individual evaluation model. [Figure 7] It is a table diagram showing an example of the data structure of individual evaluation conditions. [Figure 8] It is an image diagram showing an example of screen output in the learning stage of a machine learning model. [Figure 9] It is a flowchart showing an example of a work flow for calculating an error and updating a model of a machine learning model. [Figure 10] It is a flowchart showing an example of a work flow for explaining the intention in the actions and plans output by a machine learning system. [Figure 11] It is a table diagram showing an example of the data structure of scenario selection conditions. [Figure 12] It is an image diagram showing an example of screen output of a machine learning system evaluation result. [Figure 13] It is a block diagram showing an example of the system configuration (hardware) of a machine learning system evaluation device using comparison with a user plan. [Figure 14] It is a flowchart showing an example of a work flow for explaining a machine learning system using comparison with a user plan. [Figure 15] It is an image diagram showing an example of screen output for explaining a machine learning system using comparison with a user plan. [Figure 16]This is a block diagram illustrating the schematic configuration of the embodiment. [Modes for carrying out the invention]

[0019] Embodiments will be described in detail with reference to the drawings. However, the present invention is not to be construed as being limited to the embodiments described below. It will be readily apparent to those skilled in the art that the specific configuration can be modified without departing from the spirit or intent of the present invention.

[0020] In the configurations of the embodiments described below, the same reference numerals are used in common across different drawings for identical parts or parts having similar functions, and redundant explanations may be omitted.

[0021] When there are multiple elements with the same or similar function, they may be described using the same symbol but with different subscripts. However, if there is no need to distinguish between multiple elements, the subscript may be omitted in the description.

[0022] In this specification, notations such as "Part 1," "Part 2," and "Part 3" are used to identify components and do not necessarily limit their number, order, or content. Furthermore, the numbers used to identify components are used on a context-by-context basis, and a number used in one context does not necessarily indicate the same component in another context. Moreover, this does not prevent a component identified by one number from also performing the function of a component identified by another number.

[0023] The positions, sizes, shapes, and ranges of each component shown in drawings, etc., may not represent their actual positions, sizes, shapes, and ranges in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the positions, sizes, shapes, and ranges disclosed in drawings, etc.

[0024] The publications, patents, and patent applications cited herein constitute part of the description herein.

[0025] In this specification, elements expressed in the singular form shall include the plural form unless otherwise clearly indicated in the context.

[0026] In the following discussion, we will primarily focus on reinforcement learning systems that develop pre-emptive plans to appropriately allocate resources such as personnel in order to minimize damage from anticipated natural disasters. However, each method can be broadly applied to general reinforcement learning problems that output actions and plans (since a plan is a scheduled action, the two together are sometimes simply referred to as actions) in accordance with the state observed from the environment, such as the action selection of robots and game AI, the operation control of trains and automobiles, and employee shift scheduling.

[0027] The information processing device, machine learning method, and information processing method proposed in the embodiment include an agent part that outputs actions and plans in accordance with observed states from an environment with state transitions based on conditions such as probability; a part in which the user specifies the conditions for state transitions to be divided and evaluated; a part that estimates the value of actions and plans for each future state transition divided by the specified conditions; a part that processes questions from the user; a part that selects a state transition corresponding to the question processing result and calculates the future state and reward; and a part that generates an explanation of the intent of the actions and plans using the obtained information.

[0028] With this configuration, even in problem settings where there are a great many state transitions in the actions and plans output by the AI, by evaluating the value of each state transition divided according to user-specified conditions, specific future scenarios assumed by the AI ​​based on the user's interests can be presented, and useful information can be obtained to interpret the intentions of the AI's actions.

[0029] Hereinafter, several embodiments of the present invention will be described with reference to the drawings. However, these embodiments are merely examples of how to realize the present invention and do not limit the technical scope of the present invention. It will be readily apparent to those skilled in the art that the specific configuration can be modified without departing from the spirit or intent of the present invention.

[0030] In the configuration of the invention described below, identical or similar components or functions are denoted by the same reference numerals, and redundant descriptions are omitted.

[0031] The positions, sizes, shapes, and ranges of each component shown in the drawings, etc., may not represent the actual positions, sizes, shapes, and ranges, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not limited to the positions, sizes, shapes, and ranges, etc., disclosed in the drawings, etc. Hereinafter, embodiments of the present invention will be described with reference to the drawings. [Examples]

[0032] Figure 1 shows an example configuration of a machine learning system evaluation device that implements an embodiment of the present invention. The machine learning system evaluation device consists of a storage device 1001, a processing device 1002, an input device 1003, and an output device 1004.

[0033] The storage device 1001 is a general-purpose device for persistently storing data, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), and contains plan information 1010, an expected value evaluation model 1020 which is a machine learning model that evaluates the expected value of multiple state transitions for the plan output by the AI, an individual evaluation model 1030 which is a machine learning model that divides and evaluates the plan output by the AI ​​for each state transition based on user-specified conditions, and plan explanation information 1040. Note that the storage device 1001 may reside on the cloud or an external server rather than on a terminal like the other devices, and it is acceptable for the data to be accessible via a network.

[0034] The planning information 1010 includes a planning generation agent 1011 that outputs a plan according to the observed state of the environment, environment data 1012 (see Figure 2) that stores information about the environment, feature data 1013 (see Figure 3) that serves as input data for the agent, the plan 1014 (see Figure 4) output by the agent, environment transition conditions 1015 (see Figure 5) that specify the state transition conditions of the environment, model training data 1016 that serves as input data for training each machine learning model, and the evaluation result of the plan by the evaluation model 1017.

[0035] The plan explanation information 1040 includes individual evaluation conditions 1041, which are conditions for dividing and evaluating the AI's plan for each state transition; user question data 1042 regarding the AI's plan; scenario selection conditions 1043, which store the conditions for the state transitions specified based on the question; and answer data 1044, which are answers to the question.

[0036] The processing unit 1002 is a general-purpose computer and contains a machine learning model processing unit 1050, an environment processing unit 1060, a plan explanation processing unit 1070, a screen output unit 1080, and a data input unit 1090, all of which are stored in memory as software programs.

[0037] The plan explanation processing unit 1070 includes an individual evaluation processing unit 1071 that processes the individual evaluation model 1030, a question processing unit 1072 that processes user question data 1042 and scenario selection conditions 1043, and an explanation generation unit 1073 that generates answer data 1044 for the user.

[0038] The screen output unit 1080 is used to convert the plan 1014 and response data 1044 into a format that can be displayed on a screen.

[0039] The data input unit 1090 is used to set parameters and questions from the user.

[0040] The input device 1003 is a general-purpose input device for a computer, such as a mouse, keyboard, or touch panel.

[0041] The output device 1004 is a device such as a display, and displays information for user interaction through the screen output unit 1080. However, if it is not necessary for a human to confirm the evaluation results of the machine learning system (for example, if the evaluation results are directly passed to another system), an output device may not be provided.

[0042] The above configuration may be made up of a single device, or any part of the device may be made up of other computers connected via a network. In this embodiment, functions equivalent to those made up of software can also be realized with hardware such as FPGA (Field Programmable Gate Array) and ASIC (Application Specific Integrated Circuit).

[0043] Figure 2 shows an example of environmental data 1012. Environmental data 1012 consists of data that does not change over time 1012C, data that changes over time 1012V, and machine learning parameters 1012P. Here, we consider an example of planning the appropriate allocation of resources for power transmission and distribution in areas 1 to 3.

[0044] The time-unchanging data 1012C is a database consisting of categories 21 representing the category information of each data item, and their corresponding values ​​22. For example, it records the number of facilities such as power plants in each area, and the distance between areas.

[0045] The time-varying data 1012V is a database consisting of 23 steps representing time cross-sections, 24 categories of data items, and 25 values. As an example, it records power demand and temperature for each area that vary over time.

[0046] The machine learning parameter database 1012P consists of 26 categories and 27 values ​​for each parameter used in machine learning.

[0047] Environmental data 1012 is, for example, information entered by a user or obtained from a predetermined information processing device. Furthermore, the data format is not limited to table data; it may also be, for example, image information or calculation formulas.

[0048] Figure 3 shows an example of feature data 1013. Feature data 1013 is a database consisting of a category 31 and its value 32 for each feature. Feature data 1013 is generated by the environment processing unit 1060 based on the environment data 1012, and the values ​​32 are in a data format (mainly numerical or categorical data) that can be input to the plan generation agent 1011. In this specification, "state" refers to the feature data 1013 at each time step.

[0049] Figure 4 shows an example of plan 1014. Plan 1014 is a database consisting of a step number 41 representing the target time step, a category 42 for the plan item, and its value 43. Plan 1014 is output for each time step by the plan generation agent 1011. In this example, plan 1014 is a plan to appropriately allocate resources (personnel) for power transmission and distribution in a certain area.

[0050] Figure 5 shows an example of environmental transition conditions 1015. Environmental transition conditions 1015 is a database consisting of a step number 51 representing the target time step, a category 52 of transition condition items, and their values ​​53. Environmental transition conditions 1015 are defined by probabilities and conditional expressions and are reflected in the feature data 1013 of the next time step by the environmental processing unit 1060. In this specification, "occurrence of an event" refers to the occurrence of an environmental transition. In this example, the environmental transition conditions show the probability of a power outage for each area at each step.

[0051] The following describes the operating process of the machine learning system evaluation device. This embodiment can be broadly divided into a learning phase and an explanation phase.

[0052] Figure 6 is a flowchart illustrating an example of the learning process for the plan generation agent 1011, the expected value evaluation model 1020, and the individual evaluation model 1030. In this embodiment, learning data is accumulated by repeating multiple episodes in which the agent outputs actions and plans according to the state observed from the environment at each time step (s603~s610), and the successive error function is calculated and the machine learning model is updated (s612) to train a highly accurate model.

[0053] The operation based on the flowchart is as follows:

[0054] Step s601: The user specifies individual evaluation criteria 1041. This can be done interactively on a GUI, by sending data from another information processing device in the form of a file, etc. Automatic classification using algorithms such as clustering is also acceptable. Details are explained in Figures 7 and 8.

[0055] Step s602: Based on the individual evaluation conditions 1041, the individual evaluation processing unit 1071 generates the individual evaluation model 1030. The number of models is determined by the conditions set in the individual evaluation conditions 1041. Examples of the individual evaluation conditions 1041 and the number of models generated therefrom are shown in Figures 7 and 8. The individual evaluation model 1030 is assumed to be a machine learning model such as a neural network. The individual evaluation model 1030 can handle the same features as the expected value evaluation model 1020 as input data, and its output is also basically structured as a scalar value called the Q value, which is stored in the evaluation result 1017, similar to the expected value evaluation model 1020. Details of the Q value are shown in Figure 9.

[0056] Step s603: Start the episode loop to accumulate training data and update the model.

[0057] Step s604: The environmental processing unit 1060 takes the data from the first time step as input from the environmental data 1012, specifically the data 1012C which does not change over time and the data 1012V which does change over time, and outputs feature data 1013 (Figure 3).

[0058] Step s605: Starts a loop that processes each time step in an episode. An episode consists of multiple time steps. The number of time steps is specified by data 1012V, which involves the time change of environmental data, and machine learning parameters 1012P. For example, an episode might cover the period from when a typhoon makes landfall until it passes, with time steps being 13:00, 14:00, 15:00, etc. Environmental transition conditions 1015 determine how the environment changes (where power outages occur) when time steps change.

[0059] Step s606: With feature data 1013 as input, the plan generation agent 1011 outputs a plan 1014. The agent is a machine learning model such as a general neural network.

[0060] Step s607: The environmental processing unit 1060 generates feature data 1013 for the next time step, taking as input the data item for the next time step from the time-varying data 1012V among the environmental data 1012, the plan 1014 output in step s606, and the environmental transition condition 1015.

[0061] Step s608: The environment processing unit 1060 calculates the reward using the feature data 1013 and the plan 1014 for the current and next time steps as input. The reward is a value that represents the benefit or penalty obtained by the plan output by the agent before and after the state transition, and is generally a scalar value. In this embodiment, this includes the cost of allocating resources such as personnel, and the amount of damage that could be reduced by appropriate allocation. The processing in steps s606 to s608 applies reinforcement learning known as Actor-Critic.

[0062] Step s609: The environment processing unit 1060 combines the feature data 1013 for the current time step and the next time step, the plan 1014, the reward value generated in step s608, and the labels corresponding to the individual evaluation conditions 1041 (see Figure 7) into a single tuple or similar. The label condition determination is based on the state transitions processed in step s607. The created training data is stored as model training data 1016 by the environment processing unit 1060.

[0063] Step s610: Repeats for the number of steps specified by the time-varying environmental data 1012V or the machine learning parameter 1012P. The number of steps may also be specified by a conditional expression. The environmental processing unit 1060 determines whether to terminate or continue.

[0064] Step s611: The environment processing unit 1060 determines whether the model update frequency condition specified by the machine learning parameter 1012P is met. If it is met, proceed to step s612; otherwise, proceed to step s613.

[0065] Step s612: Train and update the machine learning model using the accumulated data. The detailed process is described in Figure 9.

[0066] Step s613: Repeat for the number of episodes specified by the machine learning parameter 1012P. The environment processing unit 1060 determines whether to terminate or continue.

[0067] Figure 7 shows an example of individual evaluation conditions 1041. Individual evaluation conditions 1041 is a database consisting of a label 71 for each condition and a condition 72 which is the content of that condition.

[0068] Label 71 is stored in the training data in step s609 of Figure 6 and is used as an indicator to show which individual evaluation condition it corresponds to.

[0069] Condition 72 corresponds to environmental transition condition 1015, and stores information such as which event occurred and the magnitude of the impact caused by that environmental transition. Multiple environmental transitions may also apply to a single condition.

[0070] Condition 72 is specified by the user, for example, based on the previously set environment transition condition 1015. Condition 72 can be a time step-independent condition (applicable to any time step) or a condition specific to each time step. Condition 72 can be a condition tied to a specific program variable name or value (e.g., if variable A becomes greater than or equal to value X...) or a condition corresponding to environment transition condition 1015 (e.g., if the "power outage in area 1" described in the environment transition condition occurs...).

[0071] In the example in Figure 7, a condition independent of the time step is described. Therefore, if a "power outage in Area 1" occurs at any time step due to the environmental transition condition 1015, the training data will be labeled "Label 1" at step s609 according to the individual evaluation condition 1041. If the individual evaluation condition 1041 is defined for each time step, such as "power outage in Area 1 at time step 1" and "power outage in Area 1 at time step 2", then the label will be assigned according to the occurrence of the environmental transition condition at the corresponding time step.

[0072] Figure 8 shows an example of an interface where the user inputs individual evaluation conditions 1041 and a file output of the model training results. It consists of a file input section 801 for the individual evaluation conditions 1041, a button 802 to start model training, and a file 803 of the trained output model. In Figure 8, five individual evaluation models are created because five labels were specified by the individual evaluation conditions 1041 in Figure 7.

[0073] Figure 9 is a flowchart illustrating an example of the error calculation and model update process for a machine learning model performed in step s612 of Figure 6. The error function is calculated using data sampled from the training data, and each model parameter is updated. While a neural network is shown as an example of the machine learning model used, no specific model is required as long as it can be used for reinforcement learning.

[0074] Step s901: The machine learning model processing unit 1050 samples arbitrary data from the model training data 1016. The total number and conditions may be specified in the environment data 1012.

[0075] Step s902: For each sampled training data, the expected value evaluation model 1020 outputs a Q-value, taking the feature data 1013 before the state transition and the plan 1014 as input. The Q-value is a general scalar value in reinforcement learning that represents the goodness of the plan in that state, but any value that represents the goodness of the plan other than the Q-value is acceptable. The evaluated training data is stored in evaluation result 1017, associated with the Q-value. The expected value evaluation model 1020 is assumed to be generated by a known method, for example, by the environment processing unit.

[0076] Step s903: The machine learning model processing unit 1050 calculates the error function using the evaluation result 1017 and updates the model. For example, in a general Q-learning framework, if the time step before transition is t, the time after transition is t+1, the reward is R_t+1, the learning rate is γ, the state before transition is s_t, the state after transition is s_t+1, the plan is a_t, the plan for the next time step is a_t+1, and the Q value is Q, then the error function is expressed as follows.

[0077]

number

[0078] Here, Q_EX is the Q-value calculated in step s902, and Q_EX_target is the Q-value evaluated for the state s_t+1 in the next time step and the plan a_t+1 output by the plan generation agent 1011 using it as input. In general Q-learning, for the purpose of learning stabilization, the evaluation of Q_EX_target uses a target network, which is the expected value evaluation model 1020 from the model used in step s902 before it is updated. The learning rate γ is a machine learning parameter specified and included in the environment data 1012.

[0079] Step s904: The machine learning model processing unit 1050 trains the plan generation agent 1011. In typical Q-learning, the average Q-value of the data stored in the evaluation result 1017 in step s902 is multiplied by -1 and used as the error function for training. The plan generation agent continues to train in order to create plans that result in higher Q-values.

[0080] Step s905: Perform model update processing for each individual evaluation model 1030.

[0081] Step s906: The machine learning model processing unit 1050 extracts data with corresponding individual evaluation labels from the model training data 1016 sampled in step s901.

[0082] Step s907: For each sampled training data, the individual evaluation model 1030 outputs a Q-value, taking the feature data 1013 before the state transition and the plan 1014 as input. The evaluated training data is stored in the evaluation result 1017, associated with the Q-value.

[0083] Step s908: The machine learning model processing unit 1050 calculates the error function using the evaluation result 1017 and updates the individual evaluation model 1030. Generally, the process in step s903 should be performed for each individual evaluation model.

[0084] However, if the pre-update individual evaluation models are used as the target network, the network will learn to minimize the error in a different direction than the expected value evaluation model 1020. Therefore, by using the expected value evaluation model 1020 as the target network and calculating the error between the part that estimates the value for each state transition and the expected value, it becomes possible to perform Q-value decomposition at a granularity that matches the user's interests, which is the objective of this embodiment, using values ​​that maintain consistency between the expected value Q-value and the individual Q-values. In addition, since each individual evaluation model is independent, learning can be accelerated by parallel processing.

[0085] Step s909: The process ends after updating all individual evaluation models 1030. Note that for models from which no data could be sampled in step s906, the update process does not need to be performed.

[0086] Figure 10 shows an example of the explanation process of a machine learning system that utilizes the trained individual evaluation model 1030. In the explanation phase, the system processes user questions, simulates the corresponding state transitions from the Q-value vectors individually evaluated for each state transition, and displays the results. This process makes it possible to interpret what future scenarios the AI ​​expected and planned for.

[0087] Step s101: The environment processing unit 1060 generates feature data 1013 for the time step to be described from the environment data 1012. The target time step and conditions are specified by the user using the data input unit 1090 or by another information processing device.

[0088] Step s102: The design generation agent 1011 outputs a design 1014 using the feature data 1013 as input.

[0089] Step s103: The expected value evaluation model 1020 and the individual evaluation model 1030 output Q values ​​using the feature data 1013 and the plan 1014 as input. The individual evaluation model used is determined by the environment processing unit 1060, which refers to the environment data 1012 and uses only those corresponding to the state transitions that can occur at the current time step.

[0090] Step s104: The user inputs question data 1042 using the input device 1003. Question input can be done by uploading a file on the GUI using the data input unit 1090, or by inputting in natural language.

[0091] Step s105: The question processing unit 1072 takes the user's question data 1042 and the scenario selection conditions 1043 (see Figure 10) as input and selects an appropriate state transition from the Q-value vector of the individual evaluation output from the individual evaluation model 1030 in step s103.

[0092] Step s106: To simulate the selected state transition, the environment processing unit 1060 generates feature data 1013 for the next time step using the environment data 1012 and the plan 1014.

[0093] Step s107: The environment processing unit 1060 calculates the reward using the feature data 1013 and the plan 1014 for the current time step and the next time step as input.

[0094] Step s108: The explanation generation unit 1073 generates response data 1044 for the user.

[0095] Step s109: The screen output unit 1080 converts the response data 1044 and other data into a GUI format and displays it on the output device 1004 (see Figure 12).

[0096] Figure 11 shows an example of a scenario selection condition 1043. The scenario selection condition 1043 is a database consisting of user questions 111 and corresponding state transitions 112. The question processing unit 1072 can select an appropriate state transition 112 from the scenario selection condition 1043 by converting the user question data 1042 into a format corresponding to the question 111. For example, by displaying the state transition that shows the maximum Q value in response to the question "Which state transition is the most expected?", it is possible to find out the specific event in which the plan 1014 is most effective. Note that the scenario selection condition 1043 is not limited to a table data format; it can also be a conditional expression. Furthermore, the state transition 112 may be a Q value that satisfies predetermined conditions, not limited to the maximum or minimum.

[0097] Figure 12 shows an example of screen output, including response data 1044, generated by the explanation generation unit 1073. It includes an example 1201 of the outputted plan 1014, an example 1202 of the plan 1014 graphically visualized, a file input unit 1203 for the scenario selection conditions 1043, a file input unit 1204 for user question data 1042, and a button 1205 for uploading a file and starting the explanation.

[0098] Furthermore, it consists of an example 1206 of the user's question text, an answer text 1207, a Q-value vector 1208 that highlights the selected state transition by listing the Q-values ​​of multiple individual evaluation models 1030, and an example 1209 that graphically visualizes the environment and reward after the selected state transition.

[0099] First, for example 1201 and example 1202, which are displayed based on the plan 1014 output from the plan generation agent 1011, the user uploads the scenario selection conditions 1043 and question data 1042 using the file input units 1203 and 1204.

[0100] Next, the question processing unit 1072 determines the state transition 112 while comparing the question data 1042 with the scenario selection conditions 1043, and presents the answer statement 1207, Q value vector 1208, and display example 1209 as answer data 1044.

[0101] In this example, the user is asking about the most anticipated scenario, so the state transition with the highest Q value is selected and highlighted in the Q value vector 1208. Note that the planning information and response information do not need to be displayed on the screen simultaneously; they can be presented by switching between the two screens.

[0102] In this embodiment, the Q value is presented to the user, but since this value is abstract, it may not be suitable as an explanation. In such cases, in addition to outputting the Q value in step s103 of Figure 10, the environment processing unit 1060 may be used to convert the Q value into a value that is easier for the user to interpret. For example, in this embodiment, this would include resource utilization rates such as power outage recovery time and personnel. In addition, the probability values ​​of state transitions can be displayed separately, and the confidence level of the AI's plan can be estimated by calculating the uncertainty of each state transition. For uncertainty, estimation methods utilizing known ensemble learning, etc., can be used.

[0103] Furthermore, while this embodiment shows the state transitions and Q-values ​​in a single time step, interpretability can be further enhanced by presenting a series of multiple time steps. In such cases, one could repeat the explanatory process in Figure 10 any number of times, or specify conditions using environmental data 1012 or scenario selection conditions 1043.

[0104] Furthermore, while this embodiment primarily assumes that one state transition is specified for each individual evaluation model, multiple state transitions may be associated with each model. During the explanation stage, the scenario selection condition 1043 specifies which state transition to use. Multiple state transitions, not just one, may be displayed.

[0105] In addition, the obtained Q-value vector can be used not only for explanation but also as a hint for determining a strategy for additional learning aimed at improving the performance of the plan generation agent 1011. For example, if the Q-value is small for a future event that an expert would consider important, the state transitions can be displayed as response data, allowing the user to decide on a strategy to additionally learn episodes in which that event occurs. [Examples]

[0106] Figure 13 is a block diagram showing a machine learning system evaluation device of the second embodiment. In this embodiment, a method is described for improving the interpretability of the AI-generated plan 1014 by comparing it with a plan assumed by the user.

[0107] As an example for implementing the second embodiment, the apparatus shown in Figure 13, which is an extension of Figure 1, is used. Additional features from the apparatus diagram in Figure 1 include the user plan 1345 in the plan explanation information 1040 of the storage device 1001, and the user plan processing unit 1374 in the plan explanation processing unit 1070 of the processing unit 1002. The specific uses of these will be described in a later section.

[0108] Figure 14 shows an example of an explanatory process using a comparison with the user plan. Many of the processes are similar to those in Figure 10, so only the differences will be described in detail.

[0109] Steps 1401-1403: These are the same as steps 101-103 in Figure 10.

[0110] Step s1404: The user inputs the user's expected user plan 1345 in addition to the question data 1042. The data format is the same as the AI-generated plan 1014.

[0111] Step s1405: For the user plan 1345, the expected value evaluation model 1020 and the individual evaluation model 1030 output Q values.

[0112] Step s1406: The question processing unit 1072 takes the user's question data 1042 and the scenario selection condition 1043 (see Figure 10) as input, compares the Q-value vectors of the individual evaluations of the AI ​​plan 1014 and the user plan 1345, and selects an appropriate state transition. For example, in the case of the question, "Why is the AI ​​plan better than the user plan?", by selecting a state transition that shows a large Q-value in the AI ​​plan 1014 and a low Q-value in the user plan 1345, it is possible to indicate items that show a large difference in the intended future events of each plan.

[0113] Steps s1407~s1410: These are the same as steps s106~s109 in Figure 10. However, for the user plan 1345, the user plan processing unit 1374 performs the same processing as the AI-generated plan, and adds the processing results to the response data.

[0114] Figure 15 shows an example of a screen output for an explanation using a comparison with the user plan 1345. It consists of an example 1501 of the AI ​​plan 1014, an example 1502 of a graphical visualization of plan 1014, a file input section 1503 for the scenario selection conditions 1043, a file input section 1504 for the user's question data 1042, a file input section 1505 for the user's expected user plan 1345, a button 1506 for uploading a file and starting the explanation, an example 1507 of displaying the user's question text, an answer text 1508, a Q-value vector 1509 of the AI ​​plan highlighting the selected state transitions, an example 1510 of a graphical visualization of the environment after the selected state transitions and the reward of the AI ​​plan, a Q-value vector 1511 of the user plan 1345 highlighting the selected state transitions, and an example 1512 of a graphical visualization of the environment after the selected state transitions and the reward of the user plan. First, the user uploads the scenario selection conditions 1043 and question data 1042 to the outputted plan 1014. Next, the question processing unit 1072 determines the state transitions while comparing them with the scenario selection conditions 1043, and presents the response data 1044, ranging from response sentence 1508 to visualization example 1512. In this example, by selecting the state transition with the largest difference in the intended future events of each plan, information is presented to interpret the AI's intentions in response to the question, "Why is the AI's plan better than the user's plan?" Note that each item may be displayed on a separate screen.

[0115] Figure 16 is a block diagram conceptually showing the configuration of the embodiment described in Figure 10. The configuration of the embodiment applies reinforcement learning known as Actor-Critic. Actor-Critic is a reinforcement learning framework consisting of an Actor, which selects and executes an Action based on the observed state of the environment, and a Critic, which evaluates the Action selected by the Actor. The Actor optimizes its plan (Action) based on the evaluation.

[0116] In this embodiment, the actor corresponds to the plan generation agent 1011. The plan generation agent 1011 takes feature data 1013, which is created from environment data 1012, as input and generates a plan 1014. From the plan 1014, the time-varying environment data 1012V, and the environment transition conditions 1015, the environment processing unit 1060 generates feature data 1013 for the next time step (a state transition occurs).

[0117] Critic 1603 corresponds to the expected value evaluation model 1020. In the embodiment already described, the expected value evaluation model 1020 takes feature data 1013 and plan 1014 as input and outputs a Q-score 1601 that represents the goodness of the plan (action) in that state. Here, the Q-score 1601 output by the expected value evaluation model 1020 represents the expected value for all state transition functions.

[0118] In the embodiment, as described above, one or more individual evaluation models 1030 are provided to realize the functionality of XAI. The individual evaluation model 1030 is a machine learning model that divides and evaluates the plan 1014 output by the plan generation agent 1011 into state transitions based on arbitrary conditions. In other words, the individual evaluation model is a model that evaluates a fixed portion of the probabilistic state transitions that the expected value evaluation model uses as the basis for its evaluation.

[0119] While the expected value evaluation model 1020 looks at the entire set of probabilistic state transitions and evaluates the Q value in an expected way, the individual evaluation model 1030 fixes a portion of the probabilistic state transitions as occurring and evaluates the Q value in that case. Based on the Q value 1602 output by the individual evaluation model 1030, the plan explanation processing unit 1070 generates explanatory information about the plan 1014 of the plan generation agent 1011.

[0120] Since each individual evaluation model 1030 performs evaluations based on different scenarios, it is possible to determine, based on the Q-value 1602 output by the individual evaluation model 1030, which scenarios the plan 1014 output by the plan generation agent 1011 is meaningful for.

[0121] According to the above embodiment, by using an agent part that outputs actions and plans in accordance with the observed state from an environment with state transitions based on conditions such as probability, a part that specifies individual evaluation conditions for plans based on the user's interests, an individual evaluation model part that estimates the value of each future state transition, a part that processes questions from the user, a part that selects an individual evaluation model with state transitions corresponding to the processing results and calculates future states and rewards, and a part that generates an explanation of the intent of actions and plans using the obtained information, it is possible to present specific future scenarios assumed by the AI ​​in accordance with the user's interests in order to interpret the intent of actions and plans output by a reinforcement learning-based machine learning system.

[0122] According to the above embodiment, the output of the machine learning model can be easily interpreted for each scenario, enabling efficient planning, reduced energy consumption, decreased carbon emissions, prevention of global warming, and contribution to the realization of a sustainable society. [Explanation of symbols]

[0123] 1001...Memory device, 1002...Processing device, 1011...Plan generation agent, 1012...Environmental data, 1013...Feature data, 1014...Plan, 1015...Environmental transition conditions, 1016...Model training data, 1017...Evaluation results, 1020...Expected value evaluation model, 1030...Individual evaluation model, 1041...Individual evaluation conditions, 1042...Question data, 1043...Scenario selection conditions, 1044...Answer data, 1050...Machine learning model processing unit, 1060...Environment processing unit, 1071...Individual evaluation processing unit, 1072...Question processing unit, 1073...Explanation generation unit, 1080...Screen output unit, 1090...Data input unit, 1208...Q-value vector, 1345...User plan, 1374...User plan processing unit

Claims

1. an agent that outputs a response based on a state observed from an environment with probabilistic state transitions; an individual evaluation model that evaluates the response assuming that some of the probabilistic state transitions occur; a plan explanation processing unit that outputs information based on the evaluation in association with information based on the response; An information processing device having the above.

2. In claim 1, the agent and the individual evaluation model are machine learning models; the state is a feature obtained from the environment, the individual evaluation model evaluates the response using the feature amount and the response as input; Information processing device.

3. In claim 2, The individual evaluation model evaluates the response using a Q value. Information processing device.

4. In claim 2, There are multiple types of the individual evaluation models, The plan explanation processing unit has a question processing unit that receives a question from a user as an input and selects a predetermined individual evaluation model from a plurality of individual evaluation models based on the question. Information processing device.

5. In claim 2, There are multiple types of the individual evaluation models, The plan explanation processing unit has an explanation generation unit that outputs an explanation regarding an individual evaluation model in which the evaluation satisfies a predetermined condition. Information processing device.

6. In claim 2, There are multiple types of the individual evaluation models, The plan explanation processing unit has an explanation generation unit that simultaneously displays evaluations of multiple types of the individual evaluation models. Information processing device.

7. In claim 3, The plan explanation processing unit has an explanation generation unit that converts the Q value into another numerical value and outputs it, Information processing device.

8. In claim 2, further comprising a user plan processing unit that processes a user plan consisting of data in the same format as the response; the user plan processing unit causes the individual evaluation model to evaluate the user plan using the feature amount and the user plan as input; The plan explanation processing unit further outputs information based on the evaluation of the user plan. Information processing device.

9. A machine learning method for machine learning the agent of claim 2, comprising: When learning the agent, the individual evaluation model, and the expected value evaluation model that evaluates the Q value as an expected value by looking at the entire probabilistic state transition, using learning data to learn the agent and the expected value evaluation model; The individual evaluation model is trained using only a portion of the training data. Machine learning methods.

10. The machine learning method of claim 9, The agent and the expectation evaluation model are trained by reinforcement learning using an actor-critic method. Machine learning methods.

11. The machine learning method of claim 10, When learning the individual evaluation model, an output of the expected value evaluation model is used. Machine learning methods.

12. The machine learning method of claim 11, By calculating an error between the output of the individual evaluation model and the output of the expected value evaluation model, the individual evaluation model is trained while ensuring consistency of the output. Machine learning methods.

13. An information processing method executed by an information processing device including: a first learning model that receives a feature based on an environment involving a stochastic state transition and outputs a response; and a second learning model that evaluates the response assuming that a part of the stochastic state transition is determined, a first step of inputting the feature to the first learning model and causing the first learning model to output a response; a second step of inputting the feature and the response into the second learning model to obtain an evaluation value of the response; a third step of outputting information based on the evaluation value in association with the response; An information processing method that performs the above.

14. 14. An information processing method according to claim 13, preparing a plurality of the second learning models; Execute at least one of outputting information based on an evaluation value of the second learning model that satisfies a condition specified by a user, and outputting information based on the second learning model that has output an evaluation value that satisfies a condition specified by a user. Information processing methods.

15. 14. An information processing method according to claim 13, training the first learning model by reinforcement learning using an actor-critic method; training the second learning model using only data having the determined state transition among the learning data used in training the first learning model; Information processing methods.

Images