Learning device, learning method, and program

Abstract

This learning device comprises: a model acquisition means for acquiring, through performing learning using a dataset in which states of environment where an agent executes behaviors, behaviors that can be executed in the states, and next states where the behaviors have been performed in the states are associated with each other, a model that receives input of a state and a behavior and outputs a next state; a feedback information acquisition means for acquiring, on the basis of the obtained model, feedback information that is information for use in training the model or training a new model for receiving input of states and behaviors and outputting next states; and a measure management means for learning a measure indicating a behavior of the agent according to the state, using the model obtained through training using the feedback information.

Description

Learning device, display device, learning method, display method, and recording medium

[0001] The present invention relates to a learning device, a display device, a learning method, a display method, and a recording medium.

[0002] Control learning for a control target may be performed offline. Here, offline learning refers to learning using data obtained in advance. For example, Patent Document 1 describes a technique for adjusting the parameters of a dynamics model by performing offline iterative learning control using motion data obtained by making an actual robot arm trace a circular trajectory and motion data obtained by making a simulator to which a dynamics model of the robot arm is applied trace a circular trajectory.

[0003] Japanese Patent Application Publication No. 2020-032481

[0004] When learning control of a controlled object using previously obtained data, if there is a state for which sufficient data is not available among the possible states of the controlled object, the accuracy of learning control in that state may be low. It is preferable to be able to improve the accuracy of learning control in that state, even for states for which sufficient data is not available.

[0005] An example of an object of the present invention is to provide a learning device, a display device, a learning method, a display method, and a recording medium that can solve the above-mentioned problems.

[0006] According to a first aspect of the present invention, a learning device includes: a model acquisition means for acquiring a model that takes a state and an action as input and outputs a next state through learning using data that links the state of the environment in which an agent performs an action, the actions that can be performed in that state, and the next state when the action is performed in that state; a feedback information acquisition means for acquiring feedback information based on the acquired model, which is information used for learning the model or for learning a new model that takes the state and the action as input and outputs the next state; and a policy management means for learning a policy that indicates the behavior of the agent according to the state, using the model acquired through learning using the feedback information.

[0007] According to a second aspect of the present invention, a display device includes display means for displaying, for each item included in information indicating a next state output by a model simulating an environment in which an agent performs an action in response to input of information indicating a state and information indicating an action, an evaluation index value of the accuracy of that item in the information indicating the next state. Items included in the information are also referred to as information items.

[0008] According to a third aspect of the present invention, a display device includes a display means for displaying, for each item included in information indicating a next state that is output by a model simulating an environment in which an agent performs an action in response to input of information indicating a state and information indicating an action, an evaluation index value for the accuracy of that item in the information indicating the next state.

[0009] According to a fourth aspect of the present invention, a display device includes display means for displaying an evaluation index value of the accuracy of time-series data for each of a plurality of time-series data of a state in an environment and an action of the agent, the time-series data being time-series data of inputs and outputs of a model simulating an environment in which an agent performs an action.

[0010] According to a fifth aspect of the present invention, a learning method includes a computer learning using data that links the state of the environment in which an agent performs an action, the actions that can be performed in that state, and the next state when the action is performed in that state, thereby obtaining a model that takes the state and the action as input and the next state as output, and based on the obtained model, obtaining feedback information that is information used for learning the model or for learning a new model that takes the state and the action as input and the next state as output, and using the model obtained by learning using the feedback information to learn a policy that indicates the action of the agent depending on the state.

[0011] According to a sixth aspect of the present invention, a display method includes a computer displaying, for each item included in information indicating a next state that is output by a model that simulates an environment in which an agent performs an action in response to input of information indicating a state and information indicating an action, an evaluation index value of the accuracy of that item in the information indicating the next state.

[0012] According to a seventh aspect of the present invention, a display method includes displaying, by a computer, an evaluation index value of accuracy of time-series data for each of a plurality of time-series data of a state in an environment and an action of the agent, the time-series data being time-series data of input and output of a model simulating an environment in which an agent performs an action.

[0013] According to an eighth aspect of the present invention, a recording medium is a recording medium having recorded thereon a program for causing a computer to execute the following steps: acquiring a model that takes a state and an action as input and outputs a next state through learning using data that links the state of the environment in which an agent performs an action, the actions that can be performed in that state, and the next state when the action is performed in that state; acquiring feedback information based on the acquired model, which is information used for learning the model or for learning a new model that takes a state and an action as input and outputs the next state; and learning a policy that indicates the agent's action depending on the state, using the model acquired through learning using the feedback information.

[0014] According to a ninth aspect of the present invention, a recording medium is a recording medium having recorded thereon a program for causing a computer to execute the following: displaying, for each item included in information indicating a next state that is output by a model simulating an environment in which an agent performs an action in response to input of information indicating a state and information indicating an action, an evaluation index value for the accuracy of that item in the information indicating the next state.

[0015] According to a tenth aspect of the present invention, a recording medium is a recording medium having recorded thereon a program for causing a computer to execute the following: displaying an evaluation index value for the accuracy of time-series data for each of a plurality of time-series data of a state in an environment and an action of the agent, the time-series data being time-series data of input and output of a model that simulates an environment in which an agent performs an action.

[0016] According to the present invention, when learning control of a control target using data obtained in advance, it is expected that the accuracy of control learning in a state where sufficient data is not available can be improved.

[0017] 1 is a diagram illustrating an example of the configuration of a learning system according to an embodiment. FIG. 1 is a diagram illustrating an example of the configuration when a data collection device according to an embodiment acquires data from a control target. FIG. 2 is a diagram illustrating an example of the flow of data related to a data collection device according to an embodiment. FIG. 3 is a diagram illustrating an example of the flow of data when a learning device according to an embodiment learns a model of a control target and learns a policy πθ. FIG. 4 is a diagram illustrating an example of a processing procedure when a learning device according to an embodiment learns a model of a control target and learns a policy πθ. FIG. 5 is a diagram illustrating an example of a system configuration when a control target is in operation according to an embodiment. FIG. 6 is a diagram illustrating an example of a hopper. FIG. 7 is a diagram illustrating an example of the configuration of data indicating a state according to an embodiment. FIG. 8 is a diagram illustrating an example of the configuration of data indicating an action according to an embodiment. FIG. 9 is a diagram illustrating an example of input / output of data in a model and a policy when a learning device according to an embodiment generates a pseudo trajectory. FIG. 10 is a diagram illustrating an example of a display screen of the importance of elements of a state displayed by a display unit according to an embodiment. FIG. 11 is a diagram illustrating an example of a display screen of the importance of a pseudo trajectory displayed by a display unit according to an embodiment. １ 1 is a diagram showing an example of an editing screen for a pseudo trajectory after correction by a user, which is displayed by a display unit according to an embodiment. １ 10 is a diagram illustrating an example of a setting screen for weights for elements of the pseudo trajectory τ displayed by the display unit according to the embodiment. ３ 10 is a diagram illustrating an example of an editing screen of the pseudo trajectory τ displayed by the display unit according to the embodiment. ３ FIG. 1 is a diagram showing an example of a setting screen for weights for elements. FIG. 2 is a diagram showing another example of the configuration of a learning device according to an embodiment. FIG. 3 is a diagram showing an example of the configuration of a display device according to an embodiment. FIG. 4 is a diagram showing another example of the configuration of a display device according to an embodiment. FIG. 5 is a diagram showing an example of a processing procedure in a learning method according to an embodiment. FIG. 6 is a diagram showing an example of a processing procedure in a display method according to an embodiment. FIG. 7 is a diagram showing another example of a processing procedure in a display method according to an embodiment. FIG. 8 is a schematic block diagram showing the configuration of a computer according to at least one embodiment.

[0018] Embodiments of the present invention will be described below, but the following embodiments do not limit the scope of the invention. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. FIG. 1 is a diagram illustrating an example of the configuration of a learning system according to an embodiment. In the configuration illustrated in FIG. 1, the learning system 1 includes a data collection device 300 and a learning device 100. The learning device 100 includes a communication unit 110, a display unit 120, an operation input unit 130, a memory unit 180, and a processing unit 190. The memory unit 180 includes a data memory unit 181, a model memory unit 182, and a policy memory unit 183. The processing unit 190 includes a data management unit 210, a learning unit 220, an analysis unit 230, and a feedback information acquisition unit 240. The learning unit 220 includes a model management unit 221 and a policy management unit 222.

[0019] The learning system 1 learns a control method for a control object. Specifically, a data collection device 300 acquires data from the control object in advance. The learning device 100 acquires a model of the control object using the acquired data, and then uses the acquired model to learn the control method.

[0020] Here, "before" refers to before the learning device 100 learns the control method for the control object. As will be described later in the description of "environment," the data collection device 300 may acquire data from the operating environment of the control object in addition to the control object. The learning device 100 may also construct a model that includes the operating environment of the control object in addition to the control object.

[0021] The control object, which is the object for which the learning device 100 learns a control method, is not limited to a specific object. Various controllable objects can be the control object. For example, the control object may be equipment such as a plant or a power plant, a system such as a production line in a factory, or a standalone device. Alternatively, the control object may be a moving object such as an automobile, an airplane, a ship, or a self-propelled mobile robot.

[0022] The learning of a control method for a control target performed by the learning device 100 can be considered as a type of reinforcement learning. Reinforcement learning here is machine learning that learns a policy, which is a behavioral rule of an agent that takes an action in an environment, based on a state observed in the environment and a reward that represents an evaluation of the state or action.

[0023] When a controlled object itself operates in accordance with control rules, the controlled object corresponds to an example of an agent, the operation of the controlled object corresponds to an example of an action, and the operation rules correspond to an example of a policy. For example, when the controlled object is a chemical plant and a control mechanism is incorporated into the chemical plant and operates automatically or semi-automatically, the chemical plant corresponds to an example of an agent, the operation of the chemical plant corresponds to an example of an action, and the operation rules for the automatic or semi-automatic operation of the chemical plant correspond to an example of a policy.

[0024] When a control device that controls the control object is provided separately from the control object, the control device corresponds to an example of an agent, the control of the control object performed by the control device corresponds to an example of an action, and the control rules correspond to an example of a policy. For example, when the control object is a chemical plant and a control device that controls the chemical plant is provided externally to the chemical plant, the control device corresponds to an example of an agent, the control of the chemical plant performed by the control device corresponds to an example of an action, and the control rules for the control device to control the chemical plant correspond to an example of a policy.

[0025] In both cases where the controlled object itself operates according to a control rule and where a control device that controls the controlled object is provided separately from the controlled object, the controlled object, or the controlled object and its operating environment, are examples of the environment. That is, the state of the controlled object may be the subject of state observation, or in addition to the state of the controlled object, the state of the operating environment of the controlled object may also be the subject of state observation. Furthermore, the value of the reward may be obtained by observing the state, or may be obtained by calculation, etc.

[0026] For example, if the control target is a chemical plant, the chemical plant or the chemical plant and its operating environment are examples of environments in reinforcement learning. Examples of states in reinforcement learning include the state of the chemical plant, such as the values ​​of pressure sensors and flow rate sensors installed in the chemical plant, or the state of the chemical plant and the operating environment of the chemical plant, such as the temperature of the air surrounding the chemical plant.

[0027] Another example of behavior in reinforcement learning is a control command value for a chemical plant, such as a PID (Proportional-Integral-Differential) control command value for the opening degree of a predetermined valve. A measurable value, such as the production amount of an end product such as ethylene or gasoline measured by a sensor, may be used as the reward value. Alternatively, a value obtained by calculation, such as calculating the production amount of an end product from the consumption amount of raw materials, may be used as the reward value without providing a sensor for measuring the production amount of the end product.

[0028] The user may be a chemical plant designer, operator, or practitioner. The model of the controlled object may be a simulator used in the operation of the chemical plant. Control rules for controlling the chemical plant are an example of a policy in reinforcement learning.

[0029] The learning device 100 uses the interaction between the simulator and the measures to construct a pseudo trajectory, which is data indicating a time series of control over the chemical plant and the state of the chemical plant. For example, the learning device 100 can improve the accuracy of the simulator by having the user return feedback information on the generated pseudo trajectory.

[0030] Furthermore, the learning device 100 can automatically operate a chemical plant by using a policy constructed using the improved simulator. Furthermore, by presenting the generated pseudo trajectories to practitioners, it can also provide support for creating operation plans for chemical plants.

[0031] Hereinafter, the controlled object, or the controlled object and its operating environment, will also be referred to as the "environment" and represented by p. The state of the controlled object, or the state of the controlled object and the state of the operating environment of the controlled object, will also be referred to as the "state" and represented by s. The behavior of the controlled object, or the control of the controlled object, will also be referred to as the "action" and represented by a. The operating rule that prescribes the behavior of the controlled object, or the control rule that prescribes the control of the controlled object, will also be referred to as the "policy" and represented by π. The policy that is the target of learning by the learning device 100 will be referred to as π. θ It is expressed as:

[0032] "π θ "θ" in " is the policy π θ The parameter values ​​in the model representing the information gathering strategy π are shown below. β To distinguish it from θ " is written as ". Measure π θ may be configured as a function that receives an input of a state s and outputs an action a. In this case, the policy π θ is expressed as in equation (1).

[0033]

[0034] Furthermore, hereinafter, time will be expressed in time steps of a fixed time Δt, and will be expressed as time step 0, time step 1, time step 2, .... Note that the time Δt may be different for each time step. A time step determined as a reference time step, such as the time step at which the controlled object starts operating or the current time step T, is expressed as time step 0. When the current time step T is used as the reference, time step t can also be expressed as "(current time step T) + t". Each step in the time step is also referred to as each time step.

[0035] Also, it is assumed that state observation is performed at each time step, and the state at time step t (t is an integer t≧0) is denoted by s ｔ The state s at time step t+1 is ｔ＋１ is the state s at time step t. ｔ It is also called the next state of the

[0036] Furthermore, the learning device 100 determines the policy π for determining an action for each time step. θ The action at time step t is assumed to be a ｔ The learning device 100 also calculates the policy π θ The reward value used for learning is obtained at each time step. The reward at time step t is defined as r ｔ Reward r ｔ is the action a at time step t ｔ This value can be said to indicate the evaluation of the quality of the product.

[0037] Strategy π θ In the learning of ０ +r １ +r ２ + ... is the objective function, and the policy π is chosen so that the evaluation indicated by this objective function becomes higher. θ Alternatively, the reward may be discounted as time passes, and the cumulative reward r ０ +αr １ +α ２ r ２ + ... (where 0 < α < 1) may be used as the objective function. In other words, the cumulative reward can be said to be a value indicating an evaluation of the quality of the behavior at each time step. In this case, the higher the cumulative reward (e.g., the larger the value of the cumulative reward), the higher the quality of the behavior, and the lower the cumulative reward (e.g., the smaller the value of the cumulative reward), the lower the quality of the behavior. The learning device 100 may be configured to use a reward that indicates a higher evaluation as the value increases. Alternatively, the learning device 100 may be configured to use a reward that indicates a higher evaluation as the value decreases (i.e., so-called loss).

[0038] As described above, data collection device 300 acquires data in advance from the control target. Fig. 1 shows an example in which data collection device 300 is configured as a device separate from learning device 100. In this case, data collection device 300 may be configured using a computer such as a personal computer (PC) or a workstation. Alternatively, data collection device 300 may be a part of learning device 100.

[0039] FIG. 2 is a diagram showing an example of the configuration when the data collection device 300 acquires data from a control target. As shown in FIG. 2, the control target is also referred to as a control target 810. A system with the configuration shown in FIG. 2 is also referred to as a data collection system 2. FIG. 2 shows an example of the case where the data collection device 300 transmits data to the learning device 100 after completing acquisition of data from the control target 810. In this case, the data collection device 300 does not need to be connected to the learning device 100 for communication while acquiring data.

[0040] Alternatively, the data collection device 300 may transmit data to the learning device 100 while acquiring data from the control target 810. In this case, the data collection device 300 may be connected to both the control target 810 and the learning device 100 for communication at the same time.

[0041] 3 is a diagram showing an example of the flow of data related to the data collection device 300. In the example shown in Fig. 3, the data collection device 300 performs an action a when the state of the environment p is s. s' is the next state of state s (i.e., the state at the next time step), and the action a causes a transition from state s to the next state s'.

[0042] Then, the data collecting device 300 acquires the value of the reward r at the time step when the next state s' is reached and the observed value of the next state s'. The data collecting device 300 may be equipped with a sensor to observe the state and acquire the observed value. Alternatively, the control object 810 as the environment p may be equipped with a sensor to observe the state, and the data collecting device 300 may acquire the observed value of the state from the control object 810. Furthermore, the value indicating the state may include a value obtained by calculation. The value indicating the state is an example of information indicating the state.

[0043] A value indicating a state is also simply referred to as a state. For example, obtaining a value indicating a state is also referred to as obtaining a state. A value of a reward is also simply referred to as a reward. For example, obtaining a value of a reward is also referred to as obtaining a reward. A value indicating an action is also simply referred to as an action. For example, obtaining a value indicating an action is also referred to as obtaining an action.

[0044] As described above, FIG. 3 shows an example in which the data collection device 300 performs action a. However, action a may also be performed by an entity other than the data collection device 300. For example, an operator may operate a chemical plant, which is an example of environment p, and the data collection device 300 may record the operations performed by the operator and the state of the chemical plant for each time step. In this case, the operations performed by the operator for each time step correspond to an example of action a, and the state of the chemical plant at the start of the operation corresponds to an example of state s. Furthermore, the state of the chemical plant after the operation is completed (specifically, the state of the chemical plant in the next time step) corresponds to an example of next state s'.

[0045] As described above, the reward r may be obtained by observing the environment p, or the reward r may be calculated by the data collection device 300 or the learning device 100. The rule for determining the action a when the data collection device 300 acquires data from the control target is called a data collection strategy, and π β The data collection strategy may be, for example, a function that receives an input of a state s and determines an action a. In this case, the data collection strategy π β As in the case of the above formula (1), β(a|s)".

[0046] The data collection device 300 generates data that combines a state s, an action a, a reward r, and a next state s' for each time step, and transmits the generated data to the learning device 100. Data that combines a state, an action, a reward, and a next state is also referred to as quadruple data, and these four pieces of data are expressed in parentheses "( )". For example, quadruple data of a state s, an action a, a reward r, and a next state s' is also expressed as "(s, a, r, s')". The quadruple data is an example of data that links the state of the environment in which an agent performs an action, the actions that can be performed in that state, the reward that represents the quality of the action, and the next state when the action is performed in that state.

[0047] The data collection device 300 may be configured to repeatedly acquire data from the control object 810 until a predetermined number of quadruplets of data are obtained, so that the learning device 100 may acquire and store the predetermined number of quadruplets of data from the data collection device 300.

[0048] In the learning device 100, the data management unit 210 stores the quadruple data transmitted by the data collection device 300 in the model storage unit 182. This set of quadruple data is referred to as dataset D ｅｎｖ It can also be written as:

[0049] The learning device 100 may calculate the reward r. In this case, the data collection device 300 transmits triplet data that combines a state s, an action a, and a next state s' to the learning device 100. The data collection device 300 may transmit data that does not include the next state to the learning device 100 in the form of time-series data, and the learning device 100 may insert the next state.

[0050] As described above, the learning device 100 acquires a model of the control object 810 using data acquired from the control object 810 by the data collection device 300, and uses the acquired model to learn a control method. The model of the control object 810 corresponds to an example of a model of the environment p. The model of the control object 810 acquired by the learning device 100 is also referred to as a model of the environment p.

[0051] Here, it is conceivable that sufficient data for certain states and actions cannot be obtained from the data acquired by the data collection device 300 from the control target 810. For example, when the data collection device 300 acquires data during actual operation of the control target 810, it is conceivable that data regarding states and actions other than the states that appeared during actual operation and the actions taken during actual operation in response to those states cannot be obtained.

[0052] When the learning device 100 acquires a model of the control object 810 through learning using this data, it is conceivable that the accuracy of the model will be low for states and actions other than those shown in the data. θ When learning the policy π, the state and action for which the model is not accurate cannot be learned with sufficient accuracy. θ However, it is possible that appropriate actions cannot be suggested.

[0053] For example, even if there is a more preferable control method than the control method currently being used for the control target 810, the measure π θ Furthermore, the state of the control object 810 when the preferred control method is executed and the control from that state may not be learned because the accuracy of the model of the control object 810 is low, and the policy π θ It is possible that learning will not be possible.

[0054] Therefore, the learning device 100 receives input of information based on the user's knowledge and uses the information to learn a model of the control object 810. As the accuracy of the model of the control object 810 improves, the policy π θ It is expected that the accuracy of the control object 810 will be improved. Here, it is assumed that the user is an expert on the control object 810, for example.

[0055] The learning device 100 analyzes the output data of the model so that the user can input information that is effective in improving the accuracy of the model of the control object 810. Based on the analysis results, the learning device 100 presents the user with information indicating which of the items related to the information input by the user have the greatest impact on improving the accuracy of the model. The information input by the user to the learning device 100 is also referred to as feedback information regarding the information presented to the user by the learning device 100, or simply as feedback information. Feedback information can be considered information for reducing the discrepancy between the environment and the model of the environment.

[0056] The information presented to the user by the learning device 100 and the information input by the user to the learning device 100 are not limited to a specific type of information. For example, the learning device 100 may present to the user information indicating the accuracy of the model output for each of multiple items included in the state. This information can be interpreted as information indicating the importance of each item in terms of improving the accuracy of the model. For items with low accuracy in the model output, updating the model to improve the accuracy of that item is expected to result in a more accurate model.

[0057] The user refers to information indicating the accuracy of the model output for each item included in the state, and inputs, for example, constraint expressions that the model's input and output must satisfy for items with low accuracy. It is expected that the learning device 100 will be able to obtain a more accurate model by learning the model using the input constraint expressions.

[0058] Alternatively, the learning device 100 may acquire multiple pieces of time-series data of the input and output of the model of the control object 810, and present information indicating the accuracy of each piece of time-series data to the user. Each piece of time-series data of the input and output of the model of the control object 810 is also referred to as a pseudo trajectory. A set of pseudo trajectories is referred to as a dataset D ｍｏｄｅｌ It can also be written as:

[0059] The user refers to the information indicating the accuracy of the pseudo trajectories and, for example, corrects pseudo trajectories with low accuracy. The corrected pseudo trajectories can be said to be data indicating the correct answer that the model should output. When the learning device 100 uses all of the multiple pseudo trajectories to train a model, it is expected that a more accurate model can be obtained by using the corrected pseudo trajectories rather than the uncorrected pseudo trajectories.

[0060] The information indicating the accuracy of each pseudo trajectory can be interpreted as information indicating the importance of each pseudo trajectory in terms of improving the accuracy of the model. It is expected that a more accurate model can be obtained by the user correcting a pseudo trajectory with low accuracy rather than correcting a pseudo trajectory that is originally highly accurate. The learning device 100 may be configured using a computer such as a personal computer or a workstation.

[0061] The communication unit 110 communicates with other devices. For example, the communication unit 110 receives data obtained from the control target 810 and transmitted by the data collection device 300. The display unit 120 has a display screen, such as a liquid crystal panel or an LED (Light Emitting Diode) panel, and displays various images. For example, the display unit 120 displays the information presented to the user by the learning device 100 described above. The display unit 120 is an example of a display means.

[0062] The operation input unit 130 includes input devices such as a keyboard and a mouse, and receives user operations. For example, the operation input unit 130 receives user operations to input the above-described feedback information. The operation input unit 130 is an example of an input unit.

[0063] The storage unit 180 stores various data. The storage unit 180 is configured using a storage device included in the learning device 100. The data storage unit 181 stores data used for learning the model of the control object 810 and the policy π θ In particular, the data storage unit 181 stores data set D ｅｎｖand Dataset D ｍｏｄｅｌ The data set D is stored. ｅｎｖ and Dataset D ｍｏｄｅｌ is an example of data used for learning a model of the control object 810. ｅｎｖ and Dataset D ｍｏｄｅｌ In addition to learning the model of the control object 810, the policy π θ It may also be used for learning.

[0064] The model storage unit 182 stores a model of the control target 810. In particular, the model storage unit 182 stores a plurality of models of the control target 810. In the following, an example will be described in which the model storage unit 182 stores two models, and these two models will be referred to as p^ ０ and p^ １ However, the model storage unit 182 may store three or more models.

[0065] Model p^ ０ and p^ １ Any of the models p^ may be configured as a function that receives a state and an action in that state as input and outputs the next state. ０ The input state is represented by s, the action is represented by a', and the model p^ ０ The next state output by is s'' ０ Expressed as, model p^ ０ can be expressed as in equation (2).

[0066]

[0067] Model p^ １ The input state is represented by s, the action is represented by a', and the model p^ １ The next state output by is s'' １ Expressed as, model p^ １ can be expressed as in equation (3).

[0068]

[0069] The accuracy of the model output can be evaluated using the multiple models stored in the model storage unit 182. When the same state and action are input to multiple models and the outputs of the multiple models are similar, the accuracy of these models for this input can be evaluated as relatively high. On the other hand, when the same state and action are input to multiple models and there is a large variance in the outputs of the multiple models, the accuracy of these models for this input can be evaluated as relatively low.

[0070] As an index showing the magnitude of the variation in the outputs of the multiple models, for example, the variance of the outputs of the multiple models can be used. ０ The next state s'' output by ０ And, model p^ １ The next state s'' output by １ and are collectively referred to as the next state s'', and the next state s'' ０ and the next state s'' １ The variance of the above is denoted as Var(s''). The variance Var(s'') is expressed as in equation (4).

[0071]

[0072] s'' ａｖｇ is the next state s'' ０ and the next state s'' １ The learning device 100 uses the reward that reflects this variance Var(s'') to implement the policy π θ For example, the learning device 100 may learn the policy π using the reward r″ shown in Equation (5). θ The learning may be performed.

[0073]

[0074] "r(s, a')" represents the reward before the variance Var(s'') is reflected. The reward r(s, a') indicates the evaluation of taking action a' in state s from the viewpoint of the agent's behavioral goal. According to the reward r'', the policy π θ The model p^ for the next state s'' due to the action a' output by ０ and p^ １When the accuracy of s is low, the evaluation indicated by the reward r″ is low. θ By learning the model p^ ０ and p^ １ A policy π with a relatively low probability of outputting an action a' that transitions to the next state with a low accuracy of θ From this point of view, the model p^ ０ and p^ １ Due to the accuracy of the policy π θ It is expected that this will reduce the degree to which the accuracy of the

[0075] On the other hand, strategy π θ The action a' output by the model p^ ０ and p^ １ As described above, by limiting the action a' to a next state with low accuracy, the policy π θ Therefore, as described above, the learning device 100 presents to the user information indicating which of the items related to the information input by the user have a large impact on improving the accuracy of the model, and accepts input of feedback information by the user.

[0076] The policy storage unit 183 stores the policy π θ Remember.

[0077] Processing unit 190 performs various processes by controlling each unit of learning device 100. The functions of processing unit 190 are performed, for example, by a CPU (Central Processing Unit) included in learning device 100 reading and executing a program from storage unit 180.

[0078] The data management unit 210 manages the data stored in the data storage unit 181. For example, the data management unit 210 manages the data stored in the data storage unit 181 by storing the quadruple data received by the communication unit 110 from the data collection device 300 as a data set D ｅｎｖ Store in.

[0079] The data management unit 210 also manages the model of the control target 810 and the policy π θThe data set D stored in the data storage unit 181 is a set of quadruple data generated using ｍｏｄｅｌ When the learning device 100 acquires multiple pseudo trajectories as described above, the data management unit 210 stores the pseudo trajectories in a data set D ｍｏｄｅｌ Store in.

[0080] When the learning device 100 obtains a more accurate model of the control object 810 using the feedback information and generates quadruple data using the model, the data management unit 210 stores the generated quadruple data in the storage unit 180. The data management unit 210 stores the newly obtained data in the already obtained data set D ｍｏｄｅｌ Alternatively, the data management unit 210 may overwrite the already obtained data set D ｍｏｄｅｌ The data management unit 210 may store the newly obtained data in the data set D. ｍｏｄｅｌ or the data set D ｍｏｄｅｌ Alternatively, a different data set may be generated.

[0081] The learning unit 220 learns a model of the control object 810 and calculates the policy π θ The learning unit 220 also learns the model of the control object 810 and the policy π θ Generate the quadruple data using

[0082] The model management unit 221 learns the model of the control target 810. Specifically, the model management unit 221 learns the model of the control target 810. ｅｎｖ Using the model p^ ０ Learning and modeling p １ Furthermore, when the user inputs feedback information using the operation input unit 130, the model management unit 221 learns the model p^ by reflecting the obtained feedback information. ０ Learning and modeling p １ Repeat the learning process.

[0083] The model management unit 221 manages the already obtained model p^ ０and p^ １ Alternatively, the model management unit 221 may perform learning of these models so as to update the already obtained model p^. ０ and p^ １ Without using the model p^ ０ Learning and modeling p １ The learning may be restarted from the beginning.

[0084] That is, the model management unit 221 may use the feedback information to further train an already obtained model. Alternatively, the model management unit 221 may use the feedback information to train a new model. The model management unit 221 corresponds to an example of a model acquisition means.

[0085] The policy management unit 222 θ In particular, the policy management unit 222 uses data of quadruples indicating input and output data of the model obtained by learning that reflects feedback information to learn the policy π θ As a result, the policy management unit 222 learns the data set D ｅｎｖ For states and actions for which sufficient data could not be obtained using only the strategy π θ In this respect, a more accurate policy π θ The policy management unit 222 is an example of a policy management means.

[0086] The analysis unit 230 generates information indicating which of the above-mentioned items related to the information input by the user has a large effect on improving the accuracy of the model. For example, as described above for the learning device 100, the analysis unit 230 generates information indicating which of the above-mentioned items related to the information input by the user has a large effect on improving the accuracy of the model p^ ０ and p^ １ For example, the analysis unit 230 may calculate an evaluation index value of the accuracy of each item in the next state output by the model p^ ０ and the value of that item in the output of the model p^ １The variance between the value of the item in the output of the item and the value of the item in the output of the item may be calculated as an evaluation index value for the accuracy of the item. In this case, the items included in the information indicating the state correspond to examples of items related to information input by the user.

[0087] Furthermore, as described above with respect to the learning device 100, the analysis unit 230 may calculate an evaluation index value for the accuracy of each of the multiple pseudo trajectories. For example, the analysis unit 230 may calculate an evaluation index value for the accuracy of the model p^ for each time step in the pseudo trajectory. ０ The next state by the model p^ １ The analysis unit 230 may then calculate the next state by summing or averaging the variances of the next states for each time step for all time steps in one pseudo trajectory as an evaluation index value for the accuracy of the pseudo trajectory. In this case, the pseudo trajectory corresponds to an example of an item related to information input by the user. The analysis unit 230 corresponds to an example of analysis means.

[0088] The feedback information acquisition unit 240 acquires feedback information. Specifically, the feedback information acquisition unit 240 reads feedback information input by the user using the learning device 100. As described above, the analysis unit 230 analyzes the output of the model of the control object 810 and presents the analysis results to the user, and the feedback information acquisition unit 240 acquires feedback information input by the user with reference to the analysis results. In this respect, it can be said that the feedback information acquisition unit 240 acquires feedback information based on the output of the model of the control object 810. When the analysis unit 230 presents the analysis results to the user, the feedback information acquisition unit 240 may externally prompt (for example, prompt the user) to input feedback information. The feedback information acquisition unit 240 corresponds to an example of feedback acquisition means.

[0089] As described above with respect to the learning device 100, the feedback information acquisition unit 240 may acquire feedback information indicating constraints that must be satisfied by the input data and output data of the model of the control object 810 in order to improve the accuracy of the model. In particular, the feedback information acquisition unit 240 may acquire feedback information indicating constraints on items that are evaluated as having a relatively low accuracy among items included in the next state output by the model of the control object 810.

[0090] In this case, an item with a relatively low accuracy rating may be, for example, an item with a rating lower than the average (or median) of the ratings for multiple items including that item. Alternatively, an item with a relatively low accuracy rating may be an item with the lowest rating among a selected portion of multiple items. Alternatively, an item with a relatively low accuracy rating may be an item that satisfies a predetermined criterion. In this case, the "predetermined criterion" refers to the criterion for determining whether the rating is low. For example, the "predetermined criterion" may be that the rating is below a threshold.

[0091] The user refers to the evaluation index values ​​for each item included in the state displayed by the display unit 120 and inputs feedback information using the operation input unit 130. In this respect, it can be said that the feedback information acquisition unit 240 acquires feedback information input by a user operation accepted by the operation input unit 130 after the display unit 120 starts displaying the evaluation index values ​​for each item included in the state.

[0092] Furthermore, as described above with respect to the learning device 100, the feedback information acquisition unit 240 may acquire feedback information indicating a correction to a pseudo trajectory. In particular, the feedback information acquisition unit 240 may acquire feedback information indicating a correction to a pseudo trajectory whose accuracy is evaluated as being relatively low.

[0093] The user refers to the evaluation index value for each pseudo trajectory displayed by the display unit 120 and inputs feedback information using the operation input unit 130. In this respect, it can be said that the feedback information acquisition unit 240 acquires feedback information input by a user operation accepted by the operation input unit 130 after the display unit 120 starts displaying the evaluation index value for each pseudo trajectory.

[0094] FIG. 4 shows the process in which the learning device 100 learns a model of the control object 810 and the policy π θ 4 is a diagram illustrating an example of the flow of data when learning is performed. In the example shown in FIG. 4, the data management unit 210 ｅｎｖ The model management unit 221 reads out the quadruple data (s, a, r, s') from the model manager 221 and outputs it to the model management unit 221 and the policy management unit 222. The model management unit 221 uses this quadruple data to create the model p^ ０ Learning and modeling p １ The policy management unit 222 uses this quadruple data to learn the policy π θ Here, the policy π θ The learning method for dataset D ｍｏｄｅｌ Strategy π for generating θ The learning method is not limited to a specific one as long as it can acquire the policy π θ The learning may be performed.

[0095] Dataset D ｅｎｖ After the learning using the model p̂ is completed, the data management unit 210 ０ and p^ １ and strategy π θ and obtain a quadruple (s, a', r'', s'') by ｍｏｄｅｌ Specifically, the policy management unit 222 stores a certain state s in the policy π θ to obtain the action a′, and outputs the state s and the action a′ to the model management unit 221 .

[0096] The policy management unit 222 determines the policy π θ The state s to be input to the data set D is not limited to a specific one.ｅｎｖ The state s may be read from any of the quadruple data included in the model manager 221 and output to the policy manager 222. Alternatively, the policy manager 222 may arbitrarily generate the state s within the range of possible actions of the control object 810. When the learning device 100 uses a pseudo trajectory, the policy manager 222 uses the next state s'' output by the model manager 221 as the state s in the next time step.

[0097] The model management unit 221 manages the state s and the action a′ in the model p̂ ０ and the next state s'' ０ The model management unit 221 also acquires the state s and the action a′ as a model p̂ １ and the next state s'' １ Then, the model management unit 221 outputs the next state s″ and the reward r″ to the policy management unit 222.

[0098] Next state s'' is next state s'' ０ and the next state s'' １ For example, the model management unit 221 may set the next state s'' as the next state s''. ０ Alternatively, the model management unit 221 may select the next state s'' as the next state s''. １ Alternatively, the model management unit 221 may select the next state s''. ０ and the next state s'' １ The average of these may be calculated and used as the next state s''.

[0099] As described above, the analyzer 230 determines the next state s''. ０ and s'' １ To this end, the learning unit 220 calculates the variance between the next state s'' and the next state r'' for the four-tuple data (s, a', r'', s''). ０ and s'' １ may be linked to each other.

[0100] Alternatively, as shown in the above formula (5), the model management unit 221 may use the next state s'' when calculating the reward r''.０ and s'' １ The learning unit 220 calculates the variance Var(s'') between the next state s'' and ０ and s'' １ Alternatively, the variance Var(s'') may be linked to the quadruple of data (s, a', r'', s''). Note that a component other than the model manager 221 may calculate the reward r''. For example, the policy manager 222 may calculate the reward r''.

[0101] Alternatively, the learning unit 220 may set the next state s'' instead of the next state s''. ０ and s'' １ In this case, when the next state is required, each unit of the learning device 100 may calculate the next state s'', for example. ０ The next state s'' is selected. ０ and s'' １ The next state is acquired according to a predetermined method for acquiring the next state from the

[0102] D ｍｏｄｅｌ After a predetermined amount of quadruple data has been accumulated, the analysis unit 230 analyzes the output of the model of the control target 810. As the analysis result, the analysis unit 230 generates information indicating which of the items related to the information input by the user have the greatest impact on improving the accuracy of the model. The analysis unit 230 then presents the obtained analysis result to the user by displaying it on the display unit 120.

[0103] The user generates feedback information by referring to the analysis results by the analysis unit 230. The user performs an input operation of the feedback information on the operation input unit 130. The feedback information acquisition unit 240 reads the feedback information from a signal output by the operation input unit 130 in response to a user operation, and outputs the feedback information to the model management unit 221.

[0104] The model management unit 221 uses the obtained feedback information to ０ and p^ １ As described above, the model management unit 221 learns the already obtained model p^ ０ and p^１ Alternatively, the already obtained model p^ may be updated. ０ and p^ １ Alternatively, learning may be performed from the beginning without using the above.

[0105] The policy manager 222 uses the newly obtained model p^ ０ and p^ １ Using the strategy π θ The policy management unit 222 learns the already obtained policy π θ Alternatively, the already obtained policy π θ Alternatively, learning may be performed from the beginning without using the above.

[0106] FIG. 5 shows the process in which the learning device 100 learns a model of the control object 810 and the policy π θ FIG. 10 is a diagram illustrating an example of a procedure for a process of learning the above.

[0107] (Step S1) The model management unit 221 ｅｎｖ Based on the model p^ ０ and p^ １ Specifically, the model management unit 221 constructs a dataset D ｅｎｖ Based on the quadruple data (s, a, r, s') contained in ０ and p^ １ Search for the model p^ ０ and p^ １ For example, a known supervised learning method can be used to search for the model p̂ ０ or p^ １ When a state s and an action a are input, the next state s' is outputted. This is also referred to as estimating the next state s' from the state s and the action a.

[0108] In addition, the policy management unit 222 ｅｎｖ Based on the policy π θ As mentioned above, the strategy π θ The learning method for dataset D ｍｏｄｅｌ Strategy π for generating θThe learning method is not limited to a specific one as long as it can acquire the above information. After step S1, the process proceeds to step S2.

[0109] (Step S2) The data management unit 210 ｅｎｖ The data management unit 210 randomly extracts a quadruple of data (s, a, r, s') from the data management unit 222. The data management unit 210 outputs the extracted quadruple of data to the policy management unit 222. After step S2, the process proceeds to step S3.

[0110] (Step S3) The policy management unit 222 assigns the state s included in the quadruple data (s, a, r, s′) extracted in step S2 to the policy π θ to obtain the behavior a'. Note that the obtained behavior a' may be different from the behavior a included in the data of the quadruple. After step S3, the process proceeds to step S4.

[0111] (Step S4) The model management unit 221 converts the state s and the action a′ in step S3 into a model p^ ０ and the next state s'' ０ In addition, the model management unit 221 obtains the state s and the action a′ in step S3 as a model p^ １ and the next state s'' １ Get.

[0112] (Step S5) The model management unit 221 calculates the reward r'' based on the above formula (5). After step S5, the process proceeds to step S6.

[0113] (Step S6) The data management unit 210 stores the state s, the action a′, the reward r″, and the next state s″ in the form of a quadruple (s, a′, r″, s″) as a dataset D ｍｏｄｅｌ After step S6, the process proceeds to step S7. When the learning device 100 handles a pseudo trajectory, the learning unit 220 may set the next state s'' as a new state s and repeat the processes of steps S3 to S6. In this case, the learning unit 220 repeats the processes of steps S3 to S6 the number of times as many times as the number of time steps in the pseudo trajectory, and then the process proceeds to step S7.

[0114] (Step S7) The learning unit 220 uses the data set D ｍｏｄｅｌ and D ｅｎｖ Using this, policy π θ Update the policy π θ For example, a known reinforcement learning technique can be used to update the parameter . After step S7, the process proceeds to step S8.

[0115] (Step S8) The analysis unit 230 calculates the model p^ ０ and p^ １ For example, the analysis unit 230 performs an analysis to efficiently improve the next state s'' for each item included in the state as described above. ０ The value of that item in １ Alternatively, the analysis unit 230 may calculate the variance between the value of the item in the next state s″ for each time step in the pseudo trajectory as described above. ０ and s'' １ Alternatively, the variance of the pseudo trajectory may be calculated for each time step, and the variances calculated for all time steps in the pseudo trajectory may be summed up. After step S8, the process proceeds to step S9.

[0116] (Step S9) The display unit 120 displays the analysis results by the analysis unit 230. Then, the feedback information acquisition unit 240 acquires feedback information based on the user operation received by the operation input unit 130. As described above, the user is, for example, an expert on the control target 810. After step S9, the process proceeds to step S10.

[0117] (Step S10) The model management unit 221 ｅｎｖ and D ｍｏｄｅｌ Based on the feedback information, the model p̂ ０ and p^ １ For example, when a constraint expression related to an item included in a state is input as feedback information, the model management unit 221 updates the data set D ｅｎｖ The quadruple data contained in the dataset D ｍｏｄｅｌFor each data quadruple in ０ is set to output the next state s′′ for the input of the state s and the action a included in the data quadruple, and to satisfy the constraints indicated by the feedback information. ０ The model management unit 221 learns the parameter values ​​of the model p^ １ Similarly, parameter values ​​are learned for the model p^ ０ and p^ １ For example, a known constrained supervised learning method can be used to learn the parameter values ​​of . After step S10, the process proceeds to step S11.

[0118] (Step S11) The learning unit 220 learns the model p^ ０ and p^ １ It is determined whether a predetermined condition is satisfied as the termination condition for updating the model p^. ０ and p^ １ The termination condition for updating the model p̂ is not limited to a specific method. ０ and p^ １ The condition for ending the update may be whether or not the processing of steps S2 to S11 in FIG. 4 has been repeated a predetermined number of times.

[0119] Alternatively, the model p^ ０ and p^ １ The termination condition for updating the model p^ ０ and p^ １ Alternatively, the condition may be whether the accuracy of the model p^ is evaluated to be higher than a predetermined evaluation. ０ and p^ １ The termination condition for updating is policy π θ The condition may be whether the evaluation of the item is higher than a predetermined evaluation.

[0120] If the learning unit 220 determines that the termination condition is not met (step S11: NO), the process returns to step S2. On the other hand, if the learning unit 220 determines that the termination condition is met (step S11: YES), the learning device 100 ends the process of FIG.

[0121] 6 is a diagram showing an example of a system configuration during operation of a control target 810. The system shown in FIG. 6 is also referred to as a control system 3. In the configuration shown in FIG. 6, the control device 400 controls the learning device 100 to learn the policy π θ The learning device 100 may function as the control device 400. Alternatively, the control device 400 may be provided separately from the learning device 100, and the control device 400 may control the control target 810 using the policy π θ may be stored.

[0122] Next, the processing performed by the learning device 100 will be described using an example in which a policy for controlling a hopper is learned. FIG. 7 is a diagram showing an example of a hopper. In the example shown in FIG. 7, a hopper 910 includes a top 911, thighs 912, legs 913, and feet 914. The connection portions of these parts are configured as joints whose angles can be adjusted, and each joint is provided with a rotor for adjusting the angle.

[0123] Rotor 921 adjusts the angle between top of head 911 and thigh 912. Rotor 921 is also referred to as thigh rotor 921. Rotor 922 adjusts the angle between thigh 912 and leg 913. Rotor 922 is also referred to as leg rotor 922. Rotor 923 adjusts the angle between leg 913 and foot 914. Rotor 923 is also referred to as foot rotor 923.

[0124] The hopper 910 is a mobile robot, and obtains thrust by changing its posture due to the movement of each rotor. The hopper 910 must be moved without tipping over. As shown in FIG. 7 , the x-axis is set on the plane on which the hopper 910 moves, and the z-axis is set perpendicular to this plane. The plane on which the hopper 910 moves is also referred to as the moving plane.

[0125] 8 is a diagram showing an example of the structure of data indicating a state s. In the example shown in FIG. 8, the state s is represented by an 11-dimensional numerical vector. The elements of the state s are represented as element s ｅ０ , s ｅ１ , ..., s ｅ１０Each of these elements corresponds to an example of an item included in a state. However, the number of elements in a state s is not limited to a specific number.

[0126] element s ｅ０ indicates the z-coordinate value of the top of the head 911, i.e., the height of the top of the head 911. ｅ１ indicates the angle of the top of the head 911 relative to the plane of movement. ｅ２ denotes the angle of the thigh 912 relative to the plane of motion. ｅ３ denotes the angle of the leg 913 relative to the plane of movement. ｅ４ indicates the angle of the foot 914 relative to the plane of movement.

[0127] element s ｅ５ indicates the x-component of the velocity of the top of the head 911. ｅ６ indicates the z-component of the velocity of the top of the head 911. ｅ７ indicates the angular velocity of the top of the head 911 relative to the plane of movement. ｅ８ denotes the angular velocity of the thigh 912 relative to the plane of motion. ｅ９ denotes the angular velocity of the leg 913 relative to the plane of movement. ｅ１０ denotes the angular velocity of the foot 914 relative to the plane of movement.

[0128] 9 is a diagram showing an example of the structure of data representing behavior a. In the example shown in FIG. 9, behavior a is represented by a three-dimensional numerical vector. The elements of behavior a are represented as element a ｅ０ , a ｅ１ , a ｅ２ However, the number of elements of the action a is not limited to a specific number.

[0129] Element a ｅ０ represents the torque applied to the thigh rotor 921. ｅ１ represents the torque applied to the rotor 922 of the leg. ｅ２ represents the torque applied to the rotor 923 of the foot.

[0130] Furthermore, the above formula (5) is used as the formula for calculating the reward r'', and the value of the "r(s, a')" part of formula (5) is calculated based on formula (6).

[0131]

[0132] The right side of equation (6) "s ｅ５ - | a ｅ０ |-|a ｅ１ |-|a ｅ２ |" indicates the difference obtained by subtracting the sum of the magnitudes of the torques applied to the thigh rotor, leg rotor, and foot rotor from the speed of the top 911 of the hopper 910 in the x coordinate. Equation (6) indicates that the faster the speed at which the hopper 910 moves in the positive direction of the x coordinate, the higher the evaluation, and also indicates that the smaller the total power consumption to operate the rotors, the higher the evaluation.

[0133] FIG. 10 shows the model p̂ when the learning device 100 generates a pseudo trajectory. ０ and strategy π θ 10 is a diagram illustrating an example of input and output of data in the data set D. ｅｎｖ A plurality of quadruplets of time series data obtained by operating the hopper 910 are stored in the memory 910. The quadruplets of time series data are called trajectories.

[0134] The state of the hopper 910 at time step t (t is an integer t≧0) is denoted by s ｔ The time step 0 is the start of the operation of the hopper 910, and the initial state of the hopper 910 is expressed as s ０ The behavior of the hopper 910 at time step t is expressed as a ｔ and the reward r at time step t is expressed as r ｔ It is expressed as:

[0135] The data management unit 210 manages the data set D ｅｎｖ From the initial state s of the hopper 910 in the trajectory ０ and outputs it to the policy management unit 222 and the model management unit 221. The policy management unit 222 reads out the initial state s ０ Policy π θ Enter and act a ０ The obtained action a ０ to the model management unit 221.

[0136] The model management unit 221 manages the state s ０ and Action A０ Model p^ ０ Enter and act １ 10 shows the model p̂ ０ The model management unit 221 uses the state output by the １ to the policy management unit 222. The policy management unit 222 outputs the state s １ Policy π θ Enter and act a １ The obtained action a １ to the model management unit 221.

[0137] In this way, for time step t, the policy manager 222 calculates the state s ｔ Policy π θ Enter and act a ｔ The obtained action a ｔ to the model management unit 221. The model management unit 221 outputs the state s ｔ and Action a ｔ Model p^ ０ and enter the state s ｔ＋１ and obtain the resulting state s ｔ＋１ The strategy management unit 222 and the model management unit 221 continue the action a until a predetermined termination condition is met, for example, the hopper 910 reaches the destination or falls over. ｔ Obtaining and state s ｔ＋１ Repeat the acquisition.

[0138] Furthermore, the model management unit 221 manages the model p^ for each time step. ０ The state s to be input to ｔ and Action a ｔ The same state as ｔ and Action a ｔ Model p^ １ Also input the model p^ １ Actions in case of ｔ＋１ Also, the learning unit 220 acquires the state s ｔ and Action A ｔ and are input into the reward function shown in equation (6) to obtain the reward r ｔ The learning unit 220 obtains the reward r ｔAnd, model p^ ０ and p^ １ Each state s ｔ＋１ and into equation (5) to obtain the reward r'' at time step t. ｔ The learning unit 220 acquires the quadruple data (s ｔ , a ｔ , r'' ｔ , s ｔ＋１ ) to the data management unit 210. The data management unit 210 accumulates the quadruple data output by the learning unit 220 as time-series data to generate a pseudo trajectory.

[0139] The content of the analysis by the analysis unit 230 and the information input by the user as feedback information are not limited to specific ones, and can be various depending on, for example, the control of the control target. Below, the following cases will be described as examples: (1) a case where the analysis result indicates the accuracy of each element included in the state predicted by the model, and the feedback information indicates a constraint equation related to the input and output of the model, and (2) a case where the analysis result indicates the accuracy of each of multiple pseudo trajectories, and the feedback information indicates a corrected value of the pseudo trajectory, or information on whether the value indicated in the pseudo trajectory is correct.

[0140] (1) The analysis results indicate the accuracy of each element included in the state predicted by the model, and the feedback information indicates constraints on the input and output of the model.

[0141] 5, the analysis unit 230 calculates, for each element of the state, the accuracy of that element in the model output based on Equation 7. The accuracy of an element in the model output is also referred to as the importance of that element.

[0142]

[0143] Let i = 0, 1, ..., 10, and the element s of the state s ｅ０ , s ｅ１ , ..., s ｅ１０ , s ｅ＿ｉ Also, s ｔ＋１，ｅ＿ｉ is the element s at time step t+1 ｅ＿ｉ I(s) indicates the importance of ｅ＿ｉ) is the element s ｅ＿ｉ The analysis unit 230 calculates the importance of the model p̂ for each time step based on the formula (7). ０ Element s in the next state output by ｅ＿ｉ And, model p^ １ Element s in the next state output by ｅ＿ｉ The variance of the pseudo trajectory is calculated, and the variance is summed for all time steps in the pseudo trajectory to obtain the importance I(s ｅ＿ｉ )

[0144] Importance I(s ｅ＿ｉ ) is the model p^ ０ Element s in the next state output by ｅ＿ｉ This corresponds to an example of an evaluation index value for the accuracy of (1). Equation (7) can be interpreted as indicating that the more ambiguous an element is among the elements of the state predicted by the model, the more important it is. Among the elements of the state predicted by the model, the more ambiguous an element is among the elements with low prediction accuracy by the model.

[0145] Note that ∞ in Σ in formula (7) indicates that the number of time steps in the pseudo trajectory is arbitrary. The analysis unit 230 sums up the variances for each time step up to the last time step in the pseudo trajectory. When there are multiple pseudo trajectories, the analysis unit 230 further calculates the importance I(s) shown in formula (7) for all the pseudo trajectories. ｅ＿ｉ ) may be summed.

[0146] Alternatively, the analysis unit 230 may calculate the importance of each element of the state in the model output based on equation (8) instead of equation (7).

[0147]

[0148] In equation (8), the model p̂ ０ and model p^ １ The variance of the element in the output of ０ (s ｔ＋１，ｅ＿ｉ |a ｔ , s ｔ ), p^ １ (s ｔ＋１，ｅ＿ｉ|a ｔ , s ｔ )) and the cumulative reward after that time step, Σ ｔ’＝ｔ ∞ r ｔ’ " Multiply by

[0149] Equation (8) can be interpreted as indicating that, among the elements of the state predicted by the model, the more ambiguous the element and the more it contributes to the cumulative reward, the more important it is. Regarding equation (8), when there are multiple pseudo trajectories, the analysis unit 230 further calculates the importance I(s) shown in equation (8) for all the pseudo trajectories. ｅ＿ｉ ) may be summed.

[0150] In this way, the analysis unit 230 calculates the model p̂ ０ and p^ １ The analysis unit 230 calculates the importance of each element of the state output by the analysis unit 230 and outputs the calculated importance as the analysis result. However, the importance calculated by the analysis unit 230 is not limited to those shown in the above formulas (7) and (8), and various values ​​can be used.

[0151] 11 is a diagram showing an example of a display screen of the importance of elements of a state displayed by the display unit 120. In the example of FIG. 11, the display unit 120 displays the identification information "s ｅ０ "," "s ｅ１ ", ..., "s ｅ１０ ", along with a description of each element and the importance of each element calculated by the analysis unit 230.

[0152] The user can refer to the display screen shown in Fig. 11 and determine that the greater the importance value of an element, the more important it is to improve the accuracy. ｅ０ The most important thing is to improve the accuracy of the element s ｅ２ It is important to improve the accuracy of element s ｅ３ It can be determined that improving the accuracy of is important. For example, consider a case where a user is considering introducing one of the following two constraint equations into the model search. The first constraint equation considered by the user is shown as equation (9).

[0153]

[0154] Equation (9) is a function of the z-coordinate value of the top of the head 911 at time step t, the velocity in the z-axis direction, and the coefficient c １ represents the z-coordinate value of the top of the head 911 at time step t+1. The second constraint equation considered by the user is expressed as equation (10).

[0155]

[0156] Equation (10) is a function of the angle of the thigh 912 at time step t, and the torque applied to the rotor of the thigh 912 is multiplied by a coefficient c ２ The constraint condition is that the angle calculated by adding the value multiplied by t represents the angle of the thigh 912 at time step t+1. Since both equations (9) and (10) require investigation of coefficient values, it is assumed that the user is considering adopting only one of these two constraint equations. In this case, the user can select one of the two constraint equations by referring to the importance calculated by the analysis unit 230.

[0157] Equation (9) is the state element s ｅ０ and s ｅ６ , the user can determine the importance I(s) of these two elements. ｅ０ ) = 1 and I(s ｅ６ ) = 0, the importance of equation (9) is calculated as 1. On the other hand, equation (10) calculates the importance of the state element s ｅ２ , the user can see the state element s ｅ２ Importance I(s ｅ２ ) = 0.5 is set as the importance of equation (10). Since the importance of equation (9) is greater than the importance of equation (10), the user adopts equation (9). １ The value of is obtained, for example, by actual measurement, and is set in equation (9).

[0158] The user may generate a constraint expression by focusing on an element with a high degree of importance. For example, the user may generate a constraint expression by focusing on an element s with the highest degree of importance. ｅ０It may be decided to generate a constraint equation for the z coordinate value of the top of the head 911 by focusing on:

[0159] In step S9 of FIG. 5, the user inputs the selected equation (9) as feedback information to learning device 100, and feedback information acquisition unit 240 acquires the input feedback information.

[0160] In step S10 of FIG. 5, the model management unit 221 calculates the coefficient c １ The equation (9) in which the value of is set is used as the model p^ ０ and p^ １ In addition to the constraints when searching for the model, the model management unit 221 searches for the model p^ based on the formula (11). ０ A search for the following may be performed.

[0161]

[0162] In equation (11), the model p^ obtained by the calculation of the right-hand side is ０ In other words, model ^p is substituted into model p^ ０ argmin is a function that outputs a parameter value that minimizes the value of the objective function. The model management unit 221 calculates "-logp^(s ｔ＋１ |s ｔ , a ｔ ) + (s ｔ＋１，ｅ０ -s ｔ，ｅ０ -s ｔ，ｅ６ *c １ ) ２ A search is performed for the model p^ so that the value of " becomes smaller, and the obtained model p^ is called the model p^ ０ "*" indicates multiplication.

[0163] "-logp^(s ｔ＋１ |s ｔ , a ｔ ) is the dataset D ｅｎｖ The higher the likelihood of the state-action pair included in "-logp^(s ｔ＋１ |s ｔ , a ｔ) is the model p^ in state s ｔ and Action A ｔ The next state that receives the input and outputs is the data set D ｅｎｖ The next state s ｔ＋１ The closer it is to , the smaller the value becomes.

[0164] "(s ｔ＋１，ｅ０ -s ｔ，ｅ０ -s ｔ，ｅ６ *c １ ) ２ ” means that the model p^ is in state s ｔ and Action A ｔ The value of the model p^ decreases as the degree to which the next state to be output in response to the input of p^ increases and the equation (9) adopted as the constraint equation is satisfied. ０ Similarly to the above case, the model management unit 221 calculates the model p^ based on the formula (12). １ A search for the following may be performed.

[0165]

[0166] The model management unit 221 stores the data set D ｅｎｖ For example, the model management unit 221 may search for a model using the data set D ｅｎｖ The state, s, shown in the quadruple contained in ｔ , action a ｔ , next state s ｔ＋１ , state s ｔ Element e of ０ is ｔ，ｅ０ , state s ｔ Element e of ６ is ｔ，ｅ６ , and the next state s ｔ＋１ Element e of ０ is ｔ＋１，ｅ０ Applying Equation (11) and Equation (12) to the model p̂ ０ and p^ １ Alternatively, the model management unit 221 may search for the data set D ｅｎｖ In addition to or instead of the data set D ｍｏｄｅｌ A model may be searched for based on the above.

[0167] The user may input multiple constraint equations to the learning device 100. In this case, the model management unit 221 may calculate the importance of each of the constraint equations and weight the constraint equations based on the calculated importance. For example, the model management unit 221 may create a model p^ based on equation (13). ０ A search for the following may be performed.

[0168]

[0169] ReLU stands for Ramp Function. ｔ＋１，ｅ２ -s ｔ，ｅ２ -c ２ *a ｔ，ｅ０ The value of "s" is 0 if the formula (10) is true, and is 0 if the formula (10) is not true. ｔ＋１，ｅ２ -s ｔ，ｅ２ -c ２ *a ｔ，ｅ０ ", that is, the value obtained by subtracting the right side of equation (10) from the left side.

[0170] "(ReLU(s ｔ＋１，ｅ２ -s ｔ，ｅ２ -c ２ *a ｔ，ｅ０ )) ２ ” means that the model p^ is in state s ｔ and Action A ｔ If the next state output in response to the input satisfies the equation (10) adopted as the constraint equation, then it becomes 0, and if it does not satisfy the equation (10), then the smaller the degree of deviation, the smaller the value becomes.

[0171] α １ is "(s ｔ＋１，ｅ０ -s ｔ，ｅ０ -s ｔ，ｅ６ *c １ ) ２ " is the weighting coefficient for α １ is "(ReLU(s ｔ＋１，ｅ２ -s ｔ，ｅ２ -c ２ *a ｔ，ｅ０ )) ２" is a weighting coefficient for the equation (9). For example, the model management unit 221 calculates the importance of the equation (9) as 1 and the importance of the equation (10) as 0.5, as in the above. Then, the model management unit 221 normalizes the calculated importance to obtain α １ = 1 / (1 + 0.5), and α ２ = 0.5 / (1 + 0.5).

[0172] By using the weighting coefficients exemplified in Equation (13), the model management unit 221 can more strongly influence the model search of the more important constraint expressions set by the user, which is expected to result in a highly accurate model.

[0173] The learning device 100 may be configured to eliminate the need for user input of feedback information during model learning and policy learning. For example, the storage unit 180 may be configured to pre-store constraint equations based on user knowledge, such as those exemplified by the above equations (9) and (10). Then, in step S9 of FIG. 5 , the feedback information acquisition unit 240 may calculate the importance of each constraint equation as described above and select the constraint equation with the highest importance. Alternatively, the feedback information acquisition unit 240 may select multiple constraint equations based on the importance of each constraint equation, such as selecting a predetermined number of constraint equations in descending order of importance.

[0174] (2) When the analysis result indicates the accuracy of each of a plurality of pseudo trajectories, and the feedback information indicates the corrected value of the pseudo trajectory or information on whether the value indicated in the pseudo trajectory is correct.

[0175] In the following, an example will be described in which the learning unit 220 generates pseudo trajectories six times. ０ , τ １ , ..., τ ５ However, the number of pseudo trajectories generated by the learning unit 220 is not limited to a specific number, as long as it is two or more.

[0176] The jth (j is an integer 0≦j≦5) pseudo trajectory τ ｊThe state and action at time step t included in ｔ，τｊ , a ｔ，τｊ In step S8 of Fig. 5, the analysis unit 230 calculates the importance of each pseudo trajectory based on, for example, equation (14).

[0177]

[0178] I(τ ｊ ) is the pseudo-trajectory τ ｊ The analysis unit 230 calculates the importance of the model p^ for each pseudo trajectory and for each time step based on the equation (14). ０ The next state s ｔ＋１ , τ ｊ And, model p^ １ The next state s ｔ＋１ , τ ｊ The variance of the pseudo trajectory is calculated, and the variance is summed up for all time steps in the pseudo trajectory to obtain the importance I(τ ｊ ) The importance I(τ ｊ ) is an example of an evaluation index value for the accuracy of time series data.

[0179] Equation (14) can be interpreted as indicating that the more ambiguous the state predicted by the model is, the more important the pseudo trajectory is. A pseudo trajectory in which the state predicted by the model is ambiguous overall can be said to be a pseudo trajectory in which the prediction accuracy by the model is low when all time steps included in the pseudo trajectory are considered comprehensively.

[0180] As explained for equation (7), the ∞ in Σ in equation (14) also indicates that the number of time steps in the pseudo trajectory is arbitrary. The analysis unit 230 sums up the variances for each time step up to the last time step in the pseudo trajectory. However, the importance calculated by the analysis unit 230 is not limited to that shown in equation (14) above, and various values ​​are possible.

[0181] 12 is a diagram showing an example of a display screen of the importance of the pseudo trajectories displayed by the display unit 120. In the example of FIG. 12, the display unit 120 displays the identification information "τ ０ ", "τ １ ", ... "τ ５ ", the importance of each pseudo trajectory, and a button icon for accepting a user operation to request the display of an editing screen for each pseudo trajectory.

[0182] The user can determine that the greater the importance value of a pseudo trajectory, the more important it is to improve the accuracy by referring to the display screen shown in FIG. １ The most important thing is to improve the accuracy of the pseudo-trajectory τ ３ The improvement of the accuracy of the pseudo trajectory τ is important. ５ The pseudo trajectories can be prioritized such that improving the accuracy of the pseudo trajectories is important.

[0183] If correcting all pseudo trajectories is too much for the user, the user can select the pseudo trajectories to be corrected based on their importance. １ and τ ３ It has been decided to amend the following.

[0184] In step S9 of FIG. 5, the user corrects the pseudo trajectory, and the feedback information acquisition unit 240 acquires feedback information indicating the correction of the pseudo trajectory.

[0185] FIG. 13 shows the pseudo trajectory τ displayed by the display unit 120. １ 12 is a diagram showing an example of an editing screen for the pseudo trajectory τ １ When the button icon shown in the row is pressed, the edit screen of FIG. 13 is displayed.

[0186] In the example of FIG. 13, the display unit 120 displays the pseudo trajectory τ １For each time step, the values ​​of each element of state s and each element of action a are displayed. The user corrects the pseudo trajectory by correcting the values ​​of the displayed state elements. The display unit 120 displays the values ​​of the elements of action a as reference information for the user to obtain the correct values ​​of the state elements. Alternatively, the values ​​of the elements of action a may also be subject to correction by the user.

[0187] 14 is a diagram showing an example of an editing screen of a pseudo trajectory after correction by the user, displayed by the display unit 120. Fig. 14 shows an example in which a correction operation by the user is performed on the editing screen shown in Fig. 13. The user edits the element s of the state s at time step 3. ｅ０ and s ｅ２ The user corrects the values ​​of these elements. Here, it is assumed that the user has knowledge of the correct values, such as being able to calculate the values ​​of these elements. In the example of Figure 14, the display unit 120 displays the corrected values ​​with underlines.

[0188] Trajectory τ ｊ The state s of the control object 810 at time step t is ｔ，τｊ The state after the user makes the correction to s ｔ，τｊ，ｆ In addition, weights based on the reliability of the pseudo trajectory elements are introduced. The pseudo trajectory elements here are the state elements shown in the pseudo trajectory. When behavior is also subject to user modification, the behavior elements shown in the pseudo trajectory are also referred to as pseudo trajectory elements.

[0189] The user sets weight values ​​based on the user's own judgment as to whether the elements of the pseudo trajectory are correct. For example, the user sets a weight value of 1 for an element whose value is judged to be correct. The user also sets a weight value of 0 for an element whose value is judged to be unknown. The user also sets a weight value of -1 for an element whose value is judged to be incorrect. For a pseudo trajectory that has been corrected, the user sets a weight value for the corrected pseudo trajectory. The weight setting value also corresponds to an example of feedback information.

[0190] FIG. 15 shows the pseudo trajectory τ １ 15 shows an example of a screen for setting weights for elements of the time step 3. FIG. 15 shows an example of weights set by the user for the values ​​of the elements shown in FIG. 14. The user can set weights for the element s ｅ０ The corrected value of element s ｅ２ The corrected value is determined to be correct, and the weight value is set to 1.

[0191] On the other hand, the user ｅ１ , s ｅ３ , s ｅ４ , ..., s ｅ１０ The model management unit 221 determines that each value is correct or incorrect and sets the weight value to 0. This weight is used when updating the model using feedback information. For example, by using this weight, the model management unit 221 can filter out element values ​​that the user has determined to be correct or incorrect.

[0192] FIG. 16 shows the pseudo trajectory τ displayed by the display unit 120. ３ 12 is a diagram showing an example of an editing screen for the pseudo trajectory τ ３ When a button icon shown in the row is pressed, the editing screen of FIG. 16 is displayed. ３ For the above, the user does not modify the element values ​​but only sets the weights.

[0193] FIG. 17 shows the pseudo trajectory τ displayed by the display unit 120. ３ 17 shows an example of a screen for setting weights for elements of the time step 3. FIG. 17 shows an example of weights set by the user for the values ​​of the elements shown in FIG. 16. The user can set weights for the element s ｅ４ , s ｅ９ Each value is judged to be correct and the weight value is set to 1.

[0194] On the other hand, the user ｅ０ , s ｅ２ , s ｅ５ , s ｅ７ , s ｅ８ , s ｅ１０ The user determines that the values ​​of the elements s are correct or incorrect and sets the weight to 0. ｅ１ , s ｅ３ , s ｅ６ Each value is judged to be incorrect and the weight value is set to -1.

[0195] 5, the model management unit 221 searches for a model using the obtained feedback information. The model management unit 221 searches for a model p^ based on Equation (15). ０ A search for the following may be performed.

[0196]

[0197] State S ｔ’＋１，τｊ，ｆ For state s ｔ’＋１，τｊ If no corrections have been made to state s ｔ’＋１，τｊ The value of ｔ’＋１，τｊ，ｆ Also, the state s ｔ’＋１，τｊ，ｆ If only some of the elements included in ｔ’＋１，τｊ The value of that element in is used as is.

[0198] "w ｔ’，τｊ " is the trajectory τ ｊ The state s of the control object 810 at time step t′ shown in ｔ’，τｊ In equation (15), the weight "wｔ’，τｊ " is expressed as a horizontal vector. Also, the model p^ is ｔ’，τｊ and Action a ｔ’，τｊ For the input, the corrected trajectory τ ｊ The next state s shown in ｔ’＋１，τｊ，ｆ The likelihood of outputting "p'^(s ｔ’＋１，τｊ，ｆ |s ｔ’，τｊ , a ｔ’，τｊ ) is expressed as a vertical vector for each element of the state.

[0199] "w ｔ’，τｊ ・p'^(s ｔ’＋１，τｊ，ｆ |s ｔ’，τｊ , a ｔ’，τｊ ) the weight vector "w ｔ’，τｊ " and likelihood vector "p'^(s ｔ’＋１，τｊ，ｆ |s ｔ’，τｊ , a ｔ’，τｊ )) is taken. Therefore, for elements whose weight value is set to 1, the value of this inner product is calculated as follows: ｊ The state s shown in ｔ’＋１，τｊ，ｆ The higher the likelihood of outputting the value of that element in , the larger the value.

[0200] On the other hand, elements whose weight value is set to 0 are filtered out in the calculation of the value of this dot product formula. In other words, for elements whose weight value is set to 0, the value output by the model p^ does not affect the value of this dot product formula.

[0201] In addition, for the element whose weight value is set to −1, the model p̂ calculates the corrected pseudo trajectory τ ｊ The state s shown in ｔ’＋１，τｊ，ｆ The higher the likelihood of outputting the value of that element in , the smaller the value.

[0202] The model management unit 221 calculates the model p̂ using equation (15). ０By performing a search for the pseudo trajectory, the user's modifications to the pseudo trajectory and weight settings are reflected in the search for the model. In this respect, it is expected that the accuracy of the obtained model will be high. However, it is not essential for the user to set weights. If the user does not set weights, the model management unit 221 may set all weight values ​​to 1 and perform a search for the model.

[0203] Model p^ ０ Similarly to the above case, the model management unit 221 calculates the model p^ based on the equation (16). １ A search for the following may be performed.

[0204]

[0205] The model management unit 221 stores the data set D ｅｎｖ Alternatively, the model management unit 221 may search for a model using the data set D ｅｎｖ In addition to or instead of the data set D ｍｏｄｅｌ A model may be searched for based on the above.

[0206] As described above, the model management unit 221 learns using data that associates the state of the environment in which the agent performs an action, the action that can be performed in that state, and the next state when that action is performed in that state, and generates a model p^ ０ and p^ １ Based on the obtained model, the feedback information acquisition unit 240 acquires feedback information, which is information used for learning the model or for learning a new model that takes a state and an action as input and a next state as output. The policy management unit 222 learns policies that indicate the behavior of the agent depending on the state by learning using the model obtained using the feedback information.

[0207] According to the learning device 100, the model management unit 221 uses a previously obtained data set D ｅｎｖIn this case, when sufficient data is not available, the feedback information can be used to compensate for the lack of information, and it is expected that a relatively accurate model can be obtained. When the policy management unit 222 uses this model to learn the policy (learn the control of the control object 810), it is expected that the accuracy of the control learning in the state where sufficient data is not available can be improved.

[0208] The feedback information acquisition unit 240 also acquires the feedback information indicating constraints that must be satisfied by the input data and output data of the model being trained. The model management unit 221 uses the constraints to search for a model. The learning device 100 can obtain a model in which the constraints are reflected in the relationship between the input data and the output data, and in this respect, it is expected that a relatively accurate model can be obtained.

[0209] Furthermore, for each item included in the information indicating the state, the analysis unit 230 calculates an evaluation index value for the accuracy of that item in the next state to be output by the obtained model. The feedback information acquisition unit 240 acquires feedback information indicating constraint conditions for items with relatively low accuracy evaluations. According to the learning device 100, for items included in the information indicating the state for which the accuracy of the values ​​output by the model is relatively low, the accuracy is improved based on the constraint conditions, and in this respect, it is expected that the accuracy of the model can be efficiently improved.

[0210] The display unit 120 also displays an evaluation index value for the accuracy of the item in the next state output by the obtained model. The operation input unit 130 accepts a user operation to input feedback information. After the display unit 120 starts displaying the evaluation index value, the feedback information acquisition unit 240 acquires the feedback information input by the user operation accepted by the operation input unit 130.

[0211] According to the learning device 100, the model management unit 221 searches for a model using feedback information, which allows the user's knowledge according to the accuracy of the model to be reflected in the model search. In this respect, the learning device 100 is expected to be able to obtain a relatively accurate model relatively efficiently.

[0212] Furthermore, the feedback information acquisition unit 240 acquires feedback information indicating modifications to the input / output data of the obtained model. The model management unit 221 uses the input / output data reflecting the modifications to train the model. The learning device 100 is expected to obtain a relatively accurate model because it trains the model using the input / output data reflecting the modifications.

[0213] The analysis unit 230 also calculates an evaluation index value for the accuracy of the time-series data for each of the multiple time-series data of the input and output of the obtained model. The feedback information acquisition unit 240 acquires feedback information indicating corrections to time-series data with a relatively low evaluation of accuracy.

[0214] According to the learning device 100, among multiple pieces of time-series data, time-series data with a relatively low evaluation of accuracy is corrected. The model management unit 221 uses the corrected time-series data to learn the model, which is expected to improve the accuracy of the model relatively efficiently.

[0215] The display unit 120 also displays an evaluation index value for the accuracy of the time-series data for each of the multiple pieces of time-series data for input and output of the obtained model. The operation input unit 130 accepts a user operation for inputting feedback information. After the display unit 120 starts displaying the evaluation index values, the feedback information acquisition unit 240 acquires the feedback information input by the user operation accepted by the operation input unit 130.

[0216] According to the learning device 100, the model management unit 221 uses feedback information to learn a model, and thus the model can be learned using data whose values ​​have been modified by the user for time-series data with relatively low accuracy among multiple time-series data. In this respect, the learning device 100 is expected to be able to relatively efficiently obtain a model with relatively high accuracy.

[0217] In addition, the display unit 120 displays, for each item included in the information indicating the next state that is output by the model simulating the environment in which the agent performs an action in response to input of information indicating the state and information indicating the action, an evaluation index value for the accuracy of that item in the information indicating the next state.

[0218] With learning device 100, the user can refer to the index values ​​displayed by display unit 120 and input feedback information to learning device 100 to improve items with low accuracy in the model output. In this respect, learning device 100 is expected to be able to efficiently improve the accuracy of the model.

[0219] Furthermore, the display unit 120 displays an evaluation index value for the accuracy of time-series data for each of multiple pieces of time-series data of the state of the environment and the agent's actions, which is time-series data of the input and output of a model that simulates the environment in which the agent performs its actions. The learning device 100 allows the user to select and correct time-series data with low accuracy by referring to the index values ​​displayed by the display unit 120. In this respect, the learning device 100 is expected to be able to efficiently improve the accuracy of the model.

[0220] FIG. 18 is a diagram illustrating another example of the configuration of a learning device according to an embodiment. In the configuration illustrated in FIG. 18 , the learning device 610 includes a model acquisition unit 611, a feedback information acquisition unit 612, and a policy management unit 613. In this configuration, the model acquisition unit 611 acquires a model that takes a state and an action as input and outputs a next state through learning using data linking the state of the environment in which an agent performs an action, actions executable in that state, and the next state when the action is performed in that state. Based on the acquired model, the feedback information acquisition unit 612 acquires feedback information, which is information used for learning the model or for learning a new model that takes a state and an action as input and outputs the next state. The policy management unit 613 learns a policy indicating the agent's behavior according to the state through learning using the model acquired using the feedback information. The model acquisition unit 611 corresponds to an example of model acquisition means. The feedback information acquisition unit 612 corresponds to an example of feedback information acquisition means. The policy management unit 613 corresponds to an example of policy management means.

[0221] According to the learning device 610, when the model acquisition unit 611 does not have enough data from the data obtained in advance, it is expected that the model acquisition unit 611 can compensate for the lack of information with feedback information, and obtain a relatively accurate model. When the policy management unit 613 uses this model to learn policies, it is expected that the accuracy of control learning in a state where sufficient data is not available can be improved.

[0222] The model acquisition unit 611 can be realized using, for example, the functions of the model management unit 221 in Fig. 1. The feedback information acquisition unit 612 can be realized using, for example, the functions of the feedback information acquisition unit 240 in Fig. 1. The policy management unit 613 can be realized using, for example, the functions of the policy management unit 222 in Fig. 1.

[0223] Fig. 19 is a diagram showing an example of the configuration of a display device according to an embodiment. In the configuration shown in Fig. 19, display device 620 includes a display unit 621. In this configuration, display unit 621 displays, for each item included in information indicating a next state that is output by a model simulating an environment in which an agent performs an action in response to input of information indicating a state and information indicating an action, an evaluation index value for the accuracy of that item in the information indicating the next state. Display unit 621 corresponds to an example of display means.

[0224] With the display device 620, a user can refer to the index values ​​displayed by the display unit 621 and input feedback information to the display device 620 to improve items with low accuracy in the model output. In this respect, the display device 620 is expected to be able to efficiently improve the accuracy of the model. The display unit 621 can be realized, for example, using the functions of the display unit 120 in FIG. 1 .

[0225] Fig. 20 is a diagram showing another example of the configuration of a display device according to an embodiment. In the configuration shown in Fig. 20, a display device 630 includes a display unit 631. In this configuration, the display unit 631 displays an evaluation index value for the accuracy of time-series data for each of a plurality of pieces of time-series data of the state of the environment and the behavior of the agent, which is time-series data of the input and output of a model that simulates the environment in which the agent performs its behavior. The display unit 631 corresponds to an example of display means.

[0226] The display device 630 allows a user to select and correct time-series data with low accuracy by referring to the index values ​​displayed by the display unit 631. In this respect, the display device 630 is expected to be able to efficiently improve the accuracy of the model. The display unit 631 can be realized, for example, using the functions of the display unit 120 in FIG. 1 .

[0227] 21 is a diagram illustrating an example of a processing procedure in a learning method according to an embodiment. The learning method illustrated in FIG. 21 includes acquiring a model (step S611), acquiring feedback information (step S612), and learning a policy (step S613).

[0228] In acquiring a model (step S611), the computer acquires a model in which the state and the action are input and the next state is output, based on data linking the state of the environment in which the agent performs an action, the action that can be performed in that state, and the next state when that action is performed in that state.

[0229] In obtaining feedback information (step S612), the computer obtains feedback information for obtaining a more accurate model based on the output of the obtained model. In learning a policy (step S613), the computer learns a policy that indicates the behavior of the agent depending on the state, using the model obtained using the feedback information.

[0230] According to the learning method shown in Fig. 21, when the previously obtained data is insufficient, the lack of information can be compensated for by the feedback information, and it is expected that a relatively accurate model can be obtained. By learning a policy using this model, it is expected that the accuracy of control learning in a state where sufficient data is not available can be improved.

[0231] Fig. 22 is a diagram showing an example of a processing procedure in a display method according to an embodiment. The display method shown in Fig. 22 includes performing display (step S621). In performing display (step S621), a computer displays, for each item included in information indicating a next state that is output by a model simulating an environment in which an agent performs an action in response to input of information indicating a state and information indicating an action, an evaluation index value for the accuracy of that item in the information indicating the next state.

[0232] According to the display method shown in Fig. 22, a user can refer to the displayed index values ​​and input feedback information to the computer to improve items with low accuracy in the model output. In this respect, the display method shown in Fig. 22 is expected to efficiently improve the accuracy of the model.

[0233] 23 is a diagram showing another example of a procedure of processing in a display method according to an embodiment. The display method shown in FIG. 23 includes performing display (step S631). In performing display (step S631), a computer displays an evaluation index value of the accuracy of time-series data for each of a plurality of pieces of time-series data of the state of the environment and the behavior of the agent, which is time-series data of the input and output of a model that simulates the environment in which the agent performs its behavior.

[0234] According to the display method shown in Fig. 23, a user can select and correct time-series data with low accuracy by referring to the displayed index values. In this respect, the display method shown in Fig. 23 is expected to efficiently improve the accuracy of the model.

[0235] 24 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment. In the configuration shown in FIG. 24, a computer 700 includes a CPU 710, a main memory device 720, an auxiliary memory device 730, an interface 740, and a non-volatile recording medium 750.

[0236] One or more of the learning device 100, learning device 610, display device 620, and display device 630, or a portion thereof, may be implemented in computer 700. In this case, the operation of each of the above-described processing units is stored in the auxiliary storage device 730 in the form of a program. CPU 710 reads the program from auxiliary storage device 730, loads it into main storage device 720, and executes the above-described processing in accordance with the program. Furthermore, CPU 710 allocates storage areas in main storage device 720 corresponding to each of the above-described storage units in accordance with the program. Communication between each device and other devices is performed by interface 740, which has a communication function and performs communication under the control of CPU 710. Furthermore, interface 740 has a port for non-volatile storage medium 750, and reads information from non-volatile storage medium 750 and writes information to non-volatile storage medium 750.

[0237] When learning device 100 is implemented in computer 700, the operations of processing unit 190 and each of its units are stored in the form of a program in auxiliary storage device 730. CPU 710 reads the program from auxiliary storage device 730, loads it into main storage device 720, and executes the above-described processing in accordance with the program.

[0238] Furthermore, the CPU 710 allocates storage areas corresponding to the storage unit 180 and each unit thereof in the main storage device 720 in accordance with the program. Communication by the communication unit 110 is performed by the interface 740 having a communication function and performing communication under the control of the CPU 710. Display of images by the display unit 120 is performed by the interface 740 having a display device and displaying images under the control of the CPU 710. Reception of user operations by the operation input unit 130 is performed by the interface 740 having an input device and receiving user operations.

[0239] When the learning device 610 is implemented in the computer 700, the operations of the model acquisition unit 611, the feedback information acquisition unit 612, and the policy management unit 613 are stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0240] Furthermore, CPU 710 allocates a storage area in main memory 720 for learning device 610 to perform processing in accordance with the program. Communication between learning device 610 and other devices is performed by interface 740, which has a communication function and operates under the control of CPU 710. Interaction between learning device 610 and a user is performed by interface 740, which has a display device and an input device, displaying various images under the control of CPU 710 and accepting user operations.

[0241] When the display device 620 is implemented in the computer 700, its operation is stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0242] Furthermore, the CPU 710 allocates a storage area in the main storage device 720 for the display device 620 to perform processing in accordance with the program. Communication between the display device 620 and other devices is performed by the interface 740, which has a communication function and operates under the control of the CPU 710. Display of images by the display unit 621 is performed by the interface 740 having a display device and displaying images under the control of the CPU 710. Reception of user operations on the display device 620 is performed by the interface 740 having an input device and receiving user operations.

[0243] When the display device 630 is implemented in the computer 700, its operation is stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0244] Furthermore, the CPU 710 allocates a storage area in the main storage device 720 for the display device 630 to perform processing in accordance with the program. Communication between the display device 630 and other devices is performed by the interface 740, which has a communication function and operates under the control of the CPU 710. Display of images by the display unit 631 is performed by the interface 740 having a display device and displaying images under the control of the CPU 710. Reception of user operations on the display device 630 is performed by the interface 740 having an input device and receiving user operations.

[0245] One or more of the above-described programs may be recorded on nonvolatile recording medium 750. In this case, interface 740 may read the programs from nonvolatile recording medium 750. Then, CPU 710 may directly execute the programs read by interface 740, or may temporarily store the programs in main storage device 720 or auxiliary storage device 730 and then execute them.

[0246] Note that a program for executing all or part of the processing performed by learning device 100, learning device 610, display device 620, and display device 630 may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be loaded into a computer system and executed to perform the processing of each unit. Note that the term "computer system" here includes hardware such as an operating system (OS) and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, read-only memories (ROMs), and compact disc read-only memories (CD-ROMs), as well as storage devices such as hard disks built into computer systems. The program may be designed to implement part of the aforementioned functions, or may be capable of implementing the aforementioned functions in combination with a program already recorded on the computer system.

[0247] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0248] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0249] (Supplementary Note 1) A learning device comprising: a model acquisition means for acquiring a model that takes a state and an action as input and outputs a next state through learning using data that links the state of the environment in which an agent performs an action, the actions that can be performed in that state, and the next state when that action is performed in that state; a feedback information acquisition means for acquiring, based on the acquired model, feedback information that is information used for learning the model or for learning a new model that takes the state and the action as input and outputs the next state; and a policy management means for learning a policy that indicates the agent's behavior according to the state, using the model acquired through learning using the feedback information.

[0250] (Supplementary Note 2) The learning device according to Supplementary Note 1, wherein the feedback information acquisition means acquires the feedback information indicating constraint conditions that must be satisfied by input data and output data of a model to be learned, and the model acquisition means searches for a model using the constraint conditions.

[0251] (Supplementary Note 3) The learning device described in Supplementary Note 2 further comprises an analysis means for calculating, for each item included in the information indicating a state, an evaluation index value for the accuracy of that item in the next state output by the obtained model, and the feedback information acquisition means acquires the feedback information indicating constraint conditions for items with a relatively low evaluation of accuracy.

[0252] (Supplementary Note 4) The learning device described in Supplementary Note 3 further comprises: a display means for displaying the evaluation index value; and an input means for accepting a user operation for inputting the feedback information, wherein the feedback information acquisition means acquires the feedback information input by the user operation accepted by the input means after the display means starts displaying the evaluation index value.

[0253] (Supplementary Note 5) The learning device according to Supplementary Note 1, wherein the feedback information acquisition means acquires the feedback information indicating a modification to the obtained input / output data of the model, and the model acquisition means performs model learning using the input / output data in which the modification is reflected.

[0254] (Supplementary Note 6) The learning device according to Supplementary Note 5, further comprising an analysis means for calculating an evaluation index value of the accuracy of the time series data for each of the plurality of time series data of the input and output of the obtained model, wherein the feedback information acquisition means acquires the feedback information indicating a correction for time series data with a relatively low evaluation of accuracy.

[0255] (Supplementary Note 7) The learning device described in Supplementary Note 6 further comprises: a display means for displaying the evaluation index value; and an input means for accepting a user operation for inputting the feedback information, wherein the feedback information acquisition means acquires the feedback information input by the user operation accepted by the input means after the display means starts displaying the evaluation index value.

[0256] (Supplementary Note 8) A display device comprising a display means for displaying, for each item included in information indicating a next state output by a model simulating an environment in which an agent performs an action in response to input of information indicating a state and information indicating an action, an evaluation index value of the accuracy of that item in the information indicating the next state.

[0257] (Supplementary Note 9) A display device comprising: a display means for displaying an evaluation index value of accuracy of time-series data for each of a plurality of time-series data of a state in an environment and an action of the agent, the time-series data being time-series data of inputs and outputs of a model that simulates an environment in which an agent performs an action.

[0258] (Supplementary Note 10) A learning method including the steps of: a computer learning using data that links the state of the environment in which an agent performs an action, the actions that can be performed in that state, and the next state when that action is performed in that state, thereby obtaining a model that takes the state and the action as input and the next state as output; based on the obtained model, obtaining feedback information that is information used for learning the model or for learning a new model that takes the state and the action as input and the next state as output; and using the model obtained using the feedback information to learn a strategy that indicates the action of the agent depending on the state.

[0259] (Supplementary Note 11) A display method including a computer displaying, for each item included in information indicating a next state that is output by a model that simulates an environment in which an agent performs an action in response to input of information indicating a state and information indicating an action, an evaluation index value of the accuracy of that item in the information indicating the next state.

[0260] (Supplementary Note 12) A display method including: a computer displays an evaluation index value of the accuracy of time-series data for each of a plurality of time-series data of states in an environment and actions of an agent, the time-series data being time-series data of inputs and outputs of a model that simulates an environment in which an agent performs an action.

[0261] (Supplementary Note 13) A recording medium having recorded thereon a program for causing a computer to execute the following: acquiring a model in which the state and action are input and the next state is output by learning using data linking the state of the environment in which an agent performs an action, the actions that can be performed in that state, and the next state when that action is performed in that state; acquiring feedback information based on the acquired model, which is information used for learning the model or for learning a new model in which the state and action are input and the next state is output; and learning a strategy that indicates the action of the agent depending on the state, using the model acquired by learning using the feedback information.

[0262] (Supplementary Note 14) A recording medium having recorded thereon a program for causing a computer to execute the following: for each item included in information indicating a next state that is output by a model that simulates an environment in which an agent performs an action in response to input of information indicating a state and information indicating an action, displaying an evaluation index value of the accuracy of that item in the information indicating the next state.

[0263] (Supplementary Note 15) A recording medium having recorded thereon a program for causing a computer to execute the following: displaying an evaluation index value for the accuracy of time-series data for each of a plurality of time-series data of a state in an environment and an action of the agent, the time-series data being time-series data of inputs and outputs of a model that simulates an environment in which an agent performs an action.

[0264] The present invention may be applied to a learning device, a display device, a learning method, and a recording medium.

[0265] 1 Learning system 2 Data collection system 3 Control system 100, 610 Learning device 110 Communication unit 120, 621, 631 Display unit 130 Operation input unit 180 Memory unit 181 Data memory unit 182 Model memory unit 183 Strategy memory unit 190 Processing unit 210 Data management unit 220 Learning unit 221 Model management unit 222, 613 Strategy management unit 230 Analysis unit 240, 612 Feedback information acquisition unit 300 Data collection device 400 Control device 620, 630 Display device 611 Model acquisition unit

Claims

1. a model acquisition means for acquiring a model in which a state and an action are input and a next state is output by learning using data in which an environmental state in which an agent executes an action, an action executable in that state, and a next state when that action is executed in that state are linked; a feedback information acquisition means for acquiring feedback information, which is information used for learning the model based on the obtained model, or for learning a new model in which a state and an action are input and a next state is output; a policy management means for learning a policy indicating an action of the agent according to a state by using a model obtained by learning using the feedback information; A learning device comprising:

2. the feedback information acquisition means acquires the feedback information indicating constraint conditions that should be satisfied by input data and output data of a model to be trained; The model acquisition means searches for a model using the constraint conditions. The learning device according to claim 1 .

3. Further, an analysis means is provided for calculating, for each item included in the information indicating the state, an evaluation index value of the accuracy of the item in the next state output by the obtained model; the feedback information acquisition means acquires the feedback information indicating a constraint condition related to an item having a relatively low evaluation of accuracy; The learning device according to claim 2 .

4. A display means for displaying the evaluation index value; an input means for receiving a user operation for inputting the feedback information; Further equipped with the feedback information acquisition means acquires the feedback information input by a user operation accepted by the input means after the display means starts displaying the evaluation index value. The learning device according to claim 3.

5. the feedback information acquisition means acquires the feedback information indicating a modification to the input / output data of the obtained model; the model acquisition means performs model learning using the input / output data in which the correction is reflected. The learning device according to claim 1 .

6. The method further includes an analysis means for calculating an evaluation index value of the accuracy of the time-series data for each of the obtained multiple time-series data of the input and output of the model, The feedback information acquisition means acquires the feedback information indicating a correction to the time series data having a relatively low evaluation of accuracy. The learning device according to claim 5 .

7. A display means for displaying the evaluation index value; an input means for receiving a user operation for inputting the feedback information; Further equipped with the feedback information acquisition means acquires the feedback information input by a user operation accepted by the input means after the display means starts displaying the evaluation index value. The learning device according to claim 6.

8. The computer By learning using data that links the state of the environment in which the agent performs an action, the actions that can be performed in that state, and the next state when that action is performed in that state, a model is obtained in which the state and action are input and the next state is output. Based on the obtained model, feedback information is obtained, which is information used to learn the model or to learn a new model in which the state and the action are input and the next state is output; learning a policy indicating an action of the agent according to a state by using a model obtained by learning using the feedback information; A learning method that includes:

9. On the computer, A model is obtained by learning using data that links the state of the environment in which the agent performs an action, the actions that can be performed in that state, and the next state when that action is performed in that state, and the model takes the state and the action as input and the next state as output; Obtaining feedback information based on the obtained model, which is information used to learn the model or to learn a new model in which the state and the action are input and the next state is output; learning a policy indicating an action of the agent according to a state by using a model obtained by learning using the feedback information; A program for executing the above.