Learning device, control system, learning method and program

Abstract

In the present invention, a learning device generates a policy, which is a rule for determining a to-be-controlled action, on the basis of a reward value that indicates an evaluation of the action in a state in which a time step is traced back in an episode representing a learning period, and the learning device determines the to-be-controlled action on the basis of the policy.

Description

Learning device, control system, input / output device, learning method, and recording medium

[0001] The present disclosure relates to a learning device, a control system, an input / output device, a learning method, and a recording medium.

[0002] One type of machine learning is reinforcement learning (see, for example, Patent Document 1).

[0003] Japanese Patent Application Publication No. 2022-182581

[0004] It is preferable to be able to efficiently perform reinforcement learning using simulation.

[0005] An example of an object of the present disclosure is to provide a learning device, a control system, an input / output device, a learning method, and a recording medium that can solve the above-mentioned problems.

[0006] According to a first aspect of the present disclosure, a learning device includes a policy generation means for generating a policy, which is a decision rule for the behavior of a controlled object, based on a reward value, which is a value indicating an evaluation of the behavior of the controlled object at a time step back within an episode representing a learning period, and a behavior decision means for deciding the behavior of the controlled object based on the policy.

[0007] According to a second aspect of the present disclosure, a control system includes a learning device and a control device, wherein the learning device includes a policy generation means for generating a policy, which is a decision rule for the behavior of the controlled object, based on a reward value, which is a value indicating an evaluation of the behavior of the controlled object at a time step back within an episode representing a learning period, and a behavior decision means for deciding the behavior of the controlled object based on the policy, and the control device controls the controlled object based on the policy obtained using the learning device.

[0008] According to a third aspect of the present disclosure, the input / output device includes a state presentation means for presenting to a user the state of the environment in which the controlled object is acting, and a state designation receiving means for receiving a user operation for specifying a state in which a simulator that simulates the environment needs to be interrupted.

[0009] According to a fourth aspect of the present disclosure, a learning method includes generating a policy, which is a decision rule for the behavior of a controlled object, based on a reward value, which is a value indicating an evaluation of the behavior of the controlled object at a time step back within an episode representing a learning period, and determining the behavior of the controlled object based on the policy.

[0010] According to a fifth aspect of the present disclosure, a recording medium stores a program for causing a computer to generate a policy, which is a decision rule for the behavior of a controlled object, based on a reward value, which is a value indicating an evaluation of the behavior of the controlled object at a time step back within an episode representing a learning period, and to determine the behavior of the controlled object based on the policy.

[0011] According to the present disclosure, reinforcement learning using simulation can be performed relatively efficiently.

[0012] 1 is a diagram illustrating an example of a configuration of a control system according to some embodiments of the present disclosure. FIG. 2 is a diagram illustrating an example of a configuration of a learning device according to some embodiments of the present disclosure. FIG. 3 is a diagram illustrating an example of an instantaneous reward value from which a cumulative reward value is calculated. FIG. 4 is a diagram illustrating an example of a cumulative reward value. FIG. 5 is a diagram illustrating a first example of a probability distribution set for a reset destination candidate according to some embodiments of the present disclosure. FIG. 6 is a diagram illustrating a second example of a probability distribution set for a reset destination candidate according to some embodiments of the present disclosure. FIG. 7 is a diagram illustrating a third example of a probability distribution set for a reset destination candidate according to some embodiments of the present disclosure. FIG. 8 is a diagram illustrating an example of data input / output in a learning device according to some embodiments of the present disclosure. FIG. 9 is a diagram illustrating an example of a processing procedure performed by a learning device according to some embodiments of the present disclosure. FIG. 10 is a diagram illustrating another example of a configuration of a learning device according to some embodiments of the present disclosure. FIG. 11 is a diagram illustrating another example of a configuration of a control system according to some embodiments of the present disclosure. FIG. 12 is a diagram illustrating an example of a configuration of an input / output device according to some embodiments of the present disclosure. FIG. 13 is a diagram illustrating an example of a processing procedure in a learning method according to some embodiments of the present disclosure. FIG. 14 is a diagram illustrating an example of a configuration of a computer according to at least one embodiment of the present disclosure.

[0013] Embodiments of the present disclosure will be described below, but the following embodiments do not limit the disclosure according to the claims. Furthermore, not all combinations of features described in the embodiments are necessarily essential to the disclosed solution. FIG. 1 is a diagram showing an example of the configuration of a control system according to some embodiments of the present disclosure. In the configuration shown in FIG. 1, the control system 1 includes a learning device 100, a control device 200, and a control target 910.

[0014] The control system 1 is a system that learns control of a control object 910 and controls the control object 910 based on the learning results. The learning device 100 learns control of the control object 910. In particular, the learning device 100 learns control of the control object 910 by reinforcement learning using a simulation.

[0015] Reinforcement learning here is machine learning that learns a policy, which is the behavioral rule of an agent that takes an action in an environment, based on a reward that represents an evaluation of the action. The state of the environment is also simply called the state. The environment here may include an agent. Therefore, the state here may include the state of the agent.

[0016] At each step in reinforcement learning, the learning device 100 determines an action based on a policy under the current state, simulates the determined action, and calculates the next state, which is the state at the next step. The learning device 100 also calculates a reward value (reward value) based on the obtained next state, and updates the policy based on the calculated reward value. The policy update here can be considered as policy generation. In other words, the learning device 100 can be considered as generating a policy based on past policies.

[0017] Hereinafter, a step in reinforcement learning will be referred to as a time step or simply as a step. Hereinafter, time will be represented by a time step. The reinforcement learning method used by the learning device 100 is not limited to a specific type of method. For example, the learning device 100 may learn control of the control object 910 based on a known reinforcement learning method such as Q-learning or SARSA.

[0018] For example, for each episode, the learning device 100 repeats the processing for each step until the episode ends. An episode here is a unit of time in reinforcement learning, and is a time interval from when an agent starts a series of actions until when the series ends. For example, the learning device 100 repeats the processing for each step in the episode until a predetermined condition for ending the episode is met.

[0019] Furthermore, if the learning device 100 determines that a predetermined condition for suspending a simulation is satisfied, it suspends the simulation even in the middle of the episode. The condition for suspending a simulation is also referred to as a suspension condition. Suspending a simulation can also be said to be suspending the episode being executed. This allows the learning device 100 to automatically suspend execution of the episode when the learning device 100 falls into a state where policy updating is not expected even if the currently executed episode is continued, or when the learning device 100 falls into a state where policy updating is expected to make little progress. Policy updating can be considered as policy improvement. Policy updating can be considered as progress in reinforcement learning.

[0020] The learning device 100 that has suspended the execution of an episode may go back a time step within the episode and resume the execution of the episode. Alternatively, the learning device 100 that has suspended the execution of an episode may start the execution of another episode. The learning device 100 can perform reinforcement learning more efficiently than when the execution of the episode is continued even when the state in which the policy update is not expected or when the state in which the policy update is expected to progress slowly is reached.

[0021] Furthermore, the learning device 100 stores the state at each of a plurality of time steps during the execution of an episode. When the learning device 100 suspends the execution of an episode, it selects one of the time steps for which the state is stored, returns to the selected time step, and resumes the execution of that episode. Specifically, the learning device 100 resumes processing for each time step from the state of the selected time step.

[0022] Here, if we consider the case where an episode is executed from the beginning each time, it is conceivable that the beginning of the episode is a part where learning has already progressed sufficiently, and policy updating will not proceed any further. The learning device 100 is expected to be able to perform reinforcement learning more efficiently by selecting a time step in the middle of an episode, not just the beginning, as the destination to go back to in the episode. The time step to go back to in the episode is also referred to as the reset destination. The reset destination candidates are also referred to as reset destination candidates. A reset destination candidate is, for example, an executed time step whose state is stored as a time step at which the simulation can be resumed within the episode that was being executed until the simulation was interrupted. Interrupting the execution of a simulation and changing the state in the simulation to the reset destination state is also referred to as resetting the simulation.

[0023] The control device 200 controls the control object 910 based on a policy obtained by learning by the learning device 100. The control object 910 is not limited to a specific one, and various control objects can be learned for controlling the control object 910 using reinforcement learning. For example, the control object 910 may be equipment such as a plant or a power plant, a system such as a production line in a factory, or a standalone device. Alternatively, the control object 910 may be a moving object such as an automobile, a railroad vehicle, an airplane, a ship, or a self-propelled mobile robot, or a transportation system such as a railway or air traffic control system. The control object 910 may be configured as a part of the control system 1 or may be configured external to the control system 1.

[0024] During learning, the learning device 100 is sufficient, and the control device 200 and the control object 910 may not be required. Furthermore, during control execution, the control device 200 and the control object 910 may be sufficient, and the learning device 100 may not be required. All of the learning device 100, the control device 200, and the control object 910, or a combination of two of them, may be configured integrally. For example, the learning device 100 and the control device 200 may be configured as a single device. Furthermore, the control device 200 and the control object 910 may be configured as a single device.

[0025] Fig. 2 is a diagram showing an example of the configuration of learning device 100. In the configuration shown in Fig. 2, learning device 100 includes a communication unit 110, a display unit 120, an operation input unit 130, a storage unit 180, and a processing unit 190. The processing unit 190 includes an action determination unit 191, a simulation unit 192, a measure generation unit 193, a reset determination unit 194, and a restart state determination unit 195.

[0026] The communication unit 110 communicates with other devices. For example, the communication unit 110 may receive various information for performing a simulation from the other devices. The communication unit 110 may also transmit a policy obtained by learning to the control device 200.

[0027] Display unit 120 has a display screen such as a liquid crystal panel or an LED (Light Emitting Diode) panel, and displays various images. For example, display unit 120 may display information related to the learning performed by learning device 100, such as the progress of the learning performed by learning device 100.

[0028] The operation input unit 130 includes input devices such as a keyboard and a mouse, and receives user operations. For example, the operation input unit 130 may receive a user operation to instruct the start of learning control for the control target 910. The storage unit 180 stores various data. For example, the storage unit 180 may store a snapshot of the state in the simulation. The storage unit 180 is configured using a storage device included in the learning device 100.

[0029] The processing unit 190 controls each unit of the learning device 100 to perform various processes. The functions of the processing unit 190 are performed, for example, by a CPU (Central Processing Unit) included in the learning device 100 reading and executing a program from the storage unit 180. The behavior decision unit 191 decides the behavior of the control object 910 in reinforcement learning. The behavior decision unit 191 decides the behavior of the control object 910 based on a policy. The behavior decision unit 191 is an example of a behavior decision means.

[0030] The simulation unit 192 simulates the environment. In particular, the simulation unit 192 simulates the action determined by the action determination unit 191 and calculates the next state, which is the state after the action. Here, the control object 910 in the simulation can be considered as an agent or a part thereof, and the control object 910 may be included in the environment that is the target of the simulation. The state of the control object 910 may be included in the state in the simulation. The simulation unit 192 corresponds to an example of simulation means. The simulation unit 192 may be configured as part of the learning device 100, or may be configured external to the learning device 100.

[0031] The policy generation unit 193 updates the policy based on the reward value. As described above, the reward value is a value indicating an evaluation for an action. A policy is a decision rule for an action. As described above, the policy update here can be considered as the generation of a policy. The policy generation unit 193 may update the policy based on an instantaneous reward value, or may update the policy based on a cumulative reward value. The instantaneous reward value is a value indicating an evaluation for an action at each step, calculated based on the state, action, and next state, or a part of these, for each step. The cumulative reward value is a value calculated by accumulating instantaneous reward values ​​for multiple steps. When calculating the cumulative reward value, the instantaneous reward value may be multiplied by a coefficient value such as a forgetting coefficient. Alternatively, the policy generation unit 193 may update the policy based on a value function value (also referred to as value). The value function value is a predicted value of the cumulative reward value, such as an expected value of the cumulative reward value at the end of an episode.

[0032] The instantaneous reward value, cumulative reward value, and value function value are all examples of reward values. The learning device 100 may use a reward value in which a larger value indicates a better evaluation, or a reward value in which a smaller value indicates a better evaluation. The following describes an example in which the learning device 100 uses a reward value in which a larger value indicates a better evaluation. The combination of the behavior decision unit 191 and the policy generation unit 193 is also referred to as an agent processing unit.

[0033] When the reset determination unit 194 determines that the interruption condition is met, it interrupts the simulation by the simulation unit 192. Specifically, the reset determination unit 194 interrupts the simulation being performed by the simulation unit 192. As described above, the interruption condition is a condition that is determined in advance as a condition for interrupting the simulation of the behavior of the control target 910. The reset determination unit 194 is an example of a reset determination means.

[0034] As described above for the learning device 100, the reset decision unit 194 can automatically interrupt the execution of an episode when the current episode reaches a state where policy updates are not expected to proceed even if the episode continues, or when the current episode reaches a state where policy updates are not expected to progress smoothly.

[0035] The reset decision unit 194 that has suspended the execution of an episode may go back a time step within the episode and restart the simulation by the simulation unit 192 from the state of the time step that was gone back. Alternatively, the reset decision unit 194 that has suspended the execution of an episode may select another episode and cause the simulation unit 192 to perform a simulation of the selected episode. This allows the learning device 100 to perform reinforcement learning more efficiently than if the execution of the episode is continued even when the learning device 100 falls into a state where policy updating is not expected or when policy updating is expected to make little progress.

[0036] The restart state determination unit 195 determines a reset destination when the simulation by the simulation unit 192 is interrupted. The simulation unit 192 restarts the simulation from the reset destination state determined by the restart state determination unit 195. The restart state determination unit 195 corresponds to an example of restart state determination means.

[0037] As described above, it is conceivable that policy updates will not necessarily progress at the beginning of an episode. As described above for the learning device 100, by allowing the resumption state determination unit 195 to select not only the beginning of an episode but also any time step in the middle as the destination to go back in the episode, it is expected that the learning device 100 will be able to perform reinforcement learning more efficiently.

[0038] When the reset determination unit 194 suspends the execution of an episode, the simulation unit 192 may resume the simulation from a predetermined state, such as the initial state of the episode. In this case, the learning device 100 may not be configured to include the resume state determination unit 195.

[0039] The reset determination unit 194 may determine whether to interrupt the simulation by the simulation unit 192 based on a change in the cumulative reward value. FIG. 3 is a diagram showing an example of an instantaneous reward value that is the basis for calculating the cumulative reward value. The horizontal axis of the graph in FIG. 3 indicates the time step. The vertical axis indicates the instantaneous reward value. In the example of FIG. 3, the instantaneous reward value can take both positive and negative values. The larger the instantaneous reward value (e.g., the larger the positive value and the magnitude), the better the evaluation indicated by the instantaneous reward value. The smaller the instantaneous reward value (e.g., the larger the negative value and the magnitude), the worse the evaluation indicated by the instantaneous reward value. In the example of FIG. 3, the instantaneous reward value is positive at time step t11, and the time steps following time step t11 are consecutive time steps in which the instantaneous reward value is negative.

[0040] Fig. 4 is a diagram showing an example of a cumulative reward value. The horizontal axis of the graph in Fig. 4 represents the time step. The vertical axis represents the cumulative reward value. Fig. 4 shows the cumulative reward value calculated by accumulating the instantaneous reward values ​​shown in Fig. 3 from the start of the episode. The cumulative reward value reaches a local maximum at time step t11, and from the time step after t11 onwards, the cumulative reward value continues to decrease.

[0041] In a state where the cumulative reward value is small, such as the state at time step t12, it is considered that an event has occurred that causes the evaluation indicated by the reward value to be poor. For example, if the control object 910 is a railway, it is considered that the intervals between trains have become short, causing a disruption to the train schedule.

[0042] If an event occurs that causes the evaluation indicated by the reward value to be poor, continuing to execute the episode as is will not improve the evaluation indicated by the reward value, and it is considered that policy update will progress more if the execution of the episode is restarted by going back a time step. For example, if the control object 910 is a railway and train intervals are short, causing a schedule disruption, it is considered that the evaluation indicated by the reward value will not improve until the short train intervals are resolved. It is considered that the poor evaluation indicated by the reward value will not progress in updating the policy by the policy generation unit 193.

[0043] In this case, rather than continuing to execute the episode as is, it is expected that policy updating will progress more if the execution of the episode is restarted by going back to a time step before the occurrence of an event that caused the evaluation indicated by the reward value to be poor, such as a time step before the train intervals became shorter. In the example of Figure 4, in the time step after time step t11, where the cumulative reward value continues to decrease, it is considered that an event that caused the evaluation indicated by the reward value to be poor has already occurred. In this case, it is expected that policy updating will progress more if the execution of the episode is restarted by going back to time step t11 or an earlier time step.

[0044] Therefore, the reset decision unit 194 may decide whether to suspend the simulation by the simulation unit 192 based on a change in the cumulative reward value. For example, when the reset decision unit 194 determines that the evaluation indicated by the cumulative reward value has worsened below a predetermined threshold, the reset decision unit 194 may decide to suspend the simulation by the simulation unit 192. In the example of Fig. 4, the reset decision unit 194 may compare the magnitude of the amount of decrease of the cumulative reward value from the maximum value at time step t11 with a threshold d11, and may decide to suspend the simulation by the simulation unit 192 at or after the end of time step t12 when the magnitude of the amount of decrease is greater than threshold d11.

[0045] In this case, the reset determination unit 194 may update the threshold value so that the extent of deterioration in evaluation indicated by the threshold value of the cumulative reward value increases as the simulation by the simulation unit 192 progresses. In the example of Fig. 4, the reset determination unit 194 may update the value of the threshold value d11 so that the value of the threshold value d11 increases as the time step progresses.

[0046] The starting point from which the reset determination unit 194 calculates the amount of decrease in the cumulative reward value is not limited to the maximum value of the cumulative reward value. For example, the reset determination unit 194 may calculate the amount of decrease from a cumulative reward value of 0 and compare it with a threshold. Alternatively, the reset determination unit 194 may calculate the amount of decrease from the maximum value of the cumulative reward value and compare it with a threshold.

[0047] Here, it is considered that in the early stages of an episode or in the early stages of reinforcement learning, learning has not progressed and the reward value does not become large in many cases. In this case, by having the reset decision unit 194 set the threshold to a relatively small value and suspending the execution of the episode earlier, it is expected that the action decision unit 191 will try various actions and will be able to find an action (or series of actions) that will increase the reward value at a relatively early stage.

[0048] Alternatively, the reset determination unit 194 may determine to interrupt the simulation by the simulation unit 192 when a deterioration in the evaluation indicated by the cumulative reward value continues for a predetermined number of time steps or more, based on an increase or decrease in the cumulative reward value. In the example of Fig. 4, if the number of time steps for interrupting the simulation by the simulation unit 192 is set to 7, the reset determination unit 194 may determine to interrupt the simulation by the simulation unit 192 at or after the end of time step t12 when the cumulative reward value has decreased seven times consecutively from time step t11 at which the cumulative reward value is maximized.

[0049] The reset determination unit 194 may determine whether to interrupt the simulation by the simulation unit 192 based on a combination of the amount of decrease in the cumulative reward value and the number of time steps in which the cumulative reward value has continuously decreased. For example, the reset determination unit 194 may determine to interrupt the simulation by the simulation unit 192 when the magnitude of the amount of decrease in the cumulative reward value becomes larger than a predetermined threshold and when the number of time steps in which the cumulative reward value has continuously decreased reaches a predetermined number or more.

[0050] The reset determination unit 194 may determine whether to suspend the simulation by the simulation unit 192 based on the value function value in addition to or instead of the cumulative reward value. The value function value is a predicted value of the cumulative reward value, and it is considered that there is a positive correlation between the value function value and the cumulative reward value. By having the reset determination unit 194 determine whether to suspend the simulation by the simulation unit 192 based on the value function value, the simulation by the simulation unit 192 will be suspended when an event occurs that would cause the evaluation indicated by the reward value to be poor, just as in the case where the determination is based on the cumulative reward value, and it is expected that the update of the policy will progress.

[0051] As in the case of the cumulative reward value described above, when reset determination unit 194 determines, based on an increase or decrease in the value function value, that the evaluation indicated by the value function value has deteriorated below a predetermined threshold, it may suspend the simulation by simulation unit 192. In this case, as in the case of the cumulative reward value described above, reset determination unit 194 may update the threshold so that the extent of deterioration in the evaluation indicated by the threshold of the value function value increases as the simulation by simulation unit 192 progresses.

[0052] Furthermore, as in the case of the cumulative reward value described above, reset determination unit 194 may be configured to determine, based on an increase or decrease in the value function value, that if a deterioration in the evaluation indicated by the value function value continues for a predetermined number of time steps or more, to suspend the simulation by simulation unit 192. As in the case of the cumulative reward value described above, reset determination unit 194 may be configured to determine whether or not to suspend the simulation by simulation unit 192, based on a combination of the amount of decrease in the value function value and the number of time steps in which the value function value has continuously decreased.

[0053] The reset determination unit 194 may interrupt the simulation by the simulation unit 192 if it determines that the current state (the state at the current time step) is similar to one or more states that are preset as states requiring interruption to a certain degree or more. For example, if the control object 910 is a railway, the storage unit 180 stores multiple patterns of states in which train intervals are short and disruption is occurring as states requiring interruption. The reset determination unit 194 then compares the current state with each of the states requiring interruption and determines whether the current state is similar to any of the states requiring interruption. If it determines that the current state is similar to any of the states requiring interruption, the reset determination unit 194 decides to interrupt the simulation by the simulation unit 192.

[0054] The method by which the reset determination unit 194 determines whether the current state and the state requiring interruption are similar is not limited to a specific method. For example, the reset determination unit 194 may calculate the feature quantities of each of the two states as vectors and calculate the similarity of the vectors, such as cosine similarity. The reset determination unit 194 may then compare the calculated similarity with a threshold and determine that the two states are similar if the similarity is equal to or greater than the threshold. Alternatively, a machine learning model that determines whether two states are similar may be prepared, and the reset determination unit 194 may use this machine learning model to determine whether the current state and the state requiring interruption are similar.

[0055] When learning device 100 collects states requiring interruption, it may present the states to the user and accept the user's designation of the state requiring interruption. For example, reset determination unit 194 may cause display unit 120 to display some state, such as the current state or a past state in the simulation. Then, reset determination unit 194 may accept a user operation via operation input unit 130 to instruct whether or not to register the displayed state as a state requiring interruption.

[0056] In this case, the combination of the reset determination unit 194 and the display unit 120 corresponds to an example of a state presenting means. The combination of the reset determination unit 194 and the operation input unit 130 corresponds to an example of a state designation accepting means. If the control object 910 is a railway, the display unit 120 may display the current train operation status in the simulation in the form of a diagram. Then, when the user determines that the displayed operation status indicates a situation requiring interruption, the user may perform a user operation indicating this on the operation input unit 130, and the reset determination unit 194 may detect that the user operation has been performed.

[0057] Learning device 100 is an example of an input / output device in that it includes reset determination unit 194, display unit 120, and operation input unit 130. Alternatively, the input / output device may be configured as a device separate from learning device 100. For example, a terminal device of learning device 100 may have a function to display the status according to instructions from learning device 100 and a function to accept a user operation specifying a status requiring interruption and notify learning device 100 of the status.

[0058] The restart state determination unit 195 may select, from among the reset destination candidates in the episode that was being executed until the simulation was interrupted, a reset destination candidate that corresponds to a time step with the fewest number of time steps between the time when the simulation was interrupted and the time when the change in evaluation, indicated by an increase or decrease in the cumulative reward value, has turned from improvement to deterioration, or a time step earlier than that. In other words, the restart state determination unit 195 may select a reset destination candidate that corresponds to a time step in which the evaluation is maximally good immediately before the interruption of the simulation, or a time step earlier than that.

[0059] 4, if the reset determination unit 194 interrupts the simulation by the simulation unit 192 at time step t12, time step t11 corresponds to the time step at which the cumulative reward value is maximized immediately before the simulation is interrupted (time step t12). The restart state determination unit 195 may select a reset destination candidate corresponding to time step t11 or an earlier time step.

[0060] In time steps after the time step in which the evaluation was most recently maximally good when the simulation was interrupted, the cumulative reward value is continuously decreasing, and it is thought that an event has occurred that is a factor in causing the evaluation indicated by the reward value to be bad. By having the restart state determination unit 195 select a reset destination candidate that corresponds to the time step in which the evaluation was most recently maximally good when the simulation was interrupted or an earlier time step, it is expected that the possibility of selecting a time step before the occurrence of an event that is a factor in causing the evaluation indicated by the reward value to be bad will increase.

[0061] Alternatively, the restart state determination unit 195 may select one of all reset destination candidates in the episode that was being executed until the simulation was interrupted as selection targets. In this way, if a decrease in the cumulative reward value does not necessarily indicate the occurrence of an event that causes the evaluation indicated by the reward value to be a bad evaluation, the restart state determination unit 195 may be able to select a later reset destination candidate (a reset destination candidate that is closer to the time the simulation was interrupted).

[0062] The restart state determination unit 195 may select one of the multiple reset destination candidates according to a probability distribution set for the multiple reset destination candidates. Here, when an event that causes the reward value to be evaluated poor occurs, from the perspective of returning to a time step before the event occurred, it is considered to return to as early a time step as possible within the episode (a time step closest to the beginning of the episode). On the other hand, from the perspective of reducing the number of learning repetitions early in the episode, it is considered to return to as late a time step as possible among the time steps already executed within the episode. Thus, there is a trade-off between improving the efficiency of reinforcement learning by avoiding a state in which an event that causes the reward value to be evaluated poor occurs and improving the efficiency of reinforcement learning by reducing the number of learning repetitions early in the episode.

[0063] Therefore, the restart state determination unit 195 probabilistically selects one of the multiple reset destination candidates. This makes it possible to avoid restarting the execution of the episode from the beginning every time the execution of the episode is interrupted. Furthermore, if an event that causes the evaluation indicated by the reward value to be poor has already occurred in the selected reset destination candidate, it is expected that by interrupting the episode one or more times, it will be possible to select a reset destination candidate that occurred before the event that causes the evaluation indicated by the reward value to be poor occurred.

[0064] The restart state determination unit 195 may select one of the multiple reset destination candidates according to a uniform probability distribution. This allows the restart state determination unit 195 to select a reset destination candidate in response to the control object 910 for which the approximate number of time steps to go back to reach a state before the occurrence of an event that causes the evaluation indicated by the reward value to be poor is unknown.

[0065] FIG. 5 is a diagram showing a first example of a probability distribution set for reset destination candidates. The horizontal axis of the graph in FIG. 5 represents time steps. The vertical axis represents probability. In the example of FIG. 5, the reset determination unit 194 suspends the simulation by the simulation unit 192 at a time step after time step t25, and five time steps from time steps t21 to t25 are reset destination candidates. The restart state determination unit 195 sets three time steps from time steps t21 to t23 out of these five reset destination candidates as reset destination candidates to be selected, and sets a uniform probability distribution for these reset destination candidates. In accordance with the set probability distribution, the restart state determination unit 195 selects each of the three reset destination candidates with a one-third probability.

[0066] Alternatively, the restart state determination unit 195 may set a probability distribution for the reset destination candidates so that the greater the deterioration in evaluation at the time the simulation is interrupted, as indicated by an increase or decrease in the cumulative reward value, the more likely it is to select a reset destination candidate with a larger number of time steps between the time the simulation is interrupted and the reset destination candidate, and select one of the reset destination candidates according to the set probability distribution.

[0067] Fig. 6 is a diagram showing a second example of a probability distribution set for reset destination candidates. The horizontal axis of the graph in Fig. 6 represents time steps. The vertical axis represents probability. In the example of Fig. 6, the reset determination unit 194 suspends the simulation by the simulation unit 192 at a time step after time step t25, and five time steps from time steps t21 to t25 are set as reset destination candidates. The resume state determination unit 195 sets three time steps from time steps t21 to t23 out of these five reset destination candidates as reset destination candidates to be selected, and sets probability distributions for these reset destination candidates.

[0068] 6 shows an example in which the magnitude of the decrease in the cumulative reward value when the simulation is interrupted is relatively small, and a higher probability is set for a reset destination candidate that is closer to the interruption time (i.e., a reset destination candidate with fewer time steps between it and the interruption time). The restart state determination unit 195 selects one of the three reset destination candidates in accordance with the set probability distribution. This makes it relatively easy for the restart state determination unit 195 to select a reset destination candidate that is closer to the interruption time (i.e., a reset destination candidate with fewer time steps between it and the interruption time).

[0069] FIG. 7 is a diagram showing a third example of a probability distribution set for reset destination candidates. The horizontal axis of the graph in FIG. 7 represents time steps. The vertical axis represents probability. In the example of FIG. 7, the reset determination unit 194 suspends the simulation by the simulation unit 192 at a time step after time step t25, and five time steps from time steps t21 to t25 are set as reset destination candidates. The resume state determination unit 195 sets three time steps from time steps t21 to t23 out of these five reset destination candidates as reset destination candidates to be selected, and sets probability distributions for these reset destination candidates.

[0070] 7 shows an example in which the magnitude of the decrease in the cumulative reward value when the simulation is interrupted is relatively large. In the example of FIG. 7, a higher probability is set for time step t21, which is a reset destination candidate that is relatively close to the time of interruption, than in the case of FIG. 6. Furthermore, a lower probability is set for time step t23, which is a reset destination candidate that is relatively close to the time of interruption, than in the case of FIG. 6. The restart state determination unit 195 selects one of the three reset destination candidates in accordance with the set probability distribution. This makes it relatively easy for the restart state determination unit 195 to select a reset destination candidate that is far from the time of interruption (i.e., a reset destination candidate that has many time steps between it and the time of interruption).

[0071] Here, it is considered particularly desirable to avoid a situation in which the cumulative reward value decreases suddenly. To increase the likelihood of avoiding a situation in which the cumulative reward value decreases suddenly, it is considered to increase the number of time steps going back to when the simulation was interrupted. On the other hand, it is considered that a situation in which the cumulative reward value decreases gradually is less desirable to avoid than a situation in which the cumulative reward value decreases suddenly. In this case, it is considered to relatively reduce the number of time steps going back to when the simulation was interrupted in order to reduce the repetition of learning in the early stages of an episode. It is expected that the learning device 100 can efficiently perform reinforcement learning by having the restart state determination unit 195 change the reset destination candidates that are easy to select depending on the magnitude of the increase or decrease in the cumulative reward value.

[0072] 8 is a diagram showing an example of data input / output in the learning device 100. In the example of Fig. 8, the action decision unit 191 decides on an action for the control object 910 based on a policy, and outputs the decided action to the simulation unit 192. The simulation unit 192 simulates the action decided by the action decision unit 191 to calculate a state (next state), and outputs the calculated state to the policy generation unit 193.

[0073] The policy generation unit 193 calculates a reward value based on the state acquired from the action determination unit 191, and updates the policy based on the calculated reward value. Depending on the reward value, the policy generation unit 193 may not update the policy (leave it as is). Furthermore, the policy generation unit 193 outputs the calculated reward value to the reset determination unit 194 and the restart state determination unit 195.

[0074] The reset decision unit 194 decides, based on the reward value, whether to suspend the simulation by the simulation unit 192. If the reset decision unit 194 decides to suspend the simulation, the restart state decision unit 195 decides a reset destination based on the reward value. Then, the reset decision unit 194 and the restart state decision unit 195 output an instruction to suspend the simulation and the reset destination to the simulation unit 192 and the measure generation unit 193.

[0075] Here, there are no particular limitations on the reward value used by the policy generator 193, the reward value used by the reset determiner 194, and the reward value used by the restart state determiner 195. The reward value used by the policy generator 193, the reward value used by the reset determiner 194, and the reward value used by the restart state determiner 195 may be any of an instantaneous reward value, a cumulative reward value, a value function value, or other reward values, or a combination of these.

[0076] Furthermore, the reward value used by the policy generator 193, the reward value used by the reset determiner 194, and the reward value used by the resume state determiner 195 may be the same or different. For example, the policy generator 193 may output an instantaneous reward value to the reset determiner 194, and the reset determiner 194 may calculate a cumulative reward value based on the instantaneous reward value.

[0077] When an interrupt instruction is received, the simulation unit 192 interrupts the simulation and returns the state of the simulation to the specified reset destination. Also, when an interrupt instruction is received, the policy generation unit 193 returns either the policy or the reward value, or both, to the values ​​at the specified reset destination. Note that the policy generation unit 193 may not change either the policy or the reward value when the simulation is interrupted. In this case, the reset determination unit 194 and the resume state determination unit 195 may not output an interrupt instruction and a reset destination to the policy generation unit 193.

[0078] 9 is a diagram showing an example of the procedure of processing performed by the learning device 100. In the processing shown in FIG. 9, the simulation unit 192 sets up a simulation (step S101). At the start of an episode, the simulation unit 192 sets the state of the episode specified in that episode. Next, the processing unit 190 executes one step of reinforcement learning (step S102). Then, the reset determination unit 194 determines whether an interruption condition is met (step S103).

[0079] If the reset determination unit 194 determines that the interruption condition is met (step S103: YES), the restart state determination unit 195 determines a reset destination (step S111). After step S111, the process returns to step S101. In this case, in step S101, the simulation unit 192 sets the state in the simulation to the reset destination state.

[0080] On the other hand, if the reset determination unit 194 determines that the interruption condition is not met in step S103 (step S103: NO), the processing unit 190 determines whether the episode end condition is met (step S121). If the processing unit 190 determines that the episode end condition is not met (step S121: NO), the processing returns to step S102.

[0081] On the other hand, if it is determined in step S103 that the termination condition for the episode is met (step S121: YES), the processing unit 190 determines whether the termination condition for reinforcement learning is met (step S131). If it is determined that the termination condition for reinforcement learning is not met (step S131: NO), the processing unit 190 selects the next episode (step S141). After step S141, the processing returns to step S101. In this case, the simulation unit 192 sets the state of the episode specified in the episode selected by the processing unit 190. On the other hand, if it is determined in step S131 that the termination condition for reinforcement learning is met (step S131: NO), the learning device 100 ends the processing of FIG. 9.

[0082] As described above, the policy generation unit 193 generates a policy, which is a decision rule for behavior, based on a reward value, which is a value indicating an evaluation of the behavior of the control object 910 at a time step going back within an episode representing a learning period. The behavior decision unit 191 decides the behavior of the control object based on the policy.

[0083] The learning device 100 is expected to be able to perform reinforcement learning more efficiently by going back in time steps within an episode. For example, the learning device 100 may be able to avoid the occurrence of a factor that would cause a poor evaluation indicated by a reward value by going back in time steps that are earlier than the time step in which that factor occurred. This is expected to enable the learning device 100 to more quickly generate a policy that will improve the evaluation of the entire episode.

[0084] Furthermore, the reset decision unit 194 suspends the simulation when it is determined that a predetermined condition for suspending the simulation of the behavior of the control object 910 is satisfied. The policy generation unit 193 generates a policy based on a reward value calculated based on the simulation of the behavior of the control object 910. The learning device 100 can perform reinforcement learning more efficiently than when the execution of an episode is continued even when the device falls into a state where policy updating is not expected or when policy updating is expected to proceed slowly.

[0085] Furthermore, the simulation unit 192 simulates the behavior of the control object 910 and calculates the state of the environment in which the control object behaves. The learning device 100 is expected to be able to perform reinforcement learning more efficiently by going back in time steps in the simulation.

[0086] Furthermore, the reset decision unit 194 decides whether to suspend the simulation based on a change in the cumulative reward value, which is the cumulative value of the instantaneous reward values. This allows the reset decision unit 194 to decide whether to suspend the simulation based not only on the state at one time step but also on the change in the state. In this respect, the learning device 100 is expected to be able to appropriately decide whether to suspend the simulation.

[0087] Furthermore, the reset decision unit 194 decides to suspend the simulation if it determines, based on the increase or decrease in the cumulative reward value, that the evaluation indicated by the cumulative reward value has deteriorated below a predetermined threshold. According to the learning device 100, it is possible to determine whether to suspend the simulation by the simple process of comparing the cumulative reward value with a threshold.

[0088] Furthermore, the reset determination unit 194 updates the threshold value of the cumulative reward value so that the extent of deterioration in the evaluation indicated by the threshold value increases as the simulation progresses. As described above, in the early stages of an episode or the early stages of reinforcement learning, it is conceivable that there are many cases in which learning is not progressing and the reward value does not increase. In this case, the reset determination unit 194 sets the threshold value to a relatively small value and interrupts the execution of the episode earlier, which is expected to encourage the action determination unit 191 to try various actions and find an action (or series of actions) that will increase the reward value at a relatively early stage. In this respect, the learning device 100 is expected to be able to perform reinforcement learning efficiently.

[0089] Furthermore, the reset decision unit 194 decides to suspend the simulation if the deterioration of the evaluation indicated by the cumulative reward value continues for a predetermined number of time steps or more, based on the increase or decrease of the cumulative reward value. According to the learning device 100, it is possible to determine whether to suspend the simulation by the simple process of counting the number of steps in which the cumulative reward value continuously decreases.

[0090] Furthermore, the reset decision unit 194 decides whether to interrupt the simulation based on a value function value, which is a predicted value of a cumulative reward value, which is a cumulative value of instantaneous reward values. This allows the reset decision unit 194 to decide whether to interrupt the simulation based not only on the state at one time step, but also on changes in the state. In this regard, the learning device 100 is expected to be able to appropriately decide whether to interrupt the simulation.

[0091] Furthermore, if the reset decision unit 194 determines, based on the increase or decrease in the value function value, that the evaluation indicated by the value function value has deteriorated below a predetermined threshold, it decides to suspend the simulation. The learning device 100 can determine whether to suspend the simulation by the simple process of comparing the value function value with a threshold.

[0092] Furthermore, the reset determination unit 194 updates the threshold value of the value function value so that the extent of deterioration in the evaluation indicated by the threshold value increases as the simulation progresses. As described above, in the early stages of an episode or the early stages of reinforcement learning, it is conceivable that there are many cases in which learning is not progressing and the reward value does not increase. In this case, by the reset determination unit 194 setting the threshold value to a relatively small value and suspending the execution of the episode earlier, it is expected that the action determination unit 191 will try various actions and will be able to find an action (or series of actions) that will increase the reward value at a relatively early stage. In this respect, the learning device 100 is expected to be able to perform reinforcement learning efficiently.

[0093] Furthermore, based on the increase or decrease in the value function value, reset decision unit 194 decides to suspend the simulation if the deterioration of the evaluation indicated by the value function value continues for a predetermined number of time steps or more. Learning device 100 can determine whether to suspend the simulation by the simple process of counting the number of steps in which the value function value continuously decreases.

[0094] Furthermore, the reset decision unit 194 decides to suspend the simulation if it determines that the current state of the environment in which the controlled object 910 acts, as calculated in the simulation, is similar to one or more states that are preset as states requiring suspension, to a certain degree or more. The designer of the learning device 100 only needs to prepare samples of states that require suspension of the simulation, and does not need to design rules for determining whether or not to suspend the simulation. In this respect, the learning device 100 reduces the burden on the designer.

[0095] Furthermore, the combination of the reset decision unit 194 and the display unit 120 presents the status to the user. The combination of the reset decision unit 194 and the operation input unit 130 accepts a user operation specifying a status requiring interruption. The learning device 100 can acquire a sample of a status requiring interruption of a simulation in response to a user's designation. In this respect, the learning device 100 reduces the burden on the designer of the learning device 100. Furthermore, the user can reflect their own judgment as to whether or not the simulation needs to be interrupted in the learning device 100.

[0096] Furthermore, the restart state determination unit 195 determines the state when the simulation is restarted. As described above, it is conceivable that the policy update will not necessarily progress at the beginning of an episode. As described above for the learning device 100, the restart state determination unit 195 can select not only the beginning of an episode but also an intermediate time step as the destination to go back in the episode, which is expected to enable the learning device 100 to perform reinforcement learning more efficiently.

[0097] In addition, the restart state determination unit 195 selects one of the reset destination candidates based on the probability distribution set for the reset destination candidates, which are executed time steps whose states are stored as time steps at which the simulation can be restarted within the episode that was being executed until the simulation was interrupted.

[0098] As described above, when an event that causes the reward value to be evaluated poor occurs, it is possible to return to as early a time step as possible within the episode (a time step closest to the beginning of the episode) from the perspective of returning to a time step before the event occurred. On the other hand, from the perspective of reducing the number of learning repetitions early in the episode, it is possible to return to as late a time step as possible among the time steps already executed within the episode. In this way, there is a trade-off between improving the efficiency of reinforcement learning by avoiding a state in which an event that causes the reward value to be evaluated poor occurs and improving the efficiency of reinforcement learning by reducing the number of learning repetitions early in the episode.

[0099] Therefore, by having the restart state determination unit 195 probabilistically select one of the multiple reset destination candidates, it is possible to avoid restarting the execution of the episode from the beginning every time the execution of the episode is interrupted. Furthermore, if an event that causes the evaluation indicated by the reward value to be poor has already occurred in the selected reset destination candidate, it is expected that by repeating the interruption of the episode one or more times, it will be possible to select a reset destination candidate that occurred before the event that causes the evaluation indicated by the reward value to be poor occurred.

[0100] The reset determination unit 194 selects one of the multiple reset destination candidates according to a uniform probability distribution. The learning device 100 can select a reset destination candidate in response to the control object 910 for which the approximate number of time steps to go back to reach a state before the occurrence of an event that caused the evaluation indicated by the reward value to be poor is unknown.

[0101] In addition, the reset decision unit 194 sets a probability distribution for multiple reset destination candidates so that the greater the deterioration in evaluation at the time the simulation is interrupted, as indicated by an increase or decrease in the cumulative reward value, the more likely it is to select a reset destination candidate with a larger number of time steps between the time the simulation is interrupted and the reset destination candidate, and selects one of the reset destination candidates according to the set probability distribution.

[0102] As described above, it is considered particularly desirable to avoid a situation in which the cumulative reward value decreases suddenly. To increase the likelihood of avoiding a situation in which the cumulative reward value decreases suddenly, it is considered to take a large number of time steps back to when the simulation was interrupted. On the other hand, it is considered that a situation in which the cumulative reward value decreases gradually is less desirable to avoid than a situation in which the cumulative reward value decreases suddenly. In this case, it is considered to relatively reduce the number of time steps back to when the simulation was interrupted in order to reduce the repetition of learning in the early stages of an episode. It is expected that the learning device 100 can efficiently perform reinforcement learning by having the restart state determination unit 195 change the reset destination candidates that are easy to select depending on the magnitude of the increase or decrease in the cumulative reward value.

[0103] In addition, the reset decision unit 194 selects, from among the reset destination candidates, a reset destination candidate that corresponds to the time step with the fewest number of time steps between the time when the simulation was interrupted and the time step in which the change in evaluation indicated by the increase or decrease in the cumulative reward value has turned from improvement to deterioration, or a time step earlier than that.

[0104] In time steps after the time step in which the evaluation was most recently maximally good when the simulation was interrupted, the cumulative reward value is continuously decreasing, and it is thought that an event has occurred that is a factor in causing the evaluation indicated by the reward value to be bad. By having the restart state determination unit 195 select a reset destination candidate that corresponds to the time step in which the evaluation was most recently maximally good when the simulation was interrupted or an earlier time step, it is expected that the possibility of selecting a time step before the occurrence of an event that is a factor in causing the evaluation indicated by the reward value to be bad will increase.

[0105] FIG. 10 is a diagram illustrating another example of the configuration of a learning device according to some embodiments of the present disclosure. In the configuration illustrated in FIG. 10 , the learning device 610 includes a policy generation unit 611 and an action determination unit 612. In this configuration, the policy generation unit 611 generates a policy, which is a decision rule for behavior, based on a reward value, which is a value indicating an evaluation of the behavior of the control target at a time step going back within an episode representing a learning period. The action determination unit 612 determines the behavior of the control target based on the policy. The policy generation unit 611 corresponds to an example of a policy generation means. The action determination unit 612 corresponds to an example of a behavior determination means.

[0106] The learning device 610 is expected to be able to perform reinforcement learning more efficiently by going back in time steps within an episode. For example, the learning device 610 may be able to avoid the occurrence of a factor that would cause a poor evaluation indicated by a reward value by going back in time steps that are earlier than the time step in which that factor occurred. This is expected to enable the learning device 610 to more quickly generate a policy that will improve the evaluation of the entire episode.

[0107] The measure generator 611 can be realized, for example, by using the functions of the measure generator 193 in Fig. 1. The action determiner 612 can be realized, for example, by using the functions of the action determiner 191 in Fig. 1.

[0108] 11 is a diagram illustrating another example of a configuration of a control system according to some embodiments of the present disclosure. In the configuration illustrated in FIG. 11, a control system 620 includes a strategy generator 622 and an action determiner 623.

[0109] With this configuration, the policy generation unit 622 generates a policy, which is a decision rule for behavior, based on a reward value, which is a value indicating an evaluation of the behavior of the controlled object at a time step going back within the episode representing the learning period. The behavior decision unit 623 decides the behavior of the controlled object based on the policy. The policy generation unit 622 is an example of a policy generation means. The behavior decision unit 623 is an example of a behavior decision means.

[0110] The control system 620 is expected to be able to perform reinforcement learning more efficiently by going back in time steps within an episode. For example, the control system 620 may be able to avoid the occurrence of a factor that would cause a poor evaluation indicated by a reward value by going back in time steps that are earlier than the time step in which that factor occurred. This is expected to enable the control system 620 to more quickly generate a policy that will improve the evaluation of the entire episode.

[0111] The measure generator 622 can be realized, for example, by using the functions of the measure generator 193 in Fig. 1. The action determiner 623 can be realized, for example, by using the functions of the action determiner 191 in Fig. 1.

[0112] Fig. 12 is a diagram illustrating an example of the configuration of an input / output device according to some embodiments of the present disclosure. In the configuration illustrated in Fig. 12, the input / output device 630 includes a state presenting unit 631 and a state designation receiving unit 632. In this configuration, the state presenting unit 631 presents to the user the state of the environment in which the controlled object is behaving. The state designation receiving unit 632 receives a user operation that specifies a state that requires the simulator that simulates the environment to be suspended. The state presenting unit 631 corresponds to an example of a state presenting means. The state designation receiving unit 632 corresponds to an example of a state designation receiving means.

[0113] The input / output device 630 can acquire a sample of a state in which a simulation needs to be interrupted in response to a user's designation. The input / output device 630 reduces the burden on the designer of the input / output device 630. Furthermore, the user can reflect their own judgment as to whether or not a simulation needs to be interrupted in the execution of the simulation. The state presentation unit 631 can be realized, for example, using the functions of the reset determination unit 194 and the display unit 120 in FIG. 1 . The state designation reception unit 632 can be realized, for example, using the functions of the reset determination unit 194 and the operation input unit 130 in FIG. 1 .

[0114] 13 is a diagram illustrating an example of a processing procedure in a learning method according to some embodiments of the present disclosure. The learning method illustrated in FIG. 13 includes generating a policy (step S611) and determining an action (step S612).

[0115] In generating a policy (step S611), the computer generates a policy, which is a decision rule for the behavior, based on a reward value, which is a value indicating an evaluation of the behavior of the controlled object at a time step going back in time within the episode representing the learning period. In determining an action (step S612), the computer determines the behavior of the controlled object based on the policy.

[0116] The learning method shown in Fig. 13 is expected to enable more efficient reinforcement learning by going back in time steps within an episode. For example, the learning method shown in Fig. 13 may be able to avoid the occurrence of a factor that would cause a poor evaluation indicated by a reward value by going back in time steps that are earlier than the time step in which that factor occurred. As a result, the learning method shown in Fig. 13 is expected to be able to more quickly generate a policy that improves the evaluation of the entire episode.

[0117] 14 is a diagram illustrating an example of a computer configuration according to at least one embodiment of the present disclosure. In the configuration illustrated in FIG. 14, a computer 700 includes a CPU 710, a main memory device 720, an auxiliary memory device 730, an interface 740, and a non-volatile recording medium 750.

[0118] One or more of the learning device 100, control device 200, learning device 610, learning device 621, control device 626, and input / output device 630, or a portion thereof, may be implemented in the computer 700. In this case, the operation of each of the above-described processing units is stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program. The CPU 710 also allocates storage areas in the main storage device 720 corresponding to each of the above-described storage units in accordance with the program. Communication between each device and other devices is performed by the interface 740, which has a communication function and performs communication under the control of the CPU 710. The interface 740 also has a port for the nonvolatile storage medium 750, and reads information from the nonvolatile storage medium 750 and writes information to the nonvolatile storage medium 750.

[0119] When learning device 100 is implemented in computer 700, the operations of processing unit 190 and each of its units are stored in the form of a program in auxiliary storage device 730. CPU 710 reads the program from auxiliary storage device 730, loads it into main storage device 720, and executes the above-described processing in accordance with the program.

[0120] Furthermore, the CPU 710 allocates a storage area for the storage unit 180 in the main storage device 720 in accordance with the program. Communication with other devices by the communication unit 110 is performed by the interface 740 having a communication function and operating under the control of the CPU 710. Display of images by the display unit 120 is performed by the interface 740 having a display device and displaying various images under the control of the CPU 710. Reception of user operations by the operation input unit 130 is performed by the interface 740 having an input device and receiving user operations under the control of the CPU 710.

[0121] When the control device 200 is implemented in the computer 700, its operation is stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0122] Furthermore, the CPU 710 allocates a storage area in the main storage device 720 for the control device 200 to perform processing in accordance with the program. Communication between the control device 200 and other devices is performed by the interface 740, which has a communication function and operates under the control of the CPU 710. Interaction between the control device 200 and a user is performed by the interface 740, which has an input device and an output device, presenting information to the user via the output device under the control of the CPU 710 and accepting user operations via the input device.

[0123] When the learning device 610 is implemented in the computer 700, the operations of the policy generation unit 611 and the action decision unit 612 are stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0124] Furthermore, CPU 710 allocates a storage area in main memory 720 for learning device 610 to perform processing in accordance with the program. Communication between learning device 610 and other devices is performed by interface 740, which has a communication function and operates under the control of CPU 710. Interaction between learning device 610 and a user is performed by interface 740, which has an input device and an output device, presenting information to the user via the output device under the control of CPU 710 and accepting user operations via the input device.

[0125] When the learning device 621 is implemented in the computer 700, the operations of the policy generation unit 622 and the action decision unit 623 are stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0126] Furthermore, the CPU 710 allocates a storage area in the main storage device 720 for the learning device 621 to perform processing in accordance with the program. Communication between the learning device 621 and other devices is performed by the interface 740, which has a communication function and operates under the control of the CPU 710. Interaction between the learning device 621 and a user is performed by the interface 740, which has an input device and an output device, presenting information to the user via the output device under the control of the CPU 710 and accepting user operations via the input device.

[0127] When the control device 626 is implemented in the computer 700, its operation is stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0128] Furthermore, the CPU 710 allocates a storage area in the main storage device 720 for the control device 626 to perform processing in accordance with the program. Communication between the control device 626 and other devices is performed by the interface 740, which has a communication function and operates under the control of the CPU 710. Interaction between the control device 626 and a user is performed by the interface 740, which has an input device and an output device, presenting information to the user via the output device under the control of the CPU 710 and accepting user operations via the input device.

[0129] When the input / output device 630 is implemented in the computer 700, the operations of the state presentation unit 631 and the state designation reception unit 632 are stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0130] Furthermore, the CPU 710, in accordance with the program, allocates a storage area in the main storage device 720 for the input / output device 630 to perform processing. Communication between the input / output device 630 and other devices is performed by the interface 740, which has a communication function and operates under the control of the CPU 710. Interaction between the input / output device 630 and a user is performed by the interface 740, which has an input device and an output device, presenting information to the user via the output device under the control of the CPU 710 and accepting user operations via the input device.

[0131] One or more of the above-described programs may be recorded on nonvolatile recording medium 750. In this case, interface 740 may read the programs from nonvolatile recording medium 750. Then, CPU 710 may directly execute the programs read by interface 740, or may temporarily store the programs in main storage device 720 or auxiliary storage device 730 and then execute them.

[0132] Note that a program for executing all or part of the processing performed by learning device 100, control device 200, learning device 610, learning device 621, control device 626, and input / output device 630 may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be loaded into a computer system and executed to perform the processing of each unit. Note that the term "computer system" here includes hardware such as an operating system (OS) and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, read-only memories (ROMs), and compact disc read-only memories (CD-ROMs), as well as storage devices such as hard disks built into computer systems. The program may be designed to implement part of the aforementioned functions, or may be capable of implementing the aforementioned functions in combination with a program already recorded on the computer system.

[0133] The embodiments of this disclosure have been described in detail above with reference to the drawings, but the specific configuration is not limited to this embodiment, and includes designs within the scope that do not deviate from the gist of this disclosure.

[0134] Some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0135] (Supplementary Note 1) A learning device comprising: a policy generation means for generating a policy, which is a decision rule for the behavior of a controlled object, based on a reward value, which is a value indicating an evaluation of the behavior of the controlled object at a time step going back within an episode representing a learning period; and a behavior decision means for deciding the behavior of the controlled object based on the policy.

[0136] (Supplementary Note 2) The learning device according to Supplementary Note 1 further comprises a reset decision means for suspending the simulation when it is determined that a predetermined condition for suspending the simulation of the behavior of the controlled object is satisfied, and the policy generation means generates the policy based on the reward value calculated based on the simulation.

[0137] (Supplementary Note 3) The learning device according to Supplementary Note 2, further comprising: a simulation means for performing the simulation to calculate a state of an environment in which the controlled object behaves.

[0138] (Supplementary Note 4) The learning device according to Supplementary Note 2 or Supplementary Note 3, wherein the reset decision means decides whether to interrupt the simulation based on a change in an accumulated reward value, which is an accumulated value of instantaneous reward values.

[0139] (Supplementary Note 5) The learning device according to Supplementary Note 4, wherein the reset decision means decides to interrupt the simulation when it is determined, based on an increase or decrease in the cumulative reward value, that an evaluation indicated by the cumulative reward value has deteriorated below a predetermined threshold.

[0140] (Supplementary Note 6) The learning device according to Supplementary Note 5, wherein the reset determination means updates the threshold value of the cumulative reward value so that a range of deterioration in evaluation indicated by the threshold value increases as the simulation progresses.

[0141] (Supplementary Note 7) The learning device according to any one of Supplementary Notes 4 to 6, wherein the reset decision means decides to interrupt the simulation when a deterioration in the evaluation indicated by the cumulative reward value continues for a predetermined number of time steps or more, based on an increase or decrease in the cumulative reward value.

[0142] (Supplementary Note 8) The learning device according to any one of Supplementary Notes 2 to 7, wherein the reset decision means decides whether to interrupt the simulation based on a value function value that is a predicted value of a cumulative reward value that is a cumulative value of instantaneous reward values.

[0143] (Supplementary Note 9) The learning device according to Supplementary Note 8, wherein the reset decision means decides to interrupt the simulation when it is determined, based on an increase or decrease in the value function value, that an evaluation indicated by the value function value has deteriorated below a predetermined threshold.

[0144] (Supplementary Note 10) The learning device according to Supplementary Note 9, wherein the reset decision means updates the threshold value of the value function value so that a range of deterioration in evaluation indicated by the threshold value increases as the simulation progresses.

[0145] (Supplementary Note 11) The learning device according to Supplementary Note 9 or Supplementary Note 10, wherein the reset decision means decides to interrupt the simulation when a deterioration in the evaluation indicated by the value function value continues for a predetermined number of time steps or more, based on an increase or decrease in the value function value.

[0146] (Supplementary Note 12) The learning device described in any one of Supplementary Notes 2 to 11, wherein the reset decision means decides to interrupt the simulation when it determines that the current state of the environment in which the controlled object behaves, as calculated in the simulation, is similar to one or more states that are pre-set as states requiring interruption to a certain extent or more.

[0147] (Supplementary Note 13) The learning device according to Supplementary Note 12, further comprising: a state presenting means for presenting the state of the environment to a user; and a state designation receiving means for receiving a user operation for designating the state requiring interruption.

[0148] (Supplementary Note 14) The learning device according to any one of Supplementary Notes 2 to 13, further comprising: a restart state determination means for determining a state of an environment in which the controlled object will act when restarting an interrupted simulation.

[0149] (Supplementary Note 15) The learning device according to Supplementary Note 14, wherein the restart state determination means selects one of the reset destination candidates based on a probability distribution set for the reset destination candidates, which are executed time steps whose states are stored as time steps at which the simulation can be restarted within the episode that was being executed until the simulation was interrupted.

[0150] (Supplementary Note 16) The learning device according to Supplementary Note 15, wherein the restart state determination means selects one of a plurality of reset destination candidates in accordance with a uniform probability distribution.

[0151] (Appendix 17) The learning device described in Appendix 15, wherein the restart state determination means sets a probability distribution for a plurality of reset destination candidates such that the greater the deterioration in evaluation at the time the simulation is interrupted, which is indicated by an increase or decrease in the cumulative reward value, which is the cumulative value of the instantaneous reward values, the more likely it is to select a reset destination candidate with a larger number of time steps between the time the simulation is interrupted and the reset destination candidate, and selects one of the reset destination candidates according to the set probability distribution.

[0152] (Supplementary Note 18) The learning device described in any one of Supplementary Notes 14 to 17, wherein the restart state determination means selects a reset destination candidate corresponding to a time step with the fewest number of time steps between the time when the simulation was interrupted and the time step where the change in evaluation indicated by the increase or decrease in the cumulative reward value, which is the cumulative value of the instantaneous reward values, has turned from improvement to deterioration, from among the reset destination candidates which are executed time steps whose state is stored as a time step at which the simulation can be restarted within the episode that was being executed until the simulation was interrupted, or a time step earlier than that.

[0153] (Supplementary Note 19) A control device that controls a control target based on a policy obtained using the learning device according to any one of Supplementary Notes 1 to 18.

[0154] (Supplementary Note 20) A control system comprising a learning device and a control device, wherein the learning device comprises: a policy generation means for generating a policy, which is a decision rule for the behavior of a controlled object, based on a reward value, which is a value indicating an evaluation of the behavior of the controlled object at a time step going back within an episode representing a learning period; and an action decision means for deciding the behavior of the controlled object based on the policy, and the control device controls the controlled object based on the policy obtained using the learning device.

[0155] (Supplementary Note 21) An input / output device comprising: a state presenting means for presenting to a user the state of an environment in which a controlled object acts; and a state designation receiving means for receiving a user operation for designating a state in which a simulator that simulates the environment needs to be interrupted.

[0156] (Supplementary Note 22) A learning method including: a computer generating a policy, which is a decision rule for the behavior of a controlled object, based on a reward value, which is a value indicating an evaluation of the behavior of the controlled object at a time step back within an episode representing a learning period; and determining the behavior of the controlled object based on the policy.

[0157] (Appendix 23) A recording medium storing a program for causing a computer to execute the following: generating a policy, which is a decision rule for the behavior of a controlled object, based on a reward value, which is a value indicating an evaluation of the behavior of the controlled object at a time step going back within an episode representing a learning period; and determining the behavior of the controlled object based on the policy.

[0158] This application claims priority based on Japanese Patent Application No. 2023-103973, filed on June 26, 2023, the disclosure of which is incorporated herein by reference in its entirety.

[0159] The present disclosure may be applied to a learning device, a control system, an input / output device, a learning method, and a recording medium.

[0160] 1, 620 Control system 100, 610, 621 Learning device 110 Communication unit 120 Display unit 130 Operation input unit 180 Storage unit 190 Processing unit 191, 612, 623 Action decision unit 192 Simulation unit 193, 611, 622 Policy generation unit 194 Reset decision unit 195 Restart state decision unit 200, 624 Control device 630 Input / output device 631 State presentation unit 632 State designation reception unit 910 Control target

Claims

1. A policy generation means that generates a policy, which is a rule for determining the behavior, based on a reward value, which is a value indicating an evaluation of the controlled behavior in a state where time steps are reversed within an episode representing the learning period. Action determination means for determining the action of the controlled object based on the policy, A learning device equipped with the following features.

2. A reset determination means determines that a predetermined condition for interrupting the simulation of the controlled object's behavior has been met, and interrupts the simulation. Furthermore, The policy generation means generates the policy based on the reward value calculated based on the simulation. The learning device according to claim 1.

3. The reset determination means updates the threshold value, which is the cumulative value of instantaneous reward values, so that the magnitude of deterioration in evaluation indicated by the threshold value increases as the simulation progresses. If it determines that the evaluation indicated by the cumulative reward value has deteriorated below a predetermined threshold, it decides to interrupt the simulation. The learning device according to claim 2.

4. The reset determination means updates the threshold value, which is the predicted value of the value function value (which is the cumulative value of instantaneous reward values), so that the magnitude of deterioration in evaluation indicated by the threshold value increases as the simulation progresses. If it determines that the evaluation indicated by the value function value has deteriorated below a predetermined threshold, it decides to interrupt the simulation. The learning device according to claim 2.

5. Restart state determination means for determining the state of the environment in which the controlled object acts when resuming an interrupted simulation. The learning device according to claim 2, further comprising:

6. The restart state determination means selects one of the reset destination candidates based on a probability distribution set for a reset destination candidate, which is an executed time step in the episode that was running up to the time the simulation was interrupted, in which the state is stored as a time step in which the simulation can be restarted. The learning device according to claim 5.

7. The restart state determination means selects one of several reset destination candidates according to a uniformly distributed probability distribution. The learning device according to claim 6.

8. Equipped with a learning device and a control device, The learning device is A policy generation means that generates a policy, which is a rule for determining the behavior, based on a reward value, which is a value indicating an evaluation of the controlled behavior in a state where time steps are reversed within an episode representing the learning period. Action determination means for determining the action of the controlled object based on the policy, Equipped with, The control device performs control on the controlled object based on the strategy obtained using the learning device. Control system.

9. Computers Based on the reward value, which is a value indicating the evaluation of the controlled behavior in the time step backward within the episode representing the learning period, a policy, which is the rule for determining the behavior, is generated. The action of the controlled object is determined based on the policy. A learning method that includes this.

10. On the computer, Based on the reward value, which is a value indicating the evaluation of the controlled behavior in the time step backward within the episode representing the learning period, a policy, which is a rule for determining the behavior, is generated. The action of the controlled object is determined based on the policy, A program to execute.