Learning method, learning device, learning program, control method, control device, and control program

Abstract

To provide a learning method for preventing learning from converging to a local optimum.SOLUTION: A learning method according to the embodiment, wherein a probability distribution indicating a probability density or a distribution of the probability of selecting a plurality of actions is obtained based on current observation data and a control parameter, a first action among the plurality of actions is selected based on the probability distribution, and a control object is made to execute the first action, and next observation data observed after a control object performs a first action and a first reward is received, the probability density or probability of the first action is obtained from a probability distribution, the first reward is corrected from the probability density or probability, and a control parameter is updated based on the current observation data, the first action, the next observation data, and the corrected first reward. The smaller the probability density or probability, the larger the first reward is corrected.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] Embodiments of the present invention relate to a learning method, a learning device, a learning program, a control method, a control device, and a control program. [Background technology]

[0002] A known machine learning method (also called a reinforcement learning method) is one in which an action of the controlled object 10 is selected, the controlled object 10 is made to perform the selected action, and the action of the controlled object 10 corresponding to the performed action is evaluated, and this process is repeated. For example, reinforcement learning methods are applied to the control of the actions of mobile objects such as automobiles, robots, and drones, or movable objects such as robotic arms. In reinforcement learning methods, the learning may converge to a local optimum in the early stages of learning. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Special Publication No. 2021-513128 [Overview of the project] [Problems that the invention aims to solve]

[0004] The object of the present invention is to provide a learning method, a learning device, and a learning program that prevent learning from converging to a local optimum, and a control method, a control device, and a control program that use the learned parameters. [Means for solving the problem]

[0005] The learning method according to this embodiment is The first step is to receive the current observational data, A second step involves determining a probability distribution that shows the probability density or probability distribution of multiple actions being selected, based on the current observation data and control parameters. A third step of selecting a first action from among the multiple actions based on the aforementioned probability distribution, A fourth step involves causing the controlled object to perform the first action, A fifth step in which the controlled object receives the next observed data and a first reward after performing the first action, A sixth step of determining the probability density or probability of the first action from the probability distribution and the first action, A seventh step of correcting the first reward from the probability density or the probability, The system comprises an eighth step of updating the control parameters based on the current observation data, the first action, the next observation data, and the corrected first reward. The seventh step corrects the first reward such that the smaller the probability density or the probability, the larger the first reward. [Brief explanation of the drawing]

[0006] [Figure 1] A block diagram illustrating an example of a learning device and a controlled object that implement the learning method according to the first embodiment. [Figure 2] A diagram illustrating an example of a robotic arm as a controlled object. [Figure 3] A flowchart illustrating an example of a control parameter learning method performed by the learning device according to the first embodiment. [Figure 4] A diagram illustrating an example of a neural network that calculates parameters for a probability distribution. [Figure 5] A diagram illustrating an example of the relationship between the probability density of an action being selected and the second reward. [Figure 6] A diagram illustrating an example of a neural network that calculates an estimated cumulative reward. [Figure 7] A diagram illustrating the problems with common reinforcement learning methods. [Figure 8] A diagram illustrating an example of the effects of the control parameter learning method according to the first embodiment. [Figure 9] A block diagram illustrating an example of a control device and a controlled object that implement the control method according to the first embodiment. [Figure 10]A flowchart illustrating an example of a control method performed by the control device according to the first embodiment. [Figure 11] A flowchart illustrating an example of a control parameter learning method performed by the learning device according to the second embodiment. [Figure 12] A diagram illustrating an example of a neural network that calculates probability distributions. [Figure 13] A diagram illustrating an example of the relationship between the probability of an action being selected and the second reward. [Figure 14] A flowchart illustrating an example of a control parameter learning method performed by the learning device according to the third embodiment. [Figure 15] A diagram illustrating an example of a neural network that calculates behavioral value. [Figure 16] A flowchart illustrating an example of a learning method according to the fourth embodiment. [Modes for carrying out the invention]

[0007] The embodiments will be described below with reference to the drawings. The following description exemplifies devices and methods for realizing the technical concept of the embodiments, and the technical concept of the embodiments is not limited to the structure, shape, arrangement, material, etc. of the components described below. Modifications that a person skilled in the art can easily conceive of are naturally included within the scope of disclosure. In order to make the description clearer, the size, thickness, planar dimensions, or shape of each element may be schematically represented in the drawings with modifications from the actual embodiments. Multiple drawings may include elements with different dimensional relationships or ratios. In multiple drawings, the same reference numeral may be used for corresponding elements to omit redundant descriptions. Some elements may be given multiple names, but these examples of names are merely illustrative and do not preclude the use of other names for these elements. Similarly, elements that do not have multiple names may also be given other names. In the following description, "connection" may include not only direct connections but also connections via other elements.

[0008] This embodiment will be described in detail below with reference to the drawings.

[0009] [First Embodiment] Figure 1 is a block diagram illustrating an example of a learning device 30 and a controlled object 10 that implement the learning method according to the first embodiment.

[0010] The learning device 30 is electrically connected to the controlled object 10. The electrical connection between the learning device 30 and the controlled object 10 may be wired or wireless. If the controlled object 10 is a mobile device such as an automobile, robot, or drone, the learning device 30 and the controlled object 10 may be connected wirelessly.

[0011] The learning device 30 uses a reinforcement learning method that probabilistically controls behavior and optimizes control parameters. Examples of reinforcement learning methods include actor-critic methods and SARSA methods, and any of these methods can be used in the first embodiment. Here, as an example, the first embodiment using the actor-critic method will be described.

[0012] The learning device 30 receives various information from the controlled object 10 regarding the state of the controlled object 10 itself and the state of the environment surrounding the controlled object 10. Using this information, it selects an action that the controlled object 10 should take and causes the controlled object 10 to execute the selected action. The learning device 30 learns the control parameters so that the controlled object 10 can select an appropriate action according to the state of the controlled object 10 itself and the state of the environment surrounding the controlled object 10. To evaluate whether an appropriate action has been selected, the learning device 30 receives a reward for performing the action. The reward indicates whether the action was appropriate or not. The learning device 30 learns the action selection of the controlled object 10 so that if the reward to be obtained in the future by performing the action is large, that action will be selected more often in that situation, and if the reward is small, that action will be selected less often in that situation, and reflects the learning results in the values ​​of the control parameters.

[0013] The actions of the controlled object 10 include both continuous and discrete actions. The first embodiment is applicable to either type of action. Here, as an example, the first embodiment for controlling the controlled object 10, which includes continuous actions, is described.

[0014] Figure 2 illustrates an example of a robot arm 52 as the controlled object 10. Figure 2(a) shows a top view of the robot arm 52, and Figure 2(b) shows a side view of the robot arm 52. The robot arm 52 is equipped with an end effector 54 at its tip. The end effector 54 grasps the item 56. An example of continuous action control of the robot arm 52 is the vertical displacement, lateral displacement, and longitudinal displacement of the end effector 54 during the period of one action execution. Any multiple of these actions may be combined to form a single action.

[0015] Returning to the explanation of Figure 1, the learning device 30 includes a processor 32 such as a CPU, a non-volatile storage device 34 that stores programs executed by the processor 32 and various data, a main memory 36 that stores programs and data read from the storage device 34 or various data generated during the learning process, a transmission unit 38 that supplies drive signals and control signals to the controlled object 10, a receiving unit 40 that receives observation data from the controlled object 10, an input unit 42 such as a keyboard, and a display unit 44 such as an LCD. The learning device 30 is also called a computer. The program stored in the storage device 34 includes a program for executing the control parameter learning method. This program is read from the storage device 34 and loaded into the main memory 36.

[0016] The learning device 30 may be directly connected to the controlled object 10 and implemented as a standalone device that performs learning about one controlled object 10, or it may be located on a network and configured to perform learning about multiple controlled objects 10 via the network.

[0017] The controlled object 10 includes a processor 12 such as a CPU, a non-volatile storage device 14 that stores programs executed by the processor 12 and various data, a main memory 16 that stores programs and data read from the storage device 14 or various data generated during the learning process, sensors 18 that detect the state of the controlled object 10 itself and the state of the environment surrounding the controlled object 10, a drive unit 20 that drives each movable part of the controlled object 10, a transmission unit 22 that supplies observation data regarding the state of the controlled object 10 itself and the state of the environment surrounding the controlled object 10 to the learning device 30, and a receiver unit 24 that receives drive signals and control signals from the learning device 30. The sensors 18 include rotation sensors attached to the movable parts that detect the state of the movable parts, and sensors that detect the surrounding conditions such as acceleration sensors, gyro sensors, infrared sensors, and cameras.

[0018] The learning device 30 and the controlled object 10 may be configured to operate synchronously. The action selection cycle of machine learning is fixed, and the controlled object 10 may send observation data to the learning device 30 at each action selection cycle, thereby matching the learning cycle of the learning device 30 with the action selection cycle. Alternatively, the controlled object 10 may send observation data to the learning device 30 after an action has been performed and before the next action is performed. Or, the transmission unit 22 may send observation data to the learning device 30 continuously or at very short intervals (shorter intervals than the action selection cycle).

[0019] The controlled object 10 is not limited to the robot arm 52; it can be anything, and the first embodiment is applicable to any controlled object 10 that performs a continuous action. Furthermore, the controlled object 10 may be a real machine, or it may be a simulator that performs the same actions as a real machine.

[0020] Figure 3 is a flowchart illustrating an example of a control parameter learning method performed by the learning device 30 according to the first embodiment.

[0021] The control parameter learning method includes step S102 of receiving current observation data, step S104 of calculating a probability distribution π1(a) based on the current observation data and the control parameter Θ, step S106 of selecting an action a ,

[0022] , , ,

[0025] , , , ,

[0026] , , ,

[0024] , , ,

[0023] , based on the probability distribution π1(a), step S108 of causing the control target 10 to execute the action a t step S110 of receiving the next observation data and the first reward r1, step S112 of obtaining the probability density π1(a t from which the action a t is selected, step S114 of calculating the second reward r21 from the probability density π1(a t ), step S116 of updating the value of the control parameter Θ based on the current observation data, the action a t ), the next observation data, the first reward r1, and the second reward r21, and step S118 of determining whether to end the learning.

[0022] Details of each step will be described below.

[0023] (Step S102 of receiving current observation data) The learning device 30 receives the current observation data O t .

[0024] The observation data O t is a set of values representing the state of the control target 10 and the environment surrounding it. The values representing the state of the control target 10 and the environment surrounding it are, for example, values indicating the position and orientation of the control target 10, attribute information such as the position and size of objects existing around the control target 10, the presence or absence and existence probability of objects at each position around the control target 10, or values acquired by the sensor 18. The learning device 30 acquires these values from the control target 10 and the environment surrounding it.

[0025] The observation data may include the history of actions executed in the past.

[0026] ​​The controlled object 10 and its surrounding environment may be the actual controlled object 10 and its surrounding actual environment, or they may be a simulated controlled object 10 and its surrounding simulated environment that mimics them.

[0027] (Step S104: Calculating the probability distribution) The learning device 30 uses the current observation data O t Based on the value of the control parameter Θ, the probability distribution π1(a) is calculated. The probability distribution π1(a) is the distribution of the probability density in which the action with value a is selected. As the probability distribution, known probability distributions such as the normal distribution, beta distribution, and truncated normal distribution can be used. Specifically, the learning device 30 uses the current observation data O t The probability distribution π1(a) is calculated by determining the set of parameters α of the probability distribution π1(a) based on the value of the control parameter Θ. For example, if a normal distribution is used as the probability distribution, its parameters are the mean and the variance.

[0028] The learning device 30 may use a neural network to calculate the parameters of the probability distribution. Figure 4 is a diagram illustrating an example of a neural network that calculates the parameters of the probability distribution. For the sake of explanation, Figure 4 shows how the current observed data contains four values ​​O1, O2, O3, and O4, and three parameters α1, α2, and α3 are calculated, but the number of values ​​in the observed data and the number of parameters are not limited to these numbers. The control parameter Θ is a set of variables that change the input and output characteristics of the neural network that calculates the parameters α1, α2, and α3 of the probability distribution. The control parameter Θ includes the weights of the neural network, etc.

[0029] Convolutional neural networks, recurrent neural networks, softmax functions, and other methods may be used in combination to calculate the parameters of the probability distribution.

[0030] The parameters of the probability distribution may be calculated using a combination of methods, such as normalizing the input and output or adding randomness to the input and output characteristics. The value of the control parameter Θ is stored in the main memory 36.

[0031] (Step S106: Selecting an action) The learning device 30 performs action a based on the probability distribution π1(a). t Select. Specifically, the learning device 30 probabilistically extracts an action according to the probability distribution π1(a), and this extracted action is called action a t By doing so, action a t Select this option.

[0032] (Step S108: Causing the controlled object to perform an action) The learning device 30 performs action a t The controlled object 10 is instructed to perform the action corresponding to the action a selected by the control parameter learning method according to the first embodiment. t The corresponding action is performed. Alternatively, the current action a performed by the control parameter learning method according to the first embodiment is executed. t The learning device 30 may be configured by the user of the learning device 30 so that the controlled object 10 performs the corresponding action.

[0033] (Step S110: Receive the next observation data and the first reward) The learning device 30 uses the following observational data O t+1 And you receive the first reward, r1.

[0034] Next observation data O t+1 This is observational data representing the state of the controlled object 10 and its surrounding environment after the controlled object 10 has performed the action corresponding to the selected action. The state of the controlled object 10 and its surrounding environment corresponding to the next observational data may be the state of the controlled object 10 and its surrounding environment corresponding to the current observational data at a point in time one action execution cycle later than the present. That is, the current observational data O t If we consider this as a set of values ​​representing the state of the controlled object 10 and its surrounding environment at time t=t0, then the following observation data O t+1 This may be a set of values ​​representing the state of the controlled object 10 and its surrounding environment at time t=t0+1.

[0035] The first reward r1 is the reward obtained when the controlled object 10 performs an action corresponding to the action selected by the control parameter learning method according to the first embodiment. The first reward r1 indicates whether the performed action was appropriate or not. The learning device 30 learns the action selection of the controlled object 10 so that if the reward to be obtained in the future by performing the action is large, that action will be selected more often in that situation, and if the reward is small, that action will be selected less often in that situation. When the robot arm 52 is controlled so that the end effector 54 approaches the item 56, the first reward r1 is large when the end effector 54 is close to the item 56, and small otherwise. The value of the first reward r1 may be a value given by the controlled object 10 or its surrounding environment, or it may be a value given by the user of the learning device 30 according to the quality of the action and its results. The first reward r1 is given by the current observation data O t The time t=t0 corresponds to the next observation data O t+1 The reward may also be obtained during the period between the time t=t0+1 corresponding to that time.

[0036] (Step S112 to calculate probability density) The learning device 30 executes step S112 in parallel with step S108. The learning device 30 uses the probability distribution π1(a) calculated in step S104 and the action a selected in step S106. t Therefore, action a t The probability density of selection is π1(a t ) is sought. Specifically, the learning device 30 determines action a t The value of the probability distribution π1(a) in, i.e., a=a t The value of the probability distribution π1(a) in action a t The probability density of selection is π1(a t By doing so, action a t The probability density of selection is π1(a t )

[0037] (Step S114 to calculate the second reward) The learning device 30 calculates the probability density π1(a) obtained in step S112.t The second reward r21 is calculated from ). Figure 5 shows action a t The probability density of selection is π1(a t This figure illustrates an example of the relationship between the probability density π1(a) and the second reward r21 calculated in step S114. The learning device 30 calculates the probability density π1(a) t The second reward r21 is calculated such that the smaller ) is, the larger the second reward r21 becomes. The learning device 30 may calculate the second reward r21 using a lookup table that has been set in advance so that the desired input / output relationship can be obtained, or the probability density π1(a t The second reward r21 may be calculated using a function like the following, where the smaller ) is, the larger the second reward r21 becomes. The lookup table or function is stored in main memory 36.

[0038] r²¹ = -β·logπ¹(a t ) Equation 1 In Equation 1, β is any positive constant.

[0039] (Step S116: Update control parameters) After the completion of steps S110 and S114, the learning device 30 processes the current observation data O received in step S102. t , action a selected in step S106 t Based on the next observation data Ot+1 received in step S110, the first reward r1, and the second reward r21 calculated in step S114, the value of the control parameter Θ is updated.

[0040] The learning device 30 can update the control parameters in a similar manner to how the parameters used to calculate the probability distribution in known actor-critic methods are updated, by using the sum of the first reward r1 and the second reward r21 as the reward in known actor-critic methods.

[0041] Specifically, the learning device 30 estimates the cumulative reward V obtained from the present to the future. t The selected current action a tBased on the difference between the estimated cumulative reward R obtained from the present to the future as a result of the execution of the function, the value of the control parameter Θ is updated as follows:

[0042] Θ = Θ + η·∇ Θ logπ1(a t )·(RV t ) Equation 2 In equation 2, ∇ Θ logπ1(a t ) is behavior a t The probability density of selection is π1(a t This is the gradient of the logarithm of ) with respect to the control parameter Θ, where η is the learning rate.

[0043] ∇ Θ logπ1(a t ) is behavior a t This corresponds to an update direction of the control parameter Θ that increases the probability density of the selected action a. t The estimated cumulative reward R obtained from the present to the future as a result of the execution of is the estimated cumulative reward V obtained from the present to the future. t If it is greater than, the learning device 30 performs action a t The control parameter Θ is updated so that the probability density of selecting action a increases. Conversely, action a t The estimated cumulative reward R obtained from the present to the future as a result of the execution of is the estimated cumulative reward V obtained from the present to the future. t If it is smaller than, the learning device 30 performs action a t The control parameter Θ is updated so that the probability density of selection decreases.

[0044] Similar to the process by which the learning device 30 calculates a set of parameters α for the probability distribution based on the observed data and the value of the control parameter Θ, it also calculates an estimated value V of the cumulative reward to be obtained from the present to the future. t current observation data O t and the parameter Θ of the state value function vIt is calculated based on the value of . Figure 6 is a diagram illustrating an example of a neural network that calculates an estimated cumulative reward. The state value function is an estimate of the cumulative reward that can be obtained from the time the observation data was obtained to the future, based on the observed data (i.e., with the observed data as input). t This is a function that calculates (outputs) the state value function parameter Θ. v This parameter determines the input-output characteristics of the state value function. Current observed data O t Based on this, the estimated value calculated by this state value function is V t And the following observation data O t+1 Based on this, the estimated value calculated by this state value function is V t+1 The learning device 30, similar to calculating the set of parameters α of the probability distribution, uses a neural network to estimate the cumulative reward V that can be obtained from the present to the future. t You may calculate this.

[0045] The learning device 30 estimates the cumulative reward R to be obtained from the present to the future as a result of performing the selected action, which is the sum of the first reward r1 and the second reward r21, and estimates the reward V to be obtained from the next state to the future. t+1 Based on the coefficient γ, the following formula is used to calculate it.

[0046] R = (r1 + r21) + γ·V t+1 formula 3 The coefficient γ is also called the discount rate.

[0047] The learning device 30 estimates the cumulative reward V that can be obtained from the present to the future. t Similar to the calculation process, V is an estimated value of the reward that can be obtained over time from the following state. t+1 The next observation data O t+1 and the parameter Θ of the state value function v It is calculated based on the value of [the value].

[0048] The learning device 30 controls the parameter Θ of the state value function. v The value is also the learning rate η v Update it as follows:

[0049] Θ V = Θ V -η·∇ ΘV (R - V t ) 2 Equation 4 As described above, the control parameter learning method according to the first embodiment includes: a step S102 of receiving current observation data; a step S104 of calculating a probability distribution π1(a) based on the current observation data and the control parameter Θ; a step S106 of selecting an action a based on the probability distribution π1(a); a step S108 of causing the control target 10 to execute the action a; a step S110 of receiving the next observation data and the first reward r1; a step S112 of obtaining the probability density π1(a) of the action a being selected from the probability distribution π1(a); a step S114 of calculating a second reward r21 from the probability density π1(a); and a step S116 of updating the value of the control parameter Θ based on the current observation data, the action a, the next observation data, the first reward r1, and the second reward r21. These steps are performed for each cycle of action execution to perform learning of the control parameter in action control of a robot, a mobile body, or the like. t and the next observation data, the first reward r1, and the second reward r21 to update the value of the control parameter Θ. These steps are performed for each cycle of action execution to perform learning of the control parameter in action control of a robot, a mobile body, or the like.

[0050] Effect of the First Embodiment FIG. 7 is a diagram for explaining the problems of a general reinforcement learning method. FIG. 8 is a diagram for explaining an example of the effect in the control parameter learning method according to the first embodiment.

[0051] Generally, in reinforcement learning, when the estimated value R of the cumulative reward obtained from the present to the future due to the execution of a certain action a is relatively large, the control parameter is updated so that the probability density of the action a being selected becomes large. Conversely, when the estimated value R of the cumulative reward obtained from the present to the future due to the execution of a certain action a is relatively small, the action a t is updated so that the control parameter is such that the probability density of the action a t being selected becomes large. Conversely, when the estimated value R of the cumulative reward obtained from the present to the future due to the execution of a certain action a t is relatively small, the action a t ​​​​​​​​​​​​The control parameter Θ is updated to reduce the probability density of the action being selected. However, the estimated cumulative reward obtained from the present to the future as a result of performing a certain action is an estimate based on information obtained up to a certain point in time. Therefore, in the early stages of learning, when there is little information already obtained, the estimate for a certain action may be unfairly underestimated. In such cases, as shown in Figure 7, general reinforcement learning methods calculate an unfairly low probability density for the selection of that action. Furthermore, it becomes difficult to obtain information about the reward obtained as a result of performing that action, and it becomes difficult to correct the estimated cumulative reward obtained as a result of performing that action. As a result, learning converges to a local optimum, leading to inaccurate learning.

[0052] In contrast, according to the learning method of the first embodiment, even if the probability density of selecting a certain action is calculated to be unreasonably small in the initial stage of learning, as shown in Figure 8, a second reward due to the small probability density of selecting that action is added to the first reward, thereby correcting the unreasonably small probability density of selecting that action. As a result, the learning does not converge to a local optimum and inaccurate learning is avoided.

[0053] Furthermore, according to the first embodiment, since it is not necessary to calculate the entropy gradient of the probability distribution, the control parameters can be accurately learned based on the probability distribution even for control targets with probability distributions for which it is difficult to calculate the entropy gradient.

[0054] Furthermore, according to the first embodiment, since there is no need to significantly change the control parameter update method compared to known reinforcement learning methods, it is easy to avoid inaccurate learning in various known reinforcement learning methods.

[0055] Next, a control device, a control method, and a program that use the control parameters learned by the learning method according to the first embodiment will be described.

[0056] Figure 9 is a block diagram illustrating an example of a control device 60 and a controlled object 10 that execute the control method according to the first embodiment. The control device 60 comprises a processor 62, a storage device 64, a main memory 66, a transmission unit 68, a reception unit 70, an input unit 72, and a display unit 74. The processor 62, storage device 64, main memory 66, transmission unit 68, reception unit 70, input unit 72, and display unit 74 are almost identical to the processor 32, storage device 34, main memory 36, transmission unit 38, reception unit 40, input unit 42, and display unit 44 of the learning device 30. The difference is that the storage device 34 stores the learning program, while the storage device 64 stores the control program.

[0057] Figure 10 is a flowchart illustrating an example of a control method executed by the control device 60 according to the first embodiment. The control device 60 implements the control method by executing a control program stored in the storage device 64.

[0058] The control method involves steps S202 to receive the current observation data, step S204 to calculate the probability distribution π1(a) based on the current observation data and the control parameter Θ, and taking action a based on the probability distribution π1(a). t Step S206 to select and the controlled object 10 to perform action a t It consists of step S208, which causes the operation to be performed, and step S210, which determines whether or not to terminate the control.

[0059] In step S202, the control device 60 processes the current observation data O t The data is received. Step S202 may be the same as step S102, which receives the current observation data in the control parameter learning method.

[0060] In step S204, the control device 60 processes the current observation data O t Based on the values ​​of the control parameters Θ learned by the control parameter learning method according to the first embodiment, a probability distribution π1(a) is calculated. Step S204 may be the same as step S104, which calculates the probability distribution in the control parameter learning method.

[0061] In step S206, the control device 60 selects an action a based on the probability distribution π(a). Step S206 may be the same as step S106 for selecting an action in the control parameter learning method. Alternatively, in step S206, the control device 60 may select an action a by taking the average value of the action a under the probability distribution π1(a) as the action a t Or, in step S206, the control device 60 may select an action a by taking the most frequent value of the action a under the probability distribution π1(a) as the action a t as the action a t Or, in step S206, the control device 60 may select an action a by taking the most frequent value of the action a under the probability distribution π1(a) as the action a t as the action a t

[0062] [Second Embodiment] As the first embodiment, a learning device, a learning method, a learning program, and a control device, a control method, and a control program using the learned control parameters in an actor-critic method when the control target 10 executes continuous actions have been described. The second embodiment relates to a learning device, a learning method, a learning program, and a control device, a control method, and a control program using the learned control parameters in an actor-critic method when the control target 10 executes discrete actions. Since the learning device according to the second embodiment is the same as the learning device 30 according to the first embodiment, and the control device according to the second embodiment is the same as the control device 60 according to the first embodiment, the illustration of the learning device and the control device according to the second embodiment is omitted.

[0063] When the control target 10 is a robot arm 52 (FIG. 2), examples of discrete action control include "turn the arm 52 to the right", "turn the arm 52 to the left", "fix the arm 52" shown in FIG. 2(a), and "move the end effector 54 to the back side", "move the end effector 54 to the front side", "raise the end effector 54", "lower the end effector 54", "close the end effector 54 and grasp the item 56", "open the end effector 54 and release the item 56" shown in FIG. 2(b). Any combination of these multiple actions may be used as one action.​

[0064] Figure 11 is a flowchart illustrating an example of a control parameter learning method performed by the learning device according to the second embodiment.

[0065] The control parameter learning method involves steps S302 to receive the current observation data, step S304 to calculate the probability distribution π2(a) based on the current observation data and the control parameter Θ, and then taking action a based on the probability distribution π2(a). t Step S306 to select and the controlled object 10 to perform action a t Step S308 to execute, step S310 to receive the next observation data and the first reward r1, and the probability distribution π2(a) and action a t Action a t The probability that is selected is π2(a t Step S312 to find the probability π²(a t Step S314 calculates the second reward r22 from ), and the current observation data, action a t The process consists of step S316, which updates the value of the control parameter Θ based on the next observation data, the first reward r1, and the second reward r22, and step S318, which determines whether or not to terminate the learning process.

[0066] The following explains each step in detail.

[0067] (Step S302: Receiving current observation data) The learning device 30 uses the current observation data O t The step of receiving the current observation data, S302, may be the same as the step of receiving the current observation data, S102, according to the first embodiment.

[0068] (Step S304: Calculating the probability distribution) The learning device 30 uses the current observation data O t Based on the values ​​of the control parameter Θ, the probability distribution π²(a) is calculated. The probability distribution π²(a) is calculated for each action a=a performed on the controlled object 10. a a b This is the probability distribution of which of the following is selected.

[0069] The learning device 30 may calculate the probability distribution π²(a) using a neural network. Figure 12 is a diagram illustrating an example of a neural network that calculates the probability distribution. For the sake of explanation, Figure 12 shows that the current observation data contains four values ​​O1, O2, O3, and O4, and three actions a a a b、 a c The probability for each of π2(a a ), π2(a b ), π2(a c This shows how a probability distribution consisting of ) is calculated, but the number of values ​​and actions included in the observed data are not limited to these numbers. The control parameter Θ is a set of variables that change the input and output characteristics of the neural network that calculates the probability distribution. The control parameter Θ includes the weights of the neural network, etc.

[0070] Convolutional neural networks, recurrent neural networks, softmax functions, and other methods may be used in combination to calculate the probability distribution.

[0071] To calculate the probability distribution, a combination of methods such as normalizing the input and output, or adding randomness to the input and output characteristics, may be used.

[0072] (Step S306: Selecting an action) The learning device 30 performs action a based on the probability distribution π²(a). t Select. Specifically, the learning device 30 probabilistically extracts an action according to the probability distribution π2(a), and this extracted action is called action a t By doing so, action a t Select this option.

[0073] (Step S308: Cause the controlled object 10 to perform an action) The learning device 30 performs action a t The controlled object 10 is instructed to perform the action corresponding to the action a selected by the control parameter learning method according to the second embodiment. tThe corresponding action is executed. Alternatively, the current action a executed by the control parameter learning method according to the second embodiment is executed. t The learning device 30 may be configured by the user of the learning device 30 so that the controlled object 10 performs the corresponding action.

[0074] (Step S310: Receive the next observation data and the first reward) The learning device 30 uses the following observational data O t+1 The first reward r1 is received. Step S310, which receives the next observation data and the first reward, may be the same as step S110, which receives the next observation data and the first reward, according to the first embodiment.

[0075] (Step S312 to calculate probability) The learning device 30 executes step S312 in parallel with step S308. The learning device 30 uses the probability distribution π2(a) calculated in step S304 and the action a selected in step S306. t Therefore, action a t The probability that is selected is π2(a t ) is sought. Specifically, the learning device 30 determines action a t The value of the probability distribution π²(a) in, i.e., a = a t The value of the probability distribution π2(a) in action a t The probability that is selected is π2(a t By doing so, action a t The probability that is selected is π2(a t )

[0076] (Step S314 to calculate the second reward) The learning device 30 calculates the probability π²(a) obtained in step S312. t The second reward r22 is calculated from ). Figure 13 shows action a t The probability that is selected is π2(a t This is a diagram illustrating an example of the relationship between the probability π²(a) and the second reward r²². The learning device 30 calculates the probability π²(a) tThe second reward r22 is calculated such that the smaller ) is, the larger the second reward r22 becomes. The learning device 30 may calculate the second reward r22 using a lookup table that has been set in advance so that the desired input / output relationship can be obtained, or it may use probability π2(a t The second reward r22 can also be calculated using a function like the following equation, where the smaller ) is, the larger the second reward r22 becomes.

[0077] r²² = -β·logπ²(a t ) Equation 5 In equation 5, β is any positive constant.

[0078] (Step S316: Update control parameters) After the completion of steps S310 and S314, the learning device 30 receives the current observation data O in step S302. t , action a selected in step S306 t , the next observation data O received in step S310 t+1 Based on the first reward r1 and the second reward r22 calculated in step S314, the value of the control parameter Θ is updated.

[0079] The learning device 30 can update the control parameters in a similar manner to how the parameters used to calculate the probability distribution in known actor-critic methods are updated, by using the sum of the first reward r1 and the second reward r22 as the reward in known actor-critic methods.

[0080] The learning device 30 estimates the cumulative reward V that can be obtained from the present to the future. t The selected current action a t Based on the difference between the estimated cumulative reward R obtained from the present to the future as a result of the execution of the function, the value of the control parameter Θ is updated as follows:

[0081] Θ = Θ + η·∇ Θ logπ2(a t )·(RV t ) Equation 6 In equation 6, ∇Θlogπ2(at ) is behavior a t The probability that is selected is π2(a t The gradient of the logarithm of ) with respect to the control parameter Θ, where η is the learning rate.

[0082] ∇ Θ logπ2(a t ) is behavior a t This corresponds to the update direction of the control parameter Θ that increases the probability of selecting action a. t The estimated cumulative reward R obtained from the present to the future as a result of the execution of is the estimated cumulative reward V obtained from the present to the future. t If it is greater than, the learning device 30 performs action a t The control parameter Θ is updated so that the probability of action a is selected increases. Conversely, action a t The estimated cumulative reward R obtained from the present to the future as a result of the execution of is the estimated cumulative reward V obtained from the present to the future. t If it is smaller than, the learning device 30 performs action a t Update the parameter Θ so that the probability of selecting is reduced.

[0083] Similar to the process by which the learning device 30 calculates the probability distribution π²(a) based on the observed data and the value of the control parameter Θ, it also calculates an estimated value V of the cumulative reward to be obtained from the present to the future. t current observation data O t , and the parameter Θ of the state value function v The calculation is based on the value of . The learning device 30 uses a neural network, similar to the calculation of the probability distribution π²(a), to estimate the cumulative reward V that can be obtained from the present to the future. t You may calculate this.

[0084] The learning device 30 estimates the cumulative reward R to be obtained from the present to the future as a result of performing the selected action, using the sum of the first reward r1 and the second reward r22, and the estimated reward V to be obtained from the next state to the future. t+1 Based on this, the calculation is performed as follows:

[0085] R = (r1 + r22) + γ·V t+1 formula 7 In Equation 7, γ is a coefficient (also called the discount rate).

[0086] The learning device 30 estimates the cumulative reward V that can be obtained from the present to the future. t Similar to the calculation process, V is an estimated value of the reward that can be obtained in the future from the following state. t+1 The next observation data O t+1 and the parameter Θ of the state value function v It is calculated based on the value of [the value].

[0087] The learning device 30 controls the parameter Θ of the state value function. v The value of is also updated as shown in the following equation.

[0088] Θ V =Θ V -η v ·∇ ΘV (RV t ) 2 formula 8 In equation 8, η v This is the coefficient (also called the learning rate).

[0089] As explained above, the control parameter learning method according to the second embodiment includes the steps of receiving current observation data (S302), calculating the probability distribution π2(a) based on the current observation data and the control parameter Θ (S304), and taking action a based on the probability distribution π2(a). t Step S306 to select, action a to the controlled object 10 t Step S308 to execute, step S310 to receive the next observation data and the first reward r1, probability distribution π2(a) and action a t Action a t The probability that is selected is π2(a t Step S312 to find the probability π²(a t Step S314 calculates the second reward r22 from ) and the current observation data and action a tThen, step S316 is performed at each action execution cycle to update the value of the control parameter Θ based on the next observation data, the first reward r1, and the second reward r22, thereby learning the control parameters for the action control of robots and mobile objects.

[0090] According to the learning method of the second embodiment, even when the controlled object 10 performs discrete actions, if the probability of a certain action being selected is calculated to be unreasonably small in the initial stage of learning, a second reward due to the small probability of that action being selected is added to the first reward, thereby correcting the unreasonably small probability of that action being selected. As a result, inaccurate learning is avoided.

[0091] Furthermore, in the second embodiment, since it is not necessary to calculate the entropy gradient of the probability distribution, the control parameters can be accurately learned based on the probability distribution even for control targets with probability distributions for which calculating the entropy gradient is difficult.

[0092] Furthermore, since the second embodiment does not require significant changes to the control parameter update method compared to known reinforcement learning methods, it is easy to avoid inaccurate learning in various known reinforcement learning methods.

[0093] The control device, control method, and program that use the control parameters learned by the learning method according to the second embodiment are configured in the same way as the control device, control method, and program according to the first embodiment. The processing flow of the control method according to the second embodiment is the same as in Figure 10.

[0094] The control device 60 receives the current observation data O t Receive current observation data O t The step of receiving the data may be the same as step S302 in the control parameter learning method, which involves receiving the current observation data.

[0095] The control device 60 receives the current observation data O tBased on the values ​​of the control parameters Θ learned by the control parameter learning method, a probability distribution π2(a) is calculated. The step of calculating the probability distribution π2(a) may be the same as the step S304 in the control parameter learning method for calculating the probability distribution.

[0096] The control device 60 performs action a based on the probability distribution π²(a). t Select this. Action a t The step of selecting the action may be the same as the step of selecting the action in the control parameter learning method S306. Alternatively, the control device 60 may set the mean of action a under the probability distribution π2(a) to action a t By doing so, action a t Alternatively, the control device 60 may select the mode of action a under the probability distribution π2(a) as action a t By doing so, action a t You may choose this option.

[0097] [Third Embodiment] The first and second embodiments describe a learning device, learning method, and learning program that use the actor-critic method as an example of a reinforcement learning method, and a control device, control method, and control program that use the learned control parameters. The third embodiment describes a learning device, learning method, and learning program that use the SARSA method as an example of a reinforcement learning method, and a control device, control method, and control program that use the learned control parameters. Similar to the actor-critic method, the SARSA method is applicable to both controlled objects that perform continuous actions and controlled objects that perform discrete actions.

[0098] The learning device according to the third embodiment is the same as the learning device 30 according to the first embodiment, and the control device according to the third embodiment is the same as the control device 60 according to the first embodiment; therefore, the illustration of the learning device and control device according to the third embodiment is omitted.

[0099] Figure 14 is a flowchart illustrating an example of a control parameter learning method performed by the learning device according to the third embodiment.

[0100] The control parameter learning method involves: step S402 receiving the current observation data; step S404 calculating a probability distribution π1(a) or π2(a) based on the current observation data and the control parameter Θ; and taking action a based on the probability distribution π1(a) or π2(a). t In step S406, select action a to the controlled object 10. t Step S408 to execute, step S410 to receive the next observation data and the first reward r1, probability distribution π1(a) or π2(a) and action a t Action a t The probability density of selection is π1(a t ) or probability π²(a t Step S412 to find the probability density π1(a t Calculate the second reward r21 from ) or the probability π2(a t Step S414 calculates the second reward r22 from the current observation data and action a t The process consists of step S416, which updates the value of the control parameter Θ based on the next observation data, the first reward r1, and the second reward r21 or r22, and step S418, which determines whether or not to terminate the learning process.

[0101] When the controlled object 10 performs a continuous action, the probability distribution π1(a) and the probability density π1(a) are given. t ), the second reward r21 is calculated. When the controlled object 10 performs discrete actions, the probability distribution π2(a), probability π2(a t ), the second reward r22 is calculated.

[0102] The following explains each step in detail.

[0103] (Step S402: Receiving current observation data) The learning device 30 uses the current observation data O t The step of receiving the current observation data, S402, may be the same as the step of receiving the current observation data, S102, according to the first embodiment, or the step of receiving the current observation data, S302, according to the second embodiment.

[0104] (Step S404: Calculating the probability distribution) The learning device 30 uses the current observation data O t Based on the value of the control parameter Θ, the learning device calculates a probability distribution π1(a) or π2(a). The learning device calculates the probability distribution π1(a) when the controlled object 10 performs continuous actions, and calculates the probability distribution π2(a) when the controlled object 10 performs discrete actions.

[0105] Specifically, the learning device 30 first processes the current observation data O t Based on the value of the control parameter Θ, the action value Q(a) for action a is calculated. The action value Q(a) represents an estimate of the cumulative reward that will be obtained from the present to the future as a result of performing action a.

[0106] The learning device 30 may calculate the action value Q(a) using a neural network. Figure 15 illustrates an example of a neural network that calculates the action value Q(a) when the controlled object 10 performs discrete actions. For the sake of explanation, Figure 15 shows that the current observation data contains four values ​​O1, O2, O3, and O4, and three action values ​​Q(a) a ), Q(a b ), Q(a c This shows how the action value is calculated, but the number of values ​​in the observed data and the number of action values ​​are not limited to these numbers. The control parameter Θ is a set of variables that change the input and output characteristics of the neural network that calculates the action value. The control parameter Θ includes the weights of the neural network, etc.

[0107] Convolutional neural networks, recurrent neural networks, softmax functions, and other methods may be used in combination to calculate action values.

[0108] The calculation of behavioral value may involve a combination of methods, such as normalizing inputs and outputs or adding randomness to input and output characteristics.

[0109] Figure 15 shows an example of calculating action value using a neural network when controlling discrete actions. For calculating action value using a neural network when controlling continuous actions, the Normalized Advantage Functions method (GU Shixiang et al., “Continuous Deep Q-Learning with Model-based Acceleration”, International conference on Machine Learning. PMLR, 2016, p.2829-2838) can be used.

[0110] The learning device 30 then calculates a probability distribution π(a) based on the action value Q(a) for action a. When controlling discrete actions, the learning device 30 calculates a probability distribution π2(a) from the action value Q(a) using the softmax function as shown in the following equation.

[0111]

number

[0112] In Equation 9, K is the number of actions.

[0113] When controlling a series of actions, the learning device 30 calculates the probability distribution π1(a) as shown in the following equation.

[0114]

number

[0115] (Step S406: Selecting an action) The learning device 30 performs action a based on the probability distribution π1(a) or π2(a). t Select the action. Step S406, in which an action is selected, may be the same as the action selection step S106 according to the first embodiment, or the action selection step S306 according to the second embodiment.

[0116] (Step S408: Cause the controlled object 10 to perform an action) The learning device 30 performs action a t The controlled object 10 is made to perform the corresponding action. Step S408, which causes the controlled object 10 to perform the action, may be the same as step S108, which causes the controlled object 10 to perform the action according to the first embodiment, or step S308, which causes the controlled object 10 to perform the action according to the second embodiment.

[0117] (Step S410: Receiving the next observation data and the first reward r1) The learning device 30 uses the following observational data O t+1 The first reward r1 is received. Step S410, which receives the next observation data and the first reward r1, may be the same as step S110, which receives the next observation data and the first reward r1 according to the first embodiment, or step S310, which receives the next observation data and the first reward r1 according to the second embodiment.

[0118] (Step S412 to determine probability density or probability) The learning device 30 executes step S412 in parallel with step S408. The learning device 30 uses the probability distribution π1(a) or π2(a) calculated in step 404 and the action a selected in step S406. t Action a t The probability density π1(a) or probability π2(a) of the selected option is calculated. The learning device 30 calculates the probability density in the same manner as in step S112 of the first embodiment, and calculates the probability in the same manner as in step S312 of the second embodiment.

[0119] (Step S414 to calculate the second reward) The learning device 30 calculates the probability density π²(a) obtained in step S412. t ) or probability π²(a t) from which a second reward r21 or r22 is calculated. Step S414 for calculating the second reward may be the same as step S114 for calculating the second reward r21 according to the first embodiment when controlling a continuous action, and may be the same as step S314 for calculating the second reward r22 according to the second embodiment when controlling a discrete action.

[0120] (Step S416: Update control parameters) After the completion of steps S410 and S414, the learning device 30 receives the current observation data O in step S402. t , action a selected in step S406 t , the next observation data O received in step S410 t+1 The value of the control parameter Θ is updated based on the first reward r1 and the second reward r21 or r22 calculated in step S414.

[0121] Specifically, the learning device 30 can use a parameter update method similar to the method used to update the parameters used to calculate the action value in a known SARSA method, by adding the second reward r21 or r22 to the first reward r1 and using that value as the reward in a known SARSA method.

[0122] The learning device 30 first selects the action in step S406, and then uses the following observation data O t+1 Based on the control parameter Θ, the next action a t+1 Select this option.

[0123] The learning device 30 then processes the current behavior a t The value of the action related to Q(a t ), the sum of the first reward r1 and the second reward r21 or r22, the next action a t+1 The value of the action related to Q(a t+1 Based on ), and the coefficient γ, the value of the control parameter Θ is updated as follows:

[0124] Θ V =Θ V -η·∇ Θ((r1+r21(or r22))+γ·Q(a t+1 )-Q(a t )) 2 Formula 11 In Equation 11, γ and η are coefficients, where γ is also called the discount rate and η is also called the learning rate.

[0125] As explained above, the control parameter learning method according to the third embodiment includes the steps of receiving current observation data in step S402, calculating a probability distribution π1(a) or π2(a) based on the current observation data and the control parameter Θ in step S404, and taking action a based on the probability distribution π1(a) or π2(a). t In step S406, select action a to the controlled object 10. t Step S408 to execute the next observation data and the first reward r1, probability distribution π1(a) or π2(a) and action a t Action a t The probability density of selection is π1(a t ) or probability π²(a t Step S412 to find the probability density π1(a t ) or probability π²(a t Step S414 calculates the second reward r21 or r22 from the current observation data and action a t Step S416, which updates the value of the control parameter Θ based on the next observation data and the first reward r1 and the second reward r21 or r22, is performed at each cycle of action execution, thereby learning the control parameters for the action control of robots, mobile objects, etc.

[0126] The control device, control method, and program that use the control parameters learned by the learning method according to the third embodiment are configured in the same way as the control device, control method, and program according to the first embodiment. The processing flow of the control method according to the third embodiment is the same as in Figure 10.

[0127] The control device 60 receives the current observation data O tThe step of receiving the current observation data may be the same as step S402 in the control parameter learning method for receiving the current observation data.

[0128] The control device 60 receives the current observation data O t Based on the control parameters Θ learned by the control parameter learning method, a probability distribution π1(a) or π2(a) is calculated. The step of calculating the probability distribution may be the same as step S404 in the control parameter learning method.

[0129] The control device 60 performs action a based on the probability distribution π1(a) or π2(a). t Select the action. The step of selecting the action may be the same as the step of selecting the action S406 in the control parameter learning method. Alternatively, the control device 60 selects the mean of action a under the probability distribution π1(a) or π2(a) as action a t By doing so, action a t Alternatively, the control device 60 may select the mode of action a under the probability distribution π1(a) or π2(a) as action a t By doing so, action a t You may choose this option.

[0130] According to the learning method of the third embodiment, in the reinforcement learning method using the SARSA method, whether the controlled object 10 performs continuous actions or discrete actions, even if the probability of selecting a certain action is calculated to be unreasonably small in the initial stages of learning, a second reward due to the small probability of selecting that action is added to the first reward, thereby correcting the unreasonably small probability of selecting that action. As a result, inaccurate learning is avoided.

[0131] [Fourth Embodiment] In the first to third embodiments, the controlled object 10 was a robot arm 52, but the controlled object 10 is not limited to a robot arm and may include other movable bodies, including other manufacturing equipment. The controlled object 10 may also be a mobile body such as an automobile. Examples of controlling continuous actions in an automobile include steering angle control, acceleration control, and deceleration control. Examples of controlling discrete actions in an automobile include selecting actions such as "go straight," "change lanes to the right," or "change lanes to the left," or selecting actions such as "accelerate," "deceleration," or "maintain constant speed." Examples of mobile bodies are not limited to automobiles and may include self-propelled robots, drones, trains, and other mobile bodies.

[0132] [Fifth Embodiment] In the control parameter learning method according to the first to third embodiments, in order to compensate for the probability density or probability of a certain action being selected being calculated to be unduly small in the initial stage of learning, a second reward is added to the first reward, the smaller the probability density or probability of that action being selected, the larger the value. As a result, the unduly small probability density or probability of that action being selected is corrected. In the fifth embodiment, instead of correcting the first reward r1 by adding the second reward r21 or r22 to the first reward r1, the first reward r1 is corrected by multiplying it by a correction coefficient w. That is, in the fifth embodiment, the control parameters are updated based on w × r1 instead of r1 + r21 (or r22) as in the first to third embodiments. The correction coefficient w is predetermined such that r1 + r21 (or r22) = w × r1 depending on the probability density or probability of the action being selected. The correction coefficient w is stored in the main memory 36.

[0133] An example of the learning device according to the fourth embodiment is the same as the learning device 30 according to the first embodiment (Figure 1). Figure 16 is a flowchart illustrating an example of the learning method according to the fourth embodiment. From step S102, which receives the current observation data, to step S110, which receives the next observation data and the first reward r1, the process is the same as the learning method according to the first embodiment (Figure 3).

[0134] When step S110 is completed, the learning device 30 corrects the first reward r1 based on the correction coefficient w (step S502). The correction coefficient w is larger as the probability density or probability decreases, similar to the second reward. As a result, the first reward is corrected so that it becomes larger as the probability density or probability decreases. This prevents the learning from converging to a local optimum and resulting in inaccurate learning, even in the early stages of reinforcement learning.

[0135] When step S502 is completed, the learning device 30 will process the current observation data O t , selected action a t , next observation data O t+1 Based on the corrected first reward w×r1, the value of the control parameter Θ is updated (step S504). Step S504 can be performed in step S116 of the learning method according to the first embodiment, step S316 of the learning method according to the second embodiment, and step S416 of the learning method according to the third embodiment by using the corrected first reward w×r1 instead of the first reward r1, or by using the corrected first reward w×r1 instead of the sum of the first reward r1 and the second reward r21 or r22 (r1+r21 (or r22)). If step S504 is performed in step S116 of the learning method according to the first embodiment, step S316 of the learning method according to the second embodiment, or step S416 of the learning method according to the third embodiment by using the corrected first reward w×r1 instead of the first reward r1, then the second reward r21 or r22 does not need to be used in step S504.

[0136] When step S504 is completed, the learning device 30 determines whether or not to terminate the learning process (step S118).

[0137] Figure 16 shows a learning method according to a fifth embodiment, which is a modification of the first embodiment. In the fifth embodiment, which is a modification of the second embodiment, the steps from step S102, which receives the current observation data in Figure 16, to step S110, which receives the next observation data and the first reward r1, are replaced by steps S302 to S310 of the learning method according to the second embodiment (Figure 11). In the fifth embodiment, which is a modification of the third embodiment, the steps from step S102, which receives the current observation data in Figure 16, to step S110, which receives the next observation data and the first reward r1, are replaced by steps S402 to S410 of the learning method according to the third embodiment (Figure 14).

[0138] The fifth embodiment also achieves the same effects as the first to third embodiments.

[0139] It should be noted that the present invention is not limited to the embodiments described above, and the components can be modified and implemented in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiments. Moreover, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0140] 10...Controlled object, 18...Sensor, 20...Drive unit, 30...Learning device, 32...Processor, 52...Robot arm, 54...End effector

Claims

1. The first step is to receive the current observational data, A second step involves determining a probability distribution that shows the probability density or probability distribution of multiple actions being selected, based on the current observation data and control parameters. A third step of selecting a first action from among the multiple actions based on the aforementioned probability distribution, A fourth step involves causing the controlled object to perform the first action, A fifth step in which the controlled object receives the next observed data and a first reward after performing the first action, A sixth step of determining the probability density or probability of the first action from the aforementioned probability distribution, A seventh step of correcting the first reward from the probability density or the probability, The system comprises an eighth step of updating the control parameters based on the current observation data, the first action, the next observation data, and the corrected first reward, The seventh step is a learning method that modifies the first reward such that the smaller the probability density or the probability, the larger the first reward.

2. The learning method according to claim 1, wherein the eighth step is to update the control parameters for each control cycle of the controlled object.

3. The learning method according to claim 1, wherein the second step involves inputting the current observation data into a neural network whose input / output characteristics change according to the control parameters, and causing the neural network to output the probability distribution.

4. The learning method according to claim 1, wherein the first reward indicates whether or not the first action was appropriate.

5. The seventh step involves correcting the first reward by adding the second reward to the first reward, The learning method according to claim 1, wherein the second reward is larger the smaller the probability density or the probability.

6. The seventh step corrects the first reward by multiplying it by a coefficient, The learning method according to claim 1, wherein the coefficient is larger the smaller the probability density or the probability.

7. We receive the current observation data, Based on the current observation data and control parameters, a probability distribution showing the probability density or probability distribution of multiple actions being selected is obtained. Based on the aforementioned probability distribution, a first action is selected from among the multiple actions. The controlled object is made to perform the first action, The controlled object receives the following observation data and the first reward after performing the first action. The probability density or probability of the first action is determined from the aforementioned probability distribution. The first reward is corrected from the probability density or the probability, The system comprises a processor that updates the control parameters based on the current observation data, the first action, the next observation data, and the corrected first reward. The processor is a learning device that compensates for the first reward by increasing the smaller the probability density or the probability.

8. On the computer, The first step is to receive the current observational data, A second step involves determining a probability distribution that shows the probability density or probability distribution of multiple actions being selected, based on the current observation data and control parameters. A third step of selecting a first action from among the multiple actions based on the aforementioned probability distribution, A fourth step involves causing the controlled object to perform the first action, A fifth step in which the controlled object receives the next observed data and a first reward after performing the first action, A sixth step of determining the probability density or probability of the first action from the aforementioned probability distribution, A seventh step of correcting the first reward from the probability density or the probability, A learning program for performing an eighth step of updating the control parameters based on the current observation data, the first action, the next observation data, and the corrected first reward, The seventh step is a learning program that modifies the first reward more as the probability density or the probability decreases.

9. The step of receiving the current observational data, A step of determining a probability distribution that shows the probability density or probability distribution of the selection of multiple actions, based on the current observation data and the control parameters updated by the learning method described in any one of claims 1 to 6, A step of selecting one of the multiple actions based on the probability distribution, A control method comprising the step of causing a controlled object to perform one of the aforementioned actions.

10. We receive the current observation data, Based on the current observation data and control parameters, a probability distribution showing the probability density or probability distribution of multiple actions being selected is obtained. Based on the aforementioned probability distribution, a first action is selected from among the multiple actions. A control device comprising a processing unit that causes a controlled object to perform the first action, The control parameters are determined based on the current observation data and the control parameters, and are updated according to a probability distribution showing the probability density or probability distribution of the selection of the multiple actions, and a reward indicating whether the first action performed by the controlled object was appropriate. The aforementioned reward is corrected so that the reward increases as the probability density or probability decreases.

11. On the computer, The first step is to receive the current observational data, A second step involves determining a probability distribution that shows the probability density or probability distribution of multiple actions being selected, based on the current observation data and control parameters. A third step of selecting a first action from among the multiple actions based on the aforementioned probability distribution, A control program that causes the controlled object to perform the first action, The control parameters are determined based on the current observation data and the control parameters, and are updated according to a probability distribution showing the probability density or probability distribution of the selection of the multiple actions, and a reward indicating whether the first action performed by the controlled object was appropriate. A control program in which the reward is corrected so that the reward increases as the probability density or probability decreases.

Images