Learning device, method for controlling the learning device, program

Abstract

To solve a problem that there are cases where it is hard to perform control corresponding to a control target changing according to conditions.SOLUTION: A learning device includes: observation means for observing a state of a control object for each observation period; deduction means for outputting an operation quantity according to data observed by the observation means to the control object; learning means for reinforcement-learning the operation quantity to be output to the control object by the deduction means; and setting means for setting a control target corresponding to at least one of the states to be output by the observation means. In the learning device, while the learning means is performing the reinforcement learning, the setting means changes the control target. Reward is calculated on the basis of the control target and the state of the object.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a learning device that sets a control target when learning control. [Background Art]

[0002] Conventionally, techniques for changing control parameters in accordance with a control target have been disclosed. For example, Patent Document 1 discloses a technique for changing an adjustment parameter without causing a large change in control output in response to a change in the control target while automatic control is in operation. Patent Document 2 discloses a technique for stopping learning when a remarkable change in a target value occurs while training a neural network compensator that generates a feedforward compensation signal, because appropriate learning cannot be performed in such a case. [Prior Art Documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Unexamined Patent Application Publication No. 2020-30567 [Patent Document 2] Japanese Unexamined Patent Application Publication No. 4-326101 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] In reinforcement learning, a learning subject called an agent observes the state of a learning target called an environment, and instructs the environment to take an action according to the state. Then, as a result of the instruction, the agent observes how the state of the environment changes after an observation cycle, and determines whether the action instructed in the initial state was appropriate based on an indicator called a reward. In reinforcement learning, the process from observing an initial state to observing the next state is performed within one observation cycle.

[0005] When applying reinforcement learning to control systems, the agent acts as the controller and the environment as the controlled object. However, for a reinforcement learning controller to be practical, it must be able to control the controlled variable so that it can keep up with changing control targets. This is because the controlled object, such as a motor, may be driven at different speeds depending on the situation. For example, when controlling speed, multiple speed targets may be used depending on the situation. However, if the system is trained using only one speed target, it will become specialized for that target. As a result, when setting other speed targets that were not used during training during inference (i.e., when actually performing speed control), the system may not be able to control the speed appropriately, and a deviation may occur between the speed target and the actual speed.

[0006] In the prior art disclosed in Patent Document 1 mentioned above, adjustment parameters are changed in response to changes in the control target, but reinforcement learning cannot learn to respond to changes in the control target.

[0007] Furthermore, while the prior art disclosed in Patent Document 2 mentioned above appropriately performs learning by stopping learning in response to significant changes in the target value, it is not effective in addressing the challenge of responding to changing control targets.

[0008] Therefore, the present invention aims to provide a reinforcement learning method that enables control in response to control targets that change according to the situation. [Means for solving the problem]

[0009] To achieve the above objectives, this Disclosure The learning device comprises: an observation means for observing the state of the controlled object at each observation period; an inference means for outputting a manipulated quantity to the controlled object according to the data observed by the observation means; a learning means for reinforcement learning the manipulated quantity output by the inference means to the controlled object; and a setting means for setting a control target corresponding to at least one of the states output by the observation means. While the learning means is performing reinforcement learning, the setting means changes the control target, and the reward is calculated based on the control target and the state of the object. The number of times observations are made by the observation means is defined as the number of observation cycles, the learning means does not perform learning until the observation means has gone through a predetermined number of observation cycles, and performs learning when the predetermined number of observation cycles has ended, and after the episode, which is the process from the initial state to going through the predetermined number of observation cycles of the controlled object, has ended, the state of the controlled object is set to a predetermined initial state, the number of observation cycles is set to 0, and then the next episode begins. It is characterized by the following. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide a reinforcement learning method that enables control in response to control targets that change according to the situation. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows the concept of reinforcement learning in Embodiment 1 of the present invention. [Figure 2] This figure shows the deep neural network used for training in Embodiment 1 of the present invention. [Figure 3] This is a block diagram showing the configuration of the reinforcement learning method in Embodiment 1 of the present invention. [Figure 4] This figure shows an example of multiple target speeds in Embodiment 1 of the present invention. [Figure 5] This figure shows the reward for learning multiple target speeds in Embodiment 1 of the present invention. [Figure 6] This figure shows an example of the response when the initial state of the motor in Embodiment 1 of the present invention is changed. [Figure 7] This flowchart shows the processing flow in Embodiment 1 of the present invention. [Figure 8] This flowchart shows the processing flow in Embodiment 2 of the present invention. [Modes for carrying out the invention]

[0012] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0013] [First Embodiment] Figure 1 is a conceptual diagram of reinforcement learning according to an embodiment of the present invention. In reinforcement learning, the learning entity is defined as the agent, and the learning target and the system influenced by the learning target are collectively defined as the environment. The agent consists of a deep neural network (DNN) that actually performs learning and an experience data storage unit that stores the learning experience. The input to the DNN is a state S, and the output of the DNN is an action A, where state S is set to information that can sufficiently represent the state of the environment, which is the learning target. Reinforcement learning observes state S and outputs action A by inputting it to the DNN. Next, the environment executes action A output by the DNN, and state S transitions to the next state, state Sa. The environment refers to a predetermined reward calculation method, calculates a reward R based on state Sa, and passes it to the agent. In reinforcement learning, the above process is performed in one observation cycle, and the obtained state S, action A, reward R, and next state Sa for one observation cycle are stored in the experience data storage unit, and the stored experience data is called up at a predetermined learning timing to perform DNN learning.

[0014] Figure 2 shows an example of the structure of the deep neural network (DNN) in Figure 1. The circles represent nodes, and the arrows connecting the nodes represent weights. At each node, the product of the weight and the output of the previous layer, plus a fixed bias, is input to a nonlinear function called an activation function, and the function output is output to the next layer. As explained in Figure 1, the input is a state S. The output is a value called a Q-value, which represents the value of each action, and Qa to Qf in Figure 2 represent the Q-values ​​of each action. The DNN predicts the value of each action in each input state S and selects the action with the largest Q-value. During training, the weights of the DNN are updated to predict the value of each action more accurately.

[0015] FIG. 3 is a block diagram illustrating the hardware configuration of the reinforcement learning apparatus 300 of the present embodiment. A communication unit 301 communicates with the outside and exchanges information with other blocks. The input / output interface 302 is an interface for a user to perform input to the machine learning apparatus 300. A DNN processing unit 303 performs deep neural network computations used for reinforcement learning. A CPU 304 executes user instructions and the like through the communication unit 301. It is assumed that a ROM 305 stores processing contents to be executed by the CPU 304. A memory 306 is for storing parameters required in the processing of the machine learning apparatus 300, and serves the role of storing values of weight matrices for matrix operations.

[0016] FIG. 4 is a diagram illustrating a case where there are three control targets in the present embodiment. In the present embodiment, the reinforcement learning target is a motor, and the constant speed control of the motor is learned. FIG. 4 illustrates an example in which three target speeds are set, and an object of the present embodiment is to realize, with a single learner, control that follows each target speed as in speed responses 401, 402, and 403.

[0017] FIG. 5 is a diagram showing a function representing a reward provided from an environment to an agent in the present embodiment, FIG. 6 is an example of a speed response when the initial state is changed, and FIG. 7 is a flowchart for reinforcement learning while switching control targets. Hereinafter, acquisition of the speed control operation of a DC motor by reinforcement learning according to the first embodiment of the present invention will be described along the flow of the flowchart of FIG. 7. In the present embodiment, the agent 101 in FIG. 1 is a reinforcement learner, and the environment 104 is a DC motor.

[0018] In S701 of FIG. 7, the environmental state S is observed. In the present embodiment, the state S is a speed deviation that is a difference between a target speed and a rotational speed at the current time. Since the rotational speed of the motor is 0 in the very first state S, the speed deviation is equal to the target speed. A plurality of target speeds are determined in advance before the start of learning, and target speed numbers are assigned as consecutive numbers. In the first processing of S701 immediately after the start of learning, the target speed of target speed number 1 is used, and after that, the target speed is used by referring to the target speed number when the processing of S701 is performed.

[0019] In S702, the state S observed in S701 is input to DNN103 in Figure 1. DNN103 outputs the Q value for each action, and inference is performed to input the action A with the largest Q value as the manipulated variable to the DC motor. The outputs Qa to Qf in Figure 2 show the Q values ​​for each action. In this embodiment, action A (manipulated variable) is the voltage input to the DC motor, and is divided from 0V to the maximum applied voltage with a predetermined resolution. For example, if the maximum applied voltage is 5.5V and the resolution is 0.5V, there are 12 patterns of actions: 0V, 0.5V, 1.0V, ..., 5.5V.

[0020] In S703, the DC motor, which is the environment 104, has the voltage output by DNN103 in S702 applied to it, and the control variable, the rotational speed, changes. The time for which the manipulated variable output in S702 is applied is determined by the observation period T, and the rotational speed after time T becomes the new state Sa.

[0021] In S704, the reward R is calculated according to a predetermined reward calculation method based on the new state Sa, and sent to agent 101.

[0022] In this embodiment, in order to acquire DC motor speed control through learning, the reward calculation method was set to give a higher reward R the smaller the speed deviation, and the reward function shown in Figure 5 was used. The reward function in Figure 5 is set so that the closer the deviation rate, calculated by dividing the rotational speed of the new DC motor transitioned in S703 by the target speed, is to 100%, the higher the reward. If the reward is calculated according to the speed deviation calculated as the difference between the target speed and the current speed rather than the deviation rate, the same reward will be calculated when the same speed deviation occurs for multiple target speeds, but in reality, they may not have the same value, so learning cannot be performed correctly.

[0023] For example, in Figure 4, suppose target speed 2 is 100 rps and target speed 3 is 50 rps, and the speed deviation is 50. In this case, with target speed 2, the current speed is 50 rps, meaning that acceleration is in progress and only a little more acceleration is needed. However, with target speed 3, the current speed is 0 rps, meaning that no acceleration has occurred and acceleration is needed. However, since the same reward value is given, action A will be evaluated as having the same value in both target speeds, and as a result, learning will not be performed correctly. Therefore, when performing reinforcement learning while switching target speeds, it is better to use the deviation rate rather than the speed deviation to perform learning more correctly.

[0024] In S705, the state S, action A, new state Sa, and reward R for one observation period T are grouped together as data for the same observation period T and stored in the experience data storage unit 102 of agent 101. This observation period is represented as a unit called a step in reinforcement learning.

[0025] In S706, it is determined whether the specified number of steps has been reached. The specified number of steps is predetermined to be sufficient to consistently output the target speed, taking into account the characteristics of the environment. If the target has been reached, the process moves to S708, where the state S, action A, new state Sa, and reward R of several random steps are read from the data stored in the experience data storage unit 102, and the weights of the DNN 103 are updated.

[0026] This section explains how to update weights in S708. Reinforcement learning outputs a Q-value representing the value of each action in an observed state S. Since the system then performs the action with the highest Q-value against the environment, the weights are updated to correctly output the Q-values ​​of each action in various states. The Q-value is updated using the reward (Equation 1).

[0027]

number

[0028] γ is a parameter called the discount rate that controls how much the future value of the action is considered. Equation 1 shows that the weights are updated by the difference between the sum of the reward R(S,A) received as a result of performing action A in state S, the largest Q value in the new state Sa to which action A was performed, and the current Q value (Q(S,A)) of action A in state S.

[0029] If the specified number of steps has not been reached in S706, the number of steps is increased by 1 in S707, the process returns to S702, the new state Sa that was reached is input to DNN103 in S703, and the next action A is output.

[0030] In S709, it is determined whether the number of episodes has reached a predetermined number (specified number of episodes). An episode is a unit representing the number of times the specified number of steps has been reached, and after the specified number of steps is reached, the state S and the number of steps are returned to their initial state. If the specified number of episodes has not been reached in S709, the process in S710 is performed. If the specified number of episodes has been reached, the learning process is terminated.

[0031] The process in S710 is explained using the flowchart in (b). In S711, the current target speed number is checked. In S712, the subsequent processing is selected using the target speed number checked in S711. If the target speed number does not match the maximum value of the target speed numbers, 1 is added to the current target speed number and that value is set as the new target speed number. If they match, there are no target speeds with a larger target speed number, so the target speed number is set to 1. In S715, the episode number is increased by 1.

[0032] Through these S710 processes, for example, if three target speeds are set before training, the target speed number will switch to 1, 2, 3, 1, 2, 3, 1, ... each time the episode switches, and the target speed will change accordingly.

[0033] Through the above process, when learning to control the speed of a DC motor using reinforcement learning, it becomes possible to perform control that tracks an arbitrary target speed.

[0034] In this embodiment, the initial rotational speed of the motor was set to 0. However, it may also be implemented as a deceleration control that follows a target speed 2, with a target speed 1 as the initial state, as shown in Figure 6, which can be set to any rotational speed.

[0035] In this embodiment, a DC motor was used as the environment, but other motors may also be used.

[0036] In this embodiment, speed control was used as the control method, but other control methods such as position control may also be used.

[0037] In this embodiment, steps S712, S713, and S714 in Figure 7(b) add 1 to the target speed number to set a new target speed number, and when the maximum value of the target speed number is reached, it returns to 1. Alternatively, one value randomly selected from among the target speed numbers up to the maximum value may be set as the target speed number.

[0038] [Second Embodiment] The process in the second embodiment of the present invention will be described below with reference to the figures.

[0039] In the first embodiment, the modification of the control objective during learning at the controller design stage and the corresponding reward were explained. In the case of a controller installed in a product, reinforcement learning of the control can be performed after it is delivered to the user, allowing for control that adapts to aging deterioration such as motor degradation and grease hardening. In this embodiment, a DC motor operating in an actual device will be used as an example.

[0040] Figure 8(a) is a flowchart showing the learning process at the user's location. Figure 8(b) is a flowchart showing the part related to changing the control target during the learning process at the user's location. Hereafter, following the flowchart in Figure 8(a), the acquisition of DC motor control by reinforcement learning in an actual device at the user's location according to the second embodiment of the present invention will be explained.

[0041] In S801 of Figure 8(a), the latest date and time are obtained.

[0042] S802 determines whether the specified number of days has elapsed using the date and time obtained by S801. The purpose of learning at the user's location in this embodiment is to respond to changes in controllability due to aging, so reinforcement learning is performed at specified intervals to correspond to the latest state of the drive system. The specified number of days is determined in advance before shipment.

[0043] In S803, if a specified number of days have elapsed, the user is asked to confirm whether it is OK to proceed with learning. Learning puts a load on the CPU, so other operations will be unavailable. Therefore, if the user has plans to perform other operations immediately, learning will not be performed. If the user selects "OK to start learning" in S803, the learning process in S804 begins. The processing content of S804 is the same as the flowchart in Figure 7(a), so a detailed explanation is omitted. The difference from Embodiment 1 is that learning to match the new state has been performed before shipment, and the weights have already been optimized to some extent.

[0044] Similar to the first embodiment, the target speed is changed in S710 in Figure 7. The processing content of S710 is explained using the flowchart in Figure 8(b). In S805, one random integer is selected from the maximum value of the target speed numbers given in S701, that is, from the number of target speeds predetermined before learning. In S806, the integer obtained in S805 is set as the target speed number. Through these processes in Figure 8(b), one random target speed is set from among the multiple target speeds predetermined before learning each time the episode switches.

[0045] Through the above process, it becomes possible to enable learning to track an arbitrary target speed in order to prevent a decrease in the controllability of the DC motor due to aging as the user uses the actual device.

[0046] In this embodiment, the latest date and time are obtained in S801, and a specified number of days is used for the determination in S802. However, any value can be used as long as it indicates that time has passed.

[0047] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist.

[0048] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

Claims

1. An observation means for observing the state of the controlled object at each observation period, An inference means that outputs a manipulated quantity corresponding to the data observed by the observation means to the target of control, The inference means includes a learning means that reinforces learning the manipulated variable output to the object of control, A setting means for setting a control target corresponding to at least one of the states output by the observation means, Equipped with, While the learning means is performing reinforcement learning, the setting means changes the control target. The reward is calculated based on the control target and the state of the target. The number of times observations are performed by the aforementioned observation means is defined as the number of observation periods. The learning means does not perform learning until the observation means has gone through a predetermined number of observation cycles. When the predetermined number of observation periods has ended, learning is performed. After an episode, which is the process from the initial state to the predetermined number of observation periods of the controlled object, has finished, the state of the controlled object is returned to a predetermined initial state. After setting the number of periods in the aforementioned observation to zero, start the next episode. A learning device characterized by the following.

2. The learning device according to claim 1, characterized in that the timing at which the setting means changes the control target is when the episode ends.

3. The learning device according to claim 1, characterized in that the reward is given in accordance with the deviation rate calculated by dividing the state observed by the observation means by the control target.

4. The learning device according to claim 1, characterized in that the initial state at the start of the episode is changed each time an episode is started.

5. The learning device according to claim 1, characterized in that the control target is selected one by one from a plurality of predetermined control targets at a predetermined timing.

6. The learning device according to claim 1, characterized in that when the setting means changes the control target at a predetermined timing, if all predetermined control targets have been changed, it again selects one by one from a plurality of predetermined control targets.

7. The learning device according to claim 1, characterized in that the control target is randomly selected from a plurality of predetermined control targets at a predetermined timing.

8. An observation step in which the state of the controlled object is observed at each observation period, An inference step in which a manipulated quantity corresponding to the observed data is output to the target of control, The learning step involves reinforcing learning the manipulated variable to be output to the control target in the inference step, A setting step in which a control target is set that corresponds to at least one of the states output in the observation step, Equipped with, While reinforcement learning is being performed in the learning step, the control target is changed in the setting step. The reward is calculated based on the control target and the state. The number of times observations are performed in the aforementioned observation step is defined as the number of observation periods. In the learning step, learning is not performed until a predetermined number of observation cycles have passed, as set in the observation step. When the predetermined number of observation periods has ended, learning is performed. After an episode, which is the process from the initial state to the predetermined number of observation periods of the controlled object, has finished, the state of the controlled object is returned to a predetermined initial state. After setting the number of periods in the aforementioned observation to zero, start the next episode. A control method for a learning device characterized by the following.

9. A computer-readable program for causing a computer to function as one of the means of a learning device according to any one of claims 1 to 7.

Images