Control apparatus, learning apparatus, control method, and learning method

Abstract

To materialize a learning machine that withstands strongly against a disturbance.SOLUTION: A control apparatus 20 includes a learned learning machine 12 that accepts input of a reference value r(t) and outputs a correction quantity o(t) of a condition variable q(t) of a robot 30, a reference value determination unit 13 that obtains, as the reference value, a sum of a product of the condition variable of the robot by a coefficient, which is larger than 0 and equal to or smaller than 1, and the correction quantity, and an output unit 14 through which the reference value is outputted to a controller 15 that determines a manipulation quantity u(t) of an object apparatus.SELECTED DRAWING: Figure 22

Description

[Technical field]

[0001] The present invention relates to a control device and a learning device that use a learning unit. [Background technology]

[0002] Collaborative robots, in which humans and robots coexist in the same environment and perform tasks, are being developed. Because robots will be operating in an environment close to humans, it is possible that they will be subjected to unexpected disturbances. Conventionally, when an unexpected disturbance is applied to a robot, the robot is immediately stopped by a control system to prevent it from going out of control and causing harm to humans.

[0003] However, if the robot suddenly stops, its operation will be interrupted, which will lead to a decrease in work efficiency.In addition, the sudden acceleration will cause the robot itself to be damaged.

[0004] Non-Patent Document 1 discloses a technique for gradually responding to unexpected disturbances by generating a reference trajectory in consideration of the current state of the robot. Also, Non-Patent Document 2 discloses a technique for drawing a predetermined trajectory even in an initial state different from the learned trajectory by using a Gaussian mixture model. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Seto & Sugihara et. al., "Nonlinear Reference Shaping with Endpoint Position Feedback for Large Acceleration Avoidance in Reaching Movement", Journal of Robotics and Mechatronics Vol.22 No.2, 2010, p.173-178 [Non-Patent Document 2] Khansari-Zadeh & Billard et. al., "Learning Stable Nonlinear Dynamical Systems With Gaussian Mixture Models", IEEE TRANSACTIONS ON ROBOTICS, VOL. 27, NO. 5, OCTOBER 2011, p.943-957 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the configuration of Non-Patent Document 1 has a problem of poor responsiveness, and the configuration of Non-Patent Document 2 has a problem of extremely high learning cost for the Gaussian mixture model.

[0007] An object of one aspect of the present invention is to realize a learning device that is resistant to disturbances. [Means for solving the problem]

[0008] In order to solve the above problems, a control device according to one embodiment of the present invention is configured to include a trained learning device that receives an input of a reference value and outputs a correction amount for a state variable of a target device, a reference value determination unit that calculates the reference value by multiplying the state variable of the target device by a coefficient greater than 0 and equal to or less than 1 and the sum of the correction amount, and an output unit that outputs the reference value to a controller that determines the operation amount of the target device.

[0009] In order to solve the above problems, a learning device according to one embodiment of the present invention is configured to include a learning device that receives an input of a reference value, uses the deviation between the reference value and a target value for a state variable of a target equipment as a teacher, and outputs a correction amount for the state variable of the target equipment, a reference value determination unit that calculates the reference value by multiplying the state variable of the target equipment by a coefficient greater than 0 and equal to or less than 1, and the correction amount, and an output unit that outputs the reference value to a controller that determines the operation amount of the target equipment.

[0010] In order to solve the above problems, a control method according to one embodiment of the present invention is a method including: a learning application step in which a reference value is input and a trained learning device that calculates a correction amount for a state variable of a target device is caused to output the correction amount; a reference value determination step in which the reference value is calculated by multiplying the state variable of the target device by a coefficient greater than 0 and equal to or less than 1 and the sum of the correction amount and the state variable of the target device by a coefficient greater than 0 and equal to or less than 1; and an output step in which the reference value is output to a controller that determines the operation amount of the target device.

[0011] In order to solve the above problems, a learning method according to one embodiment of the present invention is a method including: a learning step in which a reference value is input, the deviation between the reference value and a target value for a state variable of a target device is used as a teacher, and a correction amount for the state variable of the target device is output; a reference value determination step in which the reference value is calculated by multiplying the state variable of the target device by a coefficient greater than 0 and equal to or less than 1, and the correction amount; and an output step in which the reference value is output to a controller that determines the operation amount of the target device. Effect of the Invention

[0012] According to one aspect of the present invention, a learning device that is resistant to disturbances can be realized. [Brief description of the drawings]

[0013] [Figure 1] FIG. 2 is a block diagram showing the configuration of the main parts of the learning system. [Diagram 2] FIG. 2 is a control block diagram of the learning device. [Diagram 3] FIG. 2 is a block diagram showing the configuration of a main part of a control system. [Figure 4] FIG. 2 is a control block diagram of a control device. [Diagram 5] 13 is a graph showing a reference value when an impulse external force is applied to a learning device that has learned a step input as a target value. [Figure 6] 13 is a graph showing the manipulated variable when an impulse external force is applied to a learning device that has learned a step input as a target value. [Figure 7]13 is a graph showing state variables when an impulse external force is applied to a learning device that has learned a step input as a target value. [Figure 8] This is a graph of the reference value when an obstacle to be removed is placed on a learning machine that has learned a step input as a target value. [Figure 9] 13 is a graph showing the amount of operation when an obstacle to be removed is placed on a learning device that has learned a step input as a target value. [Figure 10] 1 is a graph of state variables when an obstacle to be removed is placed on a learning device that has learned a step input as a target value. [Figure 11] 13 is a graph of a reference value when a random disturbance is applied steadily to a learning device that has learned a step input as a target value. [Figure 12] 1 is a graph showing a manipulated variable when a random disturbance is steadily applied to a learning device that has learned a step input as a target value. [Figure 13] 1 is a graph of state variables when a random disturbance is applied steadily to a learning device that has learned a step input as a target value. [Figure 14] 13 is a graph of a reference value when stationary random observation noise is added to a learning device that has learned a step input as a target value. [Figure 15] 13 is a graph of the manipulated variable when stationary random observation noise is added to a learning device that has learned a step input as a target value. [Figure 16] 13 is a graph of state variables when stationary random observation noise is added to a learning device that has learned a step input as a target value. [Figure 17] This is the time series change of the tip position of the second link in the XY coordinate space when controlled with step input from different initial values ​​using a single learned time series change. [Figure 18]This shows the time series change in the tip position of the second link when controlled with step inputs from different initial values ​​using a single learned time series change in joint angle space. [Figure 19] This shows the time series change in the position of the tip of the second link in the XY coordinate space when the tip position of the second link is controlled to trace a predetermined trajectory from different initial values ​​using a single learned time series change. [Figure 20] This shows the time series change in the tip position of the second link when controlled to trace a specified trajectory from different initial values ​​using a single learned time series change in joint angle space. [Figure 21] FIG. 11 is a control block diagram showing another implementation of the learning system. [Figure 22] FIG. 11 is a control block diagram showing another implementation example of the control system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] [Embodiment 1] Hereinafter, an embodiment according to one aspect of the present invention (hereinafter, also referred to as "the present embodiment") will be described with reference to the drawings. In the drawings, the same or corresponding parts are designated by the same reference characters and their description will not be repeated.

[0015] (System Configuration) 1 is a block diagram showing the configuration of the main parts of a learning system 1. The learning system 1 includes a learning device 10 and a robot (target device) 30. The learning device 10 includes a state acquisition unit 11, a learning unit 12, a reference value determination unit 13, an output unit 14, and a controller 15. The learning device 10 is capable of not only learning, but also controlling the robot 30.

[0016] In the learning device 10, a state acquisition unit 11 acquires the state quantities of the robot 30 as state variables, and a learning device 12 learns target values ​​for the state variables. Then, in the learning device 10, a reference value determination unit 13 determines a reference value corresponding to a control target value actually used for control from the correction amount output by the learning device 12, and a controller 15 controls the robot 30 according to the reference value.

[0017] (Configuration of learning device 10) The state acquisition unit 11 acquires the state quantities of the robot 30 as state variables. Examples of the state variables include the joint angles and joint positions of the robot 30, or the measurement values ​​of sensors that measure external forces acting on the robot 30. Note that the state quantities may include observation noise (errors) caused by measuring instruments. The state acquisition unit 11 outputs the state variables to the reference value determination unit 13 and the controller 15.

[0018] The learning device 12 is a device that performs learning, and the type of the learning device is not particularly limited. In this embodiment, a reservoir computing (hereinafter, referred to as a reservoir) is used as the learning device 12. The learning device 12 obtains a previous reference value from the reference value determination unit 13, and outputs a new correction amount to the reference value determination unit 13.

[0019] The reference value determination unit 13 obtains a new reference value by multiplying the state variable acquired from the state acquisition unit 11 by a coefficient greater than 0 and equal to or less than 1, and adding the result to the correction amount acquired from the learning device 12. The reference value determination unit 13 outputs the new reference value r(t) to the output unit 14.

[0020] The output unit 14 temporarily stores the change in the reference value, and outputs the previous reference value to the learning device 12 and the new reference value to the controller 15 .

[0021] The controller 15 is a controller that performs control so as to reduce the deviation between the reference value and the state variable. In this embodiment, a PD controller is used as the controller 15. The controller 15 outputs the calculated operation amount to the robot 30. Examples of the operation amount include a current, a torque, and a thrust force.

[0022] (Control block diagram of learning device 10) 2 is a control block diagram of the learning device 10. The state acquisition unit 11 acquires the state quantity of the robot 30 as a state variable q(t). The state acquisition unit 11 outputs the acquired state variable q(t) to the reference value determination unit 13 and the controller 15. Note that t represents the current time, and the state variable q(t) represents a state quantity that changes with the change in time.

[0023] It is assumed that observation noise is added to the state variable q(t) acquired by the state acquisition unit 11. Here, the observation noise is noise that occurs when measuring the state, such as electrical noise of analog quantities or noise associated with discretization of digital quantities.

[0024] The learning device 12 outputs a correction amount o(t) to the reference value determination unit 13 based on the previous reference value r(t-Δt) acquired from the output unit 14. The reference value determination unit 13 multiplies the state variable q(t) by a coefficient α greater than 0 and equal to or less than 1, and adds the correction amount o(t) to obtain a reference value r(t). The output unit 14 outputs the reference value r(t) to the controller 15. Here, Δt represents the control period.

[0025]

number

[0026]

number

[0027]

number

[0028]

number

[0029] Furthermore, when the above steps are performed at time t+Δt, the reference value input to the learning device 12 is the reference value r(t) at time t. In other words, the reference value r(t) determined by the reference value determination unit 13 becomes the reference value r(t-Δt) input to the learning device 12 at the next time.

[0030] (Reservoir computing in learning device 12) The learning device 12 is a reservoir. The reservoir is a recurrent neural network with random connections of N elements. The state x of element i in the reservoir at time t is i The dynamics of (t) may be given by a firing rate model with tanh as the activation function.

number

[0031]

number

[0032] g Rec > 1, but the reference value r(t) containing the correction amount o(t), which is the network's own output, is used as the input to the reservoir, preventing the network from entering a chaotic state.

[0033] The correction amount o(t), which is the readout output, is obtained by a weighted sum of the activity of each element.

[0034]

number

[0035]

number

[0036]

number

[0037] (Learning by learning device 10) The learning device 10 learns the learning unit 12 in a state where observation noise is included, but no other disturbances are included. This is to ensure that the correction amount o(t) becomes a value related to the target value d(t) in a disturbance-free state. In other words, the ideal state is learned without learning disturbances.

[0038] (Disturbances and observation noise) The learning device 10 learns including the observation noise. This is because the observation noise will occur even when control is performed after learning. However, as shown in the formula (2), when the coefficient α=1, the control deviation e r The state variable q(t) including the observation noise at (t) is cancelled out by the term of the state variable q(t) included in the reference value r(t). Therefore, the observation noise is cancelled out, and the control deviation e r(t) is the same as the correction amount o(t). Note that even if 0<α<1, the control deviation e r The observation noise at (t) decreases.

[0039] Therefore, it is possible to realize control that is less affected by observation noise. By adjusting the value of coefficient α, it is possible to reduce the contribution of disturbances and observation noise to controller 15, and therefore the influence of these factors can be reduced.

[0040] However, the state variable q(t) containing observation noise is input to the learning device 12. Therefore, it is preferable that the learning device 12 has high noise robustness. Therefore, it is preferable that the learning device 12 is a reservoir having noise robustness.

[0041] This tendency is also the same for disturbances, which will be described later. Although the state variable q(t) changes due to the influence of disturbances, this change is offset by the reference value r(t) that includes the state variable q(t). Therefore, when the coefficient α=1, the components contributed by the disturbances are offset even in a disturbed environment, so stable control can be achieved.

[0042] [Embodiment 2] Other embodiments of the present invention will be described below. For ease of explanation, the same reference numerals are given to members having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0043] In this embodiment, the object is to perform control using the learning device 12 that has learned by the learning device 10 in embodiment 1. Note that in this embodiment, a disturbance may be applied.

[0044] (System Configuration) FIG. 3 is a block diagram showing the configuration of the main parts of the control system 2. The control system 2 includes a control device 20 and a robot 30. The control device 20 includes a state acquisition unit 11, a learning device 12, a reference value determination unit 13, an output unit 14, and a controller 15. That is, the learning device 10 may be used as the control device 20. However, the learning device 12 in the control device 20 does not need to have learning capability. That is, the result of learning by the learning device 12 of the external learning device 10 may be installed in the learning device 12 of the control device 20 and applied. That is, the learning device 12 of the control device 20 has a learned model (here, a reservoir).

[0045] In other words, the learning device 12, which serves as a reservoir, has multiple elements to which a reference value r(t) is input, the elements are randomly connected, and a correction amount o(t) is output by linearly adding the learned readout weights for the elements.

[0046] (Control block diagram of control device 20) 4 is a control block diagram of the control device 20. FIG. 4 shows the relationship between the correction amount o(t) in the learning device 12 in FIG. d The learning term by the teacher in (t) has been deleted. In other words, the learning device 12 does not perform learning, but simply outputs the calculation results based on the learned model.

[0047] Furthermore, the robot 30 is controlled by taking into account (adding) disturbance to the operation amount u(t) input to the robot 30. In other words, the application of an external force (disturbance) to the robot 30 is taken into consideration.

[0048] (Robot 30 model) In this embodiment, a two-link rigid robot arm is used as the robot 30. The robot 30 has a first link and a second link connected by a joint. The first link and the second link have common parameters, the center of gravity is located at the center of the link, and gravity is not taken into consideration. Note that time (t) is omitted.

number

number

number

[0049]

number

[0050] [Table 1] In this embodiment, one end (base end) of the first link of the robot 30 is pinned (rotatably fixed) to the base via a first joint. One end of the second link is rotatably fixed to the other end of the first link via a second joint. The control device 20 controls the angle of the first joint and the angle of the second joint of the robot 30. For example, the control device 20 controls the angle q1 of the first joint and the angle q2 of the second joint by controlling the torque u1 applied to the first link at the first joint and the torque u2 applied to the second link at the second joint. In this way, the control device 20 moves the other end (tip) of the second link to a target position. u is a vector having u1 and u2 as elements. q is a vector having q1 and q2 as elements. The target value d is a vector having the angle q1 of the first joint and the angle q2 of the second joint when the other end of the second link moves to the target position as elements. In this example, the target position does not change with time, and the target value d does not change either.

[0051] (Simulation: Impulse external force) FIG. 5 is a graph of the reference value r2(t) when an impulse external force is applied to the learning device 12 that has learned a step input as the target value d2(t). FIG. 6 is a graph of the manipulated variable u2(t) when an impulse external force is applied to the learning device 12 that has learned a step input as the target value d2(t). FIG. 7 is a graph of the state variable q2(t) when an impulse external force is applied to the learning device 12 that has learned a step input as the target value d2(t). The impulse external force was applied at a time of 500 [ms]. The impulse external force is an unknown disturbance for the control system 2. The graph with the coefficient α=0 is a comparative example. Here, the target value d(t), the reference value r(t), the operation amount u(t) and the state variable q(t) may each be a vector, and the elements thereof relating to the second link are referred to as the target value d2(t), the reference value r2(t), the operation amount u2(t) and the state variable q2(t), respectively.

[0052] As shown in Figure 5, the reference value r2(t) is constant even at time 500 [ms] when coefficient α = 0. In contrast, when coefficient α = 1, the reference value drops instantaneously at time 500 [ms]. This is because the state variable q2(t) changes suddenly due to the impulse external force.

[0053] 6, the control device 20 outputs a large manipulated variable u2(t) at time 500 [ms] when the coefficient α = 0. In contrast, the control device 20 outputs a small manipulated variable u2(t) at time 500 [ms] when the coefficient α = 1. The change in the manipulated variable at time 0 [ms] is almost the same.

[0054] As shown in Fig. 7, the state variable q2(t) drops significantly at time 500 [ms] regardless of the coefficient α. However, the drop is greater when coefficient α = 1 than when coefficient α = 0. Furthermore, the responsiveness of the state variable q2(t) to an impulse external force is almost the same whether coefficient α = 0 or α = 1. Furthermore, the responsiveness of the state variable q2(t) near time 0 [ms] is also almost the same.

[0055] Therefore, from the change in the reference value r2(t), when the coefficient α=0, even if an impulse external force is applied, the operation is not performed taking the impulse external force into consideration. Therefore, the operation amount u2(t) becomes large. In other words, when the coefficient α=0, the reference value r2(t) does not take the state variable q2(t) into consideration.

[0056] In contrast, when the coefficient α=1, the reference value r2(t) is decreased in accordance with the impulse external force, so the manipulated variable u2(t) also becomes smaller. In other words, when the coefficient α=1, the reference value r2(t) changes in accordance with the change in the state variable q2(t) due to the impulse external force, so the manipulated variable u2(t) also changes gently.

[0057] From the above, it can be seen that the influence of disturbances can be reduced by increasing the coefficient α. Also, energy is saved because the operation amount u2(t) in response to the disturbance is small. Therefore, when the coefficient α=1, for example when the operation amount u2(t) is torque, the robot 30 operates with low torque, and safety is higher than when the coefficient α=0. For example, the peak torque when the coefficient α=1 is about one tenth of the peak torque when the coefficient α=0.

[0058] Moreover, when the coefficient α is 0<α<1, the behavior of each of the coefficients α is between 0 and 1. Therefore, by adjusting the coefficient α, it is possible to adjust whether the control system 2 needs to avoid an excessively large amount of operation or needs responsiveness.

[0059] (Simulation: Obstacles) FIG. 8 is a graph of the reference value r2(t) when an obstacle that stops the second link is placed for the learning device 12 that has learned a step input as the target value d2(t). FIG. 9 is a graph of the manipulated variable u2(t) when an obstacle is placed for the learning device 12 that has learned a step input as the target value d2(t). FIG. 10 is a graph of the state variable q2(t) when an obstacle is placed for the learning device 12 that has learned a step input as the target value d2(t). The obstacle is removed at time 500 [ms]. The obstacle is an unknown disturbance for the control system 2. The graph for the coefficient α=0 is a comparative example.

[0060] As shown in Fig. 8, the reference value r2(t) is a constant value regardless of the presence or absence of an obstacle when the coefficient α = 0. In contrast, the reference value r2(t) changes (decreases) during the period when an obstacle is present when the coefficient α = 1.

[0061] As shown in Fig. 9, when the coefficient α = 0, the control device 20 outputs a constant and large manipulated variable u2(t) during the period when an obstacle is present. This is because the position does not change no matter how much the manipulated variable u2(t) is output.

[0062] When the coefficient α = 1, the control device 20 reduces the manipulated variable u2(t) during the period when an obstacle is present, more than when the coefficient α = 0. This is because the control is performed based on the reference value r2(t) that takes into account the state of the state variable q2(t).

[0063] Comparing the two, the peak torque when the coefficient α=1 is about half the peak torque when the coefficient α=0.

[0064] As shown in Figure 10, the state variable q2(t) remains constant until the obstacle is removed, and then converges to the target value d2(t) after the obstacle is removed. During this time, the coefficient α=0 converges faster than the coefficient α=1.

[0065] (Simulation: Stationary disturbance) FIG. 11 is a graph of the reference value r2(t) when a random disturbance is applied steadily to the learning device 12 that has learned a step input as the target value d2(t). FIG. 12 is a graph of the manipulated variable u2(t) when a random disturbance is applied steadily to the learning device 12 that has learned a step input as the target value d2(t). FIG. 13 is a graph of the state variable q2(t) when a random disturbance is applied steadily to the learning device 12 that has learned a step input as the target value d2(t). Gaussian noise is used for the random disturbance. The steady disturbance is an unknown disturbance to the control system 2. The graph with the coefficient α=0 is a comparative example.

[0066] As shown in Fig. 11, when the coefficient α = 0, the reference value r2(t) is a constant value of the learned step input. In contrast, when the coefficient α = 1, the reference value r2(t) is influenced by the state variable q2(t) that has changed due to the influence of the disturbance, and becomes a signal to which the disturbance has been added.

[0067] As shown in Fig. 12, when the coefficient α = 0, the control error e r In contrast, when the coefficient α=1, the manipulated variable u2(t) is calculated by subtracting the state variable q2(t) from the reference value r2(t) including the state variable q2(t), as described above, and therefore the disturbance can be offset. Therefore, when the coefficient α=1, stable control with reduced influence of disturbance can be realized. As a result, when the coefficient α=1, a relatively small manipulated variable can be realized.

[0068] As shown in Fig. 13, the state variable q2(t) becomes a signal containing noise due to the influence of disturbances. This tendency does not change significantly depending on the value of the coefficient α.

[0069] Therefore, by increasing the coefficient α, the control system 2 can reduce the influence of disturbances and realize control with a small amount of manipulation.

[0070] (Simulation: Stationary observation noise) FIG. 14 is a graph of the reference value r2(t) when a random observation noise is added steadily to the learning device 12 that has learned a step input as the target value d2(t). FIG. 15 is a graph of the manipulated variable u2(t) when a random observation noise is added steadily to the learning device 12 that has learned a step input as the target value d2(t). FIG. 16 is a graph of the state variable q2(t) when a random observation noise is added steadily to the learning device 12 that has learned a step input as the target value d2(t). Gaussian noise is used for the random observation noise. The observation noise is an unknown disturbance for the control system 2. The graph with the coefficient α=0 is a comparative example.

[0071] As shown in Fig. 14, when coefficient α = 0, reference value r2(t) is a constant value of the learned step input. In contrast, when coefficient α = 1, reference value r2(t) is influenced by state variable q2(t) that has changed due to the influence of observation noise, and becomes a signal to which a disturbance has been added.

[0072] As shown in Fig. 15, when the coefficient α = 0, the control deviation e r In contrast, when the coefficient α=1, the manipulated variable u2(t) is calculated by subtracting the state variable q2(t) from the reference value r2(t) including the state variable q2(t), as described above, and therefore the observation noise can be cancelled out. Therefore, when the coefficient α=1, stable control with reduced effects of observation noise can be achieved. As a result, when the coefficient α=1, control with a small manipulated variable can be achieved.

[0073] As shown in Fig. 16, when the coefficient α = 0, the operation amount u2(t) is a signal containing observation noise, and therefore the state variable q2(t) is also a signal containing observation noise. In contrast, when the coefficient α = 1, the operation amount u2(t) is a signal not containing observation noise, and therefore the state variable q2(t) is a stable signal.

[0074] Therefore, by increasing the coefficient α, the control system 2 can avoid an excessive amount of manipulation that reduces the influence of observation noise, and can achieve stable control with a small amount of manipulation.

[0075] (Simulation: Complex time series changes) The target value d2(t) is not limited to a step input. For example, a value that changes over time may be set as the target value d2(t). Specifically, the position of the other end of the second link is set to follow a predetermined trajectory (for example, the trajectory of a butterfly).

[0076] Even in such a case, the control system 2 can create a desired time series change by the correction amount o(t) output by the learning device 12. That is, the end points of the robot 30 can trace a predetermined trajectory. At this time, so-called disturbances such as an impulse external force, an obstacle, and observation noise may be applied to the robot 30.

[0077] When coefficient α=0, reference value r2(t) depends on time. That is, reference value r2(t) changes over time even while a disturbance is applied. Therefore, for example, if an obstacle that was installed is removed during the process, reference value r2(t) continues to change during the period in which the obstacle was installed, and the moment it is removed, the robot starts to move suddenly toward that point. During this time, operation amount u2(t) continues to change and becomes a large value.

[0078] On the other hand, when the coefficient α=1, the reference value r2(t) depends on the state. In other words, while a disturbance is applied, the reference value r2(t) does not change significantly because it is a reference value that matches the state. Therefore, for example, if an obstacle that was installed is removed midway, the reference value r2(t) for the period during which the obstacle was installed will be the reference value for changing from that state to the desired state at the next time. Therefore, the reference value r2(t) will be almost constant while the obstacle is installed, and even if it is removed, it will only start to change from that state to the desired state at the next time. In other words, when the coefficient α=1, it will not move suddenly and will operate safely. During this time, the operation amount u2(t) will be constant at a small value, which is energy-saving.

[0079] (summary) The reference value r(t) changes significantly depending on the value of the coefficient α. When coefficient α=0, the reference value r(t) is learned so as to create a time series change in the learned target value d(t). In contrast, when coefficient α=1, the time series change in the target value d(t) relative to the target value d(t-Δt) is learned. Therefore, the content being learned is significantly different, and the control results are also significantly different. When the coefficient is 0<α<1 (for example, α=0.5), the control results are a weighted mixture of the cases when coefficient α=0 and coefficient α=1.

[0080] In particular, when the coefficient α=1, the amount of operation is small. Since control is performed with a small amount of operation, for example, the torque of the robot 30 can be reduced. As a result, the control system 2 operates smoothly and is safe. Furthermore, when the coefficient α=1, even if the state variable q(t) contains so-called noise such as unknown disturbance or observation noise, the only term that enters the controller 15 is substantially the correction amount o(t), so stable control can be achieved.

[0081] Therefore, the control system 2 can achieve a smooth and adaptive response to an unknown disturbance by performing control with respect to the reference value r(t).

[0082] Generally speaking, as a method of restricting the manipulated variable u(t) so that it does not become too large, the control rule in the controller 15 may limit the manipulated variable u(t). In this case, the manipulated variable u(t) may suddenly change, and the differential value of each value may also suddenly change, resulting in discontinuous control. This may cause the robot 30 to operate with rapid acceleration, which may result in damage. In addition, discontinuous input is also given to the learner 12, which may prevent it from learning well.

[0083] On the other hand, in the control system 2, by performing control using a reservoir as the learning device 12 as in this embodiment, the direct effect of disturbances on the control deviation can be removed by explicitly adding a state variable to the reference value r(t), and an excessively large operation amount due to the effect of disturbances can be avoided. Therefore, the control system 2 can avoid generating discontinuities in each value, and therefore can realize control that is favorable for both the learning device 12 and the robot 30.

[0084] In this way, many advantages can be obtained by having the reference value determination unit 13 obtain the reference value r(t) by multiplying the state variable q(t) of the robot 30 by a coefficient greater than 0 and equal to or less than 1, and then adding the correction amount o(t).

[0085] The control system 2 uses a reservoir as the learning device 12, and the storage area of ​​the reservoir enables the control system 2 to calculate the correction amount o(t) based on the past state. As a result, the control system 2 can realize complex time series changes. Furthermore, the learning device 12 can learn and calculate the correction amount o(t) even if the state variable q(t) is mixed with observation noise, due to the noise robustness of the reservoir.

[0086] [Embodiment 3] Other embodiments of the present invention will be described below. For ease of explanation, the same reference numerals are given to members having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0087] In the second embodiment, the change in the control state according to the change in the coefficient α is shown. In this embodiment, the generalization of the learning is shown. In this embodiment, the coefficient α is set to 1. This is because it is necessary to learn the time series change depending on the state.

[0088] (Simulation: Learning Vector Fields) In the learning stage, the learning device 12 learned the time series variation of the state variable q(t) from a fixed initial value. Since the reservoir has high generalization performance, it is possible to apply a single time series variation to create other time series variations.

[0089] Fig. 17 shows the time series change in the tip position of the second link in the XY coordinate space when controlled with step inputs from different initial values ​​using a single learned time series change. Fig. 18 shows the time series change in the tip position of the second link in the joint angle space when controlled with step inputs from different initial values ​​using a single learned time series change.

[0090] 17 and 18, the dashed line represents a single time series change learned by the learning device 12. The multiple solid lines represent time series changes resulting from control using the results learned by the learning device 12, each having a different initial value. Note that the initial values ​​were set at regular intervals in the joint angle space.

[0091] 17 and 18, the control system 2 can control the reservoir to the same position as the end point of the learned time series change even when the control is started from different initial values. In other words, the reservoir has a generalization ability.

[0092] Fig. 19 shows time series changes in the tip position of the second link when controlled to draw a predetermined trajectory from different initial values ​​using a single learned time series change in the XY coordinate space. Fig. 20 shows time series changes in the tip position of the second link when controlled to draw a predetermined trajectory from different initial values ​​using a single learned time series change in the joint angle space.

[0093] 19 and 20, the dashed line represents a single time series change learned by the learning device 12. The multiple solid lines represent time series changes resulting from control using the results learned by the learning device 12, each having a different initial value.

[0094] As shown in Figures 19 and 20, it can be seen that the learning device 12 can memorize and draw a time series change in a complex trajectory (a butterfly model in Figures 19 and 20). It can also be seen that with regard to that trajectory, the control system 2 can merge with the nearest complex trajectory from the initial value and start drawing the complex trajectory from that point.

[0095] [Embodiment 4] Other embodiments of the present invention will be described below. For ease of explanation, the same reference numerals are given to members having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0096] In the first to third embodiments, a reservoir is used as the learning device 12. In the present embodiment, a case will be described in which a more general-purpose learning device is used.

[0097] Fig. 21 is a control block diagram showing another implementation example of the learning system 1. Fig. 22 is a control block diagram showing another implementation example of the control system 2. As shown in Figs. 21 and 22, the learning device 12 does not have to be a reservoir. Also, the controller 15 does not have to be a PD control device. Also, the learning system 1 and the control system 2 control the target device 30' instead of the robot 30.

[0098] (Learning Unit 12) In the learning system 1, a reference value r(t-Δt) is input to the learning device 12, and a target deviation e between the reference value r(t) and a target value d(t) for a state variable q(t) of the target device 30′ is calculated. d Using (t) as a teacher, the correction amount o(t) of the state variable q(t) of the target device 30' may be output.

[0099] In the control system 2, the learning device 12 is a learned learning device that receives the reference value r(t-Δt) and outputs a correction amount o(t) for the state variable q(t) of the target device 30'.

[0100] That is, the learning device 12 may be any learning device that can receive an input and output a desired output. For example, an extreme learning machine, deep learning, etc. are included. It is preferable that the learning device 12 can determine an output depending on past inputs in particular, that is, has a memory capacity for a time series.

[0101] (Reference value determination unit 13) The reference value determiner 13 may obtain the reference value r(t) by multiplying the state variable q(t) of the target device 30' by a coefficient greater than 0 and equal to or less than 1, and adding the correction amount o(t). In other words, the reference value r(t) obtained by the reference value determiner 13 becomes the reference value r(t-Δt) to be input to the learning device 12 at the next time.

[0102] (Controller 15) The controller 15 calculates the control error e between the reference value r(t) and the state variable q(t). r Based on (t), the manipulated variable u(t) of the target device 30' may be determined. In other words, the controller 15 may be any feedback control system. For example, the feedback control may be PID control, robust control, fuzzy control, or the like.

[0103] (Target device 30') The target device 30' controlled by the controller 15 is not limited to a robot. For example, it may be any output device such as a motor, a valve, or a heater. In particular, the target device 30' that includes disturbance or observation noise is preferable. Furthermore, even in the case of a robot, it is not limited to a robot arm, and may be a mobile robot.

[0104] Furthermore, the target value d(t), the reference value r(t), and the state variable q(t) are not limited to joint angles, but can take any state quantities such as position, velocity, acceleration, and force.

[0105] [Modifications] The learning device and the control device may include the state acquisition unit 11, the learning device 12, the reference value determination unit 13, and the output unit 14, and may not include the controller 15. In this case, the controller 15 may be a dedicated controller tailored to the robot 30.

[0106] In this case, the learning device or the control device is a control unit that acquires the state variable q(t) by the state acquisition unit 11 and outputs the reference value r(t). Note that the learning device 12 can also learn by inputting the target value d(t) to the control unit.

[0107] The control unit includes an output section 14 that outputs the reference value r(t) to a separately provided controller 15 that determines the operation amount u(t) of the robot 30 in response to an input of a control target value of the controller 15. The controller 15 separately acquires a state variable q(t) and calculates a control deviation e from the reference value r(t). r By obtaining (t), the robot 30 is controlled. Therefore, as the controller 15, a controller that has been conventionally used can be used.

[0108] 〔summary〕 The control device according to the first aspect of the present invention includes a trained learning device that receives a reference value and outputs a correction amount for a state variable of a target device, a reference value determination unit that calculates the sum of the state variable of the target device multiplied by a coefficient greater than 0 and equal to or less than 1 and the correction amount as the reference value, and an output unit that outputs the reference value to a controller that determines the operation amount of the target device. According to the above configuration, even if a disturbance is input and the state variable changes, a reference value that takes the state variable into consideration is determined. This reference value is used as a control target value in a controller that is resistant to disturbances for the target device. Therefore, the control device can respond flexibly to disturbances.

[0109] The control device according to aspect 2 of the present invention may be configured in the above aspect 1 to further include the controller, and the controller may be configured to determine the manipulated variable of the target device based on the deviation between the reference value and the state variable. With the above configuration, the controller can control the target device so as to reduce the deviation between the reference value and the state variable.

[0110] The control device according to aspect 3 of the present invention may be configured in the above aspect 1 or 2 such that the learning device has a storage capacity for time series. With the above configuration, the learning device has a storage capacity. Therefore, it is possible to calculate a correction amount taking into account a past state.

[0111] The control device according to aspect 4 of the present invention may be configured such that, in any one of aspects 1 to 3, the learning device is a reservoir. According to the above configuration, by using a reservoir as a learning device, it is possible to realize a large memory capacity and high noise robustness performance at a low learning cost.

[0112] A control device according to a fifth aspect of the present invention may be configured in any one of the first to fourth aspects above, such that the reference value determined by the reference value determination unit is the reference value to be input to the learning device at the next time. With the above configuration, the correction amount at the next time can be determined based on the past reference value. Therefore, the correction amount at the next time is determined taking into account the state variables, making it possible to perform state-dependent control.

[0113] The control device according to a sixth aspect of the present invention may be configured such that a disturbance is applied to the manipulated variable in any one of the first to fifth aspects. With the above configuration, control can be performed even in an environment where a disturbance is applied to the target device.

[0114] The control device according to a seventh aspect of the present invention may be configured such that noise is added to the state variables in any one of the above-mentioned aspects 1 to 6. With the above-mentioned configuration, even if the state variables contain noise, control can be performed by the noise robustness performance of the learning device.

[0115] A learning device according to an eighth aspect of the present invention includes a learning device that receives a reference value, uses the deviation between the reference value and a target value for a state variable of a target device as a teacher, and outputs a correction amount for the state variable of the target device, a reference value determination unit that calculates the sum of the state variable of the target device multiplied by a coefficient greater than 0 and equal to or less than 1 and the correction amount as the reference value, and an output unit that outputs the reference value to a controller that determines the operation amount of the target device. According to the above configuration, a correction amount that takes into account the state variable can be determined, and a reference value is determined accordingly. This reference value is used as a control target value in a controller that is resistant to disturbances on the target device. Therefore, the learning device can respond flexibly to disturbances.

[0116] The learning device of aspect 9 of the present invention, in accordance with aspect 8 above, may further include the controller, and the controller may be configured to determine the manipulated variable of the target equipment based on the deviation between the reference value and the state variable.

[0117] A learning device according to a tenth aspect of the present invention may be configured in the above eighth or ninth aspect, wherein the learning device has a storage capacity for time series.

[0118] A learning device according to an eleventh aspect of the present invention may be configured in any one of the eighth to tenth aspects above, wherein the learning device is a reservoir.

[0119] A learning device according to a twelfth aspect of the present invention may be configured in any one of the eighth to eleventh aspects above, such that the reference value determined by the reference value determination unit is the reference value to be input to the learning device at the next time. With the above configuration, the correction amount at the next time can be determined based on the past reference value. Therefore, the correction amount at the next time is determined taking into account the state variable, making it possible to perform state-dependent control.

[0120] The learning device according to a thirteenth aspect of the present invention may be configured in any one of the eighth to twelfth aspects above, wherein noise is added to the state variables.

[0121] A control method according to aspect 14 of the present invention is a method including a learning application step in which a reference value is input and a trained learning device that calculates a correction amount for a state variable of a target equipment is caused to output the correction amount; a reference value determination step in which the reference value is calculated by multiplying the state variable of the target equipment by a coefficient greater than 0 and equal to or less than 1 and the correction amount; and an output step in which the reference value is output to a controller that determines the operation amount of the target equipment.

[0122] A learning method according to aspect 15 of the present invention is a method including a learning step in which a reference value is input, the deviation between the reference value and a target value for a state variable of a target device is used as a teacher, and a correction amount for the state variable of the target device is output, a reference value determination step in which the reference value is calculated by multiplying the state variable of the target device by a coefficient greater than 0 and equal to or less than 1, and the correction amount, and an output step in which the reference value is output to a controller that determines the operation amount of the target device.

[0123] [Software implementation example] The functions of the learning device 10 or the control device 20 (hereinafter referred to as the "device") can be realized by a program for causing a computer to function as the device, and a program for causing a computer to function as each control block of the device (in particular, the state acquisition unit 11, the learning device 12, the reference value determination unit 13, the output unit 14 and the controller 15).

[0124] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program to realize each function described in each of the above embodiments.

[0125] The program may be non-transitory and may be recorded in one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be provided to the device via any wired or wireless transmission medium.

[0126] In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0127] [Additional Notes] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0128] 1. Learning System 2. Control System 10 Learning Device 11 Status acquisition section 12 Learning Units 13 Reference value determination unit 14 Output section 15 Controller 20 Control device 30 Robot (target equipment) 30' Target equipment

Claims

1. A trained learning device that receives a reference value and outputs a correction amount for a state variable of a target device; a reference value determination unit that determines, as the reference value, the sum of a value obtained by multiplying a state variable of the target device by a coefficient greater than 0 and equal to or less than 1 and the correction amount; an output unit that outputs the reference value to a controller that determines an operation amount of the target device; A control device comprising:

2. The controller further comprises: The control device according to claim 1 , wherein the controller determines the manipulated variable of the target device based on a deviation between the reference value and the state variable.

3. The control device according to claim 1 , wherein the learning device has a memory capacity in a time series.

4. The control device of claim 1 , wherein the learner is a reservoir.

5. The control device according to claim 1 , wherein the reference value determined by the reference value determination unit is the reference value to be input to the learning device at a next time point.

6. The control device according to claim 1 , wherein a disturbance is added to the manipulated variable.

7. The control device according to claim 1 , wherein the state variables are noise-injected.

8. a learning device to which a reference value is input, which uses a deviation between the reference value and a target value for a state variable of a target device as a teacher, and which outputs a correction amount for the state variable of the target device; a reference value determination unit that determines, as the reference value, the sum of a value obtained by multiplying a state variable of the target device by a coefficient greater than 0 and equal to or less than 1 and the correction amount; An output unit that outputs the reference value to a controller that determines an amount of operation of the target device.

9. The controller further comprises: The learning device according to claim 8 , wherein the controller determines the manipulated variable of the target device based on a deviation between the reference value and the state variable.

10. The learning device according to claim 8 , wherein the learning device has a memory capacity in a time series.

11. The learning apparatus of claim 8 , wherein the learner is a reservoir.

12. The learning device according to claim 8 , wherein the reference value determined by the reference value determination unit becomes the reference value to be input to the learning device at a next time point.

13. The learning device according to claim 8 , wherein noise is added to the state variables.

14. a learning application step of inputting a reference value and causing a trained learning device to calculate a correction amount for a state variable of a target device to output the correction amount; a reference value determining step of determining, as the reference value, the sum of a value obtained by multiplying a state variable of the target device by a coefficient greater than 0 and equal to or less than 1 and the correction amount; and an output step of outputting the reference value to a controller that determines an amount of operation of the target device.

15. a learning step in which a reference value is input, a deviation between the reference value and a target value for a state variable of a target device is used as a teacher, and a correction amount for the state variable of the target device is output; a reference value determining step of determining, as the reference value, the sum of a value obtained by multiplying a state variable of the target device by a coefficient greater than 0 and equal to or less than 1 and the correction amount; and an output step of outputting the reference value to a controller that determines an amount of operation of the target device.

Images