Learning device, learning method, and program

Abstract

A method for calculating a control input with high accuracy is provided. [Solution] The learning device comprises: a means for acquiring a differentiable model that represents the temporal change in the state of the controlled system in response to a control input to the controlled system; a means for learning a plurality of weak policies, which are models that output a control input indicating a control command value to the controlled system when the state of the controlled system is input, using PODS with the differentiable model after giving different learning conditions to each of the weak policies; and a means for calculating a meta-policy by integrating the learned plurality of weak policies.

Description

[Technical Field]

[0001] The present disclosure relates to a learning device, a learning method, and a program. [Background technology]

[0002] The time-dependent change in the state of the controlled system is expressed as the difference equation x k+1 =f(x k , a k ) or ordinary differential equation dx / dt=f(x(t), a(t)) from the current state to the desired state. Optimal control is a method for controlling a dynamic system that can be expressed by the equation dx / dt=f(x(t), a(t)) from the current state to the desired state. In optimal control, the state x at each time k is calculated according to the control specifications (the desired system state). k and the control input a, which indicates the control command value to be input to the controlled system. k and the reward function R(x k , a k ) time series sum ΣR(x k , a k ) is large. k (k=0, 1, 2...) is calculated. Methods for determining a control input sequence include Model Predictive Control (MPC) and Reinforcement Learning (RL). In addition to these methods, Non-Patent Document 1 discloses "Policy Optimization via Differentiable Simulator (PODS)" as a method for efficiently calculating a control input sequence. However, with reinforcement learning and PODS, there is a possibility that an undesirable control input a may be calculated for a state x that is not included in the data at the time of learning, which may result in a deterioration in control performance. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] MAZMora et al., “PODS: Policy Optimization via Differentiable Simulation,” Proc. of ICML 2021, pp. 7805-7817, 2021. Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present disclosure is to provide a device that determines an appropriate control input for the state of a system to be controlled.

[0005] Therefore, an object of the present disclosure is to provide a learning device, a learning method, and a program that solve the above-mentioned problems. [Means for solving the problem]

[0006] According to one aspect of the present disclosure, a learning device includes a means for acquiring a differentiable model that represents the temporal change in the state of a controlled system in response to a control input to the controlled system, a means for learning a plurality of weak policies, which are models that output a control input to the controlled system when the state of the controlled system is input, using PODS with the differentiable model after assigning different learning conditions to each of the weak policies, and a means for calculating a meta-policy by integrating the learned plurality of weak policies.

[0007] According to one aspect of the present disclosure, a learning method is provided in which a computer obtains a differentiable model that represents the temporal change in the state of a controlled system in response to a control input to the controlled system, assigns different learning conditions to each of a plurality of weak policies, which are models that output a control input to the controlled system when the state of the controlled system is input, and then learns the plurality of weak policies using PODS using the differentiable model, and calculates a meta-policy by integrating the learned plurality of weak policies.

[0008] According to one aspect of the present disclosure, the program causes a computer to function as: a means for obtaining a differentiable model that represents the temporal change in the state of a controlled system in response to a control input to the controlled system; a means for learning a plurality of weak policies, which are models that output a control input to the controlled system in response to an input of the state of the controlled system, by applying different learning conditions to each of the weak policies using PODS using the differentiable model; and a means for calculating a meta-policy by integrating the learned plurality of weak policies. [Effects of the Invention]

[0009] According to the learning device, learning method, and program of the present disclosure, it is possible to determine an appropriate control input for the state of a system to be controlled. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an example of a dynamic system. [Figure 2] FIG. 2 is a block diagram illustrating an example of a functional configuration of a learning device. [Figure 3] FIG. 10 is a diagram illustrating an outline of a learning process. [Figure 4] FIG. 1 is a diagram illustrating offline meta-policy optimization. [Figure 5] 10 is a flowchart illustrating an example of a learning process. [Figure 6] FIG. 1 is a diagram illustrating an outline of PODS. [Figure 7] FIG. 2 is a block diagram showing an example of the minimum configuration of a learning device. [Figure 8] 10 is a flowchart illustrating an example of the operation of a learning device having a minimum configuration. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, a learning device according to each embodiment of the present disclosure will be described with reference to the drawings. In the drawings used in the following description, the configuration of parts that are not related to the present disclosure may be omitted or not illustrated.

[0012] (Premise) As mentioned above, methods for determining control input sequences include model predictive control and reinforcement learning. Model predictive control is a method of determining control inputs by solving optimization problems related to control within a relatively short time period at each time point using a model f that predicts the behavior of the system constructed in advance. Reinforcement learning can be divided into model-based reinforcement learning, which explicitly learns a model of the system, and model-free reinforcement learning, which does not explicitly learn a model. Model-free reinforcement learning does not require learning a model of the system, making it useful for complex, high-dimensional systems where modeling is not realistic. However, since model-free reinforcement learning does not require learning a model of the system, it learns the optimal control policy by trial and error in a situation where no prior knowledge is available, so learning efficiency is low and it takes a long time for the learning results to converge. Model-based reinforcement learning requires the effort of modeling the system, but it avoids random search and enables efficient policy learning using prediction results. Another advantage is that once a system model is learned, it can be reused when learning other control specifications. For example, the above-mentioned x k+1 =f(x k , a k ), model-based reinforcement learning uses the state x k When input is made, the control input a according to the state is k A neural network strategy π that returns θ (a k =π θ (x k )), the time series reward sum ΣR(x k , a k ) is large. θ There is a method to update the parameter θ of the neural network without random search using the system model f. To update the parameter θ of the neural network, it is possible to update it directly using BPTT (Backpropagation through time), but this method is known to be prone to gradient explosion and gradient vanishing because it backpropagates gradient information multiple times depending on the length K of the reward time series to be evaluated (called the control horizon K).

[0013] In contrast, Non-Patent Document 1 does not directly update the parameter θ of the policy neural network like BPTT, but instead uses a model that is differentiable with respect to the control input (for example, x k+1 =f(x k , a k ) and use π θ The control input sequence A=[a0, a1, a2, . . ., a K-1 ] itself is updated in the direction of increasing the sum of rewards by the following equation (1), and the neural network policy π θ PODS, a method for learning the parameter θ, is disclosed.

[0014]

number

[0015] The flow of the PODS algorithm will be briefly explained with reference to Figure 6. For simplicity, the explanation will be given assuming one sample in the initial state. (1) Select the initial policy parameters θ and create a policy neural network π θ Also, set the initial state to x0 randomly sampled from the initial state distribution p0. (2) Differentiable model x with respect to the initial state x0 k+1 =f(x k , a k ) and π θ Using the state sequence X=[x1, x2, x3, . . ., x K ] and the control input sequence A=[a0, a1, a2, , a K-1 ] to get. (3) Reward sum V=Σ k R(x k , π θ (x k)) by the control input sequence A, dV / dA=(∂V / ∂A)·(dX / dA)+∂A / ∂X is calculated by automatic differentiation (a differentiable model is required to calculate (dX / dA) at this time), and the control input sequence A is updated using the above formula (1). In other words, the control input sequence is slightly moved in the direction in which the reward sum becomes larger than it is now, and the updated control input sequence A+ΔA=[a0', a1', a2', , a K-1 ´]=[a0+Δa0, a1+Δa1, a2+Δa2,..., a K-1 +Δa K-1 (ΔA represents dV / dA obtained by automatic differentiation.) (4) State x k The newly obtained a k +Δa k The policy π is used to output θ The parameter θ is updated by supervised learning. For example, θ is updated so that the loss function of the following equation (2) becomes smaller.

[0016]

number

[0017] Desired policy π θ The following steps (2) to (4) are repeated until the following is obtained. In this PODS, if a differentiable model of the control object is constructed, the neural network π θBecause PODS can update the parameter θ, it has the advantage of being able to learn policies more efficiently than reinforcement learning and is less susceptible to gradient explosion and gradient vanishing. On the other hand, PODS (a) presupposes the existence of an existing differentiable model. Therefore, it cannot be applied when a differentiable model does not exist. (b) Because it improves the policy near the current policy—that is, it updates the control input sequence (Equation (1)) in a direction that results in a small performance improvement—and uses this as training data to learn the parameter θ, it is prone to relying on the initially selected parameters and falling into a local optimum. (c) If the range of states x used to learn the policy parameter θ is biased (biased toward learning control near the optimal state sequence predicted by the differentiable model), inputting a state x in a range not included in the training data may result in the calculation of an abnormal control input a. (d) Because it is an algorithm that optimizes control over a control horizon K, control performance is only guaranteed within a predetermined control horizon K. These weaknesses include:

[0018] <Embodiment> (composition) 1 is a diagram illustrating an example of a dynamic system. The dynamic system 100 includes a controlled system 1, a controller 2 that controls the controlled system 1, and an optimal control input a given to the controller 2. k The learning device 10 may be included in the controller 2 or may be provided separately from the controller 2. k+1 is x k+1 =f(x k , a k ) can be predicted by the controller 2. The controller 2 predicts the state x of the controlled system 1 at time k. k and state x k is passed to the learning device 10. The learning device 10 then k The optimal control input a according to k Calculate the control input a k The controller 2 receives the control input a from the learning device 10. kis input to the controlled system 1. The controller 2 repeats this process for each control step to control the controlled system 1.

[0019] Fig. 2 is a block diagram showing an example of the functional configuration of the learning device 10. Fig. 3 is a diagram showing an overview of the learning process by the learning device 10. The learning device 10 uses a differentiable model of the system 1 to be controlled to utilize PODS to calculate a neural network policy π that outputs an appropriate control input a for the state of the system to be controlled while compensating for the weaknesses (a) to (d) of the PODS. The learning device 10 includes a differentiable model acquisition unit 11, a weak policy calculation unit 12, and a meta-policy calculation unit 13.

[0020] The differentiable model acquisition unit 11 acquires a differentiable model of the controlled system 1. The differentiable model is a model that can be expressed by a difference equation x k+1 =f(x k , a k ) and the ordinary differential equation dx / dt=f(x(t), a(t)), the internal state x of the controlled system 1 k A model that represents the time change of the control input a k The model is a model such as a simulator or neural network that is differentiable in time. If such a model already exists, the differentiable model acquisition unit 11 acquires the existing differentiable model. If no differentiable model exists, the differentiable model acquisition unit 11 constructs a differentiable model (block 21 in FIG. 3). For example, when a state x k and control input a k Alternatively, a time-series data set of x and x may be collected, or a similar data set may be collected from a simulator that simulates the controlled system 1, and the collected data set may be used as training data to construct a machine learning model that is differentiable with respect to the control input, such as a neural network, such as a Multi-layer Perceptron (MLP) or Neural Ordinary Differential Equations (Neural ODE). The constructed neural network may, for example, k and control input ak When you enter the state x k+1 In this case, the differentiable model acquisition unit 11 may model only the functions necessary for controlling the controlled system 1. Furthermore, the constructed differentiable model can be reused even if the control specifications are changed. The differentiable model acquisition unit 11 makes it possible to use PODS even if there is no existing differentiable model. This makes it possible to overcome the weakness (a) of PODS mentioned above.

[0021] The weak policy calculation unit 12 calculates the initial policy π θj Prepare multiple (j=1, 2, . . . , M) and set various initial conditions. Asynchronously (individually, independently of each other), the PODS policy π θj Repeated learning of multiple policies π θj is called a weak policy. For example, the weak policy calculation unit 12 sets various learning conditions for each weak policy, such as policy parameters, hyperparameters, number of layers, number of units, architecture, regularization, activation function, and initial state distribution (how to set the initial value of the state x). Then, the weak policy calculation unit 12 learns each weak policy using PODS. That is, the weak policy calculation unit 12 learns the current weak policy π θj Using this, we convert the initial state x into the weak policy π θj The control input a is calculated by inputting the control input a to the differentiable model to predict the state x at the next time, and the predicted state x is calculated as the weak policy π θjBy repeating this process, a time-series state sequence and a time-series control input sequence are obtained. The weak policy calculation unit 12 also obtains, by automatic differentiation, a differential value with respect to the control input sequence A of the reward sum obtained from the time-series state sequence, the time-series control input sequence, and the reward function R (Equation (1)). As a simple example, the reward function R may be configured to output a higher reward the closer the state obtained as a result of inputting the control input a to the controlled system 1 is to the ideal state. Next, the weak policy calculation unit 12 updates the control input sequence A in a direction that increases the reward sum (the direction of the differential value with respect to the control input sequence A), and calculates a plurality of weak policies π so as to output the updated control input sequence A+ΔA when the time-series state sequence is input. θj Each of these is learned by supervised learning. As a result, while general PODS tends to have bias in the learning range, the weak policy π outputs a control input a that is effective in each range of various states x. θj You can get multiple.

[0022] The meta-policy calculation unit 13 calculates a policy (referred to as a meta-policy) for outputting a control input a that is actually input to the controlled system 1, and calculates the control input a using the calculated meta-policy. More specifically, the meta-policy calculation unit 13 calculates the control input a using the asynchronously learned weak policy π θj The meta-policy π is calculated by weighting each of the (j=1, 2, . . . , M) αθ (Block 23 in FIG. 3). The meta-strategy can be expressed by the following equation (3a) or (3b): α ψ (j) (x) and α (j) is the weak policy π θj is the weight for α ψ (j) (x) and α (j) are normalized to sum to 1 over j.

[0023]

number

[0024]

number

[0025] In the case of equation (3b), the weight α (j) In the case of equation (3a), the weight α φ (j) (x) is set according to the state x. For example, if a weak policy learned using an initial state close to a certain state x1 as the learning condition is π θ1 and π θ2 If so, then π θ1 Weight α for ψ (1) and π θ2 Weight α for ψ (2) is set to a relatively large value, and other weak policies α ψ (j) A small weight α ψ (j) Regarding equation (3a), the weight α according to the state x is set. ψ (j) The method for calculating (x) will be described later. The meta-strategy calculation unit 13 calculates the meta-strategy π αθ The control input a to be actually input to the controlled system 1 is calculated using the above. The weak policy calculation unit 12 performs ensemble policy optimization to obtain a plurality of diverse policies, and the meta policy calculation unit 13 appropriately weights the plurality of weak policies π θj By combining these (meta-policy optimization), the weight of the policies that fall into local optima can be reduced, and further, each weak policy π θj It is possible to realize a policy that can calculate appropriate control inputs for a wider range of states than a single entity. Furthermore, even after a weak policy has finished controlling the control horizon K, it is possible to achieve control for a longer period than the control horizon K by switching to a weak policy that can handle the state as the initial state. This overcomes the weaknesses (b), (c), and (d) of PODS mentioned above.

[0026] <Weight of meta-strategy α ψ (j) (x) Adjustment> Next, the weight of the meta-policy α ψ (j) Here, we will explain the adjustment of (x). ψ (j) "Offline meta-policy optimization" that adjusts (x) and online α ψ (j) We will explain "online meta-policy optimization" to adjust (x). ψ (j) The adjustment of (x) is performed by the weight adjustment unit 131 included in the meta-policy calculation unit 13.

[0027] (A) Offline meta-policy optimization Weak policy π θ1 , π θ2 , , π θM The weight adjustment unit 131 adjusts the weight α of each weak policy for a certain state x. ψ (j) Model α for calculating (x) ψ (x)=[α ψ (1) (x), , α ψ (M) (x)] is learned by PODS. In other words, it is considered as a policy for calculating the weight of the weak policy for the state of the controlled system, and the policy parameter ψ is updated. Figure 4 shows the architecture of the meta-policy weight adjustment neural network. Model α ψ (x) is the neural network g ψ (x) and the Softmax function. Neural network g ψ When you input the state x into (x), g ψ (x) is the real value (weight before normalization) z j (x)(j=1, 2, . . . , M) and output z j (x) is normalized by the Softmax function, and Σα ψ (j) Weight α such that (x)=1 ψ (j) (x) (j=1, 2, . . . , M) is calculated. ψWhen learning (x) using PODS, the differentiable model representing the state transition of the controlled system can be expressed by the following equation (4).

[0028]

number

[0029] The reward sum for offline meta-policy optimization can be expressed by the following equation (5). The reward function R used to calculate the reward sum can be the same as the reward function used when the weak policy calculation unit 12 learns a weak policy using PODS. In the following equation (5), q0 represents the initial state distribution for offline meta-policy optimization, and h represents the horizon (weight adjustment horizon) for offline meta-policy optimization.

[0030]

number

[0031] PODS's training of the weak policy weights is performed as follows. For simplicity, we will explain the initial state with one sample. (1) Select the initial parameter ψ and create a neural network g ψ Also, set the initial state x0 to a state randomly selected from the initial state distribution q0 for adjusting the meta-policy weights. (2) For the initial state x0, g ψ Using the real-valued vector z0=[z0 (1) , z0 (2) , z0 (3) , , z0 (M) ] to get. (3) Initial state x0 and z0=[z0 (1) , z0 (2) , z0 (3) , , z0 (M) ] into the differentiable model of equation (4) to calculate the state x1. The calculated state x1 and z1 = [z1 (1) , z1 (2) , z1 (3) , , z1 (M)] into equation (4) to calculate the state x2. h This is repeated until the initial g ψ (x) and the weight α calculated by the Softmax function ψ Using the differentiable model of equation (4), x0~x of the controlled system h Predict the time series state up to (4) The weight sequence Z = [z0, z1, z2, . . . , z h-1 ] is calculated by automatic differentiation, Z is updated in the direction that increases the reward sum, and Z' = [z0', z1', , z h-1 ´]=[z0+Δz 0、 z1+Δz 1、···、 z h―1 +Δz h―1 ] is obtained. (Δz k (k=0, 1, ..., h-1) is obtained by automatic differentiation k ) (5) State x k The newly obtained z k g to output ´ ψ The parameter ψ of (x) is updated by supervised learning. For example, the neural network g is updated so that the loss function of the following equation (6) becomes small. ψ Update the parameter ψ of (x).

[0032]

number

[0033] The weight adjustment unit 131 adjusts the updated g ψ Repeat steps (3) to (5) using (x), update parameter ψ, and then calculate g ψ (x) is trained. This allows the neural network g to calculate appropriate weights for the state of the controlled system. ψ (x) (Model α ψ (x)) is obtained. Before actually starting the control of the control target system 1, the weight adjustment unit 131 performs the above process in advance to obtain α ψ(j) Then, the meta-policy calculation unit 13 calculates the state x (x) every moment during actual control. k The weak measure calculation unit 12 calculates the optimized control input value a by the formula (3a) according to the above. θj For each of the states x k The control input value a can be calculated by weighting according to the parameter, and improvement of control performance can be expected.

[0034] (B) Online meta-policy optimization Weak policy π θ1 , π θ2 , , π θM is assumed to be sufficiently studied. (1) The weight adjustment unit 131 calculates a real-valued vector (weight before normalization) z=[z (1) , z (2) , z (3) , , z (M) ] can be set as desired. (2) The weight adjuster 131 adjusts the state x of the controlled system online. k and calculate the state x using the following equation (7b): k+1 and predict the predicted state x k+1 Substituting into equation (7b), the state x k+2 and calculate the state x k+h h is the weight adjustment horizon, which represents the period for which predictions are made for weight adjustment. (3) The weight adjustment unit 131 adjusts the real-valued vector z representing the weight before normalization in the direction in which the evaluation value J(z) expressed by the following equation (7a) increases. As a result, the state x k The weights are adjusted depending on

[0035]

number

[0036] The weight adjustment horizon h (for example, 10 steps) is set smaller than the control horizon K (for example, 100 steps). In the calculation of the evaluation value J(z) in Equation (7a), the period of the weight adjustment horizon h is set to the weight α z (j) does not change (each π θj For example, the weight adjuster 131 predicts the state change when the weight of the current meta-policy is fixed at each time k and controlled for 10 steps using the differentiable model of equation (7b), calculates the evaluation value J(z) calculated from the state transition prediction for those 10 steps, calculates the update direction of z that will increase the evaluation value using automatic differentiation, and updates z in that direction. Then, the meta-policy calculator 13 calculates the state x of the controlled system at that time. k and the weight α obtained by normalizing z updated by the weight adjustment unit 131 using the Softmax function. ψ (j) (x k ) and using equation (3a), the control input a k The weight adjustment unit 131 executes (1) to (3) at each time and calculates α ψ (j) (x) is calculated. This allows the state x of the controlled system to be calculated. k The control input value a can be calculated by sequentially adjusting the weights using the h-step state transition prediction, which is expected to improve control performance.

[0037] (operation) FIG. 5 is a flowchart showing an example of the learning process. First, the user determines whether a differentiable model for the controlled system 1 exists (step S1). If a differentiable model does not exist, the differentiable model acquisition unit 11 constructs a differentiable model (step S2). For example, if there is only a non-differentiable simulator for the controlled system 1, a large number of data sets of control inputs to the simulator and the states of the controlled system 1 output by the simulator in response to the control inputs are collected, and a replica of the simulator is created using MLP or Neural ODE. This replica can be used as a differentiable model. If an existing differentiable model exists (step S1; Yes) or if a differentiable model has been constructed (step S2), the differentiable model acquisition unit 11 acquires the differentiable model (step S3). Next, the weak policy calculation unit 12 calculates multiple weak policies (step S4). The weak policy calculation unit 12 calculates a weak policy π of the neural network. θ We prepare multiple weak policies π by varying the conditions such as policy parameters and initial state. θj The learning condition is set as follows. At this time, in order to learn a policy that is robust with respect to the error between the actual machine and the differentiable model, learning may be performed by changing some parameters of the differentiable model. The weak policy calculation unit 12 calculates a plurality of weak policies π θj Next, the meta-policy calculation unit 13 calculates a meta-policy (step S5). The meta-policy calculation unit 13 learns the weak policy π θj When using equation (3a), the weight adjuster 131 calculates a meta-policy by equation (3a) or equation (3b) using the above. When equation (3a) is used, the weight adjuster 131 optimizes a plurality of weak policies π for each state x by offline meta-policy optimization or online meta-policy optimization. θj Next, the meta-policy calculation unit 13 calculates a control input using the meta-policy (step S6). The meta-policy calculation unit 13 substitutes the state x of the controlled system 1 into the meta-policy calculated in step S5, and calculates a control input a according to the state x. The control input a is input to the controlled system 1 by the controller 2, and the controlled system 1 is controlled. Step S6 is repeatedly executed during the control of the controlled system 1.

[0038] (effect) According to this embodiment, PODS can be used to calculate a meta-policy that outputs appropriate control inputs for the current state of the target system. This compensates for the weaknesses of PODS, such as (a) the need for a differentiable model, (b) a tendency to fall into a local optimum, (c) the possibility of calculating abnormal control inputs when inputting states in a region not included in the training data, and (d) the ability to guarantee control performance only within a pre-specified control horizon. That is, even if a differentiable model does not exist, a differentiable model can be constructed, and a meta-policy can be calculated using PODS with the constructed differentiable model. Furthermore, by learning multiple weak policies, it is possible to more easily avoid falling into a local optimum compared to learning a policy based on only one learning condition. Furthermore, by appropriately adjusting the weights of weak policies learned under various learning conditions and calculating the weighted sum, it is possible to avoid outputting abnormal control inputs even when a single weak policy is insufficiently learned. Furthermore, after control within a specified control horizon is completed, the weights of weak policies that can respond to the target system's current state as the initial state can be increased, thereby achieving control for a longer period than the specified control horizon. In addition, by calculating a weak policy using PODS, we can enjoy the advantages of PODS. For example, since we can update the policy parameter θ without collecting training data through random search, we can obtain a policy more efficiently than reinforcement learning, and gradient explosion and vanishing are less likely to occur.

[0039] The controlled system 1 to which this embodiment is applied may be any system. For example, the controlled system 1 may be a robot, various types of equipment such as a chemical plant, or an unmanned mobile object. It may also be applied to various control algorithms such as an inverted pendulum. For example, in the case of a robot, the control input a is command information for the robot's operation, and the state x is the position and posture of the robot. In the case of a chemical plant, the control input a is command information such as valve opening and pump start / stop, and the state x is the temperature, pressure, etc. at various locations in the plant. In the case of an inverted pendulum, the control input a is the force applied to the pendulum, and the state x is the angle, angular velocity, etc. of the pendulum.

[0040] FIG. 7 is a block diagram showing an example of the minimum configuration of a learning device. The learning device 800 includes an acquisition means 810 that acquires a differentiable model that represents a temporal change in the state of a controlled system; a weak policy learning means 820 that assigns different learning conditions to each of a plurality of weak policies, each of which is a model (e.g., a neural network) that receives a state of the controlled system and outputs a control input indicating a control command value for the controlled system, and then learns the plurality of weak policies by PODS using the differentiable model; and a meta-policy calculation means 830 that calculates a meta-policy by integrating the learned plurality of weak policies. The learning device 10 is an example of the learning device 800. The differentiable model acquisition unit 11 is an example of the acquisition means 810. The weak policy calculation unit 12 is an example of the weak policy learning means 820. The meta-policy calculation unit 13 is an example of the meta-policy calculation means 830.

[0041] FIG. 8 is a flowchart showing an example of the operation of a learning device having a minimum configuration. The acquisition means 810 acquires a differentiable model that represents changes in the state of the controlled system over time (step S801), the weak policy learning means 820 assigns different learning conditions to each of a plurality of weak policies, which are neural networks that input the state of the controlled system and output control inputs that indicate control command values ​​for the controlled system, and then learns the plurality of weak policies using PODS that uses the differentiable model (step S802), and the meta-policy calculation means 830 calculates a meta-policy that is a sum of the learned plurality of weak policies by integrating them (step S803). More specifically, the learned plurality of weak policies are integrated, and a weight is assigned to each of them, and they are summed to calculate the meta-policy.

[0042] Note that a part of the learning device 10, 800 in the above-described embodiments may be implemented by a computer. In this case, a program for implementing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read and executed by a computer system. Note that the "computer system" here refers to a computer system built into the learning device 10, 800, and includes hardware such as an OS (Operating System) and peripheral devices.

[0043] Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over networks like the Internet or communication lines like telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within computer systems that serve as servers or clients in such cases. Furthermore, the above-mentioned programs may be programs that realize some of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already stored in the computer system.

[0044] Furthermore, part or all of the learning devices 10 and 800 in the above-described embodiments may be realized as integrated circuits such as LSI (Large Scale Integration). Each functional unit of the learning devices 10 and 800 may be individually implemented as a processor, or part or all of them may be integrated into a processor. The integrated circuit implementation method is not limited to LSI, and may be implemented using dedicated circuits or general-purpose processors. Furthermore, if an integrated circuit implementation technology that can replace LSI emerges due to advances in semiconductor technology, an integrated circuit based on that technology may be used.

[0045] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design modifications can be made without departing from the spirit of the present invention. Furthermore, one aspect of the present invention can be modified in various ways within the scope of the claims, and embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. Furthermore, configurations in which elements described in the above embodiments and variations are substituted with elements that achieve the same effect are also included.

[0046] <Additional Notes> The learning device, learning method, and program described in the embodiments can be understood, for example, as follows.

[0047] (1) A learning device according to a first aspect includes a means for acquiring a differentiable model that represents the temporal change in the state of a controlled system in response to a control input to the controlled system; a means for learning a plurality of weak policies by PODS using the differentiable model after giving different learning conditions to each of the weak policies, which are models that output a control input indicating a control command value to the controlled system when the state of the controlled system is input; and a means for calculating a meta-policy by integrating the learned plurality of weak policies.

[0048] (2) A learning device according to a second aspect is a learning device according to (1), further comprising means for collecting time series data of the state and the control input at time k from the controlled system or a simulator of the controlled system, and constructing the differentiable model that is differentiable with respect to the control input, and that outputs the state at time k+1 when the state and the control input at time k are input.

[0049] (3) A learning device according to a third aspect is the learning device according to any one of (1) to (2), further comprising means for calculating the control input by inputting the state into the meta-policy.

[0050] (4) A fourth aspect of the learning device is the learning device according to any one of (1) to (3), wherein the means for calculating the meta-policy assigns a weight to each of the plurality of weak policies that have been learned, and calculates the meta-policy by summing the weights; The means for calculating the meta-strategy further includes means for calculating the weight according to the state.

[0051] (5) A learning device according to a fifth aspect is a learning device as described in (4), wherein the means for calculating the weight according to the state searches for a value of the weight assigned to each of the plurality of weak policies such that a high reward is calculated by a reward function configured to calculate a higher reward the closer the state is to an ideal state, when the state resulting from inputting the control input calculated by the meta policy when the weight is assigned to the weak policy is input to the controlled system.

[0052] (6) A learning device according to a sixth aspect is a learning device as described in (4), wherein the means for calculating the weight according to the state trains a neural network that outputs a weight for each of the weak policies when the state is input, and calculates the weight according to the state by inputting the state to the neural network after training.

[0053] (7) A learning device according to a seventh aspect is a learning device as described in (6), wherein the neural network is trained to calculate a higher reward using a reward function configured to calculate a higher reward the closer the state resulting from inputting the control input calculated by the meta-policy when the weights output by the neural network are applied to the controlled system is to an ideal state.

[0054] (8) A learning device according to an eighth aspect is a learning device according to any one of (1) to (7), wherein the means for learning the weak policy, for learning the weak policy by the PODS, calculates the control input by inputting an initial state to the weak policy, inputs the calculated control input to the differentiable model to predict the state at the next time, and calculates the control input by inputting the predicted state to the weak policy, by repeating the process, acquires the time-series state and the time-series control input, updates the time-series control input so that the sum of the rewards related to the time-series state becomes higher by a value obtained by differentiating, with the control input, a reward function configured to calculate a higher reward the closer the predicted state is to an ideal state, and updates the weak policy by learning so that the updated time-series control input is output when the time-series state is input.

[0055] (9) In a learning method according to the ninth aspect, a computer obtains a differentiable model that represents the temporal change in the state of a controlled system in response to a control input to the controlled system, and assigns different learning conditions to each of a plurality of weak policies, which are models that output a control input indicating a control command value to the controlled system when the state of the controlled system is input.The computer then calculates a meta-policy by integrating the plurality of weak policies using PODS using the differentiable model.

[0056] (10) The program according to the tenth aspect causes a computer to function as: a means for obtaining a differentiable model that represents the temporal change in the state of a controlled system in response to a control input to the controlled system; a means for learning a plurality of weak policies by PODS using the differentiable model after giving different learning conditions to each of the weak policies, which are models that output a control input indicating a control command value to the controlled system when the state of the controlled system is input; and a means for calculating a meta-policy by integrating the learned plurality of weak policies. [Explanation of symbols]

[0057] 10. Learning device 11. Differentiable model acquisition part 12. Weak policy calculation part 13. Meta-policy calculation part 800···Learning device 810...Acquisition method 820···Weak policy learning method 830···Meta-policy calculation method

Claims

1. means for obtaining a differentiable model representing a time-dependent change in a state of a controlled system in response to a control input to the controlled system; a means for giving different learning conditions to each of a plurality of weak policies, which are models that output a control input to the control target system when a state of the control target system is input, and then learning the plurality of weak policies by a Policy Optimization via Differentiable Simulator (PODS) using the differentiable model; a means for calculating a meta-policy by integrating the learned weak policies; A learning device comprising:

2. a means for collecting time-series data of the state and the control input at time k from the controlled system or a simulator of the controlled system, and for constructing the differentiable model that is differentiable with respect to the control input, the differentiable model outputting the state at time k+1 when the state and the control input at time k are input; The learning device according to claim 1 .

3. means for calculating the control input by inputting the state into the meta-strategy; The learning device according to claim 1 or 2, further comprising:

4. the means for calculating the meta-policy assigns a weight to each of the plurality of weak policies that have been learned and calculates a meta-policy by summing the weights; The means for calculating the meta-strategy includes: means for calculating the weight according to the state; The learning device according to claim 1 or 2, further comprising:

5. The means for calculating the weight according to the state includes: and searching for a value of weight assigned to each of the plurality of weak policies that will result in a high reward being calculated by a reward function configured to calculate a higher reward as the state resulting from inputting the control input calculated by the meta-policy with the weight assigned to the weak policies into the controlled system, the closer the state is to an ideal state. The learning device according to claim 4 .

6. The means for calculating the weight according to the state includes: training a neural network that outputs weights for each of the weak policies when the state is input, and calculating the weights according to the state by inputting the state into the trained neural network; The learning device according to claim 4 .

7. The neural network is trained to calculate a high reward using a reward function configured to calculate a higher reward as the state resulting from inputting the control input calculated by the meta-policy when the weights output by the neural network are applied to the controlled system becomes closer to an ideal state. The learning device according to claim 6.

8. The means for learning the weak policy includes, with respect to the learning of the weak policy by the PODS, The process of calculating the control input by inputting an initial state into the weak policy, inputting the calculated control input into the differentiable model to predict the state at the next time, and inputting the predicted state into the weak policy to calculate the control input is repeated to obtain the time-series state and the time-series control input, and for each predicted state, the time-series control input is updated by a value obtained by differentiating, with the control input, a reward function configured to calculate a higher reward the closer the state is to an ideal state, so that the sum of the rewards related to the time-series state becomes higher, and the weak policy is trained so that the updated time-series control input is output when the time-series state is input. The learning device according to claim 1 or 2.

9. The computer obtaining a differentiable model representing a change in a state of a controlled system over time in response to a control input to the controlled system; different learning conditions are given to each of a plurality of weak policies, which are models that output a control input to the control target system when a state of the control target system is input, and then the plurality of weak policies are learned by a Policy Optimization via Differentiable Simulator (PODS) using the differentiable model; calculating a meta-policy by integrating the learned weak policies; How to learn.

10. Computer, means for obtaining a differentiable model representing a time-dependent change in a state of a controlled system in response to a control input to the controlled system; a means for learning a plurality of weak policies, each of which is a model that outputs a control input to the control target system when a state of the control target system is input, by a Policy Optimization via Differentiable Simulator (PODS) using the differentiable model, after giving different learning conditions to each of the weak policies; a means for calculating a meta-policy by integrating the learned weak policies; A program to function as a

Images