Learning device, learning method, and learning program

Abstract

To guarantee stability of behavior in reinforcement learning.SOLUTION: A learning device including a processor for executing a program and a storage device for storing the program and training a neural network that learns a strategy for improving a first index, executes: problem generation processing of generating a convex optimization problem including a gradient function relating to a parameter update policy included in the neural network on the basis of information relating to the neural network; index generation processing of generating a second index relating to the policy on the basis of the convex optimization problem generated by the problem generation processing; and update processing of updating a parameter included in the neural network on the basis of the first index and the second index.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a learning device, a learning method, and a learning program for performing learning. [Background technology]

[0002] Reinforcement learning is a machine learning method that learns a policy through interaction with the environment to accomplish a task. In conventional reinforcement learning, the policy is typically represented by a neural network, and the policy is learned by adjusting the parameters of the neural network.

[0003] In recent years, there has been a lot of attention being paid to the application of reinforcement learning to real systems, but one problem with applying it to real systems is that the complexity of neural networks makes it difficult to adjust parameters to ensure the safety of actions.

[0004] Specifically, because neural networks have many complex parameters, it is difficult to learn strategies for determining appropriate behavior (parameter adjustment), and it is even more difficult to adjust the parameters to behave appropriately and safely.

[0005] The safety of an action means not engaging in risky behavior. For example, while driving a car at or below the legal speed limit may be appropriate, if it is dangerous to take a corner at that speed, the safety of the action cannot be guaranteed.

[0006] One of the difficulties in adjusting parameters to guarantee the safety of actions is that it is technically difficult to formulate safety, making it impossible to set safety indicators for AI (Artificial Intelligence).

[0007] For this reason, a technique has been disclosed in which a convex optimization problem is generated during learning and the result of solving the convex optimization problem is set as an index, as in Non-Patent Document 1. Non-Patent Document 1 discloses an approach to differentiation using a controlled convex program, which is a subclass of convex optimization problems used by domain-specific languages ​​(DSLs) for convex optimization.

[0008] In Non-Patent Document 1, safety is formulated as a convex optimization problem that satisfies constraints. In other words, the solution to the convex optimization problem is an index that indicates safety. In conventional reinforcement learning, the reward is set as the index to be maximized, but when Non-Patent Document 1 is applied to conventional reinforcement learning, the reward and safety indexes are maximized. Therefore, the parameters of the neural network are updated in a direction that also ensures safety, making it possible to adjust the parameters that guarantee safety.

[0009] Furthermore, claim 2 of the following Patent Document 1 states, "The computer-implemented method of claim 1, comprising: performing an optimization analysis to generate an optimization output; and compiling a command sequence using the optimization output, wherein the optimization analysis uses (i) a convex optimization solver, (ii) data representing the target state, and (iii) data representing each of the sets of responses, whereby the microwave oven executes the command sequence to heat the item to near the target state." Paragraph

[0098] states, "The training data can be used to train a neural network according to an approach in which a loss function is iteratively minimized according to stochastic gradient descent evaluation." [Prior art documents] [Patent documents]

[0010] [Patent Document 1] Special Publication No. 2019-519744 [Non-patent literature]

[0011] [Non-Patent Document 1] Agrawal, Akshay, et al, "Differentiable convex optimization layers," 2019. Summary of the Invention [Problem to be solved by the invention]

[0012] When training a neural network by including the results of a convex optimization problem in its indices, the training procedure is as follows: (1) First, a forward propagation is performed, in which a convex problem is generated from the input data, solved, and an indices are output; and (2) Next, a back propagation is performed, in which the gradient is calculated in reverse order from the output indices and trained.

[0013] However, when Non-Patent Document 1 is applied to reinforcement learning, it is not necessarily possible to obtain an optimal solution to the convex optimization problem (the best solution among the allowable solutions that satisfy all constraints) in the early stages of learning. The convex optimization problem for calculating safety has very strict constraints, so the optimal solution is not easily obtained. Since backpropagation calculations cannot be performed unless an optimal solution is obtained, Non-Patent Document 1 is not suitable for use in reinforcement learning.

[0014] Furthermore, Patent Document 1 does not mention that a convex optimization problem including a gradient function related to the parameter update policy of the neural network is solved in the training of the neural network, and reinforcement learning with guaranteed safety cannot be performed when an optimal solution to the convex optimization problem cannot be obtained.

[0015] The present invention aims to guarantee the stability of actions in reinforcement learning and to guarantee the stability of actions in convex optimization problems. [Means for solving the problem]

[0016] A learning device according to one aspect of the invention disclosed in this application comprises a processor that executes a program and a storage device that stores the program, and is a learning device for training a neural network that learns a strategy for improving a first index, wherein the processor executes the following processes: generating a convex optimization problem including a gradient function related to an update policy for parameters included in the neural network based on information about the neural network; generating a second index related to the strategy based on the convex optimization problem; and updating the parameters included in the neural network based on the first index and the second index. [Effects of the Invention]

[0017] According to a representative embodiment of the present invention, it is possible to guarantee the stability of actions in reinforcement learning and to guarantee the stability of actions in convex optimization problems. Problems, configurations, and effects other than those described above will become clear from the following description of the examples. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a block diagram illustrating an example of the hardware configuration of a learning device. [Figure 2] FIG. 2 is a block diagram illustrating an example of the functional configuration of the learning device. [Figure 3] FIG. 3 is an explanatory diagram showing an example of general forward propagation and backward propagation. [Figure 4] FIG. 4 is an explanatory diagram showing an example of forward propagation and backward propagation in the convex optimization unit. [Figure 5] FIG. 5 is an explanatory diagram illustrating an example of a matrix class. [Figure 6] FIG. 6 is an explanatory diagram showing a first example of forward propagation execution performed by the forward propagation execution unit. [Figure 7] FIG. 7 is an explanatory diagram showing matrix data of a variable matrix. [Figure 8] FIG. 8 is an explanatory diagram showing a second example of forward propagation execution performed by the forward propagation execution unit. [Figure 9] FIG. 9 is an explanatory diagram showing matrix data of a variable matrix and a constant matrix. [Figure 10A] FIG. 10A is a flowchart illustrating an example of a question generation processing procedure performed by the question generator. [Figure 10B] FIG. 10B is an explanatory diagram showing an example of a closed loop system. [Figure 11] FIG. 11 is a flowchart illustrating an example of a standard form conversion processing procedure performed by the standard form conversion unit. [Figure 12] FIG. 12 is an explanatory diagram showing an example of a solution update algorithm. [Figure 13] FIG. 13 is an explanatory diagram showing forward propagation and backward propagation for updating the variable matrix X. [Figure 14] FIG. 14 is a flowchart illustrating an example of an optimization procedure performed by the optimization unit. [Figure 15] FIG. 15 is a flowchart illustrating an example of a procedure for solution conversion processing by the solution conversion unit. [Figure 16] FIG. 16 is a flowchart illustrating an example of an index calculation process procedure performed by the index calculation unit. [Figure 17] FIG. 17 is a flowchart illustrating an example of a policy improvement processing procedure performed by the policy improvement unit. [Figure 18] FIG. 18 is a flowchart illustrating a detailed example of the policy neural network update process (step S1702) illustrated in FIG. [Figure 19] FIG. 19 is a flowchart showing a detailed example of the processing procedure of the value function neural network update processing (step S1703) shown in FIG. [Figure 20] FIG. 20 is an explanatory diagram showing an example of the input screen. [Figure 21] FIG. 21 is an explanatory diagram showing an example of the output screen. DETAILED DESCRIPTION OF THE INVENTION

[0019] <Convex optimization problem> Convex optimization problems will be explained using semidefinite programming as an example. Semidefinite programming is a type of convex optimization problem on a convex cone formed by all semidefinite matrices. Semidefinite programming is expressed by the primal problem (A) and dual problem (B) in the following equation (1).

[0020]

number

[0021] In the primal problem (A), C∈S n is a constant matrix, and indicates that the constant matrix C is a real symmetric matrix of size n.

[0022] In the primal problem (A), X∈S n is a variable matrix, and indicates that the variable matrix X is a real symmetric matrix of size n.

[0023] In the main problem (A), the operator "·" in U·V indicates the dot product of matrix U and matrix V.

[0024] In the main problem (A), S n denotes the set of real symmetric matrices of size n.

[0025] In the main problem (A), b∈R m is a constant vector, and indicates that the constant vector b is an m-th order vector.

[0026] In the dual problem (B), y∈R m is a variable vector, and indicates that the variable vector y is an m-th order vector.

[0027] In the dual problem (B), Z∈S n is a variable matrix, and indicates that the variable matrix Z is a real symmetric matrix of size n.

[0028] Constant matrix A1~A mGiven , C, and a constant vector b, a convex optimization problem with a symmetric matrix X as the variable matrix is ​​called the primal problem of a semidefinite programming problem. A variable matrix X that satisfies the constraints of the primal problem (A) is called a feasible solution to the primal problem. Furthermore, when the variable matrix X that is a feasible solution is positive definite, the feasible solution is called an interior point feasible solution.

[0029] For the main problem (A), the variable vector y1~y m A convex optimization problem with variables Z and a variable matrix Z is called the dual problem (B) of a semidefinite programming problem. i , Z) is called a feasible solution of the dual problem (B). Furthermore, when the variable matrix Z, which is a feasible solution, is positive definite, the feasible solution is called an interior point feasible solution.

[0030] The following inequality (2) holds between the objective function value of the primal problem (A) and the objective function value of the dual problem (B).

[0031]

number

[0032] If the objective function value of the primal problem (A) (the first term on the left side of the above inequality (2)) and the objective function value of the dual problem (B) (the second term on the left side of the above inequality (2)) are the same, then (X, y, Z) is the optimal solution to the semidefinite programming problem.

[0033] <Example of hardware configuration for learning device> FIG. 1 is a block diagram showing an example of the hardware configuration of a learning device. The learning device 100 includes a processor 101, a storage device 102, an input device 103, an output device 104, and a communication interface (communication IF) 105. The processor 101, the storage device 102, the input device 103, the output device 104, and the communication IF 105 are connected via a bus 106. The processor 101 controls the learning device 100. The storage device 102 serves as a working area for the processor 101. The storage device 102 is a non-transitory or temporary recording medium that stores various programs and data. Examples of the storage device 102 include a read-only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), and a flash memory. The input device 103 inputs data. Examples of the input device 103 include a keyboard, a mouse, a touch panel, a numeric keypad, a scanner, a microphone, and a sensor. The output device 104 outputs data. The output device 104 includes, for example, a display, a printer, and a speaker. The communication IF 105 connects to a network and transmits and receives data.

[0034] <Example of functional configuration of learning device 100> 2 is a block diagram showing an example of the functional configuration of the learning device 100. The learning device 100 includes an environment unit 201, a policy improvement unit 202, and a convex optimization unit 203. Specifically, the environment unit 201, the policy improvement unit 202, and the convex optimization unit 203 are realized, for example, by having the processor 101 execute a program stored in the storage device 102 shown in FIG.

[0035] Similar to conventional reinforcement learning, the environment unit 201 includes a state calculation unit 211, a reward calculation unit 212, and an environment reset unit 213. The state calculation unit 211 updates the state when an action is given. The reward calculation unit 212 calculates the reward for the selected action. The environment reset unit 213 resets the environment for each episode.

[0036] The policy improvement unit 202 includes an action determination unit 221, a parameter update unit 222, and a learning data collection unit 223. The action determination unit 221 selects an action that maximizes the value of the action calculated by the value function. The parameter update unit 222 updates the policy parameters of the policy neural network and the parameters of the value function neural network. The learning data collection unit 223 collects learning data (actions, states, rewards).

[0037] The convex optimization unit 203 solves a convex optimization problem, which is a problem of minimizing a convex function on a convex set. When the action, state, reward, and policy parameters are input, the convex optimization unit 203 outputs the solution of the convex optimization problem as an index to the policy improvement unit 202.

[0038] The convex optimization unit 203 includes a gradient function storage unit 231 , a problem generation unit 232 , a standard form conversion unit 233 , an optimization unit 234 , a solution conversion unit 235 , an index calculation unit 236 , and a matrix class 237 .

[0039] The gradient function storage unit 231 stores information for executing backpropagation (hereinafter referred to as backpropagation execution information). When learning data (action, state, reward) and policy parameters are given, the problem generation unit 232 generates an optimization problem. An optimization problem is a combination of an objective function and constraint conditions. The objective function is generated in step S1006, which will be described later, and constraint conditions are generated in step S1007.

[0040] The standard form conversion unit 233 outputs a constant matrix and a constant vector based on the objective function and constraint conditions from the problem generation unit 232. The standard form is the primal problem (A) or the dual problem (B) described in the above formula (1). The optimization problem is defined using constant matrices A and C and a constant vector b. The standard form conversion unit 233 converts the objective function and constraint conditions into constant matrices A and C and a constant vector b.

[0041] The optimization unit 234 executes a process of finding an optimal solution to the main problem (A) and the dual problem (B). Details of the optimization unit 234 will be described later with reference to FIGS.

[0042] The solution conversion unit 235 converts the solution obtained from the optimization unit 234 into a variable matrix. Details of the solution conversion unit 235 will be described later with reference to FIG.

[0043] The index calculation unit 236 calculates an index for evaluating the safety of an action based on the optimality of the solution obtained from the optimization unit 234 and the variable matrix obtained from the solution conversion unit 235. The index for evaluating the safety of an action is a real value equal to or greater than 0, and a larger index value indicates higher safety. Details of the index calculation unit 236 will be described later with reference to FIG. 16.

[0044] <Forward and backward propagation> FIG. 3 is an explanatory diagram showing an example of general forward propagation and backpropagation. Sequence 301 shows forward propagation, and sequence 302 shows backpropagation for sequence 301. In sequence 301, when a value 311 of variable x is input to function f, function f calculates a value 312 of variable y, and when a value 312 of variable y is input to function g, function g calculates a value 313 of variable z. In forward propagation, when functions f and g perform operations on input values ​​311 and 312 of variables x and y to output values ​​312 and 313 of variables y and z, the input values ​​311 and 312 of variables x and y, the output values ​​312 and 313 of variables y and z, and gradient functions f′(x) and g′(y) corresponding to the operations by functions f and g are retained. Because information for performing such backpropagation is retained, convex optimization unit 203 is capable of performing backpropagation.

[0045] The operations performed by functions f and g include operations required for neural network training, such as addition, subtraction, multiplication, and division, matrix multiplication, and tanh. Hereinafter, the expression "performing forward propagation" means that information for performing back propagation is retained at the same time as normal calculations are performed (performing calculations on inputs and outputting the results).

[0046] In the back propagation of the sequence 302, gradients 323, 322, and 321 are calculated from the output side of the forward propagation toward the input side based on the information for executing the back propagation that was held during the forward propagation.

[0047] 4 is an explanatory diagram showing an example of forward propagation and backward propagation in the convex optimization unit 203. A sequence 401 shows forward propagation, and a sequence 402 shows backward propagation for the sequence 401. In the sequence 401, a function f convex to variable x in When the value 411 is input, the function f convex is the variable y out Calculate the value 412 of the function f convex is the variable y out This is a function that outputs the value 412 of the variable y out The final output value of is L, which is an index output from the convex optimizer 203.

[0048] In forward propagation, the input variable x in For the value 411, the function f convex performs the operation and stores the variable y out When outputting the value 412 of the input variable x in The value 411 and the output variable y out The value of 412 and the function f convex The gradient function f' corresponding to the operation convex and are stored in the gradient function storage unit 231. Since the information for executing such back propagation is stored in the gradient function storage unit 231, the convex optimization unit 203 can execute back propagation.

[0049] In the backpropagation of the sequence 402, gradients 422 and 421 are calculated from the output side of the forward propagation to the input side based on the information for performing the backpropagation that has been retained.

[0050] <Matrix class> FIG. 5 is an explanatory diagram showing an example of the matrix class 237. A semidefinite programming problem is described using a constant matrix and a variable matrix. When generating a constant matrix and a variable matrix, the matrix class 237 is generated. The matrix class 237 includes matrix data 501 and a forward propagation execution unit 502. As shown in FIGS. 3 and 4, the forward propagation execution unit 502 executes forward propagation and stores information for executing back propagation in the gradient function storage unit 231.

[0051] The matrix data 501 includes indexes 511 and matrices 512. The indexes 511 identify the matrices 512 in the same row. The values ​​0 to 4 of the indexes 511 may be expressed as indexes #0 to #4. The variable matrix is ​​expressed as a linear combination of five types of matrices 512(0) to 512(4) of indexes #0 to #4.

[0052] Specifically, for example, the variable matrix H(x) can be expressed by the following equation (3).

[0053]

number

[0054] In the above formula (3), x1, x2, ..., x p is a set of p variables (p is an integer greater than or equal to 1), H0 is a constant matrix, H1, H2, …, H p is a binary matrix (it is a binary matrix when it is generated, but after forward propagation the matrix is ​​not a binary matrix).

[0055] For example, when a variable matrix X1 with 2 rows and 2 columns is generated, it is expressed as in the following equation (4).

[0056]

number

[0057] 5, variable matrix X1 is stored as a pair of variable index 511 and corresponding matrix 512. That is, on the right side of the above equation (4), variable x1 is expressed using matrix 512(1) with index #1, variable x2 is expressed using matrix 512(2) with index #2, variable x3 is expressed using matrix 512(3) with index #3, and variable x4 is expressed using matrix 512(4) with index #4. Note that constant matrix H0 corresponds to matrix 512(0).

[0058] <Example of forward propagation execution> Next, an example of forward propagation execution performed by the forward propagation execution unit 502 will be described.

[0059] 6 is an explanatory diagram showing a first execution example of forward propagation executed by the forward propagation execution unit 502. The first execution example of forward propagation shows matrix addition. A sequence 601 shows forward propagation, and a sequence 602 shows backpropagation for the sequence 601. In the sequence 601, a function Add showing addition is added with two variables M j 1 , M j 2 When the value 611,612 is entered, the function Add returns the value of the variable M j 3 When a gradient 623 is input to the gradient function Add′, the gradient function Add′ calculates gradients 621 and 622. j is the value of the index 511.

[0060] The gradients 623, 622, and 621 are values ​​output by performing backpropagation. The gradient function Add' is a gradient function corresponding to the operation by the function Add.

[0061] The forward propagation execution unit 502 executes the input variable M j 1 , M j 2 The function Add performs an operation on each value 611, 612 of the variable M j 3 When outputting the value 613 of the input variable M j 1 , M j 2 The values ​​611,612 and the output variable M j 3 The value 613 and the gradient function Add′ are stored in the gradient function storage unit 231.

[0062] For example, when a variable matrix X2 with 2 rows and 2 columns is generated, it is expressed as in the following equation (5).

[0063]

number

[0064] The addition of matrix variables X3=X1+X2 by the function Add is expressed as in the following equation (6).

[0065]

number

[0066] For example, in the first term on the right side of the above equation (6) for the variable x1 of index #1, forward propagation is performed using the following equation (7).

[0067]

number

[0068] 7 is an explanatory diagram showing matrix data of variable matrices X1 to X3. Matrix data 701 of variable matrix X1 is expressed by matrix data 501 shown in FIG. 5, as shown in equation (4) above. Matrix 712 indicates the matrix of index #j (j=0 to 4).

[0069] Matrix data 702 of variable matrix X2 is matrix 722 shown in each term on the right side of the above equation (5). Matrix 722 indicates the matrix of index #j (j=0 to 4).

[0070] Matrix data 703 of variable matrix X3 is matrix 732 calculated by adding the terms on the right side of equation (6) above. Matrix 732 indicates the matrix of index #j (j=0 to 4).

[0071] FIG. 8 is an explanatory diagram showing a forward propagation execution example 2 executed by the forward propagation execution unit 502. The forward propagation execution example 1 shows a matrix multiplication. A sequence 801 shows forward propagation, and a sequence 802 shows back propagation for the sequence 801. In the sequence 801, a function Matmul indicating a matrix multiplication is provided with two variables M j 1 , M0 2 When the values ​​811 and 812 are input, the function Matmul calculates the j 3When a gradient 823 is input to the function Matmul′, the function Matmul′ calculates gradients 821 and 822. j is the value of the index 511.

[0072] Gradients 823, 822, and 821 are values ​​output by performing backpropagation. The function Matmul' is a gradient function corresponding to the operation by the function Matmul.

[0073] The forward propagation execution unit 502 executes the input variable M j 1 , M0 2 The function Add performs an operation on each value 811, 812 of the variable M j 3 When outputting the value 813 of the input variable M j 1 , M0 2 The values ​​811,812 and the output variable M j 3 The value 813 and the gradient function Matmul′ are stored in the gradient function storage unit 231.

[0074] In matrix multiplication, at least one of the two matrices to be multiplied is a constant matrix. For example, the matrix product of the variable matrix X1 in the above formula (4) and the constant matrix C1 in the following formula (8) using the function Matmul, variable matrix X4=X1C1, is expressed by the following formula (9).

[0075]

number

[0076] For example, in the first term on the right side of the above equation (9) for the variable x1 of index #1, forward propagation is performed using the following equation (10).

[0077]

number

[0078] 9 is an explanatory diagram showing matrix data of variable matrices X1 and X4 and constant matrix C1. Matrix data 701 of variable matrix X1 is expressed by matrix data 501 shown in FIG. 5, as shown in equation (4) above. Matrix 712 indicates the matrix of index #j (j=0 to 4).

[0079] The matrix data 901 of the constant matrix C1 is the matrix shown in the above formula (8). That is, in the matrix data 912, the index #0 is the constant matrix C1, and the other indexes #1 to #4 are 0 matrices.

[0080] Matrix data 904 of variable matrix X4 is matrix 942 calculated by multiplying the matrices in the terms on the right side of equation (9) above. Matrix 942 indicates the matrix of index #j (j=0 to 4).

[0081] <Question Generation Process by Question Generator 232> 10A is a flowchart showing an example of a procedure for a problem generation process by the problem generator 232. The problem generator 232 acquires actions and policy parameters from the policy improver 202, and acquires a state and a reward from the environment unit 201 (step S1001).

[0082] The problem generator 232 models the state transition of the environment as a first-order transfer function from the state and the action (step S1002).

[0083] The problem generator 232 uses the transfer function model and the strategy to construct a closed-loop system (step S1003).

[0084] FIG. 10B is an explanatory diagram showing an example of a closed-loop system. The closed-loop system 1000 is a system in which a closed loop is formed by a policy 1101 and a transfer function 1102. When a state and a reward are given in a policy network in which policy parameters are set, the policy 1101 determines an action that maximizes the reward in a probability distribution that determines an action in the state, and outputs the action to the transfer function 1102. When an action is input, the transfer function 1102 outputs the state and the reward to the policy 1101. By constructing stability conditions for such a closed-loop system 1000, it becomes possible to calculate an index of the safety of the action.

[0085] Returning to FIG. 10A, the problem generator 232 generates a constant matrix based on the closed-loop system configured in step S1003 (step S1004).

[0086] The problem generator 232 generates a variable matrix for satisfying the conditions for ensuring the safety of the closed-loop system, that is, the safety of the behavior (step S1005).

[0087] The problem generator 232 performs forward propagation to generate an objective function related to the safety of the action, using the constant matrix generated in step S1004 and the variable matrix generated in step S1005 (step S1006).

[0088] The problem generator 232 performs forward propagation to generate constraints related to the safety of actions using the constant matrix generated in step S1004 and the variable matrix generated in step S1005 (step S1007).

[0089] The problem generator 232 generates the objective function (c T x) and the constraints generated by the forward propagation in step S1007 are output to the standard form conversion unit 233 (step S1008).

[0090] The problem generation unit 232 generates the conditions regarding the safety of the behavior as a problem (formula (11-1) below) that minimizes a linear objective function regarding the safety of the behavior under the constraint of a linear matrix inequality (formula (11-2) below) (formula (11) below).

[0091]

number

[0092] T denotes the transpose of a matrix or vector. The above formula (11-2) is the constraint generated by the forward propagation in step S1007, and is the constraint on the safety of the action in step S1007. c is a constant vector, and x is a variable vector. G0, G1, ... Gq are constant matrices, and G(x) is a matrix-valued function.

[0093] The optimization problem when there are multiple (K) linear matrix inequality constraints is as follows (Equation (12) below).

[0094]

number

[0095] In steps S1006 and S1007, the problem generator 232 uses the generated constant matrix and variable matrix to calculate c in the above formula (11). T Forward propagation is performed to calculate x and G(x). Specifically, for example, the forward propagation execution unit 502 in the matrix class 237 performs forward propagation to calculate the objective function and constraint conditions.

[0096] The constant matrices G0, G1, ... Gq are constants in a convex optimization problem, but in neural network training they are treated as parameters whose gradients should be calculated.

[0097] Although the condition for the safety of an action is expressed in the general form of the above formula (11), it is also possible to use inequalities related to matrix variables in their original form. For example, if the variable matrix X∈S n and a constant matrix A∈R n×nmay be expressed as follows using a matrix object (Equation (13) below):

[0098]

number

[0099] <Standard system conversion processing by the standard form conversion unit 233> 11 is a flowchart showing an example of a standard system conversion processing procedure by the standard form conversion unit 233. By converting to a standard form by the standard form conversion unit 233, it becomes possible to solve an optimization problem in a general-purpose manner. For example, if the user's request or the target system differs, the objective function and constraint conditions generated in steps S1006 and S1007 will change, but by converting to the standard form, it becomes possible to solve the optimization problem in the same format even if the user's request or the system differs.

[0100] The standard form conversion unit 233 acquires the objective function and the constraint conditions from the problem generation unit 232 (step S1101).

[0101] The standard form conversion unit 233 executes forward propagation to calculate the constant matrix C of the semidefinite programming problem (the above formula (1)) from the constraint conditions (step S1102).

[0102] The standard form conversion unit 233 converts the constant matrix A of the semidefinite programming problem (the above formula (1)) from the constraints. i Then, forward propagation is performed to calculate (step S1103).

[0103] The standard form conversion unit 233 executes forward propagation to calculate the constant vector b of the semidefinite programming problem (the above formula (1)) from the objective function (step S1104).

[0104] The standard form conversion unit 233 converts the constant matrix A i , the constant vector b, and the constant matrix C are output to the optimization unit 234 (step S1105).

[0105] constant matrix A i, constant vector b, constant matrix C are the matrix G in the above equation (11). i (i=0,1,...,q) and matrix c, it is calculated as follows:

[0106] constant matrix A i =G i (i=0,1,…,q) (14) constant vector b=-c (15) Constant matrix C=-G0 (16)

[0107] The above formulas (14) to (16) include the policy parameters (weights of the policy neural network) as well as the parameters of the transfer function model (weight matrix). However, the constant matrix generated from the parameters of the transfer function model is a constant in the convex optimization problem and in neural network learning, so it is not treated as a parameter for which the gradient should be calculated.

[0108] <Solution update algorithm> 12 is an explanatory diagram showing an example of a solution update algorithm. From a feasible initial solution 1201 within a feasible region 1200 or an infeasible initial solution 1202 outside the feasible region 1200, forward propagation is performed to update the solution along a central path 1204. The feasible initial solution 1201 does not have to be an admissible solution 1211.

[0109] Iteration points 1203 are output values ​​from the function in forward propagation. In Fig. 12, there are three iteration points 1203, and then an optimal solution 1212 is obtained, which indicates that forward propagation has been performed four times. At each iteration point 1203, the input value, output value, and gradient function are stored in the gradient function storage unit 231 as information for performing back propagation, thereby enabling back propagation to be performed.

[0110] 13 is an explanatory diagram showing forward propagation and backward propagation for updating the variable matrix X. In FIG. 13, a sequence 1301 shows forward propagation, and a sequence 1302 shows backward propagation for the sequence 1301. In the sequence 1301, when a function F is updated with the variable matrix X k is input, the function F is a function of the variable matrix Xk+1 The function F is calculated by the index variable matrix X k+1 k is a number indicating the iteration point 1203 in FIG. 12, and indicates the number of propagations in ascending order starting from k=1.

[0111] matrix of variables X k is the variable matrix that constitutes the k-th iteration point 1203, and the variable matrix X k+1 is a variable matrix that configures the k+1-th iteration point 1203. That is, the k-th iteration point 1203 is configured by the variable matrix X k , the variable vector y k and the variable matrix Z k The k+1th iteration point 1203 is constructed by the variable matrix X k+1 , the variable vector y k+1 and the variable matrix Z k+1 It is composed of:

[0112] In forward propagation, the input variable matrix X k The function F performs an operation on the variable matrix X k+1 When outputting, the input variable matrix X k and the output variable matrix X k+1 and a gradient function F' corresponding to the calculation using the function F are stored in the gradient function storage unit 231. Since the information for executing such back propagation is stored in the gradient function storage unit 231, the convex optimization unit 203 can execute back propagation.

[0113] In the back propagation of series 1302, gradients 1322 and 1321 are calculated from the output side to the input side of the forward propagation based on information for executing back propagation stored in gradient function storage unit 231. Note that while forward propagation and back propagation are shown for variable matrix X in Fig. 13, the same applies to variable vector y and variable matrix Z, although they are not shown.

[0114] The k+1th iteration point 1203(X k+1 ,y k+1 ,Z k+1 ) is determined to satisfy the following conditional expression (17), and the optimality δ of the solution is expressed by the following expression (18).

[0115]

number

[0116] That is, if δ=0 in the above equation (18), the k+1th iteration point 1203 (X k+1 ,y k+1 ,Z k+1 ) is the optimal solution 1212 of the semidefinite programming problem (the above formula (1)). p is the step length for updating the matrix X, and α d is the step length for updating matrix Z.

[0117] <Optimization Processing by Optimization Unit 234> 14 is a flowchart showing an example of the optimization procedure performed by the optimization unit 234. The optimization unit 234 receives the constant matrix A i , a constant vector b, and a constant matrix C (see the above formulas (14) to (16)) are obtained (step S1401).

[0118] The optimization unit 234 optimizes the constant matrix A i , constant vector b, and constant matrix C are substituted into the primal problem (A) and dual problem (B) of the above equation (1) to obtain the initial solution (X 0 ,y 0 ,Z 0 ) is selected (step S1402).

[0119] The optimization unit 234 generates an equation for approximating the points on the central path 1204 (step S1403).

[0120] The optimization unit 234 solves the equation generated in step S1403 to determine the search direction (dX k ,dy k ,dZ k ) is calculated (step S1404).

[0121] The optimization unit 234 performs forward propagation to calculate the step sizes αp and αd (step S1405).

[0122] The optimizer 234 determines the next iteration point 1203 (X k+1 ,y k+1 ,Z k+1 ) is determined so as to satisfy the conditional expression of the above equation (17) (step S1406).

[0123] The optimization unit 234 performs forward propagation to calculate the optimality δ of the solution of the above equation (18) (step S1407).

[0124] The optimization unit 234 determines whether or not an optimal solution 1212 is obtained before the predetermined number of repetitions (step S1408). If the optimal solution 1212 is not obtained before the predetermined number of repetitions (step S1408: No), the process returns to step S1403. If the optimal solution 1212 is obtained before the predetermined number of repetitions or if the predetermined number of repetitions has ended (step S1408: Yes), the optimization unit 234 outputs the optimality δ of the solution to the index calculation unit 236 (step S1409).

[0125] The optimization unit 234 outputs the updated solution (X, y, Z) to the solution conversion unit 235 (step S1410). The updated solution (X, y, Z) is the optimal solution 1212 if the optimal solution 1212 is obtained, or is the last updated iteration point 1203 (X k+1 ,y k+1 ,Z k+1 )

[0126] <Solution conversion process by the solution conversion unit 235> 15 is a flowchart showing an example of the procedure for the solution conversion process by the solution conversion unit 235. Because the solution (X, y, Z) from the optimization unit 234 is a standard form solution, the solution conversion unit 235 executes a process to return the solution (X, y, Z) to the original problem (i.e., the form of the objective function and constraint conditions generated in steps S1006 and S1007).

[0127] The solution conversion unit 235 acquires the solution (X, y, Z) from the optimization unit 234 (step S1501).

[0128] The solution conversion unit 235 performs forward propagation to convert the solution into a variable matrix used in formulating the safety (step S1502).

[0129] The solution conversion unit 235 outputs the variable matrix converted by the forward propagation in step S1502 to the index calculation unit 236 (step S1503).

[0130] <Indicator Calculation Processing by Indicator Calculation Unit 236> 16 is a flowchart showing an example of the procedure of the index calculation process by the index calculation unit 236. The index calculation unit 236 acquires the optimality δ of the solution from the optimization unit 234, and acquires the variable matrix from the solution conversion unit 235 (step S1601).

[0131] The index calculation unit 236 executes forward propagation to calculate the safety of the action using the variable matrix acquired in step S1601 (step S1602).

[0132] The index calculation unit 236 executes forward propagation to calculate an index from the safety of the action and the optimality δ of the solution obtained in the forward propagation in step S1602 (step S1603).

[0133] The index calculation unit 236 outputs the index obtained by the forward propagation in step S1603 to the policy improvement unit 202 (step S1604).

[0134] <Policy improvement processing by the policy improvement unit 202> 17 is a flowchart showing an example of a policy improvement processing procedure by the policy improvement unit 202. The policy improvement unit 202 determines an action using the action determination unit 221, and collects learning data (action, state, reward) using the learning data collection unit 223 (step S1701). The policy improvement unit 202 executes a policy neural network update process using the parameter update unit 222 (step S1702). The policy improvement unit 202 executes a value function neural network update process using the parameter update unit 222 (step S1703). The policy improvement unit 202 determines whether learning is complete (step S1704).

[0135] Specifically, for example, the policy improvement unit 202 determines that learning is complete when the number of interactions with the environment reaches a predetermined number. Here, one interaction with the environment is a process in which "the action decision unit 221 decides on an action based on the state, the action is input to the environment unit 201, and the next state and reward are output."

[0136] If the learning is not complete (step S1704: No), the process returns to step S1701. If the learning is complete (step S1704: Yes), the policy improvement process ends.

[0137] Fig. 18 is a flowchart showing a detailed example of the processing procedure of the policy neural network update process (step S1702) shown in Fig. 17. The policy neural network is a neural network that receives a state as an input and outputs an action. The policy improvement unit 202 outputs policy parameters (weights of the policy neural network) and learning data (action, state, reward) to the convex optimization unit 203 via the parameter update unit 222, and acquires an index (step S1801).

[0138] The policy improvement unit 202 performs forward propagation using the learning data (action, state, reward) via the parameter update unit 222 to calculate an objective function (for example, a surrogate objective function) that leads to maximization of the reward (step S1802). Specifically, the policy improvement unit 202 performs forward propagation with the objective function that leads to maximization of the reward as the first index and the index acquired from the convex optimization unit 203 as the second index. For example, the policy improvement unit 202 performs forward propagation with the objective function that leads to maximization of the reward as the first index and the index acquired from the convex optimization unit 203 as the second index, and calculates a value by multiplying the first index and the second index by a coefficient and adding them together.

[0139] The policy improving unit 202 executes forward propagation by the parameter updating unit 222 to calculate a loss function of the policy using the index acquired in step S1801 and the surrogate objective function obtained by executing forward propagation in step S1802 (step S1803). Specifically, for example, the policy improving unit 202 executes forward propagation by the parameter updating unit 222 to calculate a value obtained by multiplying the first index and the second index by a coefficient and adding them together, with the objective function that leads to maximization of the reward acquired by executing forward propagation in step S1802 being the first index and the index acquired in step S1801 being the second index.

[0140] The policy improving unit 202 causes the parameter updating unit 222 to perform back propagation for the forward propagation that calculates the loss function of the policy, and calculates the gradient (step S1804).

[0141] The policy improving unit 202 updates the policy parameters of the policy neural network using the gradient obtained in step S1804 through the parameter updating unit 222 (step S1805). After that, the process proceeds to the value function neural network updating process (step S1703).

[0142] Fig. 19 is a flowchart showing a detailed example of the processing procedure of the value function neural network update process (step S1703) shown in Fig. 17. The policy improvement unit 202 executes forward propagation using the learning data (action, state, reward) to calculate the loss function of the value function by the parameter update unit 222 (step S1901).

[0143] The value function is a function that evaluates the value of an action. Specifically, for example, the value function is a function that calculates an estimated value of a reward when an action is selected for a state from the environment unit 201 as the value of the action. The policy improvement unit 202 uses the action decision unit 221 to select an action that maximizes the estimated value of the reward and outputs it to the environment unit 201.

[0144] The policy improving unit 202 performs back propagation for the forward propagation that calculates the loss function of the value function, and calculates the gradient (step S1902).

[0145] The policy improving unit 202 updates the parameters of the value function neural network using the gradient obtained in step S1902 (step S1903), and then proceeds to step S1704.

[0146] <Screen example> 20 is an explanatory diagram showing an example of an input screen. The input screen 2000 is a graphic user interface that is displayed on a display, which is an example of the output device 104, and allows the user to input operations. The input screen 2000 has a first setting item 2001 to a seventh setting item 2007, and an execute button 2008.

[0147] The first setting item 2001 is an item for setting the number of interactions with the environment by a user operation. The set number of interactions is used as a criterion for determining whether to end learning in step S1704.

[0148] The second setting item 2002 is an item for setting the learning rate of the policy neural network through a user operation.

[0149] The third setting item 2003 is an item for setting the learning rate of the value function neural network through a user operation.

[0150] The fourth setting item 2004 is an item for setting, by user operation, a discount rate of the objective function that leads to maximization of the reward.

[0151] A fifth setting item 2005 is an item for setting, by user operation, a GAE (Generalized Advantage Estimator) which is a parameter of the objective function that leads to maximization of reward, that is, a generalized advantage estimation of the objective function.

[0152] A sixth setting item 2006 is used to set a parameter for adjusting the coefficient by which the safety of the action in the second index is multiplied by a user operation.

[0153] A seventh setting item 2007 is used by a user to set a parameter for adjusting a coefficient by which the optimality of the second index matrix variable is multiplied.

[0154] By pressing the execute button 2008, learning is executed using the parameters set in the first setting item 2001 to the seventh setting item 2007.

[0155] 21 is an explanatory diagram showing an example of an output screen. In the case of step S1704: Yes, the output screen 2100 displays the learning results on a display, which is an example of the output device 104. "Return" displays a numerical value indicating how much the reward has been maximized, and "Safety Metric" displays a numerical value indicating how much the index for evaluating the safety of the behavior has been maximized. The larger both numerical values ​​are, the greater the maximization.

[0156] As described above, according to this embodiment, the convex optimization unit 203 calculates the gradient by performing back propagation from the index, thereby ensuring the stability of the behavior in the convex optimization problem.

[0157] The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added to, deleted from, or replaced with other configurations.

[0158] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by a processor interpreting and executing a program that realizes each function.

[0159] Information such as programs, tables, files, etc. that realize each function can be stored in storage devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as IC (Integrated Circuit) cards, SD cards, and DVDs (Digital Versatile Discs).

[0160] In addition, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily represent all the control lines and information lines that are necessary for implementation. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0161] 100 Learning Device 101 processors 102 Storage Devices 201 Environment Department 202 Policy Improvement Department 203 Convex Optimization 211 State Calculation Unit 212 Remuneration Calculation Department 213 Environmental Reset Department 221 Policy Improvement Department 221 Action Decision-Making Department 222 Parameter Update Unit 223 Learning Data Collection Department 231 Gradient function memory 232 Problem generation part 233 Standard Form Conversion Unit 234 Optimization Department 235 Solution conversion section 236 Indicator calculation section

Claims

1. A learning device for training a neural network that learns a strategy for improving a first index, the learning device comprising: a processor that executes a program; and a storage device that stores the program, the learning device comprising: The processor: a problem generation process for generating a convex optimization problem including a gradient function related to an update policy for parameters included in the neural network based on information about the neural network; an index generation process for generating a second index for the policy based on the convex optimization problem generated by the problem generation process; an update process for updating parameters included in the neural network based on the first index and the second index; A learning device that executes the above.

2. The learning device according to claim 1 , The convex optimization problem is a semidefinite programming problem. A learning device characterized by:

3. The learning device according to claim 1 , The information about the neural network is a parameter included in the neural network. A learning device characterized by:

4. The learning device according to claim 1 , In the problem generation process, the processor constructs a closed-loop system of the policy and a transfer function model that models a state transition of an environment based on a state and an action, and generates the convex optimization problem so as to satisfy a condition that guarantees the safety of the closed-loop system. A learning device characterized by:

5. The learning device according to claim 1 , In the update process, the processor performs forward propagation to calculate a loss function of the policy using the first index and the second index, calculates a gradient by performing backpropagation with respect to the forward propagation, and updates the parameters using the gradient. A learning device characterized by:

6. A learning method executed by a learning device for training a neural network that learns a strategy for improving a first index, the learning method comprising: a processor that executes a program; and a storage device that stores the program, the learning device comprising: The processor: a problem generation process for generating a convex optimization problem including a gradient function related to an update policy for parameters included in the neural network based on information about the neural network; an index generation process for generating a second index for the policy based on the convex optimization problem generated by the problem generation process; an update process for updating parameters included in the neural network based on the first index and the second index; A learning method characterized by performing the following.

7. a learning program that causes a processor to train a neural network that learns a strategy for improving a first indicator, the processor, a problem generation process for generating a convex optimization problem including a gradient function related to an update policy for parameters included in the neural network based on information about the neural network; an index generation process for generating a second index for the policy based on the convex optimization problem generated by the problem generation process; an update process for updating parameters included in the neural network based on the first index and the second index; A learning program characterized by executing the above.

Images