Learning device, learning method, control system and program

Abstract

This learning device is provided with: a model calculation unit which calculates each of a plurality of second evaluation values that include noise using a plurality of evaluation models, each of which calculates, on the basis of both a second state resulting from a first action performed by a control target in a first state, and a second action calculated from the second state using a policy model, a second evaluation value obtained by including noise in an indicator value indicating the result of evaluating the second action in the second state; and a model update unit which updates the policy model or the parameters of the policy model on the basis of the smallest of the plurality of second evaluation values and a first evaluation value, which is an indicator value indicating the result of evaluating the first action in the first state.

Description

[Technical field]

[0001] The present invention relates to a learning device, a learning method, a control system, and program Regarding. [Background technology]

[0002] One type of machine learning technique is a reinforcement learning method called Q-learning, which determines a policy using an optimized Q-function. For example, Patent Document 1 describes a method of optimizing the maintenance range of an object requiring maintenance by executing reinforcement learning called Q-learning. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 515930 Brochure Summary of the Invention [Problem to be solved by the invention]

[0004] It is preferable that the time required for reinforcement learning be relatively short.

[0005] An object of the present invention is to provide a learning device, a learning method, a control system, and a recording medium that can solve the above-mentioned problems. [Means for solving the problem]

[0006] According to a first aspect of the present invention, a learning device includes a model calculation unit that uses a plurality of evaluation models to calculate second evaluation values ​​that include noise, the second evaluation values ​​being index values ​​that indicate evaluation results of the second action in the second state, based on a second state corresponding to a first action in a first state of a control target and a second action calculated from the second state using a policy model, and a model update unit that updates the policy model or parameters of the policy model based on the smallest second evaluation value among the plurality of second evaluation values ​​and a first evaluation value that is an index value that indicates an evaluation result of the first action in the first state.

[0007] According to a second aspect of the present invention, the control system includes a model calculation means for calculating, using a plurality of evaluation models, second evaluation values ​​each including noise, based on a second state corresponding to a first action in a first state of a controlled object and a second action calculated from the second state using a policy model, and a model updating means for updating the policy model or parameters of the policy model based on the smallest second evaluation value among the plurality of second evaluation values ​​and a first evaluation value which is an index value indicating an evaluation result of the first action in the first state.

[0008] According to a third aspect of the present invention, a learning method includes: a computer uses a plurality of evaluation models to calculate second evaluation values ​​that include noise in an index value indicating an evaluation result of the second action in a first state based on a second state corresponding to a first action in a first state of a control target and a second action calculated from the second state using a policy model; and updates the policy model or parameters of the policy model based on the smallest second evaluation value among the plurality of second evaluation values ​​and a first evaluation value that is an index value indicating an evaluation result of the first action in the first state.

[0009] According to a fourth aspect of the present invention, a recording medium is a recording medium having a program recorded therein for causing a computer to calculate, based on a second state corresponding to a first action in a first state of a control target and a second action calculated from the second state using a policy model, second evaluation values ​​including noise, the second evaluation values ​​being index values ​​indicating evaluation results of the second action in the second state, and updating the policy model or parameters of the policy model based on the smallest second evaluation value among the plurality of second evaluation values ​​and a first evaluation value which is an index value indicating evaluation results of the first action in the first state. Effect of the Invention

[0010] According to the above-mentioned learning device, control system, learning method, and recording medium, it is possible to reduce the time required for reinforcement learning. [Brief description of the drawings]

[0011] [Figure 1A] FIG. 1 is a diagram illustrating an example of the configuration of a control system according to an embodiment. [Figure 1B] FIG. 2 is a block diagram of a control system according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of an evaluation model storage device according to the embodiment. [Diagram 3] FIG. 1 is a diagram illustrating an example of the configuration of a learning device according to an embodiment. [Figure 4] 5 is a flowchart showing an example of a procedure of a process performed by the control system according to the embodiment. [Diagram 5] FIG. 2 is a diagram for explaining a model of a Q function according to an embodiment. [Figure 6] FIG. 2 is a diagram illustrating a configuration of a Q function model according to an embodiment. [Figure 7] FIG. 11 is a diagram illustrating an example of a processing procedure in which the control system of the embodiment updates a model. [Figure 8] FIG. 11 is a diagram showing a verification result in the embodiment. [Figure 9]FIG. 11 is a diagram showing a verification result in the embodiment. [Figure 10] FIG. 11 is a diagram showing a verification result in the embodiment. [Figure 11] FIG. 2 is a diagram illustrating an example of a pendulum to be controlled in the first embodiment. [Figure 12] FIG. 11 is a diagram illustrating a configuration example of a section in a VAM plant according to a second embodiment. [Figure 13] FIG. 1 is a diagram illustrating an example of the configuration of a learning device according to an embodiment. [Figure 14] FIG. 1 is a diagram illustrating an example of the configuration of a control system according to an embodiment. [Figure 15] FIG. 11 is a diagram illustrating an example of a processing procedure in a learning method according to the embodiment. [Figure 16] FIG. 1 is a schematic block diagram illustrating a configuration of a computer according to at least one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] When controlling a control object such as a chemical plant (described later in Example 2), a robot (described later in Example 3), a manufacturing device, or a transport device, the control device according to the embodiment determines the control content for the control object using reinforcement learning. The control object operates according to the control content. It can also be said that the control device operates, for example, in a control system (FIG. 1) that performs the control. As described later in Example 2, the control device according to the embodiment determines the control content for controlling, for example, a chemical plant based on a policy model calculated according to reinforcement learning. An observation device for measuring temperature, pressure, flow rate, etc. is installed in the chemical plant. The control device determines a policy model for determining the control content for each device in the chemical plant based on the measurement results measured by the observation device. Then, the control device determines the control content according to the determined policy model, and controls each device according to the determined content.

[0013] As described later in Example 3, the control device according to the embodiment determines, for example, control content for controlling a robot based on a policy model calculated according to reinforcement learning. The robot to be controlled has multiple joints. An observation device for measuring angles of the joints and the like is installed in a system for controlling the robot. The control device determines a policy model for determining control content for the robot based on the measurement results measured by the observation device. The control device then determines the control content according to the determined policy model, and controls the robot according to the determined content. The application of the control device according to the embodiment is not limited to the above-mentioned examples, and may be, for example, a manufacturing device in a manufacturing factory, a transportation device, or the like.

[0014] <Terminology and Concept Explanation> Terms and concepts for explaining the embodiments will be explained. Reinforcement learning is a method to obtain a decision rule that maximizes the expected value of the cumulative reward in a Markov decision process when the state transition probability is unknown. The decision rule is also called a policy or a control rule.

[0015] A Markov decision process represents a process in which a series of events is repeated: "When in a certain state s, action a is selected and executed according to policy π, a transition is made from state s to state s' according to state transition probability ρ(s',r|s,a), and a reward r is given." A policy may be one that computes actions probabilistically. Alternatively, a policy can be described that computes actions uniquely using a delta distribution. Policies that compute actions uniquely are called deterministic policies, and a t =π(s t ) in a deterministic policy. t Actions to be taken at t is determined to be one. t indicates the action at time t. π is a function that indicates the policy. s tdenotes the state at time t. That is, the policy is the state s t From action a at time t t It can be said that it is a model (or function) that calculates (or determines, or selects) the following:

[0016] The cumulative reward is the sum of rewards obtained over a certain period of time. For example, the cumulative reward R from time t to (t+T) is t is expressed as equation (1).

[0017]

number

[0018] γ is a real constant γ∈[0,1]. γ is also called the discount rate. r t is the reward at time t. For this cumulative reward, state s at time t t , action a t Given the state transition probability ρ and the conditional expected cumulative reward for the policy π, Q π (s t ,a t ) and defined as equation (2).

[0019]

number

[0020] Q in Equation (2) π (s t ,a t ) is called the Q function (or action-value function). E represents the expected value. Furthermore, for a state s in a state set S that includes multiple states, the policy π that maximizes the value of equation (3) is called the optimal policy.

[0021]

number

[0022] Here, action a is sampled from policy π, which is denoted as a~π(·|S).

[0023] Meanwhile, reinforcement learning using the Q-learning method determines the parameters of the Q-function so as to derive the optimal policy (optimal policy) using the Q-function. The Q-function corresponding to the optimal policy is called the optimal Q-function. A model of the Q-function and a model of the policy are prepared, and the model of the Q-function is brought closer to the optimal Q-function through learning, and the model of the policy is brought closer to the optimal policy based on the model of the Q-function. In what follows, the model of the Q-function will be called the Q-function model, and the model of the policy will be called the policy model.

[0024] For example, the value y of the Q function is expressed as in equation (4).

[0025]

number

[0026] y is also called the correct label. θ is a parameter of the policy model. φ is a parameter of the Q function model. φ with a line above it (hereafter referred to as φbar) is a target parameter for stabilizing the update of the Q function model. For the target parameter φbar, the past value of φ is basically used, and it is updated to the value of φ as needed. While the value of the parameter φ is updated during learning and the Q function using φ changes, by delaying the update of the value of the target parameter φbar relative to the update of φ, it is possible to suppress sudden fluctuations in the value of the target y, and it is expected that learning will become more stable. Updating the value of a parameter is also called updating the parameter. When the parameter of a model is updated, the model is also updated. The target parameter is updated according to the update of the parameter.

[0027] In the Q function model, the parameter φ is explicitly stated as "Q φ " Q function model Q φ The Q function shown by φIt is also called "Q φ If "φ" in "Q φ " is a Q-function model of the parameter φ. On the other hand, "Q φ If "φ" in "Q φ " is the Q function of the parameter φ.

[0028] By explicitly stating the parameter θ of the policy π, we can write θ " Policy model π θ The policy indicated by is policy π θ It is also called "π θ If "θ" in "π θ " indicates a policy model. On the other hand, "π θ If "θ" in "π θ " indicates a measure.

[0029] The Q-learning method of the embodiment provides a technique for mitigating overestimation by using multiple Q-function models.

[0030] <Configuration in the embodiment> Fig. 1A is a diagram showing an example of the configuration of a control system according to an embodiment, and Fig. 1B is a block diagram of the control system according to an embodiment.

[0031] In the configuration shown in FIG. 1A, the control system 10 includes an observer 12, a state estimation device 13, a reward calculation device 14, a control implementation device 15, a control decision device 20, a policy model storage device 21, a learning device 30, an experience storage device 31, and an evaluation model storage device 40.

[0032] The controlled object 11 is an object to be controlled. Various controllable things (e.g., a chemical plant, a robot) can be the controlled object 11. The controlled object 11 may be a part of the control system 10. Alternatively, the controlled object 11 may be an external configuration of the control system 10.

[0033] The observer 12 observes the state of the control target 11. Information output by the observer 12 is information indicating the state of the control target 11. When the control system 10 is a chemical plant, the observer 12 is, for example, a sensor such as a temperature sensor, a humidity sensor, or a pressure sensor. When the control system 10 is a robot, the observer 12 is, for example, an observation device such as an imaging device that photographs the robot and its surroundings, or a GPS (Global Positioning system) that identifies the position of the robot. The state estimation device 13 estimates the state of the controlled object 11 based on the information obtained from the observer 12 .

[0034] The control decision-making device 20 and the control execution device 15 correspond to examples of a control decision means. The control decision device 20 selects a policy model and a policy π by referring to the state estimated by the state estimation device 13 and the policy model stored in the policy model storage device 21, and performs a calculation of the policy π to output a control value. The selection of the policy model and the policy π will be described later.

[0035] The control execution device 15 controls the controlled object 11 according to the control value output by the control decision device 20 .

[0036] For example, the control decision device 20 generates a control value based on a predetermined control side so that the difference between the control target and the state estimated by the state estimation device 13 is reduced based on a control target and the state of the control target 11 relative to the control target. The state of the control target 11 may be either or both of the state detected by the observer 12 and the state estimated by the state estimation device 13. FIG. 1 illustrates a case in which the state estimated by the state estimation device 13 is used as the state of the control target 11, but is not limited thereto. The control decision device 20 may use a control target supplied from outside, may generate a control target by itself, or may use a control target that is determined in advance.

[0037] Furthermore, the control decision device 20 refers to the policy model storage device 21 to select a policy model, and uses the policy model to decide the above-mentioned predetermined control side.

[0038] The policy model storage device 21 stores a policy model that outputs a control value in response to a state input. For example, the policy model storage device 21 stores a policy model including a parameter variable θ and the value of the parameter variable θ. Hereinafter, the policy model including the parameter variable θ is referred to as a policy model body. A policy model is obtained by setting a value for the parameter variable θ in the policy model body. This policy model is registered by learning using a learning device 30 described later.

[0039] The reward calculation device 14 is used, for example, for learning by the learning device 30. The reward calculation device 14 acquires the reward according to, for example, a "score (reward) calculation rule for a state" specified by the user. However, the method by which the reward calculation device 14 acquires the reward is not limited to a specific method. As a method by which the reward calculation device 14 acquires the reward, various methods by which a reward according to a state can be acquired can be used. For example, when obtaining the reward, the reward calculation device 14 may calculate the reward using information output by the observer 12 or information output by the state estimation device 13.

[0040] When the controlled object 11 is in state s, an action a is determined based on a policy. By executing action a in state s, the state of the controlled object 11 transitions from state s to state s'. In response to this, the reward calculation device 14 calculates an index value according to the degree of goodness or badness of state s'. This index value is called a reward. In this respect, the reward can be said to be an index value that represents the goodness (or effectiveness, value, desirability) of a certain action in a certain state. In this case, the more the reward, the better the action, and the less the reward, the worse the action. Alternatively, the index value may be a penalty. In this case, the index value can be said to be an index representing the inappropriateness of a behavior in a certain state. In this case, the more the penalty, the worse the behavior, and the less the penalty, the better the behavior. The reward corresponds to an example of a first action evaluation value. The first action evaluation value here is an evaluation value of a first action in a first state. The first action evaluation value is also referred to as a first evaluation value.

[0041] The learning device 30 adds and records, for example, one by one, in the experience storage device 31, the state s output by the state estimation device 13, the action a of the controlled object due to the control value output by the control decision device 20, the reward r output by the reward calculation device 14, and the state output by the state estimation device 13 after the action a under the control of the control execution device 15 (i.e., the set of state s' after state transition (s, a, r, s'), also represented as "experience"). Here, "one by one" means, for example, every time the control execution device 15 performs control over the controlled object 11. The learning device 30 does not have to add (or record) experience in the experience storage device 31 one by one.

[0042] Then, the learning device 30 refers to the policy model storage device 21, the evaluation model storage device 40, and the experience storage device 31, and updates the policy model storage device 21 and the evaluation model storage device 40. Specifically, the learning device 30 refers to the models and experiences stored in these storage devices, and updates the parameters of these models.

[0043] Fig. 2 is a diagram showing an example of the configuration of the evaluation model storage device 40. In the configuration shown in Fig. 2, the evaluation model storage device 40 includes, for example, a first Q function model storage device 41 and a second Q function model storage device 42.

[0044] The first Q-function model storage device 41 stores the parameters φ of the first Q-function model described above. 1 The second Q-function model storage device 42 stores the parameters φ 2 Remember. Moreover, the evaluation model storage device 40 stores a Q function model body common to the first Q function model and the second Q function model. Either one or both of the first Q function model storage device 41 and the second Q function model storage device 42 may store the Q function model body. Alternatively, the evaluation model storage device 40 may have a storage area different from the first Q function model storage device 41 and the second Q function model storage device 42 and store the Q function model body.

[0045] As a result, the evaluation model storage device 40 stores two Q function models used to evaluate the performance of the policies recorded in the policy model storage device 21 and to alleviate the overestimation problem of the Q function model described above. In particular, the evaluation model storage device 40 stores the parameters of each of these two Q function models.

[0046] As described above, a case is illustrated in which a plurality of different Q function models can be configured by applying independently determined parameter values ​​to a common Q function model body. As a result, each of the plurality of Q function models outputs a unique value. In the following description, the description of using this Q function model body may be omitted, and the description may be made instead of applying each Q function model.

[0047] Fig. 3 is a diagram showing an example of the configuration of the learning device 30. In the configuration shown in Fig. 3, the learning device 30 includes an experience acquisition unit 34, a mini-batch storage device 35, a model updating unit 50, and a model calculation unit 53. The model updating unit 50 includes a Q-function model updating unit 51 and a policy model updating unit 52.

[0048] The experience acquisition unit 34 samples experiences from the experience storage device 31 according to a predetermined criterion to construct a mini-batch. When constructing a mini-batch, an index for each experience is also included. This is to make it possible to confirm which experience in the experience storage device 31 corresponds to which experience in the mini-batch. The experience acquisition unit 34 corresponds to an example of an experience acquisition means. For example, as the predetermined criterion related to the above sampling, selection criteria such as the number of experiences to be sampled, the size of the mini-batch (or the number of experiences in the mini-batch), and a predetermined priority order of sampling may be applied. The predetermined priority order may utilize a priority such as giving priority to an experience that has been sampled for a relatively short period of time.

[0049] The model of the Q function in the embodiment is configured by combining the above-mentioned multiple function models. Fig. 5 is a diagram showing an example of a model of a Q function according to an embodiment. Fig. 6 is a diagram showing the configuration of one Q function model according to an embodiment.

[0050] The Q-function model 530 includes a number of Q-function models and an evaluator 534 .

[0051] For example, the first Q-function model of multiple Q-function models is the Q-function Q φbar1 The second Q-function model is defined as Q function Q φbar2 The last M-th model of the M_Q function is the Q function Q φbarM The number M of Q function models is an integer of 2 or more and may be appropriately determined. For example, if the value of M is set to 2, two Q function models are used, and if the value of M is set to 3, three Q function models are used. In this way, a configuration using three or more Q function models is also possible depending on the value of M. In the following embodiment, the configuration using two Q function models will be mainly described to simplify the explanation. Each Q function model uses data on a certain state s and data on a certain action a as input data. Each Q function model executes a calculation using parameters of the Q function model based on the data on the state s and the data on the certain action a to calculate an evaluation value. The data on the action a is an example of data indicating policy information related to a first action, and the data on the state s is an example of data related to state information related to the first state.

[0052] For example, the above multiple Q-function models include the Q-function Q φbar1 From Q function Q φbarM For example, the calculation block 531 includes a Q function Q φbar1 is assigned to the calculation block 532, and the Q function Q φbar2 is assigned to the calculation block 533, and the Q function Q φbarM The calculation block 531 is assigned a Q function Q defined using the parameter φbar1 of the Q function model. φbar1 The calculation block 532 calculates y1 by performing the calculation of the Q function Q defined using the parameter φbar2 of the Q function model. φbar2 The calculation block 533 calculates y2 by performing the calculation of the Q function Q defined using the parameter φ φbarM Calculate yM by performing the above operation. y1 to yM are scalars.

[0053] The evaluator 534 selects the minimum value from among the calculated y1 to yM, and outputs the selection result as the target y.

[0054] In general, increasing the number of multiple Q function models increases the tendency to mitigate overestimation, but increases the computational load. In this embodiment, multiple Q function models are used, but a case where the number of Q function models can be relatively reduced in response to the above-mentioned tendency will be described. In the description of the embodiment, a case where two Q function models are used will be described as a typical example.

[0055] In addition, overestimation of the Q function is mitigated by adopting the smaller output value of the two Q function models, y1 and y2. In other words, this stabilizes model updates and shortens the time required for learning.

[0056] Next, an example of one Q function will be described with reference to Fig. 6. Here, the calculation block 531 corresponding to the first Q function model is illustrated. The calculation block 532 may also be configured in the same manner as the calculation block 531.

[0057] Q Function Q φbar1 has hidden layers HL1 to HL9. If the left side of Fig. 6 is the input side and the right side is the output side, the hidden layers HL1 to HL9 are arranged in series from the input side to the output side. The processing process by the hidden layers HL1 to HL9 proceeds from left to right in Fig. 6 along the arrows.

[0058] The hidden layer HL1 is a first weight calculation layer (Weight) that performs a first weight calculation. A vector ias, which is an input vector to the hidden layer HL1, is formed by combining a state s and an action a, each of which is represented by a vector, as shown in formula (5). The hidden layer HL1 calculates a hidden vector h from the vector ias. For example, the hidden layer HL1 calculates the hidden vector h by multiplying the transposed vector of the vector ias by the first weighting matrix W, as shown in formula (6). The subscript T is an operator of the transposed vector.

[0059]

number

[0060]

number

[0061] The hidden layer HL2 calculates some of the values ​​of each element of the input vector h using the Q function Q φbariThe hidden layer HL2 performs a calculation process including a calculation process (called a dropout calculation) that does not reflect a part of the calculation result on the evaluation value of the control object, thereby generating an output vector h'. The hidden layer HL2 is an example of a dropout calculation layer (Dropout) that performs a first dropout calculation. The dropout calculation layer performs a dropout calculation that does not reflect a part of the calculation result based on policy information (information related to a policy model) for transitioning the state of the control object and state information of the control object on the evaluation value.

[0062] For example, the hidden layer HL2 probabilistically changes some of the values ​​of each element of the input vector h to 0. The values ​​of elements that are not changed to 0 may be the same as the values ​​of each element of the vector h. More specifically, the hidden layer HL2 generates a random value (rand) ranging from 0 to 1 for each element of the vector h, and sets the value of the element to 0 if each random value falls below a preset threshold (dropout rate), and maintains the values ​​of elements that do not fall under the above. The hidden layer HL2 then outputs the result of the calculation process as a vector h'. The size of the vector h is the same as the size of the vector h'.

[0063] For example, the calculation process by the hidden layer HL2 is shown in equation (7). Dropout(·) indicates the function of the dropout calculation for the vector.

[0064]

number

[0065] Here, the i-th element of vector h is h i Let the i-th element of vector h' be h' i The above relationship is defined as the following equation (8). As shown in equation (8), the value may be replaced with 0 based on the value of a random number (rand). The replacement with 0 can be considered as noise (called the first noise) for the normal value.

[0066]

number

[0067] The hidden layer HL3 performs an operation to normalize the input vector, vector h', to generate an output vector, vector h''. The hidden layer HL3 is provided after the hidden layer HL2 that performs the dropout operation described above. The hidden layer HL3 includes a layer normalization layer that normalizes the values ​​of elements included in the output of the hidden layer HL2 (dropout operation layer) based on the output of the hidden layer.

[0068] For example, the hidden layer HL3 calculates the average and standard deviation of each element of vector h' and performs a normalization operation to generate an output vector, vector h''. The normalization operation is, for example, a process of dividing the difference between the element value and the average by the standard deviation, as shown in equation (10). The hidden layer HL3 then outputs the result of the operation as vector h''. The size of vector h is the same as the size of vector h'. The operation by the hidden layer HL3 is shown in equation (9). LayerNorm(·) is a function for the normalization operation. A more specific example of the operation formula is shown in equation (10). |h'| in equation (10) indicates the number of elements of vector h'.

[0069]

number

[0070]

number

[0071] The hidden layer HL4 performs an operation of applying an activation function to the input vector, vector h'', to generate an output vector, vector h'''. The hidden layer HL4 is provided after the hidden layer HL4 that performs the normalization operation described above. For example, the hidden layer HL4 includes an activation function operation layer that performs an operation of applying a ReLU (Rectified Linear Unit) function including a ramp function to the output of the hidden layer HL3 (normalization operation layer). The operation process by the hidden layer HL4 is shown in formula (11). ReLU(·) indicates the activation function. A more specific example of the operation formula is shown in formula (12). The hidden layer HL4 outputs the result of the operation process as vector h'''. The size of the vector h'' is the same as the size of the vector h'''. The hidden layer HL4 is an example of an identification layer that identifies the output of the layer normalization layer, which is the hidden layer HL3.

[0072]

number

[0073]

number

[0074] Next, hidden layers HL5 to HL8 will be described. Hidden layer HL5 performs the same arithmetic processing as the previously described hidden layer HL1. Hidden layer HL6 performs the same arithmetic processing as the previously described hidden layer HL2. Hidden layer HL7 performs the same arithmetic processing as the previously described hidden layer HL3. Hidden layer HL8 performs the same arithmetic processing as the previously described hidden layer HL4. The input vectors, output vectors, internal arithmetic coefficients, thresholds, sizes of input vectors and output vectors in each layer from hidden layer HL5 to hidden layer HL8 are different from those in hidden layer HL1 to hidden layer HL4.

[0075] For example, hidden layer HL5 takes vector h''' generated by hidden layer HL4 as an input vector. Hidden layer HL5 is a second weight calculation layer (Weight) that performs a second weight calculation. The processing in hidden layer HL5 is similar to the calculation processing shown in equation (6), but differs from the processing in hidden layer HL1 in that vector ias in equation (6) is vector h''', the weight matrix used in the second calculation is second weight matrix W, and vector h in equation (6) is vector (h)'. The magnitude and element values ​​of the second weight matrix W may be different from those of the first weight matrix W.

[0076] Similarly, the above-mentioned formulas can be applied to each calculation of hidden layer HL6 to hidden layer HL8. For example, the processing in hidden layer HL6 is similar to the computational processing shown in equation (7), but differs from the processing in hidden layer HL2 in that vector h in equation (7) is vector (h)' and vector h' in equation (7) is vector (h')'. The processing in hidden layer HL7 is similar to the computational processing shown in equation (9), but differs from the processing in hidden layer HL3 in that vector h' in equation (9) is vector (h')' and vector h'' in equation (9) is vector (h'')'. The processing in hidden layer HL8 is similar to the computational processing shown in equation (11), but differs from the processing in hidden layer HL4 in that vector h'' in equation (11) is vector (h'')' and vector h''' in equation (11) is vector (h''')'.

[0077] Although detailed explanations related to the application of the above-mentioned equations (8), (10), and (12) will be omitted, it is advisable to use equations (8), (10), and (12) by referring to the explanations of the above equations (7), (9), and (11). This completes the calculation of the hidden layer HL8, and the vector (h''')' is calculated to replace the vector h'''.

[0078] The hidden layer HL9 is a third weight calculation layer (Weight) that performs a third weight calculation. The input vector to the hidden layer HL9 is the vector (h''')', which is the same as the vector h''' calculated by equations (13) and (13). The hidden layer HL9 calculates a scalar value using the vector (h'')' and a predetermined weight vector. The scalar value calculated by the hidden layer HL9 is treated as the output of the Q function.

[0079]

number

[0080] The arithmetic processing according to the embodiment utilizes a Q function including the above arithmetic processing.

[0081] The model update unit 50 refers to the mini-batches stored in the mini-batch storage device 35 and updates the parameters φ 1 , φ 2 and θ. The model updating unit 50 corresponds to an example of a model updating means. As mentioned above, the parameter φ 1 are parameters of the first Q-function model. The first Q-function model storage device 41 stores the parameters φ 1 The parameter φ is stored. 2 are parameters of the second Q-function model. The second Q-function model storage device 42 stores the parameters φ 2 The policy model storage device 21 stores the parameter θ.

[0082] The Q-function model update unit 51 updates the parameter φ 1 and φ 2 The Q function model update unit 51 corresponds to an example of an evaluation model update means.

[0083] The policy model updating unit 52 updates the parameter .theta.. The policy model updating unit 52 corresponds to an example of a policy model updating means.

[0084] The model calculation unit 53 calculates the values ​​of the first Q function model, the second Q function model, and the policy model. For example, when the Q function model update unit 51 updates each of the first Q function model and the second Q function model, the model calculation unit 53 calculates the values ​​of the first Q function model, the second Q function model, and the policy model. The model calculation unit 53 corresponds to an example of a model calculation means. For example, the model calculation unit 53 uses a first Q function model and a second Q function model (Q function) that perform a weighting operation on the above-mentioned policy information and the above-mentioned state information, add noise to the result of the weighting operation, normalize the value of the operation result to which the noise has been added, identify the normalized result according to a predetermined identification rule, and generate an evaluation value indicating the learning status based on the result of the identification (identification result). The model calculation unit 53 uses the above-mentioned first Q function model and second Q function model (Q function) to generate an evaluation value indicating the learning status. This will be described in detail later.

[0085] The parameter storage unit 57 stores hyperparameters used in the learning process. The parameter acquisition unit 58 acquires the above hyperparameters and adds them to the parameter storage unit 57. The hyperparameters are parameters determined by a user or the like among the parameters used in the learning process. For example, a scenario, an operation mode, or the like is specified by a user or the like. The above scenarios and operation modes are associated with hyperparameters that can identify them. The learning device 30 uses these hyperparameters to carry out learning processing in a desired scenario and operation mode. Parameter G, which will be described later, is an example of a hyperparameter.

[0086] More specifically, for example, the parameter acquisition unit 58 accepts and acquires at least any of the following information: the number of Q functions involved in model updating, the number of layers of the calculation layer (dropout calculation layer) that adds noise in the Q function, the number of layers of the layer normalization layer that normalizes the output based on the output of the previous layer in the Q function, and the number of times the value propagation calculation is performed depending on the operation mode of the control target 11.

[0087] <Processing in the embodiment> 4 is a flowchart showing an example of a procedure of processing performed by the control system 10. The control system 10 repeatedly performs the processing of FIG. 4, the control system 10 acquires control parameters (parameters G of the learning process) associated with a scenario, an operation mode, etc. designated by a user or the like, by the learning device 30 (step S100), and stores the acquired control parameters in the parameter storage unit 57 as control parameters. The parameters G are an example of hyperparameters. Under the conditions specified by the parameters G, the control system 10 performs the following processes.

[0088] The observer 12 observes the control target 11 (step S101). For example, the observer 12 observes the control target 11 and its surrounding environment.

[0089] Next, the state estimation device 13 estimates a state regarding the control target 11 based on the observation information of the observer 12 (step S102). For example, the state estimation device 13 estimates a state that may affect the control of the control target 11, such as estimating a state including the control target 11 and its surrounding environment.

[0090] Next, the control decision device 20 determines an action to be taken in the above-mentioned estimated state according to the state estimated by the state estimation device 13 and the policy model acquired by referring to the policy model storage device 21, and calculates a control value according to the determined action (step S103). Next, the control execution device 15 executes control of the control target 11 according to the control value output by the control decision device 20 (step S104).

[0091] Next, the reward calculation device 14 refers to the state estimated by the state estimation device 13 and the control value output by the control decision device 20, and calculates a reward based on, for example, the estimated value of the state of the control target 11 and the observation result of the control result based on the above control value or the estimated result of the state (step S105). Note that, as one example of the above, the reward calculation device 14 may use the squared error between the control target value on which the control value is based and the detection value based on the observation result to calculate the reward.

[0092] Next, the learning device 30 adds and records a set of the state estimated by the state estimation device 13, the control value output by the control decision device 20, and the reward output by the reward calculation device 14 as experience in the experience storage device 31 (step S106).

[0093] Next, the learning device 30 refers to the policy model stored in the policy model storage device 21, the Q function model stored in the evaluation model storage device 40, and the experience stored in the experience storage device 31, and updates these models (step S107). Specifically, the policy model update unit 52 updates the parameter θ of the policy model stored in the policy model storage device 21. The Q function model update unit 51 updates the parameter φ of the Q function model stored in the evaluation model storage device 40. 1 and φ 2 Update. After step S107, the control system 10 ends the process of Fig. 4. As described above, the control system 10 repeats the series of processes from steps S101 to S107 again.

[0094] 7 is a diagram for explaining an example of a processing procedure in which the control system 10 according to the embodiment updates a model. The control system 10 may perform the processing in FIG. 6 by using an algorithm shown in FIG.

[0095] Step S1: The learning device 30 initializes the policy parameters θ of the policy model and the parameters φ1 and φ2 of the two dropout Q-functions, empties the reproduction buffer D, and sets the target parameters φbar1 and φbar2 using the parameters φ1 and φ2.

[0096] Step S2: The learning device 30 repeats the following process.

[0097] Step S3: The learning device 30 determines the state s iStrategies in π θ The probability π determined by θ (|s i ) based on a i Decide on the action a i The learning device 30 controls the behavior a i Reward based on results i and the next state s i+1 and generate experience data associating these pieces of information. The learning device 30 adds this experience data to the reproduction buffer D. The experience data to be added is shown in equation (14). The experience data to be added is, for example, a result obtained by observing an event that occurs in real time. The learning device 30 stores this in the experience storage device 31 as time history information. Each experience data may be assigned identification information k that allows it to be uniquely identified.

[0098]

number

[0099] Step S4: The learning device 30 updates the hyperparameter G and repeats the processes from step S5 to step S9.

[0100] Step S5: The learning device 30 extracts a specific mini-batch B associated with the hyper-parameter G from the experience data stored in the reproduction buffer D of the experience storage device 31. The extracted mini-batch B is shown in equation (15). s, a, r, and s' in this equation (15) represent the state s of the experience data included in the extracted mini-batch B. t , action a i、 reward r i , and state s t+1 and , respectively. The empirical data for mini-batch B may include a data set for the observed experiences over a given period of time.

[0101]

number

[0102] Step S6: The learning device 30 calculates the target y of the dropout Q-function based on the extracted mini-batch B according to the following formula (16). The dropout Q-function is an example of the Q-function shown in FIG. 6 above. In the following description, the dropout Q-function will be simply called the Q-function. The Q-function in the embodiments not specifically mentioned is not a general Q-function but a dropout Q-function.

[0103]

number

[0104] The second term on the right side of this formula (16) is an example of a Q-function formula to which Maximum entropy RL (reinforcement learning) is applied. The second term in parentheses in the second term on the right side is an entropy term. The magnitude of the calculation result of this entropy term is adjusted so as to give an appropriate amount of fluctuation to the so-called Q-function value of the first term in the parentheses. This makes it possible to suppress falling into a local solution compared to general reinforcement learning that uses the Q-function of the above formula (4) alone. Note that the fluctuation (noise) added by the above dropout Q-function can be defined as the first noise, and the fluctuation (noise) due to this entropy term can be defined as the second noise. The target y calculated by the above formula (16) contains the above two fluctuation components (noise).

[0105] Step S7: The learning device 30 switches the value of the discrimination variable i between 1 and 2, and performs control so as to perform the calculations of steps S8 and S9, respectively.

[0106] Step S8: The learning device 30 updates the parameters φ1 and φ2 by the steepest descent method using equation (17). The learning device 30 selects, for example, the one of the two Q functions whose value in the upper part of equation (17) is smaller. The lower part of equation (17) is an equation for updating the parameter φi using the value in the upper part of the equation.

[0107]

number

[0108] Step S9: The learning device 30 updates the target parameters φbar1 and φbar2, respectively, using the arithmetic expression shown in equation (18) and the Q network parameters φ1 and φ2, respectively, where ρ is a predetermined constant.

[0109]

number

[0110] Step S10: The learning device 30 updates the policy parameter θ by using a hill-climbing method using a gradient based on the arithmetic expression shown in equation (19), where ρ is a predetermined constant.

[0111]

number

[0112] In addition, B in equation (19) is a mini-batch of experiences sampled from an experience storage device that stores experiences. "|B|" is the size of the mini-batch. An "experience" refers to a state transition that occurred in the past. This experience is expressed as (s, a, r, s') which combines state s, action a for state s, reward r according to action a, and next state s' according to action a. The above equation (15) shows the experience (s, a, r, s') contained in mini-batch B.

[0113] Since the target y depends on the parameter φbar, which changes during training, the target y changes during the optimization of the Q-function model. Policy model π θ We assume a deterministic policy for Q φ The parameter θ is updated to output a that maximizes

[0114] When using a general Q function as in the comparative example, one of the factors that takes time to learn the Q function is a problem called the overestimation problem of the Q function. The problem with overestimation of the Q function is Qφbar(s',π θ The target parameter φbar and the synchronization source parameter φ are the part of the policy π θ The true Q-function Q as the expected cumulative reward for πθ If it is not properly approximated, π θ (s) is "not properly approximated Q φ Since the Q function model outputs a that maximizes a, an over-bias is introduced so that the output value of the Q function model is larger than the output value of the true Q function.

[0115] Therefore, in the embodiment, two Q function models are prepared, and the output values ​​are compared to adopt the smaller output value, thereby mitigating the overestimation of the Q function. In other words, this is expected to stabilize the model update and shorten the time required for learning. In the embodiment, a case will be described as an example in which a plurality of Q function models are configured by applying different parameter values ​​to the same Q function model body.

[0116] Equations (17) to (19) represent the parameters φ of the Q-function model. i In the embodiment, two parameters φ 1 , φ 2 Since there are two Q-function models, the target parameters are also φbar 1 , φ 2 is used, and the target parameter with the smaller output value is used to calculate the teacher signal.

[0117] Qφbar in equation (18) i ” is a state s′ and a policy π θ The action obtained by applying θ (s') and the Q-function model Qφbar i This shows that it applies to Qφbar i " is given a state s' and takes an action π depending on the state s'. θ (s') is obtained. In this respect, the Q-function model Qφbar i is the action π in state s'. θ It can be said that this is a model that evaluates (or estimates) the goodness (or value, effectiveness, or desirability) of (s'). Q-function model Qφbar i The value of is the action π θ It can be said to be an index of the goodness (or value, usefulness, desirability) of (s').

[0118] State s corresponds to an example of the first state. Action a corresponds to an example of the first action. State s', which is a transition destination when the control target performs action a, which is the first action, in state s, which is the first state, corresponds to an example of the second state. State s', which is the second state, is treated as a policy π θ The action obtained by applying θ (s') is an example of the second action.

[0119] Q function Qφbar i is an example of the second behavior evaluation function. The second behavior evaluation function here is a function that calculates an evaluation value of the second behavior in the second state. Q function with state s' and action π θ (s') and the applied Q function value Qφbar i is an example of a second behavior evaluation value. The second behavior evaluation value here is an evaluation value of a second behavior in a second state. The second behavior evaluation value is also referred to as a second evaluation value.

[0120] Q-function model Qφbar iis an example of the second behavior evaluation function model. The second behavior evaluation function model here is a model of the second behavior evaluation function. By determining the parameter values ​​of the second behavior evaluation function model, the second behavior evaluation function model indicates one second behavior evaluation function.

[0121] However, the evaluation means for the second behavior in the embodiment is not limited to one expressed in the form of a function (second behavior evaluation function). Various means capable of outputting an evaluation value of the second behavior for the input of the second state and the second behavior can be used as the evaluation means for the second behavior. For example, the evaluation means for the second behavior may output an evaluation value having fluctuations such as white noise. In this case, the evaluation means for the second behavior may output different evaluation values ​​for the input of the same second state and second behavior. In this way, the evaluation value of the second behavior in the second state (second evaluation value) in the embodiment includes noise. Q function Qφbar i and the Q-function model Qφbar i are each configured to add randomness to the evaluation value of the second action.

[0122] Since the evaluation means of the second behavior is not limited to one expressed in the form of a function, the evaluation model of the second behavior in the embodiment is also not limited to a model showing a function (second behavior evaluation function model). In this way, the evaluation model of the second behavior that is not limited to a model showing a function is called the second behavior evaluation model, or simply the evaluation model. Q-function model Qφbar i is also an example of a function model.

[0123] As described above, the model calculation unit 53 calculates the state s' according to the action a in the state s of the control target 11 and the policy model π θ Action π calculated using θ Based on (s'), the action π in state s' θ The Q function value Qφbar, which is the index value of (s') 1 (s',π θ (s') and Qφbar 2 (s',π θTwo Q-function models Qφbar 1 and Qφ 2 The Q function values ​​are calculated using

[0124] As described above, state s corresponds to an example of the first state. Action a corresponds to an example of the first action. State s' corresponds to an example of the second state. Action π θ (s') corresponds to an example of the second action. Q function value Qφbar 1 and Qφ 2 is an example of the second evaluation value. 1 and Qφ 2 is an example of an evaluation model.

[0125] The model update unit 50 calculates the Q function value Qφbar 1 and Qφ 2 Based on the smaller Q-function value of Qφbar and the reward r, 1 and Qφ 2 The reward r corresponds to an example of a first evaluation value that is an index value of the goodness of the action a in the state s.

[0126] In this way, the learning device 30 can estimate the evaluation of an action using a Q function with a relatively small value by learning each Q function model using a plurality of Q function models. This can mitigate overestimation of the evaluation of an action, such as overestimation of the Q function model. In this respect, the learning device 30 can shorten the time required for reinforcement learning.

[0127] This enables the learning device 30 to learn the Q function model by preferentially using experiences that result in a large error in the Q function value, and is expected to be able to efficiently improve the error. In this respect, the learning device 30 can reduce the time required for reinforcement learning.

[0128] Next, verification results of the embodiment will be described with reference to Fig. 8 to Fig. 10. Fig. 8 to Fig. 10 are diagrams showing verification results of the embodiment. The distribution in the graph shown in Figure 8 indicates the relationship between the number of interactions with the environment and the average return of reward. In the graph shown in Figure 8, the horizontal axis represents the number of interactions with the environment, and the vertical axis represents the average return of reward. The sample efficiency of the reinforcement learning device can be read from this graph in Figure 8. The shading in Figure 8 indicates the range in which the reward value varies for each interaction.

[0129] For example, the fewer the number of samplings until the average reward gain reaches a predetermined value, that is, the fewer the number of interactions with the environment, the more efficiently the learning of the reinforcement learning device is progressing. The sample efficiency shown here indicates the overall performance of the learning characteristics of the reinforcement learning device. The further to the top left in the graph, the higher the sample efficiency.

[0130] The solid line indicates the sample efficiency of the comparative example, and the dashed line indicates the sample efficiency of this embodiment, and so on. From Fig. 8, it can be seen that the result of this embodiment (dashed line) is superior to the comparative example (solid line) in terms of sample efficiency.

[0131] The graph in Fig. 9 shows the performance of reducing overestimation bias. In the graph in Fig. 9, the horizontal axis represents the number of interactions with the environment, and the vertical axis represents the average bias of the difference between the actual and estimated results. From this graph in Fig. 9, it is possible to read how much the estimated results of the reinforcement learning device deviate from the actual ones. The shading in Fig. 9 indicates the range of variation in the difference between the estimated results of the reinforcement learning device and the actual ones for each interaction.

[0132] For example, the closer the value on the vertical axis is to 0, the more accurate the estimation is, and the faster it approaches 0, the faster the performance of reducing the difference between the actual and estimated results (referred to as reduction performance) is. From Fig. 9, it can be seen that the present embodiment has better performance of reducing overestimation bias than the comparative example.

[0133] The graph in Fig. 10 shows the reduction performance of the variance of the Q function value. In the graph in Fig. 10, the horizontal axis is the number of interactions with the environment, and the vertical axis is the square root of the variance of the Q function value. From this graph in Fig. 10, it is possible to read whether the estimation of the Q function varies. The shading in Fig. 10 indicates the range in which the square root of the variance of the Q function varies for each interaction.

[0134] For example, the closer the standard deviation of the bias is to 0, the higher the performance of reducing the variance of the Q function value. From Fig. 10, it can be seen that the embodiment has a higher performance of reducing the variance of the Q function value than the comparative example while the number of interactions with the environment is small.

[0135] According to the above embodiment, the model calculation unit 53 of the learning device 30 calculates a state s' (second state) corresponding to an action a (first action) in a state s (first state) of the control object 11, and a policy π θ Action π calculated using (policy model) θ Based on (s') (second action), action π in state s' θ The Q function value Qφbar, which is an index value that indicates the evaluation result of (s') with noise added i (2nd evaluation value) is calculated by using multiple Q function models (evaluation models) to obtain a noisy Q function value Qφbar i The model update unit 50 calculates a plurality of Q function values ​​Qφbar i The smallest Q function value Qφbar among (the second evaluation value) i (the second evaluation value) and a reward r (the first evaluation value), which is an index value indicating the evaluation result of the action a in the state s, θ (policy model) or its parameter variable θ is updated.

[0136] Hereinafter, some specific application examples will be given and the embodiments will be described. EXAMPLES

[0137] FIG. 11 is a diagram illustrating an example of a pendulum to be controlled in the first embodiment. In the first embodiment, an example will be described in which the control system 10 inverts a pendulum as shown in FIG. 11. FIG. 11 is an elevation view of the pendulum as seen from the axial direction. For example, the +X axis is set to the right in FIG. 11, the +Z axis is set to the upward direction in FIG. 11, and the +Y axis is set to intersect with the surface of FIG. 11 in the depth direction. The axis of the pendulum extends in the Y direction. Pendulum 11A in FIG. 11 corresponds to an example of the controlled object 11. A motor is attached to the axis of this pendulum 11A, and the movement of pendulum 11A can be controlled by the motor. Here, the objective of the first embodiment is to learn an automatic control law (a measure for automatic control) that controls the motor to invert the pendulum 11A within a time limit of 100 seconds (position POS3 in FIG. 11) and maintain the inverted state for as long as possible.

[0138] However, the torque of this motor is not so strong that, for example, the pendulum 11A cannot be inverted by moving it directly from position POS1 to position POS3. For this reason, in order to invert the pendulum 11A at position POS1, it is first necessary to apply torque to move it to position POS2, for example, to store a certain amount of potential energy, and then apply an appropriate torque in the opposite direction to bring it to position POS3. In the first embodiment, unless otherwise specified, "π" represents the circular constant and "x" represents an angle.

[0139] In the first embodiment, the observation device 12 is a sensor that measures the angle x of the pendulum 11A. Here, the angle is the angle x around the +Y axis extending in the positive direction of the +Y axis, and the +Z axis direction is taken as the angle reference, with the clockwise rotation direction from the +Z axis direction toward the +X axis being taken as positive and the counterclockwise rotation direction being negative, thereby defining the range of the angle x of the pendulum 11A as x∈[-π,π]. Note that the position POS1 in FIG. 11 corresponds to x=-5π / 6. The position POS2 corresponds to x=5π / 12. The position POS3 corresponds to x=0.

[0140] The state s of the pendulum 11A is represented by angle x, angular velocity x', and angular acceleration x", and is written as (x, x', x"). In addition, in the first embodiment, position POS1 is set as the initial position of the pendulum 11A, and the initial angle is set to -5π / 6. The initial angular velocity and initial angular acceleration are both set to 0.

[0141] The state estimation device 13 estimates the true axis angle x, angular velocity x', and angular acceleration x" from the sensor information of the observer 12, and constructs information on the state s = (x, x', x"). The state estimation device 13 performs state estimation every 0.1 seconds, and outputs state information every 0.1 seconds. The algorithm of the state estimation device 13 is, for example, a Kalman filter.

[0142] The reward calculation device 14 receives information on the state s from the state estimation device 13 and calculates the reward function r(s)=-x 2 This reward function is designed in accordance with the purpose of the first embodiment so that the longer the handstand time is, the higher the cumulative reward is.

[0143] The control implementation device 15 receives a control value c from the control decision device 20 and controls the pendulum 11A. The control value c in the first embodiment is a voltage V applied to the motor, and the range of the control value c is [-2V, +2V]. Furthermore, the control implementation device 15 continues to apply the same voltage to the motor until it receives a new control value c. The control value c indicates the behavior a of the pendulum 11A.

[0144] Also, it is assumed that the processing of the control decision device 20 (step S103 in FIG. 4), the processing of the control execution device 15 (step S104 in FIG. 4), and the processing of the reward calculation device 14 (step S105 in FIG. 4) are completed within 0.01 seconds from the state calculation by the state estimation device 13 (step S102 in FIG. 4). As a result, it is assumed that the control value is changed 0.01 seconds after the state estimation by the state estimation device 13. The control decision interval is 0.1 seconds, the same as the state estimation interval.

[0145] The discrete time labels t=0, 1, 2, 3, ... are defined as the control start time, (control start time + 0.1 seconds), (control start time + 0.2 seconds), (control start time + 0.3 seconds), ..., respectively. The state vectors estimated for the control start time, (control start time + 0.1 seconds), (control start time + 0.2 seconds), (control start time + 0.3 seconds), ... are defined as s 0 , s 1 , s 2 , s 3 The control values ​​calculated for the control start time, (control start time + 0.1 seconds), (control start time + 0.2 seconds), (control start time + 0.3 seconds), ... are denoted as c 0 , c 1 , c 2 , c 3 , .... Control value c 0 , c 1 , c 2 , c 3 The motion of the pendulum 11A shown by a, ... is 0 , a 1 , a 2 , a 3 The reward values ​​calculated for the control start time, (control start time + 0.1 seconds), (control start time + 0.2 seconds), (control start time + 0.3 seconds), ... are denoted as r 0 , r 1 , r 2 , r 3 , ... will be written.

[0146] The control decision device 20 receives the state s from the state estimation device 13, performs calculations on the policy model by referring to the policy model stored in the policy model storage device 21, and transmits the calculation result to the control implementation device 15 as a control value c. In the first embodiment, the policy model is a fully connected neural network with two hidden layers, where the input layer receives a state s and the output layer outputs a control value c. The number of nodes per hidden layer is 256, and the tanh function is used as the activation function. All parameters of this neural network model are stored in the policy model storage device 21.

[0147] The experience memory device 31 stores the state s estimated by the state estimation device 13 at each time t. t , the control value c output by the control decision device 20 t , the reward value r output by the reward calculation device 14 t , and the state s estimated by the state estimation device 13 at the next time (t+1) t+1 A set of (s t ,c t ,r t ,s t+1 ), i.e., "experience", is recorded sequentially. As mentioned above, the control value c t is action a t Shows.

[0148] The model stored in the first Q-function model storage device 41 of the evaluation model storage device 40 and the model stored in the second Q-function model storage device 42 are both fully connected neural networks with two hidden layers, similar to the policy model, with each hidden layer having 256 nodes and using the tanh function as the activation function. However, the input layer receives a pair of a state and a control value (s, c), and the output layer outputs a value of Q(s, c).

[0149] The experience acquisition unit 34 of the learning device 30 samples new experiences and adds them to the experience storage device 31 . The learning device 30 performs the learning process according to the process shown in FIG. 4 or the algorithm shown in FIG.

[0150] According to the technology of this embodiment, in the above-mentioned "inverted pendulum" problem, a policy model for inverting the pendulum can be acquired with "fewer experiences" compared to a case where the technology of this embodiment is not used. EXAMPLES

[0151] In the second embodiment, an example will be described in which the control system 10 performs automatic control of a VAM (Vinyl Acetate Monomer) plant, which is a type of chemical plant. Here, the VAM plant simulator is the controlled object 11, but if the VAM plant simulator sufficiently reproduces reality, the controlled object 11 may be replaced with an actual VAM plant after learning the policy model. In the second embodiment, the explanation will be given on the assumption that the controlled object 11 is replaced with an actual VAM plant.

[0152] Figure 12 is a diagram showing an example of the section configuration in a VAM plant. The VAM plant is composed of seven sections that fulfill different roles. In section 1, the raw materials for VAM are mixed. In section 2, chemical reactions take place to produce VAM. In sections 3 to 5, the VAM is separated, compressed and collected. In sections 6 to 7, the VAM is distilled and precipitated. The VAM obtained through this series of processes is sold as a product.

[0153] The VAM plant in Example 2 is equipped with approximately 100 observation devices for measuring pressure, temperature, flow rate, etc., and approximately 30 PID controllers (Proportional-Integral-Differential Controllers) for adjusting pressure, temperature, flow rate, etc. In Example 2, the objective is to obtain a policy model that increases the overall profit of this VAM plant. Here, the overall profit is the product profit (VAM) minus the consumption costs (ethylene, acetic acid, oxygen, electricity, water, etc.).

[0154] The control time for the VAM plant is set to 100 hours, and the ultimate goal is to improve the cumulative total profit during this control time to a value higher than that obtained when the initial state is continued. The initial state here refers to a state in which the target values ​​of each PID controller are manually adjusted and the VAM plant as a whole reaches a steady state. This initial state is prepared in advance in the VAM plant simulator.

[0155] In the second embodiment, the observation device 12 is configured using about 100 pieces of the observation devices described above. The VAM plant simulator used can obtain important physical quantities that cannot be measured by the observation devices, but these quantities are not used. This is because the VAM plant simulator is replaced with an actual VAM plant.

[0156] The state estimation device 13 estimates physical quantities such as true temperature, pressure, flow rate, etc. from the information from the observation device 12, and constructs the state. It is assumed that state estimation is performed every 30 minutes, and state information is also output every 30 minutes. The algorithm of the state estimation device 13 is assumed to use, for example, a Kalman filter.

[0157] The reward calculation device 14 receives the state s from the state estimation device 13 and calculates the above-mentioned total profit, r(s). The calculation method complies with the VAM plant simulator. The higher the total profit, the higher the reward. The control implementation device 15 receives a control value c from the control decision device 20 and controls the VAM plant simulator. The control value c in the second embodiment is a target value of each PID controller. The control implementation device 15 maintains the same target value until it receives a new control value c. The control value c indicates the behavior a of the VAM plant.

[0158] Also, it is assumed that the processing of the control decision device 20 (step S103 in FIG. 4), the processing of the control execution device 15 (step S104 in FIG. 4), and the processing of the reward calculation device 14 (step S105 in FIG. 4) are completed within one second from the state calculation by the state estimation device 13 (step S102 in FIG. 4). As a result, it is assumed that the control value is changed one second after the state estimation by the state estimation device 13. The control decision interval is 30 minutes, the same as the state estimation interval.

[0159] The discrete time labels t=0, 1, 2, 3, ... are defined as the control start time, (control start time + 30 minutes), (control start time + 60 minutes), (control start time + 90 minutes), ..., respectively.

[0160] The control decision-making device 20, the policy model storage device 21, the learning device 30, the experience storage device 31, and the evaluation model storage device 40 are the same as those in the first embodiment, and therefore the description thereof will be omitted.

[0161] The two effects in the embodiment 2 are similar to those in the embodiment 1. As a result, a policy model for improving the overall profit can be acquired with "fewer experiences" compared to the case where the technology of the present invention is not used, and if the VAM plant simulator sufficiently reproduces reality, the same overall profit improvement can be obtained even if the policy model is applied to an actual VAM plant. EXAMPLES

[0162] In the third embodiment, a case will be described in which the control system 10 automatically controls a humanoid robot. As in the second embodiment, the third embodiment will be described with the policy model learned by simulation being applied to an actual control target. That is, the control target 11 here is a humanoid robot on a simulator, and the application of a policy obtained using the simulator to the actual humanoid robot will be considered.

[0163] In the third embodiment, the final objective is to obtain a policy model that allows the humanoid robot to continue walking on two legs without falling over within a control time of 100 seconds. The humanoid robot to be controlled has 17 joints, each of which is equipped with a motor. The observer 12 includes a sensor that measures the angle and torque of each joint, and a LIDAR (Light Detection and Ranging) mounted on the head. The simulator used can also obtain important physical quantities that cannot be measured by the observer 12, but these are not used. This is because the model is also applicable to an actual humanoid robot.

[0164] The state estimation device 13 estimates the true angle, angular velocity, angular acceleration, torque, absolute coordinates of the robot's center of gravity, center of gravity velocity, and load on each joint from the information of the observer 12, and configures the state. State estimation is performed every 0.1 seconds, and state information is also output every 0.1 seconds. The algorithm of the state estimation device 13 is, for example, a Kalman filter or SLAM (Simultaneous Localization And Mapping), etc.

[0165] The reward calculation device 14 receives as input a set (s, c, s') of the state s output by the state estimation device 13, the control value c output by the control decision device 20, and the state output by the state estimation device 13 immediately after the control value c is implemented by the control implementation device 15, i.e., the state s' after the state transition, and calculates a reward function r(s, c, s'). The control value c indicates the behavior of the robot.

[0166] The method of calculating rewards complies with OpenAI's gym. Basically, the faster the forward speed of the humanoid robot's center of gravity, the higher the reward. Also, to save as much power as possible, the stronger the torque produced by the motor, the more points are deducted. Also, bonus points are given if the center of gravity is kept high to prevent the humanoid robot from falling over.

[0167] The control execution device 15 receives the control value c from the control decision device 20 and controls the torque of the motor of each joint. In addition, the processing of the control decision device 20 (step S103 in FIG. 4), the processing of the control execution device 15 (step S104 in FIG. 4), and the processing of the reward calculation device 14 (step S105 in FIG. 4) are completed within 0.01 seconds from the state calculation of the state estimation device 13 (step S102 in FIG. 4). As a result, the control value is changed 0.01 seconds after the state estimation in the state estimation device 13. The control decision interval is 0.1 seconds, which is the same as the state estimation interval. In addition, the discrete time label t is defined according to the timing of the state estimation, as in the first embodiment.

[0168] The control decision device 20, the policy model storage device 21, the learning device 30, the experience storage device 31, and the evaluation model storage device 40 are similar to those in the first embodiment, and therefore the description thereof will be omitted here.

[0169] The two effects in Example 3 are the same as those in Example 1. As a result, a policy model for a humanoid robot to walk on two legs without falling can be acquired with "fewer experiences" compared to the case where the technology of the present invention is not used, and if the humanoid robot model sufficiently reproduces reality, a comparable improvement in overall profit can be achieved even when the policy model is applied to an actual humanoid robot.

[0170] 13 is a diagram showing an example of the configuration of a learning device according to an embodiment. In the configuration shown in FIG. In this configuration, the model calculation unit 511 calculates the second evaluation value using a plurality of evaluation models that calculate a second evaluation value that is an index value of the second action in the second state, based on the second state corresponding to the first action in the first state of the control target and the second action calculated from the second state using the policy model. The model update unit 512 updates the policy model or a parameter θ of the policy model, based on the smallest second evaluation value among the plurality of second evaluation values ​​and the first evaluation value that is an index value of the first action in the first state. The model calculation unit 511 corresponds to an example of a model calculation means, and the model update unit 512 corresponds to an example of a model update means. As described above, the model calculation unit 511 of the embodiment intentionally includes noise (fluctuation, randomness) in the second evaluation value. In other words, the model calculation unit 511 calculates the noise-containing second evaluation value by using a plurality of evaluation models that calculate second evaluation values ​​in which noise is included in an index value indicating an evaluation result of the second action in the second state based on the second state corresponding to the first action in the first state of the control target and the second action calculated from the second state using the policy model. The model update unit 512 updates the policy model or a parameter θ of the policy model based on the smallest second evaluation value among the plurality of second evaluation values ​​calculated respectively and the first evaluation value, which is an index value indicating an evaluation result of the first action in the first state. It goes without saying that, for example, the goodness of the action or other indexes described above may be applied to the evaluation of the action.

[0171] In this way, the learning device 510 can estimate the evaluation function using an evaluation function with a relatively small value by learning the evaluation function using a plurality of evaluation functions. This can mitigate overestimation of the evaluation function, such as overestimation of the Q-function model. In this respect, the learning device 510 can reduce the time required for reinforcement learning.

[0172] The model calculation unit 511 can be realized, for example, by using the functions of the model calculation unit 53 as exemplified in Fig. 3. The model update unit 512 can be realized, for example, by using the functions of the model update unit 50 as exemplified in Fig. 3. Thus, the learning device 510 can be realized by using the functions of the learning device 30 as exemplified in Fig. 3.

[0173] Fig. 14 is a diagram showing an example of the configuration of a control system according to an embodiment. In the configuration shown in Fig. 14, a control system 520 includes a model calculation unit 521, an evaluation model update unit 522, a policy model update unit 523, a control decision unit 524, and a control implementation unit 525.

[0174] In this configuration, the model calculation unit 521 calculates the second evaluation value using a plurality of evaluation models that calculate a second evaluation value that is an index value of the goodness of the second action in the second state based on the second state corresponding to the first action in the first state of the control object and the second action calculated from the second state using the policy model. The evaluation model update unit 522 updates the evaluation model based on the smallest second evaluation value among the plurality of second evaluation values ​​and the first evaluation value that is an index value of the goodness of the first action in the first state. The policy model update unit 523 updates the policy model using the evaluation models. The control decision unit 524 calculates a control value using the policy model. The control implementation unit 525 controls the control object based on the control value.

[0175] The model calculation unit 521 corresponds to an example of a model calculation means. The evaluation model update unit 522 corresponds to an example of an evaluation model update means. The policy model update unit 523 corresponds to an example of a policy model update means. The control decision unit 524 corresponds to an example of a control decision means. The control implementation unit 525 corresponds to an example of a control implementation means.

[0176] In this way, in the control system 520, by learning the evaluation function using a plurality of evaluation functions, it is possible to estimate the evaluation function using an evaluation function with a relatively small value. This makes it possible to alleviate overestimation of the evaluation function, such as overestimation of the Q function model. In this respect, the control system 520 can shorten the time required for reinforcement learning.

[0177] The model calculation unit 521 can be realized, for example, by using the function of the model calculation unit 53 as illustrated in FIG. 3. The evaluation model update unit 522 can be realized, for example, by using the function of the Q-function model update unit 51 as illustrated in FIG. 3. The policy model update unit 523 can be realized, for example, by using the function of the policy model update unit 52 as illustrated in FIG. 3. The control decision unit 524 can be realized, for example, by using the function of the control decision device 20 as illustrated in FIG. 1. The control implementation unit 525 can be realized, for example, by using the function of the control implementation device 15 as illustrated in FIG. 1. Thus, the control system 520 can be realized by using the function of the control system 10 as illustrated in FIGS. 1 to 3.

[0178] Fig. 15 is a diagram showing an example of a processing procedure in the learning method according to the embodiment. The learning method shown in Fig. 15 includes a model calculation step (step S511) and a model update step (step S512). In a model calculation step (step S511), a second evaluation value is calculated using a plurality of evaluation models that calculate a second evaluation value, which is an index value of the goodness of the second action in the second state, based on a second state corresponding to the first action in the first state of the control target and the second action calculated from the second state using a policy model. In a model update step (step S512), the evaluation model is updated based on the smallest second evaluation value among the plurality of second evaluation values ​​and the first evaluation value, which is an index value of the goodness of the first action in the first state.

[0179] In the learning method of Fig. 15, by learning the evaluation function using a plurality of evaluation functions, it is possible to estimate the evaluation function using an evaluation function with a relatively small value. This makes it possible to mitigate overestimation of the evaluation function, such as overestimation of the Q function model. In this respect, the learning method of Fig. 15 makes it possible to shorten the time required for reinforcement learning.

[0180] FIG. 16 is a schematic block diagram illustrating a configuration of a computer according to at least one embodiment. In the configuration shown in FIG. 16, a computer 700 includes a CPU 710, a main memory device 720, an auxiliary memory device 730, an interface 740, and a non-volatile recording medium 750. Any one or more of the learning device 30, the learning device 510, and the control system 520, or a part thereof, may be implemented in the computer 700. In this case, the operation of each of the above-mentioned processing units is stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, develops it in the main storage device 720, and executes the above-mentioned processing according to the program. The CPU 710 also secures a storage area corresponding to each of the above-mentioned storage units in the main storage device 720 according to the program. Communication between each device and other devices is executed by the interface 740 having a communication function and performing communication according to the control of the CPU 710. The interface 740 also has a port for the non-volatile recording medium 750, and reads information from the non-volatile recording medium 750 and writes information to the non-volatile recording medium 750.

[0181] When the learning device 30 is implemented in a computer 700, the operations of the experience acquiring unit 34, the model updating unit 50, the Q-function model updating unit 51, and the policy model updating unit 52 are stored in the form of programs in the auxiliary storage device 730. The CPU 710 reads the programs from the auxiliary storage device 730, loads them in the main storage device 720, and executes the above-mentioned processes according to the programs.

[0182] In addition, the CPU 710 allocates a memory area corresponding to the mini-batch memory device 35 in the main memory device 720 in accordance with the program. Communication between the learning device 30 and other devices is performed by the interface 740 having a communication function and operating under the control of the CPU 710.

[0183] When the learning device 510 is implemented in the computer 700, the operations of the model calculation unit 511 and the model update unit 512 are stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-mentioned processing according to the program.

[0184] Furthermore, the CPU 710 allocates a storage area in the main storage device 720 for the processing performed by the learning device 510 in accordance with the program. Communication between the learning device 510 and other devices is performed by the interface 740 having a communication function and operating under the control of the CPU 710.

[0185] When the control system 520 is implemented in the computer 700, the operations of the model calculation unit 521, the evaluation model update unit 522, the policy model update unit 523, the control decision unit 524, and the control execution unit 525 are stored in the form of programs in the auxiliary storage device 730. The CPU 710 reads the programs from the auxiliary storage device 730, loads them into the main storage device 720, and executes the above-mentioned processes according to the programs.

[0186] Furthermore, the CPU 710 allocates a storage area in the main storage device 720 for the processing performed by the control system 520 in accordance with the program. Communications between the control system 520 and other devices, such as transmission of a control signal from the control execution unit 525 to a control target, are performed by the interface 740 having a communication function and operating under the control of the CPU 710.

[0187] Any one or more of the above-mentioned programs may be recorded in non-volatile recording medium 750. In this case, interface 740 may read the program from non-volatile recording medium 750. Then, CPU 710 may directly execute the program read by interface 740, or may temporarily store the program in main storage device 720 or auxiliary storage device 730 and execute it.

[0188] Note that a program for executing all or part of the processing performed by learning device 30, learning device 510, and control system 520 may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to perform processing of each part. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs (Read Only Memory), and CD-ROMs (Compact Disc Read Only Memory), as well as storage devices such as hard disks built into computer systems. The above-mentioned program may be for realizing part of the above-mentioned functions, or may be capable of realizing the above-mentioned functions in combination with a program already recorded in the computer system.

[0189] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Industrial Applicability]

[0190] The embodiments of the present invention may be applied to a learning device, a learning method, a control system, and a recording medium. [Explanation of symbols]

[0191] 10, 520 Control System 11 Control Object 12 Observation Instruments 13 State Estimation Device 14 Reward Calculation Device 15 Control implementation device 20 Control decision device 21 Strategy Model Memory Device 30, 510 Learning device 31 Experience Memory System 34 Experience Acquisition Department 35 Mini-batch storage 40 Evaluation model storage device 41 1st Q function model storage device 42 2nd Q function model storage device 50, 512 Model Update Section 51 Q function model update section 52, 523 Policy Model Update Section 53, 511, 521 Model Calculation Section 522 Evaluation Model Update Department 524 Control Decision Unit 525 Control Implementation Department

Claims

1. a model calculation unit that calculates, based on a second state corresponding to a first action in a first state of a control target and a second action calculated from the second state using a policy model, a second evaluation value that includes noise in an index value that indicates an evaluation result of the second action in the second state, by using a plurality of evaluation models, and calculates each of the second evaluation values ​​including noise; a model updating unit that updates the policy model or parameters of the policy model based on a smallest second evaluation value among the plurality of calculated second evaluation values ​​and a first evaluation value that is an index value indicating an evaluation result of the first action in the first state; Equipped with The model calculation unit is performing a weighting calculation on the measure information related to the first action and the state information related to the first state; Adding noise to the result of the weighting operation; normalizing the value of the calculation result to which the noise has been added; Classifying the normalized result according to a predetermined classification rule, Based on the identification result, an evaluation value indicating the learning situation is generated. Learning device.

2. a model calculation unit that calculates, based on a second state corresponding to a first action in a first state of a control target and a second action calculated from the second state using a policy model, a second evaluation value that includes noise in an index value that indicates an evaluation result of the second action in the second state, by using a plurality of evaluation models, and calculates each of the second evaluation values ​​including noise; a model updating unit that updates the policy model or parameters of the policy model based on a smallest second evaluation value among the plurality of calculated second evaluation values ​​and a first evaluation value that is an index value indicating an evaluation result of the first action in the first state; Equipped with The model calculation unit is performing a weighting calculation on the measure information related to the first action and the state information related to the first state; Adding noise to the result of the weighting operation; The noise-added calculation result value is used to generate an evaluation value indicating the learning status. Standardize Learning device.

3. The model calculation unit is calculating the second evaluation value by incorporating the noise into a calculation result based on the information related to the policy model, the state information related to the first state, and the state information related to the second state; The learning device according to claim 1 or 2.

4. The model calculation unit is A calculation for incorporating the noise and a calculation for normalizing the result of the calculation for incorporating the noise are performed. The learning device according to claim 1 or 2.

5. The model calculation unit is The calculation result including the noise is normalized using layer normalization. The learning device according to claim 1 or 2.

6. The model calculation unit is performing a weighting calculation on the measure information and the state information related to the first state, Adding noise to the result of the weighting operation; Normalizing the value of the calculation result to which the noise has been added, Classifying the normalized result according to a predetermined classification rule, A Q function for generating an evaluation value indicating the learning status based on the identification result is used to generate an evaluation value indicating the learning status. The learning device according to claim 1 or 2.

7. A model calculation unit that uses a plurality of evaluation models to calculate second evaluation values ​​that include noise in index values ​​that indicate evaluation results of the second action in the second state based on a second state corresponding to a first action in the first state of a controlled object and a second action calculated from the second state using a policy model, and calculates each of the second evaluation values ​​that include noise; a model updating unit that updates the policy model or parameters of the policy model based on a smallest second evaluation value among the plurality of calculated second evaluation values ​​and a first evaluation value that is an index value indicating an evaluation result of the first action in the first state; Equipped with The number of Q functions involved in the update; and The number of layers of the operation layer to which the noise is added in the Q function; The number of layers of a layer normalization layer that normalizes the output based on the output of a previous layer in the Q function; and the number of times that the calculation of value propagation is performed according to the operation mode of the control target. Equipped with The model calculation unit is Executing the value propagation calculation using the received information Learning device.

8. a model calculation means for calculating, based on a second state corresponding to a first action in a first state of a control target and a second action calculated from the second state using a policy model, a second evaluation value obtained by adding noise to an index value indicating an evaluation result of the second action in the second state, by using a plurality of evaluation models, each of which calculates a second evaluation value including noise; a model updating means for updating the policy model or a parameter of the policy model, based on a smallest second evaluation value among the plurality of calculated second evaluation values ​​and a first evaluation value which is an index value indicating an evaluation result of the first action in the first state; Equipped with The model calculation means performing a weighting calculation on the measure information related to the first action and the state information related to the first state; Adding noise to the result of the weighting operation; normalizing the value of the calculation result to which the noise has been added; Classifying the normalized result according to a predetermined classification rule, Based on the identification result, an evaluation value indicating the learning situation is generated. Control system.

9. a model calculation means for calculating, based on a second state corresponding to a first action in a first state of a control target and a second action calculated from the second state using a policy model, a second evaluation value obtained by adding noise to an index value indicating an evaluation result of the second action in the second state, by using a plurality of evaluation models, each of which calculates a second evaluation value including noise; a model updating means for updating the policy model or a parameter of the policy model, based on a smallest second evaluation value among the plurality of calculated second evaluation values ​​and a first evaluation value which is an index value indicating an evaluation result of the first action in the first state; Equipped with The model calculation means performing a weighting calculation on the measure information related to the first action and the state information related to the first state; Adding noise to the result of the weighting operation; The noise-added calculation result value is used to generate an evaluation value indicating the learning status. Control system.

10. a model calculation means for calculating, based on a second state corresponding to a first action in a first state of a control target and a second action calculated from the second state using a policy model, a second evaluation value obtained by adding noise to an index value indicating an evaluation result of the second action in the second state, by using a plurality of evaluation models, each of which calculates a second evaluation value including noise; a model updating means for updating the policy model or a parameter of the policy model, based on a smallest second evaluation value among the plurality of calculated second evaluation values ​​and a first evaluation value which is an index value indicating an evaluation result of the first action in the first state; Equipped with The number of Q functions involved in the update; and The number of layers of the operation layer to which the noise is added in the Q function; The number of layers of a layer normalization layer that normalizes the output based on the output of a previous layer in the Q function; and the number of times that the calculation of value propagation is performed according to the operation mode of the controlled object. Equipped with The model calculation means Executing the value propagation calculation using the received information Control system.

11. The computer calculating, based on a second state corresponding to a first action in a first state of a control target and a second action calculated from the second state using a policy model, the second evaluation values ​​including noise being calculated by using a plurality of evaluation models that calculate second evaluation values ​​by including noise in index values ​​indicating evaluation results of the second action in the second state; updating the policy model or a parameter of the policy model based on a smallest second evaluation value among the plurality of calculated second evaluation values ​​and a first evaluation value that is an index value indicating an evaluation result of the first action in the first state; performing a weighting calculation on the measure information related to the first action and the state information related to the first state; Adding noise to the result of the weighting operation; normalizing the value of the calculation result to which the noise has been added; Classifying the normalized result according to a predetermined classification rule, generating an evaluation value indicating a learning situation based on the identification result; Learning methods including:

12. The computer calculating, based on a second state corresponding to a first action in a first state of a control target and a second action calculated from the second state using a policy model, the second evaluation values ​​including noise being calculated by using a plurality of evaluation models that calculate second evaluation values ​​by including noise in index values ​​indicating evaluation results of the second action in the second state; updating the policy model or a parameter of the policy model based on a smallest second evaluation value among the plurality of calculated second evaluation values ​​and a first evaluation value that is an index value indicating an evaluation result of the first action in the first state; performing a weighting calculation on the measure information related to the first action and the state information related to the first state; Adding noise to the result of the weighting operation; The value of the calculation result to which the noise is added is used to generate an evaluation value indicating a learning situation. Learning methods including:

13. The computer calculating, based on a second state corresponding to a first action in a first state of a control target and a second action calculated from the second state using a policy model, the second evaluation values ​​including noise being calculated by using a plurality of evaluation models that calculate second evaluation values ​​by including noise in index values ​​indicating evaluation results of the second action in the second state; updating the policy model or a parameter of the policy model based on a smallest second evaluation value among the plurality of calculated second evaluation values ​​and a first evaluation value that is an index value indicating an evaluation result of the first action in the first state; The number of Q functions involved in the update; and The number of layers of the operation layer to which the noise is added in the Q function; The number of layers of a layer normalization layer that normalizes the output based on the output of a previous layer in the Q function; and the number of times the value propagation calculation is performed according to the operation mode of the control target. performing the value propagation calculation using the received information; and Learning methods including:

14. On the computer, calculating, based on a second state corresponding to a first action in a first state of a control target and a second action calculated from the second state using a policy model, each of the second evaluation values ​​including noise, using a plurality of evaluation models that calculate second evaluation values ​​by incorporating noise into index values ​​that indicate evaluation results of the second action in the second state; updating the policy model or parameters of the policy model based on a smallest second evaluation value among the plurality of calculated second evaluation values ​​and a first evaluation value that is an index value indicating an evaluation result of the first action in the first state; performing a weighting operation on the policy information related to the first action and the state information related to the first state, adding noise to a result of the weighting operation, normalizing a value of the operation result to which the noise has been added, classifying the normalized result in accordance with a predetermined classification rule, and generating an evaluation value indicating a learning status based on the classification result; A program for executing the above.

15. On the computer, calculating, based on a second state corresponding to a first action in a first state of a control target and a second action calculated from the second state using a policy model, each of the second evaluation values ​​including noise, using a plurality of evaluation models that calculate second evaluation values ​​by incorporating noise into index values ​​that indicate evaluation results of the second action in the second state; updating the policy model or parameters of the policy model based on a smallest second evaluation value among the plurality of calculated second evaluation values ​​and a first evaluation value that is an index value indicating an evaluation result of the first action in the first state; performing a weighting calculation on the measure information related to the first action and the state information related to the first state, adding noise to a result of the weighting calculation, and normalizing a value of the calculation result to which the noise has been added; A program for executing the above.

16. On the computer, calculating, based on a second state corresponding to a first action in a first state of a control target and a second action calculated from the second state using a policy model, each of the second evaluation values ​​including noise, using a plurality of evaluation models that calculate second evaluation values ​​by incorporating noise into index values ​​that indicate evaluation results of the second action in the second state; updating the policy model or parameters of the policy model based on a smallest second evaluation value among the plurality of calculated second evaluation values ​​and a first evaluation value that is an index value indicating an evaluation result of the first action in the first state; Accepting at least any of the following information: the number of Q functions involved in the update; the number of layers of an operation layer to which the noise is added in the Q function; the number of layers of a layer normalization layer in the Q function that normalizes the output based on the output of the previous layer; and the number of times that a value propagation operation is performed according to the operation mode of the controlled object; performing the value propagation calculation using the received information; and A program for executing the above.

Images